Importing mouth-cue files

OpenFaceFX can write stepped mouth-cue files for the indie 2D ecosystem (Rhubarb, Papagayo, Moho — see Exporters); this module reads them back, so a studio sitting on a Rhubarb/Papagayo library or a hand-timed Moho mouth track can bring it into OpenFaceFX to coarticulate, retarget, layer gestures/events, condition and re-export (issue #44). Because the project owns both halves, each parser is the verified inverse of shipping code in export_cues.

Import a cue file with the from-cues command; the format is auto-detected by extension and first line:

python -m openfacefx from-cues mouth.tsv  -o track.json      # Rhubarb TSV
python -m openfacefx from-cues mouth.xml  -o track.anim      # Rhubarb XML -> Unity
python -m openfacefx from-cues mouth.dat  --fps 24 -o track.json   # Moho / OpenToonz
python -m openfacefx from-cues mouth.pgo  --coarticulate -o track.json  # Papagayo

The result is an ordinary stepped FaceTrack — one [0, 1] channel per viseme, sil in the gaps — that flows unchanged through every track exporter and --retarget. --coarticulate re-solves the hard steps through the existing dominance blend to smooth them.

Formats & frames

Format	Extension	Time base	Shape vocabulary
Rhubarb TSV / XML / JSON	.tsv / .xml / .json	seconds (%.2f)	Rhubarb A–H/X
Moho / OpenToonz	.dat	1-based frames (default 24 fps)	Preston-Blair, or Rhubarb A–H/X
Papagayo-NG	.pgo	1-based frames (fps in file)	Preston-Blair

Rhubarb files are seconds-based and reconstructed on a 100 fps grid (hundredths land exactly on frames). The rate-less Moho .dat defaults to 24 fps (override with --fps); Papagayo .pgo carries its own rate. Frame decoding inverts export_cues._frame_at / _to_frames: seconds = (frame - 1) / fps.

Shape → viseme

Shape IDs map back to viseme channels through :data:RHUBARB_TO_VISEME and :data:PRESTON_BLAIR_TO_VISEME, which are derived from the forward retarget presets (retarget.PRESETS) so they can never drift out of sync. When several visemes collapse onto one shape (e.g. many consonants → Rhubarb B), the inverse picks the first viseme in canonical VISEMES order — a representative that, by construction, retargets straight back to the same shape, so a stepped track re-exports to the identical cue file.

The extended shapes G (f/v), H (tongue-up) and X (idle) invert directly to FF / nn / sil for full fidelity. A shape the inverse table does not know is routed through the documented RHUBARB_EXTENDED_FALLBACK (G→A, H→C, X→A) and reported; if it still cannot be resolved it is a clear error — never silently dropped.

Round-trip guarantee

Each parser is tested against its writer:

• Rhubarb TSV / XML / JSON round-trip byte-identically (write → import → write reproduces the file).
• Moho .dat and Papagayo .pgo reach a byte-exact idempotent fixed point and preserve the collapsed (shape, frame-boundary) cue sequence exactly. A byte difference on the first pass can only be a redundant duplicate switch the writer's frame quantisation emits when it drops an intermediate run that collides at the lower rate — it holds the same mouth, carries no animation, and dominant_cues (which merges equal adjacent shapes) structurally cannot re-emit it, so the importer collapses it into the canonical run.

The imported track validates through io_export.from_dict/to_dict and re-exports through Unity / Godot / Live2D / cues / CSV / JSON unchanged. (.lip is phoneme-based, not track-based, and is outside the cue-import path — cues carry visemes, not phonemes.)

Deterministic; stdlib + numpy only (xml.etree, json, re, string splitting). A purely additive command and module: no existing command's output changes.

Blendshape-weight CSV (ARKit / Live Link Face)

A lot of face animation lives as per-frame blendshape-weight CSV — Apple ARKit's 52 coefficients recorded by Epic's Live Link Face app, or exported by capture tools and DCCs. from-csv reads two layouts, auto-detected from the header row:

python -m openfacefx from-csv track.csv -o out.json                 # OpenFaceFX long
python -m openfacefx from-csv capture.csv --fps 60 -o out.anim       # wide, Live Link Face
python -m openfacefx from-csv capture.csv --timecode-col Timecode -o out.json

• OpenFaceFX long CSV time,channel,value — the exact inverse of io_export.write_csv; a byte-clean round-trip (read_csv(write_csv(track)) reconstructs the channels and keyframes).
• Wide per-frame CSV — one row per frame, one column per blendshape name, with an optional leading Timecode / BlendShapeCount header as Live Link Face emits. A Timecode column (SMPTE HH:MM:SS:FF, sized by --fps) or the row index (i / fps) drives the timeline, and each column is RDP-thinned via reduce_to_track into sparse keys.

Channel names are kept verbatim in rig space (jawOpen, mouthSmileLeft, …) and values are clamped to [0, 1] — a column carrying out-of-range values (e.g. a non-blendshape head-rotation angle) is clamped and reported. Because the forward viseme→ARKit map is many-to-one, this deliberately does not recover visemes: it brings the raw rig-space channels in so they can be conditioned (--smooth/--lag), layered and re-exported through Unity / Godot / Live2D. The from-csv command is kept separate from from-cues — blendshape weights and mouth-shape cues are different sources. Library callers get read_csv(path, *, fps, timecode_col). numpy + stdlib (csv), deterministic across Python 3.9/3.13.

openfacefx.importers

Import stepped mouth-cue files back into a FaceTrack (issue #44).

The verified inverse of :mod:openfacefx.export_cues. That module writes the stepped mouth-shape lists the indie 2D lip-sync ecosystem reads; this one reads them back, so a studio sitting on a Rhubarb / Papagayo / Moho library can bring it into OpenFaceFX to coarticulate, retarget, layer gestures/events, condition and re-export. Because we own both halves, each parser is the exact inverse of shipping code:

• Rhubarb Lip Sync TSV / XML / JSON -- parse_rhubarb_{tsv,xml,json}
• Moho / OpenToonz switch data (.dat) -- parse_moho_dat
• Papagayo-NG (.pgo) -- parse_pgo

Each parser yields a canonical list of (start, end, shape) second-intervals. Rhubarb files are seconds-based (%.2f); Moho .dat / Papagayo .pgo are 1-based truncated frames, inverting export_cues._frame_at / _to_frames (.dat has no stored rate, default 24; .pgo carries its own). Shape IDs map back to viseme channels by inverting the very retarget presets the writers use (:data:RHUBARB_TO_VISEME / :data:PRESTON_BLAIR_TO_VISEME, derived from retarget.PRESETS so they can never drift); an extended/unknown shape routes through the documented RHUBARB_EXTENDED_FALLBACK and is reported, never silently dropped. The result is a stepped viseme :class:FaceTrack (reduce_to_track, one [0, 1] channel per viseme, sil in the gaps) that flows unchanged through every exporter and --retarget; --coarticulate re-solves it through the dominance blend to smooth the hard steps.

Deterministic, stdlib + numpy only (csv-style split, xml.etree, json, re). A new, purely additive command/module: no existing output changes.

Interval = Tuple[float, float, str] module-attribute

RHUBARB_TO_VISEME: Dict[str, str] = _invert_preset(PRESETS['rhubarb']) module-attribute

PRESTON_BLAIR_TO_VISEME: Dict[str, str] = dict(_invert_preset(PRESETS['preston_blair'])) module-attribute

detect_format(path: str, text: str) -> str

Pick a parser by file extension, then confirm/override by the first line (so a mislabelled extension still routes correctly).

Source code in src/openfacefx/importers.py

def detect_format(path: str, text: str) -> str:
    """Pick a parser by file extension, then confirm/override by the first line
    (so a mislabelled extension still routes correctly)."""
    head = text.lstrip()[:64]
    if head.startswith("MohoSwitch1"):
        return "dat"
    if head.startswith("lipsync version"):
        return "pgo"
    fmt = _EXT_FORMAT.get(os.path.splitext(path)[1].lower())
    if fmt is not None:
        return fmt
    if head.startswith("<?xml") or head.startswith("<rhubarb"):
        return "xml"
    if head.startswith("{"):
        return "json"
    return "tsv"

parse_rhubarb_tsv(text: str) -> Tuple[List[Interval], float]

Source code in src/openfacefx/importers.py

def parse_rhubarb_tsv(text: str) -> Tuple[List[Interval], float]:
    rows: List[Tuple[float, str]] = []
    for ln in _nonblank(text):
        parts = ln.split("\t")
        if len(parts) != 2:
            raise ValueError(f"rhubarb tsv: expected 'start<TAB>shape', got {ln!r}")
        rows.append((float(parts[0]), parts[1].strip()))
    return _switch_intervals(rows)

parse_rhubarb_xml(text: str) -> Tuple[List[Interval], float]

Source code in src/openfacefx/importers.py

def parse_rhubarb_xml(text: str) -> Tuple[List[Interval], float]:
    try:
        root = ET.fromstring(text)
    except ET.ParseError as e:
        raise ValueError(f"rhubarb xml: not well-formed ({e})") from None
    cues = root.find("mouthCues")
    intervals: List[Interval] = []
    for mc in (list(cues) if cues is not None else []):
        intervals.append((float(mc.get("start")), float(mc.get("end")),
                          (mc.text or "").strip()))
    dur_el = root.find("metadata/duration")
    dur = (float(dur_el.text) if dur_el is not None and dur_el.text
           else (intervals[-1][1] if intervals else 0.0))
    return intervals, dur

parse_rhubarb_json(text: str) -> Tuple[List[Interval], float]

Source code in src/openfacefx/importers.py

def parse_rhubarb_json(text: str) -> Tuple[List[Interval], float]:
    try:
        d = json.loads(text)
    except json.JSONDecodeError as e:
        raise ValueError(f"rhubarb json: not valid JSON ({e})") from None
    if not isinstance(d, dict) or "mouthCues" not in d:
        raise ValueError("rhubarb json: no 'mouthCues' array "
                         "(is this an openfacefx.track, not a cue file?)")
    intervals = [(float(c["start"]), float(c["end"]), str(c["value"]))
                 for c in d["mouthCues"]]
    meta = d.get("metadata") or {}
    dur = float(meta.get("duration", intervals[-1][1] if intervals else 0.0))
    return intervals, dur

parse_moho_dat(text: str, fps: float) -> Tuple[List[Interval], float]

Source code in src/openfacefx/importers.py

def parse_moho_dat(text: str, fps: float) -> Tuple[List[Interval], float]:
    rows = _nonblank(text)
    if not rows or rows[0].strip() != "MohoSwitch1":
        raise ValueError("moho .dat: first line must be 'MohoSwitch1'")
    switch: List[Tuple[float, str]] = []
    for ln in rows[1:]:
        parts = ln.split()
        if len(parts) != 2 or not parts[0].isdigit():
            raise ValueError(f"moho .dat: expected '<frame> <shape>', got {ln!r}")
        switch.append(((int(parts[0]) - 1) / fps, parts[1]))
    return _switch_intervals(switch)

parse_pgo(text: str) -> Tuple[List[Interval], float, float]

Papagayo-NG .pgo. Returns (intervals, duration, fps) -- fps and the total-frame count are stored in the header, and each phoneme line holds until the next (the last runs to the clip end).

Source code in src/openfacefx/importers.py

def parse_pgo(text: str) -> Tuple[List[Interval], float, float]:
    """Papagayo-NG ``.pgo``. Returns ``(intervals, duration, fps)`` -- fps and the
    total-frame count are stored in the header, and each phoneme line holds until
    the next (the last runs to the clip end)."""
    lines = text.splitlines()
    if not lines or not lines[0].startswith("lipsync version"):
        raise ValueError("papagayo .pgo: first line must be 'lipsync version ...'")
    try:
        fps = float(int(lines[2].strip()))
        total_frames = int(lines[3].strip())
    except (IndexError, ValueError):
        raise ValueError("papagayo .pgo: header must carry integer fps and "
                         "total-frame count on lines 3 and 4") from None
    phon = [(int(m.group(1)), m.group(2)) for m in
            (re.match(r"^\t\t\t\t(\d+) (\S+)$", ln) for ln in lines) if m]
    dur = total_frames / fps if fps else 0.0
    if not phon:
        return [], dur, fps
    starts = [(f - 1) / fps for f, _ in phon]
    dur = max(dur, starts[-1])
    intervals = [(starts[i], starts[i + 1] if i + 1 < len(phon) else dur, phon[i][1])
                 for i in range(len(phon))]
    return intervals, dur, fps

build_cue_track(viseme_intervals: List[Interval], fps: float, coarticulate: bool = False) -> FaceTrack

One [0, 1] channel per viseme, stepped over its intervals (sil in the gaps), reduced with :func:reduce_to_track. coarticulate instead re-solves synthetic segments through the dominance blend to smooth the steps.

Built on the fps grid so the cue boundaries land exactly on frames -- a re-dominant_cues reproduces the same runs (the round-trip contract).

Source code in src/openfacefx/importers.py

def build_cue_track(viseme_intervals: List[Interval], fps: float,
                    coarticulate: bool = False) -> FaceTrack:
    """One ``[0, 1]`` channel per viseme, stepped over its intervals (``sil`` in
    the gaps), reduced with :func:`reduce_to_track`. ``coarticulate`` instead
    re-solves synthetic segments through the dominance blend to smooth the steps.

    Built on the ``fps`` grid so the cue boundaries land exactly on frames -- a
    re-``dominant_cues`` reproduces the same runs (the round-trip contract)."""
    if not viseme_intervals:
        return FaceTrack(fps=fps, channels=[], target_set=None)
    if coarticulate:
        return _coarticulate(viseme_intervals, fps)
    duration = viseme_intervals[-1][1]
    n = int(round(duration * fps))
    times = np.array([i / fps for i in range(n + 1)], dtype=float)
    matrix = np.zeros((n + 1, len(VISEMES)), dtype=float)
    j = 0
    for i in range(n + 1):
        t = times[i]
        while j + 1 < len(viseme_intervals) and t >= viseme_intervals[j][1] - _EPS:
            j += 1
        matrix[i, VISEME_INDEX[viseme_intervals[j][2]]] = 1.0
    # epsilon=0: keep every step corner exactly (only collinear holds are thinned)
    # so the flattened steps survive a re-export bit-for-bit.
    return reduce_to_track(times, matrix, fps=fps, epsilon=0.0)

import_cues(path: str, *, fmt: Optional[str] = None, fps: Optional[float] = None, coarticulate: bool = False) -> Tuple[FaceTrack, List[str]]

Read a mouth-cue file into a stepped viseme :class:FaceTrack.

fmt overrides the extension/first-line auto-detection (one of tsv/xml/json/dat/pgo; json-cues is accepted as an alias of json). fps sets the frame rate for the rate-less Moho .dat (default 24); Rhubarb is seconds-based (reconstructed at 100 fps) and Papagayo carries its own rate, so fps is ignored for those. Returns (track, warnings) -- warnings lists any extended/unknown shapes that were remapped via the fallback (reported, never dropped).

Source code in src/openfacefx/importers.py

def import_cues(path: str, *, fmt: Optional[str] = None, fps: Optional[float] = None,
                coarticulate: bool = False) -> Tuple[FaceTrack, List[str]]:
    """Read a mouth-cue file into a stepped viseme :class:`FaceTrack`.

    ``fmt`` overrides the extension/first-line auto-detection (one of
    ``tsv``/``xml``/``json``/``dat``/``pgo``; ``json-cues`` is accepted as an
    alias of ``json``). ``fps`` sets the frame rate for the rate-less Moho
    ``.dat`` (default 24); Rhubarb is seconds-based (reconstructed at 100 fps) and
    Papagayo carries its own rate, so ``fps`` is ignored for those. Returns
    ``(track, warnings)`` -- ``warnings`` lists any extended/unknown shapes that
    were remapped via the fallback (reported, never dropped)."""
    with open(path, encoding="utf-8") as fh:
        text = fh.read()
    fmt = (fmt or detect_format(path, text)).replace("json-cues", "json")
    if fmt not in _FORMATS:
        raise ValueError(f"unknown cue format {fmt!r}; expected one of {_FORMATS}")
    warnings: List[str] = []
    if fmt in ("tsv", "xml", "json"):
        intervals, _dur = (parse_rhubarb_tsv(text) if fmt == "tsv" else
                           parse_rhubarb_xml(text) if fmt == "xml" else
                           parse_rhubarb_json(text))
        vocab, rfps = "rhubarb", _RHUBARB_FPS
    elif fmt == "dat":
        rfps = float(fps) if fps else _DAT_DEFAULT_FPS
        intervals, _dur = parse_moho_dat(text, rfps)
        vocab = _dat_vocab({name for _s, _e, name in intervals})
    else:  # pgo
        intervals, _dur, rfps = parse_pgo(text)
        vocab = "pb"
    _validate_intervals(intervals)
    viseme_intervals = _merge_adjacent(_map_to_visemes(intervals, vocab, warnings))
    return build_cue_track(viseme_intervals, rfps, coarticulate), warnings

openfacefx.importers_csv

Import blendshape-weight CSV into a FaceTrack (issue #45).

The CSV half of :mod:openfacefx.importers. A lot of face animation lives as per-frame blendshape weights — Apple ARKit's 52 coefficients recorded by Epic's Live Link Face app, or exported by capture tools and DCCs — and OpenFaceFX could write_csv but never read one back. Two layouts, auto-detected from the header:

• OpenFaceFX long CSV time,channel,value — the exact inverse of :func:openfacefx.io_export.write_csv; a byte-clean round-trip.
• Wide per-frame CSV — one row per frame, one column per blendshape name, with an optional leading Timecode / BlendShapeCount header as Live Link Face emits. The timecode (or the row index) converts to seconds via fps/timecode_col and each column is RDP-thinned into sparse keys with :func:openfacefx.curves.reduce_to_track.

Channel names land in rig space verbatim (jawOpen, mouthSmileLeft …) and values are clamped to [0, 1]. The forward viseme→ARKit map is many-to-one, so this deliberately does not recover visemes — it brings the raw channels in so they can be conditioned (--smooth/--lag), layered and re-exported. numpy + stdlib (csv) only, deterministic (fixed RDP thinner, stable 4-dp rounding — identical on Python 3.9/3.13), and additive.

read_csv(path: str, *, fps: float = _DEFAULT_FPS, timecode_col: Optional[str] = None, epsilon: float = 0.015) -> Tuple[FaceTrack, List[str]]

Read a blendshape-weight CSV into (FaceTrack, warnings).

Auto-detects the OpenFaceFX long time,channel,value layout (the exact inverse of :func:~openfacefx.io_export.write_csv) versus a wide per-frame layout. fps times the wide rows (frame→seconds); timecode_col names the SMPTE-timecode column (else a literal Timecode column, else the row index drives the timeline). warnings reports any column whose values were clamped into [0, 1] (e.g. a non-blendshape angle column).

Source code in src/openfacefx/importers_csv.py

def read_csv(path: str, *, fps: float = _DEFAULT_FPS,
             timecode_col: Optional[str] = None,
             epsilon: float = 0.015) -> Tuple[FaceTrack, List[str]]:
    """Read a blendshape-weight CSV into ``(FaceTrack, warnings)``.

    Auto-detects the OpenFaceFX long ``time,channel,value`` layout (the exact
    inverse of :func:`~openfacefx.io_export.write_csv`) versus a wide per-frame
    layout. ``fps`` times the wide rows (frame→seconds); ``timecode_col`` names
    the SMPTE-timecode column (else a literal ``Timecode`` column, else the row
    index drives the timeline). ``warnings`` reports any column whose values were
    clamped into ``[0, 1]`` (e.g. a non-blendshape angle column)."""
    with open(path, newline="", encoding="utf-8") as fh:
        rows = list(_csv.reader(fh))
    if not rows:
        return FaceTrack(fps=fps, channels=[], target_set=None), []
    header = [h.strip() for h in rows[0]]
    body = rows[1:]
    if [h.lower() for h in header] == _LONG_HEADER:
        return _read_long(body, fps), []
    return _read_wide(header, body, fps, timecode_col, epsilon)