Subtitles & captions (SRT / WebVTT)

Captions and lip motion should come from one source of truth so they stay in sync (issue #41). OpenFaceFX already ingests word/segment timings (anchors.parse_srt, the Azure / ElevenLabs word boundaries); this is the matching output. It is deterministic string formatting over the timing arrays the pipeline already produces — not a new alignment: word_timings pulls per-word spans from texttags.naive_word_segments, whose phoneme segments are byte-identical to the pipeline.naive_segments the viseme curves are reduced from, so the words the captions carry are timed by the very segments that drove the mouth.

# a standalone subtitle file, timed like the lip-sync would be
python -m openfacefx captions --text "Well met, traveler." --wav vo.wav -o vo.srt
python -m openfacefx captions --text "Well met, traveler." --duration 2.0 -o vo.vtt --karaoke

# or co-generate the track and its captions from one run
python -m openfacefx naive --text "Well met." --wav vo.wav -o vo.track.json --emit-captions vo.srt

# and at dialogue scale, a caption sidecar next to every naive-mode track
python -m openfacefx batch --manifest loc.csv --out tracks/ --captions srt

The pipeline is three deterministic steps:

• word_timings → per-word (token, start, end) (punctuation kept for sentence detection; a hyphenated token spans its parts);
• build_cues groups words into cues — greedily packed under a max-line-length × max-lines wrap budget (no cue exceeds it), broken at sentence-ending punctuation and audible gaps, each cue's duration extended toward a reading-speed (characters-per-second) minimum where the timeline allows and clamped so cues stay monotonic and non-overlapping;
• srt_text / vtt_text serialise to SubRip (HH:MM:SS,mmm) and WebVTT (HH:MM:SS.mmm under the WEBVTT header), with optional word-level karaoke (--karaoke, WebVTT <c> spans + inline cue timestamps that fall inside their cue span).

srt_text is the exact inverse of anchors.parse_srt: parse_srt(srt_text(cues)) recovers the same cue spans (within millisecond rounding). Output is LF-terminated UTF-8, pure stdlib.

Reading WebVTT back (parse_vtt)

parse_vtt is the read-side inverse of vtt_text/write_vtt (issue #55) — co-located here so the read and write of the karaoke format can't drift. It parses WebVTT into timing Anchors: blank-line cue blocks with HH:MM:SS.mmm (or the hour-less MM:SS.mmm) timing, ignoring the WEBVTT header, NOTE/STYLE/ REGION blocks, cue identifiers and cue settings; a karaoke cue (the inline <timestamp><c>word</c> spans this module emits) yields one anchor per word, each inside the cue span, otherwise a single cue-level anchor with inline tags stripped. parse_vtt(vtt_text(cues)) round-trips the timings within millisecond rounding — in both plain and karaoke modes. It is wired into the CLI as --anchors-format vtt (self-transcribing like srt).

openfacefx.export_captions

Subtitle / caption exporter (issue #41): SRT + WebVTT from the SAME alignment.

Captions and lip motion should come from one source of truth so they stay in sync. OpenFaceFX already ingests word/segment timings (anchors.parse_srt, the Azure / ElevenLabs word boundaries) but had no caption output. This is that output — deterministic string formatting over the timing arrays the pipeline already produces, not a new alignment: :func:word_timings pulls per-word spans from :func:openfacefx.texttags.naive_word_segments, which builds the identical [sil] + phones + [sil] sequence :func:openfacefx.pipeline. naive_segments does — so the words the captions carry are timed by the very segments the viseme curves were reduced from.

The pipeline:

• :func:word_timings -> per-word (token, start, end) (punctuation-bearing display tokens paired to the word spans; a hyphenated token spans its parts);
• :func:build_cues groups words into cues — greedily packed under a max-line-length / max-lines wrap budget, broken at sentence-ending punctuation and audible gaps, each cue's duration extended toward a reading-speed (characters-per-second) minimum and clamped so cues stay monotonic and non-overlapping;
• :func:srt_text / :func:vtt_text serialise to SubRip (HH:MM:SS,mmm) and WebVTT (HH:MM:SS.mmm behind the WEBVTT header), with optional word-level karaoke highlighting (WebVTT <c> spans + inline cue timestamps that fall inside their cue span).

:func:write_srt / :func:write_vtt (and the extension-dispatching :func:write_captions) write LF-terminated UTF-8. Pure stdlib, deterministic; srt_text is the inverse of anchors.parse_srt (a round-trip recovers the cue spans within millisecond rounding).

DEFAULT_CPS = 17.0 module-attribute

DEFAULT_MAX_LINE = 42 module-attribute

DEFAULT_MAX_LINES = 2 module-attribute

DEFAULT_GAP = 0.5 module-attribute

WordTiming = Tuple[str, float, float] module-attribute

CaptionCue

Bases: NamedTuple

One subtitle cue: a 1-based index, a [start, end] span in seconds, the wrapped display lines, and the per-word words (for karaoke).

index: int instance-attribute

start: float instance-attribute

end: float instance-attribute

lines: List[str] instance-attribute

words: List[WordTiming] instance-attribute

text: str property

format_timestamp(seconds: float, sep: str = ',') -> str

seconds -> HH:MM:SS<sep>mmm (sep="," for SRT, "." for WebVTT), rounded to the nearest millisecond and floored at zero.

Source code in src/openfacefx/export_captions.py

def format_timestamp(seconds: float, sep: str = ",") -> str:
    """``seconds`` -> ``HH:MM:SS<sep>mmm`` (``sep=","`` for SRT, ``"."`` for
    WebVTT), rounded to the nearest millisecond and floored at zero."""
    ms = int(round(max(seconds, 0.0) * 1000.0))
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return "%02d:%02d:%02d%s%03d" % (h, m, s, sep, ms)

word_timings(text: str, duration: float, g2p=None) -> List[WordTiming]

Per-word (token, start, end) from the same naive alignment the lip curves use. Display tokens keep their punctuation (so cue grouping can see sentence ends); each is paired to its [A-Za-z']+ word span(s) — a hyphenated token spans from its first part's start to its last part's end, and a pure-punctuation token (no word) contributes no timing.

Source code in src/openfacefx/export_captions.py

def word_timings(text: str, duration: float, g2p=None) -> List[WordTiming]:
    """Per-word ``(token, start, end)`` from the same naive alignment the lip
    curves use. Display tokens keep their punctuation (so cue grouping can see
    sentence ends); each is paired to its ``[A-Za-z']+`` word span(s) — a
    hyphenated token spans from its first part's start to its last part's end,
    and a pure-punctuation token (no word) contributes no timing."""
    from .g2p import G2P
    from .texttags import WORD_RE, naive_word_segments
    _, spans = naive_word_segments(text, duration, g2p or G2P())
    out: List[WordTiming] = []
    idx = 0
    for token in text.split():
        n = len(WORD_RE.findall(token))
        if n == 0:
            continue
        out.append((token, spans[idx][0], spans[idx + n - 1][1]))
        idx += n
    return out

build_cues(words: Sequence[WordTiming], *, cps: float = DEFAULT_CPS, max_line: int = DEFAULT_MAX_LINE, max_lines: int = DEFAULT_MAX_LINES, gap: float = DEFAULT_GAP) -> List[CaptionCue]

Group words into subtitle cues.

Each cue wraps into at most max_lines lines of at most max_line chars (never exceeded, bar a single unbreakable over-long word), starts at its first word and ends at max(last word end, start + chars/cps) — held long enough to read at cps characters per second — with every end clamped to the next cue's start so cues are monotonic and non-overlapping.

Source code in src/openfacefx/export_captions.py

def build_cues(words: Sequence[WordTiming], *, cps: float = DEFAULT_CPS,
               max_line: int = DEFAULT_MAX_LINE,
               max_lines: int = DEFAULT_MAX_LINES,
               gap: float = DEFAULT_GAP) -> List[CaptionCue]:
    """Group ``words`` into subtitle cues.

    Each cue wraps into at most ``max_lines`` lines of at most ``max_line`` chars
    (never exceeded, bar a single unbreakable over-long word), starts at its
    first word and ends at ``max(last word end, start + chars/cps)`` — held long
    enough to read at ``cps`` characters per second — with every end clamped to
    the next cue's start so cues are monotonic and non-overlapping."""
    groups = _group(list(words), max_line, max_lines, gap)
    spans: List[List] = []
    for g in groups:
        lines = _wrap([w[0] for w in g], max_line)
        start = g[0][1]
        chars = sum(len(ln) for ln in lines)
        end = max(g[-1][2], start + chars / cps if cps > 0 else g[-1][2])
        spans.append([start, end, lines, list(g)])
    for i in range(len(spans) - 1):                  # non-overlap: clamp to next
        if spans[i][1] > spans[i + 1][0]:
            spans[i][1] = spans[i + 1][0]
    return [CaptionCue(i + 1, s[0], s[1], s[2], s[3])
            for i, s in enumerate(spans)]

srt_text(cues: Sequence[CaptionCue]) -> str

Serialise cues as SubRip: index / HH:MM:SS,mmm --> ... / text, blank-line separated (the inverse of anchors.parse_srt).

Source code in src/openfacefx/export_captions.py

def srt_text(cues: Sequence[CaptionCue]) -> str:
    """Serialise cues as SubRip: ``index`` / ``HH:MM:SS,mmm --> ...`` / text,
    blank-line separated (the inverse of ``anchors.parse_srt``)."""
    blocks = ["%d\n%s --> %s\n%s\n" % (
        c.index, format_timestamp(c.start, ","), format_timestamp(c.end, ","),
        "\n".join(c.lines)) for c in cues]
    return "\n".join(blocks)

vtt_text(cues: Sequence[CaptionCue], *, karaoke: bool = False) -> str

Serialise cues as WebVTT (WEBVTT header, HH:MM:SS.mmm dot timestamps); karaoke emits per-word <c> spans with inline cue timestamps instead of the plain wrapped text.

Source code in src/openfacefx/export_captions.py

def vtt_text(cues: Sequence[CaptionCue], *, karaoke: bool = False) -> str:
    """Serialise cues as WebVTT (``WEBVTT`` header, ``HH:MM:SS.mmm`` dot
    timestamps); ``karaoke`` emits per-word ``<c>`` spans with inline cue
    timestamps instead of the plain wrapped text."""
    blocks = ["WEBVTT\n"]
    for c in cues:
        body = _karaoke_payload(c) if karaoke else "\n".join(c.lines)
        blocks.append("%s --> %s\n%s\n" % (
            format_timestamp(c.start, "."), format_timestamp(c.end, "."), body))
    return "\n".join(blocks)

write_srt(cues: Sequence[CaptionCue], path: str) -> None

Write cues as a SubRip .srt file (LF, UTF-8).

Source code in src/openfacefx/export_captions.py

def write_srt(cues: Sequence[CaptionCue], path: str) -> None:
    """Write ``cues`` as a SubRip ``.srt`` file (LF, UTF-8)."""
    _write_text(path, srt_text(cues))

write_vtt(cues: Sequence[CaptionCue], path: str, *, karaoke: bool = False) -> None

Write cues as a WebVTT .vtt file (LF, UTF-8).

Source code in src/openfacefx/export_captions.py

def write_vtt(cues: Sequence[CaptionCue], path: str, *,
              karaoke: bool = False) -> None:
    """Write ``cues`` as a WebVTT ``.vtt`` file (LF, UTF-8)."""
    _write_text(path, vtt_text(cues, karaoke=karaoke))

write_captions(text: str, duration: float, path: str, *, g2p=None, karaoke: bool = False, cps: float = DEFAULT_CPS, max_line: int = DEFAULT_MAX_LINE, max_lines: int = DEFAULT_MAX_LINES, gap: float = DEFAULT_GAP) -> List[CaptionCue]

Derive cues from text + duration (the shared naive alignment) and write them to path as .vtt (WebVTT) or .srt (SubRip, the default for any other extension). Returns the cues. karaoke applies to WebVTT.

Source code in src/openfacefx/export_captions.py

def write_captions(text: str, duration: float, path: str, *, g2p=None,
                   karaoke: bool = False, cps: float = DEFAULT_CPS,
                   max_line: int = DEFAULT_MAX_LINE,
                   max_lines: int = DEFAULT_MAX_LINES,
                   gap: float = DEFAULT_GAP) -> List[CaptionCue]:
    """Derive cues from ``text`` + ``duration`` (the shared naive alignment) and
    write them to ``path`` as ``.vtt`` (WebVTT) or ``.srt`` (SubRip, the default
    for any other extension). Returns the cues. ``karaoke`` applies to WebVTT."""
    cues = build_cues(word_timings(text, duration, g2p), cps=cps,
                      max_line=max_line, max_lines=max_lines, gap=gap)
    if path.lower().endswith(".vtt"):
        write_vtt(cues, path, karaoke=karaoke)
    else:
        write_srt(cues, path)
    return cues

parse_vtt(text: str) -> List[Anchor]

Parse WebVTT subtitles into timing anchors — the read-side inverse of :func:vtt_text / :func:write_vtt.

Blank-line-separated cue blocks with HH:MM:SS.mmm --> HH:MM:SS.mmm (the hour-less MM:SS.mmm is accepted too); an optional cue identifier line and trailing cue settings (position:/align:/line:…) are ignored, as are the WEBVTT header and any NOTE / STYLE / REGION blocks. A cue carrying inline <timestamp><c>word</c> karaoke spans (what the #41 writer emits) yields one anchor per word, each inside the cue span; otherwise a single cue-level anchor with the inline tags stripped. Malformed input raises ValueError.

Source code in src/openfacefx/export_captions.py

def parse_vtt(text: str) -> List[Anchor]:
    """Parse WebVTT subtitles into timing anchors — the read-side inverse of
    :func:`vtt_text` / :func:`write_vtt`.

    Blank-line-separated cue blocks with ``HH:MM:SS.mmm --> HH:MM:SS.mmm`` (the
    hour-less ``MM:SS.mmm`` is accepted too); an optional cue identifier line and
    trailing cue settings (``position:``/``align:``/``line:``…) are ignored, as
    are the ``WEBVTT`` header and any ``NOTE`` / ``STYLE`` / ``REGION`` blocks. A
    cue carrying inline ``<timestamp><c>word</c>`` karaoke spans (what the #41
    writer emits) yields one anchor **per word**, each inside the cue span;
    otherwise a single cue-level anchor with the inline tags stripped. Malformed
    input raises ``ValueError``."""
    body = text.lstrip("").strip()
    if not re.match(r"WEBVTT(\s|$)", body):
        raise ValueError("vtt: missing WEBVTT header")
    out: List[Anchor] = []
    for block in re.split(r"\r?\n[ \t]*\r?\n", body):
        lines = block.splitlines()
        head = lines[0].strip() if lines else ""
        if head.startswith(("WEBVTT", "NOTE", "STYLE", "REGION")):
            continue
        ti = next((k for k, ln in enumerate(lines) if "-->" in ln), None)
        if ti is None:
            continue                        # a non-cue block (metadata / stray id)
        stamps = list(_VTT_TIME.finditer(lines[ti]))
        if len(stamps) < 2:
            raise ValueError(f"vtt: malformed timing line {lines[ti]!r}")
        start, end = _vtt_seconds(stamps[0]), _vtt_seconds(stamps[1])
        payload = " ".join(lines[ti + 1:])
        if re.search(r"<\d{1,2}:\d{2}(?::\d{2})?\.\d{1,3}>", payload):
            out.extend(_karaoke_anchors(payload, start, end))
        else:
            cue = re.sub(r"\s+", " ", _VTT_TAG.sub("", payload)).strip()
            out.append(Anchor(cue, start, end))
    if not out:
        raise ValueError("vtt: no cues found")
    return out