Visemes, coarticulation & curves

The viseme model (the Oculus/Meta 15-viseme set and the phoneme→viseme map), the Cohen–Massaro dominance-blending coarticulation solver, the FaceTrack data model and keyframe reduction, data-driven weighted mapping, and IPA support.

openfacefx.visemes

Viseme inventory and phoneme -> viseme mapping.

A viseme is the visual mouth shape corresponding to one or more phonemes. Many phonemes are visually indistinguishable (e.g. /p/, /b/, /m/ are all a lip closure), so the mapping is many-to-one.

We ship the 15-target set popularised by the Oculus/Meta LipSync SDK because it is a widely adopted, well-documented, IP-free convention that most character rigs already provide blendshapes for. Each viseme name below is a blendshape your rig is expected to expose.

VISEMES = ['sil', 'PP', 'FF', 'TH', 'DD', 'kk', 'CH', 'SS', 'nn', 'RR', 'aa', 'E', 'I', 'O', 'U'] module-attribute

VISEME_INDEX = {name: i for i, name in (enumerate(VISEMES))} module-attribute

PHONEME_TO_VISEME = {'P': 'PP', 'B': 'PP', 'M': 'PP', 'F': 'FF', 'V': 'FF', 'TH': 'TH', 'DH': 'TH', 'T': 'DD', 'D': 'DD', 'L': 'DD', 'N': 'nn', 'NG': 'nn', 'K': 'kk', 'G': 'kk', 'HH': 'kk', 'CH': 'CH', 'JH': 'CH', 'SH': 'CH', 'ZH': 'CH', 'S': 'SS', 'Z': 'SS', 'R': 'RR', 'ER': 'RR', 'W': 'U', 'Y': 'I', 'AA': 'aa', 'AE': 'aa', 'AH': 'aa', 'AY': 'aa', 'EH': 'E', 'EY': 'E', 'IH': 'E', 'IY': 'I', 'AO': 'O', 'OW': 'O', 'OY': 'O', 'AW': 'O', 'UW': 'U', 'UH': 'U', SILENCE: 'sil'} module-attribute

phoneme_to_viseme(phoneme: str) -> str

Map a (possibly stressed) ARPAbet phoneme to a viseme name.

Source code in src/openfacefx/visemes.py

def phoneme_to_viseme(phoneme: str) -> str:
    """Map a (possibly stressed) ARPAbet phoneme to a viseme name."""
    p = strip_stress(phoneme).upper() if phoneme != SILENCE else SILENCE
    return PHONEME_TO_VISEME.get(p, "sil")

viseme_index(name: str) -> int

Source code in src/openfacefx/visemes.py

def viseme_index(name: str) -> int:
    return VISEME_INDEX[name]

openfacefx.coarticulation

Coarticulation via dominance functions (Cohen & Massaro, 1993).

Real speech is not a sequence of discrete mouth poses -- each phoneme's shape is pulled toward its neighbours. A common, well-cited way to model this is to give every phoneme segment a dominance function: a bump in time, peaked at the segment centre, that decays outward. The activation of a viseme channel at any instant is the dominance-weighted average of the targets of all nearby segments.

F_v(t) = sum_i D_i(t) * target(i, v)  /  sum_i D_i(t)

where D_i(t) = alpha_i * exp( -theta_i * |t - c_i| ) (a Laplacian bump), c_i is the segment centre, and target(i, v) is 1 if segment i maps to viseme v.

The result is smooth, overlapping viseme curves rather than hard switches.

STYLE_PRESETS: Dict[str, Dict[str, object]] = {'neutral': {}, 'whisper': {'intensity': 0.5, 'gains': {'jaw': 0.7, 'lips': 0.95, 'tongue': 0.85}}, 'mumble': {'intensity': 0.62, 'gains': {'jaw': 0.75, 'lips': 0.85, 'tongue': 0.7}}, 'tense': {'intensity': 0.95, 'gains': {'jaw': 0.8, 'lips': 1.12, 'tongue': 1.18}}, 'exaggerated': {'intensity': 1.35, 'gains': {'jaw': 1.3, 'lips': 1.2, 'tongue': 1.15}}, 'broad': {'intensity': 1.55, 'gains': {'jaw': 1.5, 'lips': 1.28, 'tongue': 1.1}}} module-attribute

CoartParams(lead: Dict[str, Tuple[float, float]] = (lambda: {'basic': (0.4, 0.45), 'jaw': (0.4, 0.45), 'lips': (0.3, 0.3), 'tongue': (0.15, 0.15)})(), short_silence: float = 0.27, closure_floor: float = 0.9, split_diphthongs: bool = True, preroll: float = 0.0, allow_negative_time: bool = False, intensity: float = 1.0, gains: Dict[str, float] = (lambda: {'basic': 1.0, 'jaw': 1.0, 'lips': 1.0, 'tongue': 1.0})(), smooth: float = 0.0, lag: float = 0.0, stress_emphasis: float = 0.0, emphasis_windows: List[Tuple[float, float, float]] = list(), jali: bool = False, jali_rules: Optional[Tuple[str, ...]] = None, jali_timing: bool = True) dataclass

Component-model tunables (FaceFX-style ca_* knobs).

lead gives per-articulator-class (lead_in, lead_out) extents in seconds — how far a segment's influence reaches before/after its centre. The "basic"/"jaw" defaults reproduce the classic symmetric model; lips and especially tongue targets are tighter, so a quick stop does not smear across neighbouring vowels.

intensity (master) and gains (per-articulator-class) are JALI-style artistic dials: after the curves are normalized, every channel's opening is scaled by intensity * gains[class] and the freed weight flows into sil (see _apply_intensity). All 1.0 is a byte-identical no-op; <1 mumbles / softens a class, >1 hyper-articulates, 0 mutes it. Enforced lip closures still win afterwards, so a whispered bilabial seals.

smooth and lag are FaceFX-style post-solve curve conditioning (:mod:openfacefx.postprocess), both default off. smooth is the sigma (seconds) of a temporal Gaussian run over the dense matrix before keyframe reduction to soften jitter; closures are re-enforced after it, so lip seals stay sharp. lag slides the reduced keyframes in time (seconds; >0 lags / <0 leads the audio) and is applied by the pipeline once curves are reduced, not here. 0.0 for both is a byte-identical no-op.

stress_emphasis (issue #18) is the lexical-stress amplitude pass: with it > 0 a vowel segment carrying an ARPABET primary-stress digit (1) has its dominance amplitude raised, secondary (2) half as much, and an explicitly unstressed vowel (0) slightly lowered, so stressed syllables win more of the blend and articulate more strongly (see _stress_gains). Because it scales the dominance — which appears in both the blend and its normalizing denominator — the partition invariant is untouched. It is a graceful no-op on inputs without stress digits (vendor/IPA timing) and, at the 0.0 default, byte-identical. Named delivery-style presets that bundle the intensity/gains dials live in STYLE_PRESETS / style_params.

lead: Dict[str, Tuple[float, float]] = field(default_factory=(lambda: {'basic': (0.4, 0.45), 'jaw': (0.4, 0.45), 'lips': (0.3, 0.3), 'tongue': (0.15, 0.15)})) class-attribute instance-attribute

short_silence: float = 0.27 class-attribute instance-attribute

closure_floor: float = 0.9 class-attribute instance-attribute

split_diphthongs: bool = True class-attribute instance-attribute

preroll: float = 0.0 class-attribute instance-attribute

allow_negative_time: bool = False class-attribute instance-attribute

intensity: float = 1.0 class-attribute instance-attribute

gains: Dict[str, float] = field(default_factory=(lambda: {'basic': 1.0, 'jaw': 1.0, 'lips': 1.0, 'tongue': 1.0})) class-attribute instance-attribute

smooth: float = 0.0 class-attribute instance-attribute

lag: float = 0.0 class-attribute instance-attribute

stress_emphasis: float = 0.0 class-attribute instance-attribute

emphasis_windows: List[Tuple[float, float, float]] = field(default_factory=list) class-attribute instance-attribute

jali: bool = False class-attribute instance-attribute

jali_rules: Optional[Tuple[str, ...]] = None class-attribute instance-attribute

jali_timing: bool = True class-attribute instance-attribute

build_viseme_curves(segments: List[PhonemeSegment], fps: float = 60.0, mapping: Optional[Mapping] = None, params: Optional[CoartParams] = None) -> tuple

Return (times, matrix) where matrix[frame, target] in [0,1].

times is a 1-D array of sample times. Without mapping, columns follow visemes.VISEMES; with a Mapping they follow mapping.target_names and any phoneme may drive several targets with fractional weights. params tunes the component coarticulation model (per-articulator lead in/out, silence absorption, closure enforcement, diphthong splitting, onset pre-roll).

Source code in src/openfacefx/coarticulation.py

def build_viseme_curves(
    segments: List[PhonemeSegment],
    fps: float = 60.0,
    mapping: Optional[Mapping] = None,
    params: Optional[CoartParams] = None,
) -> tuple:
    """Return (times, matrix) where matrix[frame, target] in [0,1].

    ``times`` is a 1-D array of sample times. Without ``mapping``, columns
    follow ``visemes.VISEMES``; with a ``Mapping`` they follow
    ``mapping.target_names`` and any phoneme may drive several targets with
    fractional weights. ``params`` tunes the component coarticulation model
    (per-articulator lead in/out, silence absorption, closure enforcement,
    diphthong splitting, onset pre-roll).
    """
    params = params or CoartParams()
    n_targets = len(mapping.targets) if mapping is not None else len(VISEMES)
    if not segments:
        return np.zeros(0), np.zeros((0, n_targets))
    segments = _preprocess(segments, params)
    if params.jali:                          # JALI duplicated-viseme merge (#19)
        from . import coart_jali
        if coart_jali.rule_enabled(params, "duplicated_merge"):
            segments = coart_jali.merge_duplicates(segments, mapping)

    t0 = segments[0].start - params.preroll
    if not params.allow_negative_time:
        t0 = max(t0, 0.0) if segments[0].start >= 0.0 else segments[0].start
    t1 = segments[-1].end
    n = max(int(round((t1 - t0) * fps)) + 1, 1)
    times = t0 + np.arange(n) / fps

    matrix = _blend(segments, times, mapping, params, fps)
    return times, matrix

style_params(name: str) -> CoartParams

A fresh :class:CoartParams for the named style in STYLE_PRESETS.

The preset's dial overrides are laid over the defaults, so unset fields keep their default (byte-identical) values; gains/lead merge onto the all-1.0 defaults, scalar fields replace. style_params("neutral") is thus a plain CoartParams(). A new instance is returned each call, so callers may mutate it (e.g. compose CLI dials on top) without disturbing the shared table. Unknown name raises KeyError (validated at the CLI boundary).

Source code in src/openfacefx/coarticulation.py

def style_params(name: str) -> CoartParams:
    """A fresh :class:`CoartParams` for the named style in ``STYLE_PRESETS``.

    The preset's dial overrides are laid over the defaults, so unset fields keep
    their default (byte-identical) values; ``gains``/``lead`` merge onto the
    all-1.0 defaults, scalar fields replace. ``style_params("neutral")`` is thus
    a plain ``CoartParams()``. A new instance is returned each call, so callers
    may mutate it (e.g. compose CLI dials on top) without disturbing the shared
    table. Unknown ``name`` raises ``KeyError`` (validated at the CLI boundary)."""
    if name not in STYLE_PRESETS:
        raise KeyError(name)
    p = CoartParams()
    for field_name, value in STYLE_PRESETS[name].items():
        if field_name in ("gains", "lead"):
            merged = dict(getattr(p, field_name))
            merged.update(value)
            setattr(p, field_name, merged)
        else:
            setattr(p, field_name, value)
    return p

openfacefx.curves

Animation curves: keyframe reduction and track containers.

The dominance model produces one dense sample per frame. Rigs and engines prefer sparse keyframes, so we thin each channel with the Ramer-Douglas-Peucker algorithm: drop samples that lie within epsilon of the straight line between their neighbours. This is lossy but perceptually safe and shrinks output a lot.

Keyframe(time: float, value: float) dataclass

time: float instance-attribute

value: float instance-attribute

Channel(name: str, keys: List[Keyframe] = list()) dataclass

name: str instance-attribute

keys: List[Keyframe] = field(default_factory=list) class-attribute instance-attribute

FaceTrack(fps: float, channels: List[Channel], target_set: List[str] = None, events: 'List[Event]' = list(), variants: 'Optional[Variants]' = None) dataclass

fps: float instance-attribute

channels: List[Channel] instance-attribute

target_set: List[str] = None class-attribute instance-attribute

events: 'List[Event]' = field(default_factory=list) class-attribute instance-attribute

variants: 'Optional[Variants]' = None class-attribute instance-attribute

duration: float property

reduce_to_track(times: np.ndarray, matrix: np.ndarray, fps: float, epsilon: float = 0.015, targets=None) -> FaceTrack

targets: optional list of mapping.Target — supplies channel names and per-target min/max clamps. Defaults to the Oculus viseme set with no clamping (identical to previous releases).

Source code in src/openfacefx/curves.py

def reduce_to_track(times: np.ndarray, matrix: np.ndarray, fps: float,
                    epsilon: float = 0.015, targets=None) -> FaceTrack:
    """``targets``: optional list of ``mapping.Target`` — supplies channel
    names and per-target min/max clamps. Defaults to the Oculus viseme set
    with no clamping (identical to previous releases)."""
    if targets is None:
        names, clamps = VISEMES, [None] * len(VISEMES)
    else:
        names = [t.name for t in targets]
        clamps = [(t.lo, t.hi) if (t.lo, t.hi) != (0.0, 1.0) else None
                  for t in targets]
    channels: List[Channel] = []
    for v, name in enumerate(names):
        col = matrix[:, v]
        if clamps[v] is not None:
            col = np.clip(col, clamps[v][0], clamps[v][1])
        if not np.any(col > 1e-3):
            continue  # channel never fires; skip entirely
        idx = _rdp(times, col, epsilon)
        keys = [Keyframe(float(times[i]), round(float(col[i]), 4)) for i in idx]
        channels.append(Channel(name, keys))
    return FaceTrack(fps=fps, channels=channels,
                     target_set=None if targets is None else list(names))

openfacefx.mapping

Data-driven phoneme -> target mapping (FaceFX-style "mapping spreadsheet").

The built-in behavior maps each phoneme to exactly one Oculus-15 viseme at weight 1.0 (visemes.PHONEME_TO_VISEME). A Mapping generalizes that: any phoneme may drive any set of named targets with fractional weights, each target may declare an articulator class (used by the coarticulation model) and min/max clamps applied before keyframe reduction.

JSON file format (validated on load)::

{
  "format": "openfacefx.mapping",
  "version": 1,
  "targets": [
    {"name": "PP", "class": "lips", "min": 0.0, "max": 1.0},
    ...
  ],
  "phonemes": { "P": {"PP": 1.0}, "AY": {"aa": 0.7, "E": 0.3}, ... }
}

Mapping.default() reproduces the built-in table exactly — running without --mapping is bit-for-bit identical to previous releases.

ARTICULATOR_CLASSES = ('basic', 'jaw', 'lips', 'tongue') module-attribute

Target(name: str, articulator: str = 'basic', lo: float = 0.0, hi: float = 1.0) dataclass

name: str instance-attribute

articulator: str = 'basic' class-attribute instance-attribute

lo: float = 0.0 class-attribute instance-attribute

hi: float = 1.0 class-attribute instance-attribute

Mapping(targets: List[Target], rows: Dict[str, Dict[str, float]] = dict(), allow_custom_symbols: bool = False, normalize: Optional[Callable[[str], str]] = None) dataclass

targets: List[Target] instance-attribute

rows: Dict[str, Dict[str, float]] = field(default_factory=dict) class-attribute instance-attribute

allow_custom_symbols: bool = False class-attribute instance-attribute

normalize: Optional[Callable[[str], str]] = None class-attribute instance-attribute

target_names: List[str] property

row(phoneme: str) -> Dict[int, float]

Target-index -> weight for a (possibly stressed) phoneme. Unknown phonemes fall back to the silence row, like the built-in map. With allow_custom_symbols the key is matched verbatim (vendor symbols carry no stress digit and are case-significant), unless a normalize hook is set, which is applied to the key first.

Source code in src/openfacefx/mapping.py

def row(self, phoneme: str) -> Dict[int, float]:
    """Target-index -> weight for a (possibly stressed) phoneme.
    Unknown phonemes fall back to the silence row, like the built-in map.
    With ``allow_custom_symbols`` the key is matched verbatim (vendor
    symbols carry no stress digit and are case-significant), unless a
    ``normalize`` hook is set, which is applied to the key first."""
    if self.allow_custom_symbols:
        key = self.normalize(phoneme) if self.normalize is not None else phoneme
        row = self.rows.get(key)
        if row is None:
            row = self.rows.get(SILENCE) or {}
    else:
        key = strip_stress(phoneme).upper() if phoneme != SILENCE else SILENCE
        row = self.rows.get(key) or self.rows.get(SILENCE) or {}
    return {self.index[n]: w for n, w in row.items()}

default() -> 'Mapping' classmethod

Source code in src/openfacefx/mapping.py

@classmethod
def default(cls) -> "Mapping":
    targets = [Target(v, _DEFAULT_CLASSES.get(v, "basic")) for v in VISEMES]
    rows = {ph: {vis: 1.0} for ph, vis in PHONEME_TO_VISEME.items()}
    return cls(targets, rows)

from_json(path: str) -> 'Mapping' classmethod

Source code in src/openfacefx/mapping.py

@classmethod
def from_json(cls, path: str) -> "Mapping":
    with open(path, encoding="utf-8") as fh:
        try:
            d = json.load(fh)
        except json.JSONDecodeError as e:
            raise ValueError(f"{path}: not valid JSON ({e})") from None
    if d.get("format") != "openfacefx.mapping" or d.get("version") != 1:
        raise ValueError(
            f"{path}: expected format 'openfacefx.mapping' version 1")
    try:
        targets = [Target(t["name"], t.get("class", "basic"),
                          float(t.get("min", 0.0)), float(t.get("max", 1.0)))
                   for t in d["targets"]]
    except (KeyError, TypeError) as e:
        raise ValueError(f"{path}: malformed targets entry ({e})") from None
    phonemes = d.get("phonemes")
    if not isinstance(phonemes, dict) or not phonemes:
        raise ValueError(f"{path}: 'phonemes' must be a non-empty object")
    # "custom_symbols": true lets a mapping file key rows by a non-ARPABET
    # alphabet (SAMPA/IPA from TTS timing sources) — matched verbatim.
    return cls(targets, phonemes,
               allow_custom_symbols=bool(d.get("custom_symbols", False)))

to_json(path: str) -> None

Source code in src/openfacefx/mapping.py

def to_json(self, path: str) -> None:
    d = {
        "format": "openfacefx.mapping",
        "version": 1,
        "targets": [
            {"name": t.name, "class": t.articulator,
             "min": t.lo, "max": t.hi}
            for t in self.targets
        ],
        "phonemes": self.rows,
    }
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(d, fh, indent=2)
        fh.write("\n")

openfacefx.ipa

Built-in IPA -> Oculus-15 mapping preset (issue #32).

Piper and Cartesia timestamp their phonemes in IPA, and espeak-ng's MBROLA .pho dumps a SAMPA variant -- neither matches the ARPABET the default mapping expects, so from-timing used to relax those sources to silence unless the user hand-wrote a custom_symbols mapping. This module ships that mapping as data: IPA_MAPPING keys the Oculus-15 targets by the IPA inventory those engines actually emit, matched through _normalize_ipa so the diacritics real dumps carry collapse onto the base symbol instead of duplicating a table row per variant.

Normalization rules (_normalize_ipa), applied to the lookup key: * primary/secondary stress marks ˈ ˌ are dropped; * length marks ː ˑ are dropped (so ɑː matches ɑ); * the MFA-style secondary-articulation modifier letters ʰ ʲ ʷ are dropped (so pʰ tʲ kʷ match p t k); * every combining mark is dropped, which folds the affricate tie bar (t͡ʃ -> tʃ, matching the plain digraph too), the dental t̪ -> t, the syllabic n̩ -> n and nasalization ẽ -> e; * ASCII ' (stress) and : (X-SAMPA length) are dropped. It is idempotent and a no-op on ARPABET (which carries none of these), so the default pipeline is untouched.

Symbol inventory is grounded in verifiable sources

• espeak-ng's phoneme guide -- affricates written as tie bars (t͡ʃ d͡ʒ), stress ˈ ˌ, length ː ˑ (espeak-ng/espeak-ng docs/phonemes.md).
• the Montreal Forced Aligner US-English phone set, which Cartesia's sonic models use verbatim -- the aj aw ej ow ɔj diphthong spellings and the pʰ pʲ pʷ tʰ tʲ tʷ kʰ kʷ secondary articulations (docs.cartesia.ai, "Specify Custom Pronunciations").
• the English IPA key most G2P/TTS front-ends follow -- the aɪ aʊ eɪ oʊ ɔɪ diphthongs and ɜ ɝ ɚ r-coloured vowels (Wikipedia Help:IPA/English).

The IPA-symbol -> viseme assignment itself is our articulatory synthesis -- the same many-to-one judgement calls visemes.PHONEME_TO_VISEME documents for ARPABET, not a figure lifted from any single source.

IPA_VOWELS = frozenset(_VOWELS) | frozenset(_SAMPA_VOWELS) module-attribute

IPA_MAPPING = Mapping([(Target(v, _DEFAULT_CLASSES.get(v, 'basic'))) for v in VISEMES], _IPA_ROWS, allow_custom_symbols=True, normalize=_normalize_ipa) module-attribute

is_ipa_vowel(symbol: str) -> bool

True if a raw IPA/SAMPA token is a vowel (monophthong, diphthong or r-coloured), consulted by the coarticulation dominance model so vendor vowels get the broad vowel bump. Normalizes first (ˈaɪ, ɑː -> aɪ, ɑ). Returns False for every ARPABET symbol, so the ARPABET path is byte-for-byte unchanged.

Source code in src/openfacefx/ipa.py

def is_ipa_vowel(symbol: str) -> bool:
    """True if a raw IPA/SAMPA token is a vowel (monophthong, diphthong or
    r-coloured), consulted by the coarticulation dominance model so vendor
    vowels get the broad vowel bump. Normalizes first (``ˈaɪ``, ``ɑː`` ->
    ``aɪ``, ``ɑ``). Returns False for every ARPABET symbol, so the ARPABET path
    is byte-for-byte unchanged."""
    return _normalize_ipa(symbol) in IPA_VOWELS

ipa_unknown_symbols(symbols: Iterable[str]) -> List[str]

QA warnings for phoneme symbols the preset can't place -- they route to silence -- one line per distinct symbol (sorted), mirroring the vendor viseme path. A lone suprasegmental (a bare stress/length mark that normalizes to empty) is a structural token, not an unknown, so it never warns.

Source code in src/openfacefx/ipa.py

def ipa_unknown_symbols(symbols: Iterable[str]) -> List[str]:
    """QA warnings for phoneme symbols the preset can't place -- they route to
    silence -- one line per distinct symbol (sorted), mirroring the vendor
    viseme path. A lone suprasegmental (a bare stress/length mark that
    normalizes to empty) is a structural token, not an unknown, so it never
    warns."""
    counts: Dict[str, int] = {}
    for s in symbols:
        norm = _normalize_ipa(s)
        if norm and norm not in _IPA_ROWS:
            counts[s] = counts.get(s, 0) + 1
    return [f"unknown IPA symbol {s!r} ({n}x) routed to silence"
            for s, n in sorted(counts.items())]