JALI coarticulation rules

JALI (Edwards et al., SIGGRAPH 2016) publishes a concrete rule set and measured onset/decay constants that extend exactly the machinery OpenFaceFX already ships (Cohen–Massaro blending, per-class timing, bilabial closure). Issue #19 adopts them as a data-driven rule table (data/jali_rules.json) over the component stage.

It is entirely opt-in. With the default (JALI off) build_viseme_curves is byte-identical to before — the whole existing suite stays green and a diff against the released wheel is identical. Turn it on, and select rules, through CoartParams:

from openfacefx import CoartParams, build_viseme_curves, JALI_RULE_IDS

# all JALI rules + empirical timing
params = CoartParams(jali=True)
# or just a couple, individually toggled (JALI_RULE_IDS lists them all)
params = CoartParams(jali=True, jali_rules=("sibilant_jaw", "lip_heavy"))
times, matrix = build_viseme_curves(segments, mapping=mapping, params=params)

What it adds (the prioritised set)

• Hard constraints (post-blend forcings over articulator classes, each toggleable): bilabial_close (lips seal), labiodental_teeth (bottom lip to teeth), sibilant_jaw (sibilants narrow the jaw), nonnasal_lip_open (a non-nasal open segment opens the lips, so a neighbour's closure can't bleed across). When JALI is on these replace the legacy lips-only closure pass.
• Habits: duplicated_merge collapses adjacent same-viseme segments across a word boundary into one hold ("po_p m_an"); lip_heavy gives the rounded/ protruded visemes (UW OW OY w S Z J C) an earlier onset and longer hold; tongue_no_lip guarantees a tongue articulation (l n t d g k ng) never pulls a lip channel.
• Empirical timing (jali_timing, on by default when JALI is on): a per-phoneme, context-dependent onset/decay lookup — onset ~120 ms before the apex, tighter after a vowel (~60 ms) than after a pause (~120 ms), with a ~150 ms lip-protrusion extension — replacing the per-class lead constants.

Data, not code

The categories (phoneme sets), constraint floors/caps and timing constants live in data/jali_rules.json so new measurements drop in without touching code. The tongue articulator class was already in the mapping schema, so no schema version bump was needed.

Deferred (flagged for a follow-up)

Two lower-value habits — short-obstruent / nasal "leave-the-jaw-untouched" and word-final anticipatory lip shape — and the NVIDIA-A2F-style tongue-channel gain/offset mapping fields (with the ARKit tongue targets) are not in this pass. Adding the gain/offset fields would bump the mapping schema version; it is left out here so the schema and shipped ARKit preset stay byte-identical.

openfacefx.coart_jali

JALI coarticulation rules over the component stage (issue #19).

JALI (Edwards et al., SIGGRAPH 2016, https://www.dgp.toronto.edu/~elf/JALISIG16.pdf) publishes a concrete rule set and measured onset/decay constants that extend the machinery OpenFaceFX already ships (Cohen–Massaro blending, per-class timing, bilabial closure). This module encodes those rules as a data-driven table (data/jali_rules.json — plain data so new measurements drop in) evaluated over articulator classes, plus the empirical timing lookup.

Everything here is opt-in: it runs only when :attr:CoartParams.jali is set, so with the default (JALI off) :func:openfacefx.coarticulation.build_viseme_curves is byte-identical to before. Each constraint/habit is individually toggleable via :attr:CoartParams.jali_rules (None = all). Implemented (the issue's prioritised set):

• hard constraints — bilabial lip closure, labiodental bottom-lip-to-teeth, sibilant jaw narrowing, non-nasal lip opening;
• habits — duplicated-viseme merge across word boundaries ("po_p m_an"), lip-heavy anticipation/hysteresis (UW/OW/OY/w/S/Z/J/C start early & end late), tongue-only visemes never influence lip channels;
• empirical timing — per-phoneme, context-dependent onset/decay (post-pause vs post-vowel; a 150 ms lip-protrusion extension) replacing the per-class lead constants when :attr:CoartParams.jali_timing is on.

Deferred (flagged, not yet implemented): the short-obstruent / nasal "leave-the-jaw-untouched" and word-final anticipatory-lip habits, and the tongue-channel gain/offset mapping fields (NVIDIA A2F style) with the ARKit tongue targets — those touch the mapping schema / shipped presets and are left for a follow-up.

RULE_IDS: Tuple[str, ...] = ('bilabial_close', 'labiodental_teeth', 'sibilant_jaw', 'nonnasal_lip_open', 'duplicated_merge', 'lip_heavy', 'tongue_no_lip') module-attribute

load_rules() -> dict cached

The JALI rule table (cached). Plain JSON data — categories, constraints and the empirical timing constants.

Source code in src/openfacefx/coart_jali.py

@lru_cache(maxsize=1)
def load_rules() -> dict:
    """The JALI rule table (cached). Plain JSON data — categories, constraints and
    the empirical timing constants."""
    from importlib import resources
    with resources.files("openfacefx").joinpath(
            "data/jali_rules.json").open(encoding="utf-8") as fh:
        return json.load(fh)

rule_enabled(params, rule_id: str) -> bool

True if rule_id is active: JALI on and either all rules (jali_rules is None) or this id is in the selected set.

Source code in src/openfacefx/coart_jali.py

def rule_enabled(params, rule_id: str) -> bool:
    """True if ``rule_id`` is active: JALI on and either all rules (``jali_rules``
    is ``None``) or this id is in the selected set."""
    return bool(getattr(params, "jali", False)) and (
        params.jali_rules is None or rule_id in params.jali_rules)

merge_duplicates(segments: List[PhonemeSegment], mapping) -> List[PhonemeSegment]

Merge adjacent segments that map to the same viseme into one longer segment (JALI's duplicated-viseme merge) — the closing /p/ of "pop" and the /m/ of "man" are one bilabial hold, not two. Silence is never merged.

Source code in src/openfacefx/coart_jali.py

def merge_duplicates(segments: List[PhonemeSegment], mapping
                     ) -> List[PhonemeSegment]:
    """Merge adjacent segments that map to the **same** viseme into one longer
    segment (JALI's duplicated-viseme merge) — the closing /p/ of "pop" and the
    /m/ of "man" are one bilabial hold, not two. Silence is never merged."""
    out: List[PhonemeSegment] = []
    for s in segments:
        if (out and _viseme(out[-1], mapping) == _viseme(s, mapping)
                and _viseme(s, mapping) != "sil"):
            p = out[-1]
            out[-1] = PhonemeSegment(p.phoneme, p.start, s.end)
        else:
            out.append(s)
    return out

timing_leads(segments: List[PhonemeSegment], params) -> Tuple[np.ndarray, np.ndarray]

Per-segment (lead_in, lead_out) influence extents (seconds) from the empirical table, replacing the per-class lead constants. Onset is context-dependent — longer after a pause, tighter after a vowel — and the lip-heavy habit extends both sides by the lip-protrusion constant.

Source code in src/openfacefx/coart_jali.py

def timing_leads(segments: List[PhonemeSegment], params
                 ) -> Tuple[np.ndarray, np.ndarray]:
    """Per-segment ``(lead_in, lead_out)`` influence extents (seconds) from the
    empirical table, replacing the per-class ``lead`` constants. Onset is
    context-dependent — longer after a pause, tighter after a vowel — and the
    lip-heavy habit extends both sides by the lip-protrusion constant."""
    t = load_rules()["timing"]
    cats = load_rules()["categories"]
    n = len(segments)
    lead_in = np.full(n, float(t["onset"]))
    lead_out = np.full(n, float(t["decay"]))
    lip_heavy = rule_enabled(params, "lip_heavy")
    for i, s in enumerate(segments):
        prev = segments[i - 1] if i > 0 else None
        if prev is None or prev.phoneme == SILENCE:
            lead_in[i] = t["post_pause_onset"]
        elif is_vowel(prev.phoneme):
            lead_in[i] = t["post_vowel_onset"]
        if lip_heavy and _phon(s) in cats["lip_heavy"]:
            lead_in[i] += t["lip_protrusion_ext"]
            lead_out[i] += t["lip_protrusion_ext"]
    return lead_in, lead_out

mask_tongue_lips(weights: np.ndarray, segments: List[PhonemeSegment], mapping, names: List[str]) -> None

Zero every tongue-class segment's weight on lip-class targets, in place — a tongue articulation (l/n/t/d/g/k/ng) must not pull the lips.

Source code in src/openfacefx/coart_jali.py

def mask_tongue_lips(weights: np.ndarray, segments: List[PhonemeSegment],
                     mapping, names: List[str]) -> None:
    """Zero every tongue-class segment's weight on lip-class targets, in place —
    a tongue articulation (l/n/t/d/g/k/ng) must not pull the lips."""
    classes = _target_classes(names, mapping)
    lip_cols = [j for j, c in enumerate(classes) if c == "lips"]
    if not lip_cols:
        return
    for i, s in enumerate(segments):
        if _seg_class(s, mapping) == "tongue":
            for j in lip_cols:
                weights[i, j] = 0.0

apply_constraints(times: np.ndarray, matrix: np.ndarray, segments: List[PhonemeSegment], mapping, weights: np.ndarray, names: List[str], params) -> None

Apply the enabled hard constraints to the dense matrix, in place. Called from :func:coarticulation._blend in place of the legacy _enforce_closures when JALI is on.

Source code in src/openfacefx/coart_jali.py

def apply_constraints(times: np.ndarray, matrix: np.ndarray,
                      segments: List[PhonemeSegment], mapping,
                      weights: np.ndarray, names: List[str], params) -> None:
    """Apply the enabled hard constraints to the dense matrix, in place. Called
    from :func:`coarticulation._blend` in place of the legacy
    ``_enforce_closures`` when JALI is on."""
    rules = load_rules()
    cats, cons = rules["categories"], rules["constraints"]
    classes = _target_classes(names, mapping)
    jaw_cols = [j for j, c in enumerate(classes) if c == "jaw"]
    lip_cols = [j for j, c in enumerate(classes) if c == "lips"]
    for i, s in enumerate(segments):
        ph = _phon(s)
        centre = (s.start + s.end) / 2.0
        sel = _selection(times, centre, max(s.dur * 0.25, 1.0 / 120.0))
        # constraints 1 & 2: bilabial / labiodental lip closure
        for cid, cat in (("bilabial_close", "bilabial"),
                         ("labiodental_teeth", "labiodental")):
            if rule_enabled(params, cid) and ph in cats[cat]:
                _force_closure(matrix, weights, i, sel, cons[cid]["floor"])
        # constraint 3: sibilants narrow the jaw across the segment
        if rule_enabled(params, "sibilant_jaw") and ph in cats["sibilant"] and jaw_cols:
            cap = cons["sibilant_jaw"]["jaw_cap"]
            span = (times >= s.start) & (times <= s.end)
            for j in jaw_cols:
                matrix[span, j] = np.minimum(matrix[span, j], cap)
        # constraint 4: non-nasal open segments open the lips (cap closed-lip
        # targets so a neighbour's closure does not bleed across)
        if (rule_enabled(params, "nonnasal_lip_open") and lip_cols
                and ph not in cats["nasal"] and ph not in cats["bilabial"]
                and ph not in cats["labiodental"]):
            cap = cons["nonnasal_lip_open"]["lip_cap"]
            for j in lip_cols:
                matrix[sel, j] = np.minimum(matrix[sel, j], cap)