JALI coarticulation rules

JALI (Edwards et al., SIGGRAPH 2016) publishes a concrete rule set and measured onset/decay constants that extend exactly the machinery OpenFaceFX already ships (Cohen–Massaro blending, per-class timing, bilabial closure). Issue #19 adopts them as a data-driven rule table (data/jali_rules.json) over the component stage.

It is entirely opt-in. With the default (JALI off) build_viseme_curves is byte-identical to before — the whole existing suite stays green and a diff against the released wheel is identical. Turn it on, and select rules, through CoartParams:

from openfacefx import CoartParams, build_viseme_curves, JALI_RULE_IDS

# all JALI rules + empirical timing
params = CoartParams(jali=True)
# or just a couple, individually toggled (JALI_RULE_IDS lists them all)
params = CoartParams(jali=True, jali_rules=("sibilant_jaw", "lip_heavy"))
times, matrix = build_viseme_curves(segments, mapping=mapping, params=params)

What it adds (the prioritised set)

• Hard constraints (post-blend forcings over articulator classes, each toggleable): bilabial_close (lips seal), labiodental_teeth (bottom lip to teeth), sibilant_jaw (sibilants narrow the jaw), nonnasal_lip_open (a non-nasal open segment opens the lips, so a neighbour's closure can't bleed across). When JALI is on these replace the legacy lips-only closure pass.
• Habits: duplicated_merge collapses adjacent same-viseme segments across a word boundary into one hold ("po_p m_an"); lip_heavy gives the rounded/ protruded visemes (UW OW OY w S Z J C) an earlier onset and longer hold; tongue_no_lip guarantees a tongue articulation (l n t d g k ng) never pulls a lip channel; short_no_jaw (#53) holds the jaw at the neighbouring level through a short obstruent or nasal so a quick stop/nasal can't dip it; and wordfinal_lip (#53) gives a word-final lip-shaped phoneme an earlier onset so its lip shape anticipates (word-final is approximated as pre-silence / utterance-final, as the phoneme stream carries no inter-word boundaries).
• Empirical timing (jali_timing, on by default when JALI is on): a per-phoneme, context-dependent onset/decay lookup — onset ~120 ms before the apex, tighter after a vowel (~60 ms) than after a pause (~120 ms), with a ~150 ms lip-protrusion extension — replacing the per-class lead constants.

Data, not code

The categories (phoneme sets), constraint floors/caps, habit thresholds and timing constants live in data/jali_rules.json so new measurements drop in without touching code. The tongue articulator class was already in the mapping schema; issue #53 adds optional per-target gain/offset fields (NVIDIA-A2F-style channel tuning), which bump the mapping schema to version 2 — version-1 files still load, the absent fields reading as the no-op defaults.

Follow-up (#53)

Issue #53 completes the two remaining JALI habits above — short_no_jaw and wordfinal_lip, both off by default and byte-identical when off — and adds the tongue-channel gain/offset mapping fields (schema v2, applied at keyframe reduction; see openfacefx.mapping and reduce_to_track). At the no-op defaults (gain=1.0, offset=0.0) a mapping is byte-identical to before.

openfacefx.coart_jali

JALI coarticulation rules over the component stage (issue #19).

JALI (Edwards et al., SIGGRAPH 2016, https://www.dgp.toronto.edu/~elf/JALISIG16.pdf) publishes a concrete rule set and measured onset/decay constants that extend the machinery OpenFaceFX already ships (Cohen–Massaro blending, per-class timing, bilabial closure). This module encodes those rules as a data-driven table (data/jali_rules.json — plain data so new measurements drop in) evaluated over articulator classes, plus the empirical timing lookup.

Everything here is opt-in: it runs only when :attr:CoartParams.jali is set, so with the default (JALI off) :func:openfacefx.coarticulation.build_viseme_curves is byte-identical to before. Each constraint/habit is individually toggleable via :attr:CoartParams.jali_rules (None = all). Implemented (the issue's prioritised set):

• hard constraints — bilabial lip closure, labiodental bottom-lip-to-teeth, sibilant jaw narrowing, non-nasal lip opening;
• habits — duplicated-viseme merge across word boundaries ("po_p m_an"), lip-heavy anticipation/hysteresis (UW/OW/OY/w/S/Z/J/C start early & end late), tongue-only visemes never influence lip channels, short obstruents/nasals leave the jaw untouched, and a word-final lip shape anticipates (#53);
• empirical timing — per-phoneme, context-dependent onset/decay (post-pause vs post-vowel; a 150 ms lip-protrusion extension) replacing the per-class lead constants when :attr:CoartParams.jali_timing is on.

The NVIDIA-A2F-style per-target tongue gain/offset fields (#53) live in the mapping schema (:mod:openfacefx.mapping) and are applied at keyframe reduction, not here.

RULE_IDS: Tuple[str, ...] = ('bilabial_close', 'labiodental_teeth', 'sibilant_jaw', 'nonnasal_lip_open', 'duplicated_merge', 'lip_heavy', 'tongue_no_lip', 'short_no_jaw', 'wordfinal_lip') module-attribute

load_rules() -> dict cached

The JALI rule table (cached). Plain JSON data — categories, constraints and the empirical timing constants.

Source code in src/openfacefx/coart_jali.py

@lru_cache(maxsize=1)
def load_rules() -> dict:
    """The JALI rule table (cached). Plain JSON data — categories, constraints and
    the empirical timing constants."""
    from importlib import resources
    with resources.files("openfacefx").joinpath(
            "data/jali_rules.json").open(encoding="utf-8") as fh:
        return json.load(fh)

rule_enabled(params, rule_id: str) -> bool

True if rule_id is active: JALI on and either all rules (jali_rules is None) or this id is in the selected set.

Source code in src/openfacefx/coart_jali.py

def rule_enabled(params, rule_id: str) -> bool:
    """True if ``rule_id`` is active: JALI on and either all rules (``jali_rules``
    is ``None``) or this id is in the selected set."""
    return bool(getattr(params, "jali", False)) and (
        params.jali_rules is None or rule_id in params.jali_rules)

merge_duplicates(segments: List[PhonemeSegment], mapping) -> List[PhonemeSegment]

Merge adjacent segments that map to the same viseme into one longer segment (JALI's duplicated-viseme merge) — the closing /p/ of "pop" and the /m/ of "man" are one bilabial hold, not two. Silence is never merged.

Source code in src/openfacefx/coart_jali.py

def merge_duplicates(segments: List[PhonemeSegment], mapping
                     ) -> List[PhonemeSegment]:
    """Merge adjacent segments that map to the **same** viseme into one longer
    segment (JALI's duplicated-viseme merge) — the closing /p/ of "pop" and the
    /m/ of "man" are one bilabial hold, not two. Silence is never merged."""
    out: List[PhonemeSegment] = []
    for s in segments:
        if (out and _viseme(out[-1], mapping) == _viseme(s, mapping)
                and _viseme(s, mapping) != "sil"):
            p = out[-1]
            out[-1] = PhonemeSegment(p.phoneme, p.start, s.end)
        else:
            out.append(s)
    return out

timing_leads(segments: List[PhonemeSegment], params) -> Tuple[np.ndarray, np.ndarray]

Per-segment (lead_in, lead_out) influence extents (seconds) from the empirical table, replacing the per-class lead constants. Onset is context-dependent — longer after a pause, tighter after a vowel — the lip-heavy habit extends both sides by the lip-protrusion constant, and the word-final habit (#53) extends the onset of a word-final lip-shaped phoneme so its lip shape anticipates.

Source code in src/openfacefx/coart_jali.py

def timing_leads(segments: List[PhonemeSegment], params
                 ) -> Tuple[np.ndarray, np.ndarray]:
    """Per-segment ``(lead_in, lead_out)`` influence extents (seconds) from the
    empirical table, replacing the per-class ``lead`` constants. Onset is
    context-dependent — longer after a pause, tighter after a vowel — the
    lip-heavy habit extends both sides by the lip-protrusion constant, and the
    word-final habit (#53) extends the onset of a word-final lip-shaped phoneme so
    its lip shape anticipates."""
    rules = load_rules()
    t = rules["timing"]
    cats = rules["categories"]
    habits = rules.get("habits", {})
    n = len(segments)
    lead_in = np.full(n, float(t["onset"]))
    lead_out = np.full(n, float(t["decay"]))
    lip_heavy = rule_enabled(params, "lip_heavy")
    wordfinal = rule_enabled(params, "wordfinal_lip")
    wf_ext = float(habits.get("wordfinal_lip", {}).get("onset_ext", 0.0))
    for i, s in enumerate(segments):
        prev = segments[i - 1] if i > 0 else None
        if prev is None or prev.phoneme == SILENCE:
            lead_in[i] = t["post_pause_onset"]
        elif is_vowel(prev.phoneme):
            lead_in[i] = t["post_vowel_onset"]
        if lip_heavy and _phon(s) in cats["lip_heavy"]:
            lead_in[i] += t["lip_protrusion_ext"]
            lead_out[i] += t["lip_protrusion_ext"]
        # word-final anticipatory lip shape (#53): a lip-shaped (lip_heavy)
        # phoneme that ends a word — next segment is silence, or it is the last
        # segment — forms its lip shape early, so extend the onset lead-in.
        # Word-final is approximated as pre-silence / utterance-final because the
        # phoneme stream carries no inter-word boundaries.
        if wordfinal and _phon(s) in cats["lip_heavy"]:
            nxt = segments[i + 1] if i + 1 < n else None
            if nxt is None or nxt.phoneme == SILENCE:
                lead_in[i] += wf_ext
    return lead_in, lead_out

mask_tongue_lips(weights: np.ndarray, segments: List[PhonemeSegment], mapping, names: List[str]) -> None

Zero every tongue-class segment's weight on lip-class targets, in place — a tongue articulation (l/n/t/d/g/k/ng) must not pull the lips.

Source code in src/openfacefx/coart_jali.py

def mask_tongue_lips(weights: np.ndarray, segments: List[PhonemeSegment],
                     mapping, names: List[str]) -> None:
    """Zero every tongue-class segment's weight on lip-class targets, in place —
    a tongue articulation (l/n/t/d/g/k/ng) must not pull the lips."""
    classes = _target_classes(names, mapping)
    lip_cols = [j for j, c in enumerate(classes) if c == "lips"]
    if not lip_cols:
        return
    for i, s in enumerate(segments):
        if _seg_class(s, mapping) == "tongue":
            for j in lip_cols:
                weights[i, j] = 0.0

apply_constraints(times: np.ndarray, matrix: np.ndarray, segments: List[PhonemeSegment], mapping, weights: np.ndarray, names: List[str], params) -> None

Apply the enabled hard constraints (and the #53 short-obstruent jaw habit) to the dense matrix, in place. Called from :func:coarticulation._blend in place of the legacy _enforce_closures when JALI is on.

Source code in src/openfacefx/coart_jali.py

def apply_constraints(times: np.ndarray, matrix: np.ndarray,
                      segments: List[PhonemeSegment], mapping,
                      weights: np.ndarray, names: List[str], params) -> None:
    """Apply the enabled hard constraints (and the #53 short-obstruent jaw habit)
    to the dense matrix, in place. Called from :func:`coarticulation._blend` in
    place of the legacy ``_enforce_closures`` when JALI is on."""
    rules = load_rules()
    cats, cons = rules["categories"], rules["constraints"]
    habits = rules.get("habits", {})
    short_max_dur = float(habits.get("short_no_jaw", {}).get("max_dur", 0.0))
    classes = _target_classes(names, mapping)
    jaw_cols = [j for j, c in enumerate(classes) if c == "jaw"]
    lip_cols = [j for j, c in enumerate(classes) if c == "lips"]
    for i, s in enumerate(segments):
        ph = _phon(s)
        centre = (s.start + s.end) / 2.0
        # searchsorted index ranges over the sorted ``times`` grid replace the
        # per-segment O(frames) boolean masks (perf pass, #59-era `short_no_jaw`
        # already used the idiom); each range is the exact same frame set.
        lo, hi = _sel_range(times, centre, max(s.dur * 0.25, 1.0 / 120.0))
        # constraints 1 & 2: bilabial / labiodental lip closure
        for cid, cat in (("bilabial_close", "bilabial"),
                         ("labiodental_teeth", "labiodental")):
            if rule_enabled(params, cid) and ph in cats[cat]:
                _force_closure(matrix, weights, i, lo, hi, cons[cid]["floor"])
        # constraint 3: sibilants narrow the jaw across the segment
        if rule_enabled(params, "sibilant_jaw") and ph in cats["sibilant"] and jaw_cols:
            cap = cons["sibilant_jaw"]["jaw_cap"]
            a = int(np.searchsorted(times, s.start, "left"))
            b = int(np.searchsorted(times, s.end, "right"))
            for j in jaw_cols:
                matrix[a:b, j] = np.minimum(matrix[a:b, j], cap)
        # habit (#53): a SHORT obstruent/nasal leaves the jaw untouched — floor the
        # jaw over the segment to the neighbouring (vowel) level so a quick
        # stop/nasal cannot dip it (the jaw is slow; it rides through). Only bites
        # when a jaw target is held either side; a no-op next to a closed jaw or on
        # the built-in map, where consonants carry no jaw weight.
        if (rule_enabled(params, "short_no_jaw") and jaw_cols
                and s.dur < short_max_dur
                and (ph in cats["obstruent"] or ph in cats["nasal"])):
            a = int(np.searchsorted(times, s.start, "left"))
            b = int(np.searchsorted(times, s.end, "right"))
            if b > a:
                blo = max(int(np.searchsorted(times, s.start)) - 1, 0)
                bhi = min(int(np.searchsorted(times, s.end)), len(times) - 1)
                for j in jaw_cols:
                    hold = min(matrix[blo, j], matrix[bhi, j])
                    matrix[a:b, j] = np.maximum(matrix[a:b, j], hold)
        # constraint 4: non-nasal open segments open the lips (cap closed-lip
        # targets so a neighbour's closure does not bleed across)
        if (rule_enabled(params, "nonnasal_lip_open") and lip_cols
                and ph not in cats["nasal"] and ph not in cats["bilabial"]
                and ph not in cats["labiodental"]):
            cap = cons["nonnasal_lip_open"]["lip_cap"]
            for j in lip_cols:
                matrix[lo:hi, j] = np.minimum(matrix[lo:hi, j], cap)