Emotion & expression layer

An additive emotion/expression layer baked over a speech-solved track (issue

38). Production rigs keep expression on a separate additive layer over lip-sync

and add it onto the base at runtime — SALSA's EmoteR blends emphasis-timed emotes over speech, and Unreal additive animation is defined as the difference between a pose and a reference (T/A) pose added onto the base. This module does the same in pure numpy: an authored emotion envelope becomes a true additive delta channel_value - reference_value, is added onto the speech-solved channels, scaled by a global intensity dial, clamped per channel and re-thinned. The result is an ordinary FaceTrack that exports through every existing exporter unchanged.

Authoring modes

An envelope (format: "openfacefx.emotion", version: 1) carries one of two input modes:

• channels — direct emotion-channel keyframes, authored on the timeline like any curve:

  { "format": "openfacefx.emotion", "version": 1, "mode": "channels",
    "reference": { "smile": 0.1 },
    "clamps":    { "smile": [0.0, 1.0] },
    "channels": {
      "smile":      [[0.0, 0.0], [0.6, 0.8], [1.4, 0.0]],
      "brow_raise": [[0.0, 0.0], [0.6, 0.5], [1.4, 0.0]]
    } }
  

• valence_arousal — a compact valence/arousal keyframe track (both in [-1, 1]) mapped through the fixed, hand-authored table below:

  { "format": "openfacefx.emotion", "version": 1, "mode": "valence_arousal",
    "va": {
      "valence": [[0.0, 0.0], [0.75, 1.0], [1.5, 0.3]],
      "arousal": [[0.0, 0.2], [0.75, 0.9], [1.5, 0.4]]
    } }
  

reference is the neutral/rest pose the additive delta is measured against; a channel absent from it rests at 0. clamps optionally bounds a channel's baked output to [lo, hi] (0 <= lo <= hi <= 1).

The valence/arousal table

Valence and arousal are each sampled at three nodes, VA_AXIS = (-1, 0, +1), and a query point is bilinearly interpolated inside the resulting 3×3 grid — a table lookup and interpolation only, no ML. Because 0 is a real node on each axis, neutral affect valence = arousal = 0 maps to an all-zero pose (a true no-op). The weights follow the circumplex model of affect and FACS: pleasant valence drives a zygomatic smile with a Duchenne cheek_raise; unpleasant valence with rising arousal drives a corrugator brow_lower (anger) over a mouth-corner-down frown; high arousal at neutral valence reads as a raised, surprised brow_raise.

VA_TABLE[channel][i_v][i_a] is the channel weight at valence = VA_AXIS[i_v] and arousal = VA_AXIS[i_a] (rows run valence −1 / 0 / +1, columns arousal −1 / 0 / +1):

channel	v−1,a−1	v−1,a0	v−1,a+1	v0,a−1	v0,a0	v0,a+1	v+1,a−1	v+1,a0	v+1,a+1
smile	0.00	0.00	0.00	0.00	0.00	0.00	0.45	0.70	0.90
cheek_raise	0.00	0.00	0.00	0.00	0.00	0.00	0.30	0.55	0.80
brow_raise	0.25	0.05	0.10	0.00	0.00	0.70	0.00	0.10	0.30
brow_lower	0.10	0.45	0.85	0.00	0.00	0.00	0.00	0.00	0.00
frown	0.70	0.55	0.55	0.00	0.00	0.00	0.00	0.00	0.00

va_to_pose(valence, arousal) returns the interpolated weights for one point; values outside [-1, 1] clamp to the nearest edge.

Baking

bake_emotion(track, envelope, *, intensity=1.0, clamps=None, eps=0.015) resamples the base curve and the delta onto a shared grid via edits.sample (the piecewise-linear delta primitive), adds base + intensity * (pose - reference), clamps to each channel's [lo, hi] and re-thins with the shared RDP thinner. Channels the base track lacks are appended and their names added to target_set; channels it already has are updated in place.

The curve exporters (JSON, CSV, Unity .anim, Godot .tres, Live2D .motion3.json) carry every channel, including the emotion channels; the mouth-only cue exporters (Rhubarb, Moho/OpenToonz, Papagayo) and the .lip writer ignore the recognised expression channels (smile, cheek_raise, brow_raise, brow_lower, frown) exactly as they ignore gesture channels, and --retarget passes them through unchanged.

Invariants

• Additive / opt-in: with intensity == 0, no emotion channels, a neutral valence/arousal track, or an exactly-zero delta on every channel, the input track is returned byte-identical.
• Deterministic: numpy interp/clip + the shared RDP thinner and a fixed table, no RNG and no wall-clock — identical on Python 3.9/3.13.
• intensity scales the delta linearly; the delta encoding round-trips (base + (pose − reference) reconstructs the target pose within float tolerance).

CLI

# generate a speech track, then bake emotion over it
python -m openfacefx naive --text "..." --duration 1.5 -o base.json
python -m openfacefx emotion base.json happy.emotion.json --intensity 1.0 -o baked.json
# --intensity 0 (or a neutral envelope) => baked.json is byte-identical to base.json

The emotion command writes any of the usual output formats (.json/.csv/.anim/.tres/.motion3.json/cue formats) and composes with --retarget/--adjust. Per-channel clamps can be set with a repeatable --clamp CHANNEL LO HI.

openfacefx.emotion

Additive emotion/expression layer baked over speech (issue #38).

Production rigs keep expression on a separate additive layer over lip-sync and add it onto the base at runtime: SALSA's EmoteR blends emphasis-timed emotes over speech (https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/), and Unreal additive animation is defined as the difference between a pose and a reference (T/A) pose added onto the base (https://mocaponline.com/blogs/mocap-news/animation-layers-guide). This module does the same in pure numpy: an authored emotion envelope becomes a true additive delta channel_value - reference_value and is added onto the speech-solved channels, scaled by a global intensity dial, clamped per channel, and re-thinned.

Two authoring modes (mirroring the two ways a writer thinks about affect):

• channels -- direct emotion-channel keyframes (smile/frown/ brow_raise ...), authored on the timeline like any curve.
• valence_arousal -- a compact valence/arousal keyframe track (both in [-1, 1]) mapped through the FIXED, hand-authored :data:VA_TABLE by bilinear interpolation. No ML: a table lookup and interpolation only, so the circumplex centre valence=arousal=0 is the neutral node and maps to an all-zero pose.

The bake (:func:bake_emotion) resamples the base curve and the delta onto a common grid via :func:openfacefx.edits.sample (piecewise-linear np.interp, the delta primitive), adds base + intensity * (pose - reference), clamps to each channel's [lo, hi] (reusing the edits clamp conventions), and re-thins with :func:openfacefx.curves._rdp. The result is an ordinary :class:~openfacefx.curves.FaceTrack that exports through every existing exporter unchanged.

Additive / opt-in: with no emotion, an all-zero delta, or intensity=0 the input track is returned untouched -- byte-identical output. Deterministic: numpy interp/clip + the shared RDP thinner and a fixed table, no RNG and no wall-clock, identical on Python 3.9/3.13. numpy + stdlib only.

FORMAT = 'openfacefx.emotion' module-attribute

VERSION = 1 module-attribute

VA_AXIS: Tuple[float, float, float] = (-1.0, 0.0, 1.0) module-attribute

VA_EMOTION_CHANNELS: Tuple[str, ...] = ('smile', 'cheek_raise', 'brow_raise', 'brow_lower', 'frown') module-attribute

VA_TABLE: Dict[str, List[List[float]]] = {'smile': [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.45, 0.7, 0.9]], 'cheek_raise': [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.3, 0.55, 0.8]], 'brow_raise': [[0.25, 0.05, 0.1], [0.0, 0.0, 0.7], [0.0, 0.1, 0.3]], 'brow_lower': [[0.1, 0.45, 0.85], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]], 'frown': [[0.7, 0.55, 0.55], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]} module-attribute

EmotionEnvelope(mode: str, channels: Dict[str, List[List[float]]] = dict(), va: Dict[str, List[List[float]]] = dict(), reference: Dict[str, float] = dict(), clamps: Dict[str, List[float]] = dict(), fps: float = 60.0) dataclass

A parsed openfacefx.emotion envelope.

Two input modes select which field carries the authored track:

• mode="channels" -- channels maps an emotion-channel name to a keyframe list [[t, v], ...] authored directly.
• mode="valence_arousal" -- va holds valence and/or arousal keyframe lists (values in [-1, 1]); :data:VA_TABLE maps them onto the channels in :data:VA_EMOTION_CHANNELS.

reference is the neutral/rest pose the additive delta is measured against (channel_value - reference_value); a channel absent from it rests at 0. clamps optionally bounds a channel's baked output to [lo, hi] (0 <= lo <= hi <= 1). fps is the grid rate the valence/arousal track is sampled on when baking.

mode: str instance-attribute

channels: Dict[str, List[List[float]]] = field(default_factory=dict) class-attribute instance-attribute

va: Dict[str, List[List[float]]] = field(default_factory=dict) class-attribute instance-attribute

reference: Dict[str, float] = field(default_factory=dict) class-attribute instance-attribute

clamps: Dict[str, List[float]] = field(default_factory=dict) class-attribute instance-attribute

fps: float = 60.0 class-attribute instance-attribute

to_dict() -> Dict

Source code in src/openfacefx/emotion.py

def to_dict(self) -> Dict:
    d: Dict = {"format": FORMAT, "version": VERSION, "mode": self.mode,
               "fps": self.fps}
    if self.mode == "channels":
        d["channels"] = self.channels
    else:
        d["va"] = self.va
    if self.reference:
        d["reference"] = self.reference
    if self.clamps:
        d["clamps"] = self.clamps
    return d

from_dict(d: Dict) -> 'EmotionEnvelope' classmethod

Parse and validate an envelope dict, raising ValueError with a clear message on any malformed field.

Source code in src/openfacefx/emotion.py

@classmethod
def from_dict(cls, d: Dict) -> "EmotionEnvelope":
    """Parse and validate an envelope dict, raising ``ValueError`` with a
    clear message on any malformed field."""
    if d.get("format") != FORMAT or d.get("version") != VERSION:
        raise ValueError(
            f"expected format {FORMAT!r} version {VERSION}, got "
            f"{d.get('format')!r} version {d.get('version')!r}")
    mode = d.get("mode")
    if mode not in _MODES:
        raise ValueError(
            f"emotion 'mode' must be one of {_MODES}, got {mode!r}")
    fps = d.get("fps", 60.0)
    if not _finite(fps) or fps <= 0.0:
        raise ValueError(f"emotion 'fps' must be a positive number, got {fps!r}")
    channels: Dict[str, List[List[float]]] = {}
    va: Dict[str, List[List[float]]] = {}
    if mode == "channels":
        channels = _validate_emotion_channels(d.get("channels"))
    else:
        va = _validate_va(d.get("va"))
    return cls(
        mode=mode, channels=channels, va=va,
        reference=_validate_reference(d.get("reference")),
        clamps=_validate_clamps(d.get("clamps")),
        fps=float(fps),
    )

va_to_pose(valence: float, arousal: float) -> Dict[str, float]

Map one (valence, arousal) point in [-1, 1]^2 to emotion-channel weights by bilinear interpolation over :data:VA_TABLE.

Deterministic and reproducible -- a table lookup and interpolation, no ML. Values outside [-1, 1] clamp to the nearest edge. va_to_pose(0, 0) is the all-zero neutral pose.

Source code in src/openfacefx/emotion.py

def va_to_pose(valence: float, arousal: float) -> Dict[str, float]:
    """Map one ``(valence, arousal)`` point in ``[-1, 1]^2`` to emotion-channel
    weights by bilinear interpolation over :data:`VA_TABLE`.

    Deterministic and reproducible -- a table lookup and interpolation, no ML.
    Values outside ``[-1, 1]`` clamp to the nearest edge. ``va_to_pose(0, 0)`` is
    the all-zero neutral pose."""
    v = np.asarray([valence], dtype=float)
    a = np.asarray([arousal], dtype=float)
    return {ch: float(_bilerp(np.asarray(VA_TABLE[ch], dtype=float), v, a)[0])
            for ch in VA_EMOTION_CHANNELS}

save_envelope(env: EmotionEnvelope, path: str) -> None

Write an envelope as pretty JSON (2-space indent, trailing newline).

Source code in src/openfacefx/emotion.py

def save_envelope(env: EmotionEnvelope, path: str) -> None:
    """Write an envelope as pretty JSON (2-space indent, trailing newline)."""
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(env.to_dict(), fh, indent=2)
        fh.write("\n")

load_envelope(path: str) -> EmotionEnvelope

Read and validate an envelope from path.

Source code in src/openfacefx/emotion.py

def load_envelope(path: str) -> EmotionEnvelope:
    """Read and validate an envelope from ``path``."""
    with open(path, encoding="utf-8") as fh:
        try:
            d = json.load(fh)
        except json.JSONDecodeError as e:
            raise ValueError(f"{path}: not valid JSON ({e})") from None
    return EmotionEnvelope.from_dict(d)

bake_emotion(track: FaceTrack, envelope, *, intensity: float = 1.0, clamps: Optional[Dict[str, Tuple[float, float]]] = None, eps: float = 0.015) -> FaceTrack

Bake an additive emotion envelope onto a speech-solved track.

envelope is an :class:EmotionEnvelope or a raw dict (parsed and validated). For each emotion channel the additive delta pose - reference is resampled onto a grid shared with the base curve, scaled by intensity, added, clamped to the channel's [lo, hi] and re-thinned with the shared RDP thinner at eps. Channels the base track lacks are appended and their names added to target_set; channels it already has are updated in place.

clamps maps channel -> (lo, hi) and overrides the envelope's per-channel clamps (default [0, 1]). intensity scales the delta linearly and must be finite and >= 0.

Additive / opt-in: with intensity == 0, no emotion channels, or an exactly-zero delta on every channel, the input track is returned unchanged -- byte-identical output. Deterministic; the result is an ordinary :class:~openfacefx.curves.FaceTrack that exports through every exporter.

Source code in src/openfacefx/emotion.py

def bake_emotion(track: FaceTrack, envelope, *, intensity: float = 1.0,
                 clamps: Optional[Dict[str, Tuple[float, float]]] = None,
                 eps: float = 0.015) -> FaceTrack:
    """Bake an additive emotion ``envelope`` onto a speech-solved ``track``.

    ``envelope`` is an :class:`EmotionEnvelope` or a raw dict (parsed and
    validated). For each emotion channel the additive delta ``pose - reference``
    is resampled onto a grid shared with the base curve, scaled by ``intensity``,
    added, clamped to the channel's ``[lo, hi]`` and re-thinned with the shared
    RDP thinner at ``eps``. Channels the base track lacks are appended and their
    names added to ``target_set``; channels it already has are updated in place.

    ``clamps`` maps ``channel -> (lo, hi)`` and overrides the envelope's per-channel
    ``clamps`` (default ``[0, 1]``). ``intensity`` scales the delta linearly and
    must be finite and ``>= 0``.

    Additive / opt-in: with ``intensity == 0``, no emotion channels, or an
    exactly-zero delta on every channel, the input ``track`` is returned
    unchanged -- byte-identical output. Deterministic; the result is an ordinary
    :class:`~openfacefx.curves.FaceTrack` that exports through every exporter."""
    if not _finite(intensity) or intensity < 0.0:
        raise ValueError(f"intensity must be a finite number >= 0, got {intensity!r}")
    if not _finite(eps) or eps < 0.0:
        raise ValueError(f"eps must be a finite number >= 0, got {eps!r}")
    env = (envelope if isinstance(envelope, EmotionEnvelope)
           else EmotionEnvelope.from_dict(envelope))
    override = _validate_clamps_param(clamps)

    if intensity == 0.0:
        return track
    deltas = _emotion_deltas(env, track.duration)
    if not deltas:
        return track

    def _clamp_for(name: str) -> Tuple[float, float]:
        c = override.get(name) or env.clamps.get(name) or [0.0, 1.0]
        return float(c[0]), float(c[1])

    base_names = {c.name for c in track.channels}
    out_channels: List[Channel] = []
    modified = False
    for ch in track.channels:
        d = deltas.get(ch.name)
        if d is None:
            out_channels.append(ch)               # untouched speech channel
            continue
        new_ch, changed = _bake_channel(ch, d, intensity, _clamp_for(ch.name), eps)
        out_channels.append(new_ch if changed else ch)
        modified = modified or changed

    added: List[Channel] = []
    for name in sorted(set(deltas) - base_names):
        ch = _bake_new_channel(name, deltas[name], intensity, _clamp_for(name), eps)
        if ch is not None:
            added.append(ch)

    if not modified and not added:
        return track                              # nothing fired -> byte-identical

    out_channels.extend(added)
    baked = FaceTrack(track.fps, out_channels,
                      _extend_target_set(track, [c.name for c in added]))
    # Carry the additive event/take layer (issue #6) through untouched.
    baked.events = list(getattr(track, "events", None) or [])
    baked.variants = getattr(track, "variants", None)
    return baked