Layered multi-track export

Engines often want to re-blend or toggle facial layers at runtime rather than receive one flattened curve set — Unreal Sequencer/Control Rig keep lip-sync and expression on separate additive tracks, and SALSA blends layers by priority (issue #39). The pipeline already produces speech / emotion / gesture on disjoint channels before concatenating them into one FaceTrack, so this is a data-reshuffle of arrays it already has plus a little metadata; the runtime mix stays the engine's job.

python -m openfacefx export-layers merged.track.json -o layered.track.json

export-layers writes the same flat track with an extra top-level layers block — so a reader that ignores the block still sees the merged track, and the default (no layers) output is unchanged.

The layers block

build_layers(track) decomposes a merged track into named sub-tracks by channel classification — gesture (the issue-#5 gesture channels), emotion (the issue-#38 VA_EMOTION_CHANNELS), and speech (everything else: visemes and any rig blendshapes). Each layer carries its channel list, a per-layer blend-weight curve ([[t, w], …], default constant 1.0) and an integer priority (engine layering order — speech 0, emotion 10, gesture 20). Empty layers are omitted.

{
  "format": "openfacefx.track", "version": 1, "fps": 60.0, "duration": 2.3,
  "channels": [ /* the flat merged track, unchanged */ ],
  "layers": [
    { "name": "speech",  "priority": 0,  "weight": [[0.0, 1.0]], "channels": [ /* visemes */ ] },
    { "name": "emotion", "priority": 10, "weight": [[0.0, 1.0]], "channels": [ /* smile, brow_raise, … */ ] },
    { "name": "gesture", "priority": 20, "weight": [[0.0, 1.0]], "channels": [ /* blink, headYaw, … */ ] }
  ]
}

Because every channel lands in exactly one layer, flatten_layers summing the layers at weight 1 reproduces the merged channels exactly — a faithful, lossless decomposition (pinned by a round-trip test). priority is engine metadata, ignored by the flatten. Events (including prosody, issue #4) remain the track's own event layer, unchanged — prosody drives notifies, not curves, so it is not a channel layer.

Invariants

• Byte-identical default: to_dict(track) with no layers (and no track.layers) is byte-for-byte what it was before — the layers key is appended only when non-empty. The entire existing test suite is the proof.
• to_dict(track, layers=…) / write_json(…, layers=…) embed the block; from_dict/read_json restore it to track.layers, so names, weights and priorities survive write_json→read_json.
• Empty/absent layers are omitted, never emitted as dead channels.

Library callers get build_layers, flatten_layers, layers_to_dict, layers_from_dict and the Layer dataclass. numpy + stdlib, deterministic across Python 3.9/3.13, additive.

openfacefx.layers

Layered multi-track export (issue #39).

Engines want to re-blend or toggle facial layers at runtime rather than receive one flattened curve set -- Unreal Sequencer/Control Rig keep lip-sync and expression on separate additive tracks, and SALSA blends layers by priority. The pipeline already produces speech / emotion / gesture on disjoint channels before they are concatenated into one :class:~openfacefx.curves.FaceTrack, so this is a data-reshuffle of arrays it already has plus a little metadata; the runtime mix stays the engine's job.

:func:build_layers decomposes a merged track into named layers by channel classification (gesture = :data:openfacefx.gestures.GESTURE_CHANNELS, emotion = :data:openfacefx.emotion.VA_EMOTION_CHANNELS, speech = everything else -- visemes and any rig blendshapes), each carrying its channel list, a per-layer blend-weight curve (default constant 1.0) and an integer priority (engine layering order). Because every channel lands in exactly one layer, :func:flatten_layers summing them at weight 1 reproduces the merged channels exactly -- a faithful decomposition. Empty layers are omitted.

The layers ride as an optional layers block on the track JSON (:func:openfacefx.io_export.to_dict with layers=); the flat channel list stays the default, so a track without layers is byte-identical to before and a reader that ignores the block still sees the merged track. Events (including prosody, issue #4) remain the track's own event layer, unchanged -- prosody drives notifies, not curves, so it is not a channel layer. numpy + stdlib, deterministic.

FULL_WEIGHT: List[List[float]] = [[0.0, 1.0]] module-attribute

Layer(name: str, channels: List[Channel] = list(), weight: List[List[float]] = (lambda: [[0.0, 1.0]])(), priority: int = 0) dataclass

One named sub-track: a normal channel list plus a blend-weight curve ([[t, w], ...], default constant 1.0) and an integer priority (engine layering order). The weight and priority are metadata for the runtime mix; OpenFaceFX only uses weight when it flattens for verification.

name: str instance-attribute

channels: List[Channel] = field(default_factory=list) class-attribute instance-attribute

weight: List[List[float]] = field(default_factory=(lambda: [[0.0, 1.0]])) class-attribute instance-attribute

priority: int = 0 class-attribute instance-attribute

build_layers(track: FaceTrack) -> List[Layer]

Decompose a merged track into named layers by channel classification.

Every channel lands in exactly one layer (gesture / emotion / the speech base), so the split is lossless; empty layers are omitted. Each layer gets a constant full-blend weight and its default priority.

Source code in src/openfacefx/layers.py

def build_layers(track: FaceTrack) -> List[Layer]:
    """Decompose a merged ``track`` into named layers by channel classification.

    Every channel lands in exactly one layer (``gesture`` / ``emotion`` / the
    ``speech`` base), so the split is lossless; empty layers are omitted. Each
    layer gets a constant full-blend weight and its default priority."""
    from .gestures import GESTURE_CHANNELS
    from .emotion import VA_EMOTION_CHANNELS
    groups = {"speech": [], "emotion": [], "gesture": []}
    for c in track.channels:
        if c.name in GESTURE_CHANNELS:
            groups["gesture"].append(c)
        elif c.name in VA_EMOTION_CHANNELS:
            groups["emotion"].append(c)
        else:
            groups["speech"].append(c)
    return [Layer(name, groups[name], [[0.0, 1.0]], priority)
            for name, priority in _LAYER_PRIORITY if groups[name]]

flatten_layers(layers: List[Layer], fps: float = 60.0) -> FaceTrack

Sum layers back into one :class:FaceTrack, each channel scaled by its layer's blend-weight curve and channels of the same name added on a shared grid. At weight 1 (the default) this reproduces the merged channels -- the faithful-decomposition check. priority is engine metadata, ignored here.

Source code in src/openfacefx/layers.py

def flatten_layers(layers: List[Layer], fps: float = 60.0) -> FaceTrack:
    """Sum ``layers`` back into one :class:`FaceTrack`, each channel scaled by its
    layer's blend-weight curve and channels of the same name added on a shared
    grid. At weight 1 (the default) this reproduces the merged channels -- the
    faithful-decomposition check. ``priority`` is engine metadata, ignored here."""
    from collections import OrderedDict
    merged: "OrderedDict[str, Channel]" = OrderedDict()
    for layer in layers:
        for ch in layer.channels:
            scaled = _scale(ch, layer.weight)
            merged[ch.name] = (_sum(merged[ch.name], scaled)
                               if ch.name in merged else scaled)
    return FaceTrack(fps=fps, channels=list(merged.values()), target_set=None)

layers_to_dict(layers: List[Layer]) -> List[dict]

Serialise layers to the JSON block: a list of {name, priority, weight, channels} (channels/weights rounded to 4 dp, exactly like track keys).

Source code in src/openfacefx/layers.py

def layers_to_dict(layers: List[Layer]) -> List[dict]:
    """Serialise layers to the JSON block: a list of ``{name, priority, weight,
    channels}`` (channels/weights rounded to 4 dp, exactly like track keys)."""
    return [{
        "name": layer.name,
        "priority": int(layer.priority),
        "weight": [[round(float(t), 4), float(w)] for t, w in layer.weight],
        "channels": [
            {"name": c.name,
             "keys": [[round(k.time, 4), k.value] for k in c.keys]}
            for c in layer.channels
        ],
    } for layer in layers]

layers_from_dict(raw) -> List[Layer]

Inverse of :func:layers_to_dict; raises ValueError on a malformed block so a corrupt layers key fails loudly rather than silently.

Source code in src/openfacefx/layers.py

def layers_from_dict(raw) -> List[Layer]:
    """Inverse of :func:`layers_to_dict`; raises ``ValueError`` on a malformed
    block so a corrupt ``layers`` key fails loudly rather than silently."""
    if not isinstance(raw, list):
        raise ValueError("'layers' must be a JSON array of layer objects")
    out: List[Layer] = []
    for i, layer in enumerate(raw):
        if not isinstance(layer, dict) or "name" not in layer:
            raise ValueError(f"layer[{i}] must be an object with a 'name'")
        channels = [
            Channel(str(c["name"]),
                    [Keyframe(float(t), float(v)) for t, v in c["keys"]])
            for c in layer.get("channels", [])
        ]
        weight = [[float(t), float(w)] for t, w in layer.get("weight", [[0.0, 1.0]])]
        out.append(Layer(str(layer["name"]), channels, weight,
                         int(layer.get("priority", 0))))
    return out