scene-spec / Manim beats drift from narration (label→wait_word mismatch + unpaced timed_play)

[jmjava]

Summary

After a full tekton-dag demo regen on current docgen (Manim-only, local timestamps, scene-spec → compile → TTS → timestamps → Manim → compose), composed videos match audio length but on-screen content often has little relation to what is being spoken.

This is not primarily A/V stream drift (ffprobe durations are close). It is semantic / beat sync failure: visuals race through a generic diagram, latch to the wrong spoken words, then freeze while narration continues on other topics.

Observed in consumer (tekton-dag docs/demos)

Segment 01 (StackDAGScene):

• Narration talks about four pipelines, Flask orchestrator, optional CRD path, etc.
• Scene boxes include labels like Demo-fe, Originator, Forwarder, Terminal that never appear in the regenerated narration.
• Compiled code waits with wait_until_word(timing_words, 146) before revealing Originator, but word index 146 in timing.json is the token "operator" (~64s) — unrelated.
• Other boxes fade in via cascading timed_play with no wait_until_word, so large chunks of the visual story finish in the first tens of seconds.

Segment 03 (HeaderPropagationScene):

• Loads timing_words but mostly uses unpaced timed_play(FadeIn(...)) cascades, then relies on the audio-length tail / compose freeze for the remaining narration.

Validate can still report PASS for av_drift / timing_sync because the muxed video is padded to the mp3 duration even when the visual story is already over.

Root causes (suspected)

1. scene-spec-generate does not enforce narration-anchored labels
   System prompt says to mirror narration beats, but the model freely invents diagram vocabulary (roles, app names) that are not spoken. There is no hard gate that every label must match a token/phrase in the segment narration (or a declared synonym list).

2. sync_row_labels_to_whisper_words(..., overwrite=True) fills bad matches
   When a label is absent from timing.json words, matching appears to attach to a wrong nearby/fuzzy token (or a leftover LLM wait_word), baking incorrect indices into compiled scenes.py.

3. Compile path allows unpaced reveals
   Boxes without a resolved wait_word still emit timed_play(FadeIn) immediately after the previous animation. That produces a short slideshow + long frozen hold, which feels “completely out of sync” even when total duration matches TTS.

4. Regen order amplifies the bug
   Typical consumer flow regenerated scene specs before fresh TTS/timing.json (or compiled against sparse timing), then rendered Manim after local timestamps. Tooling should either require timing first, or treat missing timing as a hard failure for paced scenes instead of silent unpaced fallback.

Expected behavior

• Every paced box label should be a spoken anchor from that segment’s narration (or an explicit alias map).
• wait_word indices must refer to the first occurrence of that anchor in timing.json words; unmatched labels must fail generate/compile (or force a redesign), not silently bind to unrelated tokens.
• Default compile behavior should not cascade unpaced timed_play for story beats when timing exists; prefer wait-until-word (or wait-until-segment) for each reveal.
• generate-all / docs should recommend: narration → TTS → timestamps → then scene-spec-generate/compile → Manim → compose (or auto-rerun compile after timestamps).

Proposed work

• Add validation in scene-spec-generate / scene_spec.validate / sync_row_labels_to_whisper_words: reject or warn loudly when labels do not match narration tokens.
• Fix label→word matching to avoid false positives; never invent indices for unmatched labels.
• When timing is present, require pacing for non-title reveals (or document an explicit pace: none opt-out).
• Add a fixture/test that reproduces “label not in narration → must fail” and “label matches wait_word start time”.
• Update AGENTS.md / README consumer workflow for post-TTS scene regen.
• Optional: scene-compile --retime that only rewrites wait_until_word indices from current timing.json without calling OpenAI.

Environment

• docgen: current main (incl. local timestamps engine, image elements, docgen.project hint merge)
• Consumer: jmjava/tekton-dag docs/demos full regen (19 Manim segments)
• Timestamps engine used: local (default)

Related

Consumer impact: tekton-dag demo recordings after Jul 2026 full regen are not shippable for content sync until this is addressed (or scenes are hand-constrained via much stricter hints + forced retime).

[jmjava]

Consumer preference: keep timestamps on Whisper (docgen timestamps --engine whisper / timestamps.engine: whisper), not the new default local silencedetect aligner. The Jul 2026 tekton-dag regen used local and that should be treated as a misconfiguration for this consumer until Whisper retiming is redone.