- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies joint audio-video generation of multi-shot, multi-character dialogue scenes, aiming to let visual cuts and dialogue timing be controlled independently. The screenplay is represented as a visual track and an acoustic track on a shared clock, and LTX-2.3 is fine-tuned with LoRA. Two modules are proposed: (1) dual-schedule temporal cross-attention and (2) a training-free directorial lip-sync mechanism. On self-constructed 15-second and 60-second screenplay benchmarks, the method outperforms LTX-2.3, MiniMax-H3, and MOVA in cut accuracy, dialogue timing, and silence preservation, and an ablation shows that the lip-sync mechanism reduces spurious lip coupling on listeners.
Critical strengths
- A problem setting close to real film language. Independent cuts and dialogue, reaction shots, and deliberate silences are aspects that existing joint generators rarely control explicitly.
- Clear timing-control gains with low overhead. Removing the dual-schedule masks lowers Cut F1 from 0.99 to 0.34 and Containment from 92.83% to 39.34%, and the lip-sync mechanism adds very little runtime.
Critical weaknesses
- Restricting text cross-attention by shot interval is already common in multi-shot video generation (e.g., HoloCine, which the paper cites), and the paper mainly applies the same idea to the audio branch according to dialogue timestamps. MAVIN, also cited, already routes text cross-attention by both shot and dialogue intervals in a joint audio-video model, but the paper does not discuss this routing design in relation to it. The authors should discuss the relation to these methods and clarify the differences.
- Simpler ways exist to keep a visible listener from lip-syncing to off-screen speech, such as stating in the prompt that the line is off-screen (as the official MiniMax-H3 prompting guide prescribes) or disabling audio-to-video cross-attention within that interval. The "Skip A2V" variant appears only as one qualitative example in the supplementary material, and Table 2 has no quantitative comparison with either alternative. In addition, the claim of eliminating spurious lip-sync on listeners rests only on the new TemporalBias metric, which is not reported for the baselines or real videos and is not validated by human evaluation.
- The paper does not state where the training data and the 400 test screenplays come from, or whether the training and test sets are split by film. Moreover, the model is fine-tuned on the same timestamped screenplay format used at test time, whereas the baselines receive it zero-shot, and how the screenplay is converted into each baseline's input is not explained. It is therefore unclear how much of the large margin in Table 1 comes from the method itself.
Questions for authors
- Could the authors quantitatively compare the lip-sync mechanism with (a) stating in the prompt that the line is off-screen and the listener keeps listening, and (b) disabling A2V cross-attention within off-screen intervals (the Skip A2V variant)?
- Where do the training data and test screenplays come from, and are the training and test sets split by film?
- Could the authors report TemporalBias for the baselines and for real videos?
- In Table 2, is "w/o Temporal CA" a retrained model or the same checkpoint with masks removed at inference?
- How is the screenplay converted into each baseline's input? MiniMax-H3 outputs are only 14.375 seconds long; how is the missing part handled in evaluation?
4 comments
W3, Q2, and Q4 — Data provenance, adaptation fairness, and “w/o Temporal CA”.
- Data provenance and unseen test screenplays (Q2): We curated ~30,000 15s and ~30,000 60s training sequences by merging adjacent movie shots with native audio, using Faster-Whisper for speech timestamps and Gemini 3.0 Pro for dual-track annotations. Our evaluation benchmark comprises 400 screenplays in total — 200 15s and 200 60s sequences. Each duration subset consists of 100 pipeline-curated sequences randomly held out (never seen in training) and 100 LLM-authored screenplays (ChatGPT) that have zero source-film overlap with the training corpus by construction. Découpage performs consistently across both duration subsets.
- Why gains stem from our mechanism rather than format adaptation (W3 & Q4): To answer Q4 directly: the “w/o Temporal CA” row in Table 2 uses a model retrained on the exact same 30,000 sequences and screenplay format, but without dual-schedule temporal masking. Despite full adaptation to our training data and prompt format, this retrained model achieves only 0.34 Cut F1 and 39.34% Containment — on par with the zero-shot baselines (0.18–0.28 Cut F1, 27.14%–52.68% Containment) — whereas adding Dual-Schedule Temporal CA boosts performance to 0.99 Cut F1 and 92.83% Containment. This confirms that the large margin in Table 1 comes from our proposed Dual-Schedule Cross-Attention mechanism rather than format fine-tuning.
Q5 — Baseline input conversion and MiniMax-H3 duration.
For baseline evaluation, each structured screenplay was converted into a chronological text prompt containing all visual shot descriptions, shot positions, timed dialogue lines, and sound cues for LTX-2.3 and MiniMax-H3 (and paired with the initial reference frame for MOVA I2VA). Regarding the 14.375 s duration of MiniMax-H3 in our initial submission: first, our benchmark metrics are inherently robust to minor tail-duration differences (0.625 s), as internal shot transitions (matched within a 0.5 s tolerance), scheduled dialogue turns, and scorable Sync-C face windows lie within the first 14.375 s. Second, we sincerely thank the reviewer for catching this frame-count configuration bug. We re-ran MiniMax-H3 under the exact same settings to generate full 15.0 s outputs across the 200-screenplay set and updated Table 1 in the paper and the comparison videos in the supplementary materials. As expected, the resulting scores remain virtually unchanged (e.g., H3 Cut F1 is 0.28 at full 15.0 s vs. 0.29 at 14.375 s, compared to our 0.99), confirming that this slight duration offset does not affect our conclusions.
Thank you for the detailed response and the additional experiments, which clarify the differences from HoloCine and MAVIN and the effect of the lip-sync mechanism. I agree that the proposed mechanisms are simple and lightweight, which is a practical advantage. However, I am not fully convinced that its technical contribution is sufficient, so I will keep my rating.
Thank you for considering our response and additional experiments, and for recognizing the practical advantages of our lightweight design. Our contribution centers on a dual-track formulation that makes the temporal organization and cross-modal relationships of an audiovisual narrative explicitly controllable. Independently timed visual and acoustic tracks, linked by a shared timeline and persistent character identities, allow dialogue to continue across visual transitions while the visible characters and their narrative roles change. This formulation exposes two complementary control requirements. The generator is required to follow the independently specified visual and acoustic schedules, and regulate how ongoing speech influences visible articulation as speaker visibility changes. We realize these requirements through dual-schedule temporal cross-attention and visibility-conditioned A2V residual substitution using a paired non-speech trajectory. The additional controlled comparisons and blinded human evaluation support the effectiveness of the visibility-conditioned intervention. We believe this formulation, its associated mechanisms, and the demonstrated directorial capabilities together constitute a meaningful contribution to controllable audio-visual generation. We appreciate your time and constructive feedback.
We sincerely thank Reviewer ktHN for appreciating our realistic film-language problem setting and recognizing the clear timing-control gains and minimal runtime cost of our dual-schedule and lip-sync mechanisms.
W1 — Relationship to HoloCine and concurrent work MAVIN.
Découpage is designed around the core grammar of film editing over extended 15- to 60-second horizons: treating the visual cut list and the acoustic timeline as two independently scheduled tracks on a shared clock to explicitly orchestrate J-cuts, L-cuts, reaction shots, and scripted silences.
<NO SPEECH>intervals, Découpage reduces silence leakage to 1.12%.w/o MechanismandPrompt-only) show in the general response, text routing alone cannot prevent active speech features in A2V cross-attention from driving a visible listener's mouth during an L-cut or reaction shot. Découpage leverages its explicit<NO SPEECH>mode to construct a paired non-speech trajectory at inference, substituting the A2V residual whenever the active speaker is off screen.W2 and Q1 — Controlled comparison with Prompt-only and Skip A2V, plus human evaluation.
Holding the checkpoint, audio prompt, seeds, and sampling settings fixed on our evaluation subset, we evaluated both alternatives alongside a blinded user study (Tables A and B in the General Response):
<NO SPEECH>residual reduces it further to 0.33. Simply skipping A2V completely severs cross-modal conditioning during off-screen intervals, perturbing latent continuity across boundaries and degrading control over natural listener reactions. In our blinded user study (Table B), raters preferred Découpage over both Skip A2V and w/o control by wide margins on plausible lip motion (66% vs. 12% and 7%) and natural listener reactions (58% vs. 12% and 12%). Comparison videos are included in the updated supplement.Q3 —
TemporalBiason real videos and why it is inapplicable to unaligned baselines.Evaluating 200 randomly sampled real 15-second film clips from our curation pipeline, listener/reaction windows achieve a low
TemporalBiasof 0.47, reflecting that real listeners' facial dynamics are not time-locked to off-screen syllables. Our mechanism reducesTemporalBiasfrom 1.78 down to 0.33, matching the low-coupling regime of real film clips (0.47). In contrast, computing script-windowTemporalBiason the Table 1 baselines is ill-posed because the metric presupposes that the generator actually transitions to the listener shot and keeps the dialogue within the scheduled interval. Whereas Découpage satisfies both prerequisites (0.99 Cut F1, 93.01% audio containment), baselines fail to control shot cuts and dialogue timing, so target screenplay intervals capture missed cuts or drifted audio rather than true reaction shots.