- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CastDirector, a framework for joint audio-video generation conditioned on the first frame and a script, with a focus on improving the correspondence among dialogue, actions, and visible characters. The method employs Qwen3-VL to jointly encode the first frame and the structured script, injecting these conditions into the audio-video generation branch via a newly introduced cross-attention mechanism. It also adopts a two-stage training process: generator alignment and Director adaptation. Furthermore, the authors propose a script annotation pipeline that independently extracts audio and visual evidence before fusing them for verification. Experiments on Director-Bench and a re-annotated Verse-Bench demonstrate improvements in character binding, speech accuracy, and human evaluation scores.
Critical strengths
- The task has practical application value. This paper addresses a common logical issue in the video creation process, which aligns well with real-world needs.
- The experimental design is quite thorough.
- The writing is clear and the phrasing is precise.
Critical weaknesses
See Questions
Questions for authors
- How robust is the model's character binding capability? Specifically, what level of improvement can be achieved for non-human characters, characters that are partially off-screen or occluded, or characters lacking a frontal face (e.g., those shown only in profile)?
- Can the model's script execution capability be further verified? For instance, in edge cases where the speaker of a specific line, the executor of an action, or the chronological order of two events is altered, does the generated output change accordingly? Furthermore, can the model handle more complex scenarios, such as martial arts scenes or scenes with a larger number of characters?
- While Director-Bench claims coverage of multi-language, multi-character, and non-human scenarios, it does not report the sample size, number of dialogue turns, or complexity distribution for each category. Since the automatic binding metric is only applicable to 31 cases featuring frontal human faces, the overall results likely reflect this specific subset. I recommend providing a detailed breakdown of the benchmark's composition and stratified results for key categories to clarify the scope of the generalization claims. For categories with very few samples, reporting case-by-case results would suffice, without the need for new large-scale evaluations.
- I am curious whether the introduction of this structured script improves the logical coherence of actions or the spatial consistency in the generated videos.
- What are the specific number of training steps and the computational cost (e.g., GPU hours) for the two training stages?
6 comments
Reply to Reviewer t5CX: (2/2)
Q3. Benchmark composition and category results. Thank you for the suggestion. Tables E17–E20 describe the 40 cases in the original Director-Bench evaluation:
| Scene category | Number of cases |
|---|---|
| Human interactions | 31 |
| Multi-speaker voiceover / off-screen commentary | 3 |
| Cartoon character interactions | 3 |
| Animal interactions | 3 |
| Total | 40 |
| Dialogue turns per case | Number of cases |
|---|---|
| 0 turns (actions only; no spoken dialogue) | 3 |
| 1 turn (with additional silent actions) | 7 |
| 2 turns | 13 |
| 3 turns | 10 |
| 4 turns | 4 |
| 5 turns | 3 |
| Total | 40 |
| Main characters per case (including off-screen narrators) | Number of cases |
|---|---|
| 1 character | 7 |
| 2 characters | 19 |
| 3 characters | 5 |
| 4 characters | 4 |
| 5 characters | 5 |
| Total | 40 |
| Dialogue language | Number of cases |
|---|---|
| English | 15 |
| Chinese | 8 |
| Japanese | 8 |
| Korean | 6 |
| No spoken dialogue | 3 |
| Total | 40 |
We compute Speaker Match and Turn Sync-C on the fixed subset of 31 cases with frontal views of human faces. A requested turn receives zero if it cannot be scored. In the human evaluation, Action and Audio cover all 40 cases, and Dialogue covers 37.
Table E21 reports human ratings (1–5) for the other nine cases:
| Category (cases) | Method | Action | Dialogue | Audio |
|---|---|---|---|---|
| Animals (3) | LTX-2.3 | 3.32 | N/A | 3.57 |
| Animals (3) | CastDirector | 3.88 | N/A | 4.29 |
| Cartoons (3) | LTX-2.3 | 3.36 | 3.48 | 3.51 |
| Cartoons (3) | CastDirector | 3.92 | 3.97 | 4.25 |
| Voiceover (3) | LTX-2.3 | 3.76 | 3.56 | 3.99 |
| Voiceover (3) | CastDirector | 4.05 | 4.05 | 4.45 |
| All nine cases | LTX-2.3 | 3.48 | 3.52 | 3.69 |
| All nine cases | CastDirector | 3.95 | 4.01 | 4.33 |
We average ratings within each case, then give each applicable case equal weight. Action and Audio cover all nine cases; animal clips receive Audio ratings for music and effects. Dialogue covers the six cartoon and voiceover cases. CastDirector's mean is higher on every applicable criterion in all three categories. We will add this breakdown to the appendix.
Q4. Action logic and spatial execution. Thank you for the suggestion. The Action rating measures whether the specified characters carry out their actions in the right order. CastDirector scores 4.09/5, compared with 3.67/5 for LTX-2.3 (Table 2 in the paper). In the seminar example, the speaker begins spreading their arms after the first question in CastDirector and during it in LTX-2.3. E22, E9 and E23 show the seminar, ice-cave and hallway comparisons. Together, they support better action timing, character-specific placement and scene-matched acoustics in the full system.
Q5. Training budget. Thank you for asking. We train LTX-2.3-dev (22B) with Qwen3-VL-8B on the same 1,459,594 samples (2,177 h) in both phases. We use 32 H200s and a global batch size of 128, with one sample per GPU and four accumulation steps.
Table E24 reports training steps and estimated costs:
| Phase | Optimizer updates | Time/update | Estimated hours | H200 GPU-hours |
|---|---|---|---|---|
| 1: Alignment | 8,000 | ~33 s | ~73.3 | ~2,347 |
| 2: Adaptation | 2,000 additional | ~27 s | ~15.0 | ~480 |
| Total | 10,000 | — | ~88.3 | ~2,827 |
The full run takes about 3.68 days. In Phase 1, we freeze the Director and train the LTX LoRA weights (rank/scale 64/64) and conditioning modules. In Phase 2, we continue training both and add Director LoRA (32/64). Gemma stays frozen. Runs without a Director stop at 8k updates; the frozen and adapted Director runs both reach 10k. We will report the update counts and timing estimates in the training details.
Thank you for the additional experiments. The new binding test, paired screenplay edits, and benchmark breakdown address several of my original questions. I have three remaining points:
- Matched training budget. The no-Director run stops at 8k updates, whereas the Director variants reach 10k. Could you report a comparison at matched update counts, either by extending the no-Director run to 10k or by comparing the corresponding 8k checkpoints? This would help isolate the contribution of the Director.
- Binding under challenging conditions. Could you clarify the composition and scoring protocol of E8, including whether it overlaps with the original Director-Bench? A breakdown by condition, or case-level outcomes where there are too few examples, would help interpret the aggregate improvement.
- Script execution in complex scenes. The choir and martial-arts examples are useful, but they leave open how consistently the model assigns distinct actions to the correct characters in a larger cast. Could you report outcomes on a small set of such cases, including failures? A paired example that changes only which character performs an action, while keeping the first frame and the rest of the screenplay fixed, would directly complement the speaker reassignment and event-order tests.
Reply to Reviewer t5CX: (1/2)
Q1. Character binding beyond frontal faces. Thank you for the suggestion. We added a dedicated 40-video character-binding test (E8):
CastDirector gains 22.5 percentage points on the full test set, extending the binding evidence beyond the original frontal-face subset. The Director jointly reads the first frame and screenplay, using appearance, posture and frame position to identify the intended character.
Videos E9–E12 show non-human, partly off-screen, occluded and back-facing characters, with each model's prompt. For example, the partly off-screen fighter straightens while her partner stays bowed in CastDirector; both straighten in LTX-2.3. These comparisons illustrate correct assignment of the requested action under reduced facial visibility.
Q2. Screenplay edits and more complex scenes. Thank you for the suggestion. We added paired screenplay edits, keeping the first frame and duration fixed within each pair. In E13, an unchanged line moves to a different speaker. In E14, a hand raise and book lift occur in the opposite order after reversing their order in the screenplay. These pairs demonstrate the requested speaker reassignment and event reversal.
E15/E16 show larger casts and martial-arts drills, including a choir that pauses and resumes singing and a kick to a partner's training pad. They illustrate coordinated group performance and short partner interactions beyond ordinary dialogue scenes.