acceptodds
Withdrawn from ICLR 2027

CastDirector: Directing Joint Audio-Video Generation with Grounded Screenplays

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cinematic audio-video generation must assign dialogue and actions to the intended characters while producing immersive, scene-consistent audio. Existing systems can produce coherent clips while assigning speech or actions to the wrong character or omitting requested events. We introduce CastDirector, a screenplay-driven system for coordinating character performances and cinematic soundtracks. A grounded screenplay specifies the cast, speaker identities, dialogue, actions, and camera and audio cues as ordered events. To construct training screenplays from audiovisual data, our annotation pipeline extracts transcripts and visual actions independently, then verifies and combines them with the original clip to bind dialogue and actions to characters. The proposed multimodal Director jointly encodes this screenplay and the first frame, conditioning both generation streams to connect character actions, voices, and sound events to the visible scene. We first align a pretrained LTX-2.3 generator with the Director's features, then adapt the Director with LoRA through the audio-video generation loss. Under matched screenplay content, experiments show improved speaker binding, higher human ratings for action, dialogue, and audio, and gains in audio quality and speech accuracy. Qualitative examples showcase coordinated performances and immersive audio combining character voices, environmental ambience, sound effects, and scene-appropriate music.

Primary area:
generative models
Keywords:
Joint Audio-Video Generation, Human Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_t5CXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CastDirector, a framework for joint audio-video generation conditioned on the first frame and a script, with a focus on improving the correspondence among dialogue, actions, and visible characters. The method employs Qwen3-VL to jointly encode the first frame and the structured script, injecting these conditions into the audio-video generation branch via a newly introduced cross-attention mechanism. It also adopts a two-stage training process: generator alignment and Director adaptation. Furthermore, the authors propose a script annotation pipeline that independently extracts audio and visual evidence before fusing them for verification. Experiments on Director-Bench and a re-annotated Verse-Bench demonstrate improvements in character binding, speech accuracy, and human evaluation scores.

Critical strengths
  1. The task has practical application value. This paper addresses a common logical issue in the video creation process, which aligns well with real-world needs.
  2. The experimental design is quite thorough.
  3. The writing is clear and the phrasing is precise.
Critical weaknesses

See Questions

Questions for authors
  1. How robust is the model's character binding capability? Specifically, what level of improvement can be achieved for non-human characters, characters that are partially off-screen or occluded, or characters lacking a frontal face (e.g., those shown only in profile)?
  2. Can the model's script execution capability be further verified? For instance, in edge cases where the speaker of a specific line, the executor of an action, or the chronological order of two events is altered, does the generated output change accordingly? Furthermore, can the model handle more complex scenarios, such as martial arts scenes or scenes with a larger number of characters?
  3. While Director-Bench claims coverage of multi-language, multi-character, and non-human scenarios, it does not report the sample size, number of dialogue turns, or complexity distribution for each category. Since the automatic binding metric is only applicable to 31 cases featuring frontal human faces, the overall results likely reflect this specific subset. I recommend providing a detailed breakdown of the benchmark's composition and stratified results for key categories to clarify the scope of the generalization claims. For categories with very few samples, reporting case-by-case results would suffice, without the need for new large-scale evaluations.
  4. I am curious whether the introduction of this structured script improves the logical coherence of actions or the spatial consistency in the generated videos.
  5. What are the specific number of training steps and the computational cost (e.g., GPU hours) for the two training stages?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer t5CX: (1/2)

Q1. Character binding beyond frontal faces. Thank you for the suggestion. We added a dedicated 40-video character-binding test (E8):

MethodBinding accuracy (%) ↑
LTX-2.355.0
CastDirector77.5

CastDirector gains 22.5 percentage points on the full test set, extending the binding evidence beyond the original frontal-face subset. The Director jointly reads the first frame and screenplay, using appearance, posture and frame position to identify the intended character.

Videos E9–E12 show non-human, partly off-screen, occluded and back-facing characters, with each model's prompt. For example, the partly off-screen fighter straightens while her partner stays bowed in CastDirector; both straighten in LTX-2.3. These comparisons illustrate correct assignment of the requested action under reduced facial visibility.

Q2. Screenplay edits and more complex scenes. Thank you for the suggestion. We added paired screenplay edits, keeping the first frame and duration fixed within each pair. In E13, an unchanged line moves to a different speaker. In E14, a hand raise and book lift occur in the opposite order after reversing their order in the screenplay. These pairs demonstrate the requested speaker reassignment and event reversal.

E15/E16 show larger casts and martial-arts drills, including a choir that pauses and resumes singing and a kick to a partner's training pad. They illustrate coordinated group performance and short partner interactions beyond ordinary dialogue scenes.

AuthorsOpenReview ↗

Reply to Reviewer t5CX: (2/2)

Q3. Benchmark composition and category results. Thank you for the suggestion. Tables E17–E20 describe the 40 cases in the original Director-Bench evaluation:

Scene categoryNumber of cases
Human interactions31
Multi-speaker voiceover / off-screen commentary3
Cartoon character interactions3
Animal interactions3
Total40
Dialogue turns per caseNumber of cases
0 turns (actions only; no spoken dialogue)3
1 turn (with additional silent actions)7
2 turns13
3 turns10
4 turns4
5 turns3
Total40
Main characters per case (including off-screen narrators)Number of cases
1 character7
2 characters19
3 characters5
4 characters4
5 characters5
Total40
Dialogue languageNumber of cases
English15
Chinese8
Japanese8
Korean6
No spoken dialogue3
Total40

We compute Speaker Match and Turn Sync-C on the fixed subset of 31 cases with frontal views of human faces. A requested turn receives zero if it cannot be scored. In the human evaluation, Action and Audio cover all 40 cases, and Dialogue covers 37.

Table E21 reports human ratings (1–5) for the other nine cases:

Category (cases)MethodActionDialogueAudio
Animals (3)LTX-2.33.32N/A3.57
Animals (3)CastDirector3.88N/A4.29
Cartoons (3)LTX-2.33.363.483.51
Cartoons (3)CastDirector3.923.974.25
Voiceover (3)LTX-2.33.763.563.99
Voiceover (3)CastDirector4.054.054.45
All nine casesLTX-2.33.483.523.69
All nine casesCastDirector3.954.014.33

We average ratings within each case, then give each applicable case equal weight. Action and Audio cover all nine cases; animal clips receive Audio ratings for music and effects. Dialogue covers the six cartoon and voiceover cases. CastDirector's mean is higher on every applicable criterion in all three categories. We will add this breakdown to the appendix.

Q4. Action logic and spatial execution. Thank you for the suggestion. The Action rating measures whether the specified characters carry out their actions in the right order. CastDirector scores 4.09/5, compared with 3.67/5 for LTX-2.3 (Table 2 in the paper). In the seminar example, the speaker begins spreading their arms after the first question in CastDirector and during it in LTX-2.3. E22, E9 and E23 show the seminar, ice-cave and hallway comparisons. Together, they support better action timing, character-specific placement and scene-matched acoustics in the full system.

Q5. Training budget. Thank you for asking. We train LTX-2.3-dev (22B) with Qwen3-VL-8B on the same 1,459,594 samples (2,177 h) in both phases. We use 32 H200s and a global batch size of 128, with one sample per GPU and four accumulation steps.

Table E24 reports training steps and estimated costs:

PhaseOptimizer updatesTime/updateEstimated hoursH200 GPU-hours
1: Alignment8,000~33 s~73.3~2,347
2: Adaptation2,000 additional~27 s~15.0~480
Total10,000—~88.3~2,827

The full run takes about 3.68 days. In Phase 1, we freeze the Director and train the LTX LoRA weights (rank/scale 64/64) and conditioning modules. In Phase 2, we continue training both and add Director LoRA (32/64). Gemma stays frozen. Runs without a Director stop at 8k updates; the frozen and adapted Director runs both reach 10k. We will report the update counts and timing estimates in the training details.

Thank you for the additional experiments. The new binding test, paired screenplay edits, and benchmark breakdown address several of my original questions. I have three remaining points:

  1. Matched training budget. The no-Director run stops at 8k updates, whereas the Director variants reach 10k. Could you report a comparison at matched update counts, either by extending the no-Director run to 10k or by comparing the corresponding 8k checkpoints? This would help isolate the contribution of the Director.
  2. Binding under challenging conditions. Could you clarify the composition and scoring protocol of E8, including whether it overlaps with the original Director-Bench? A breakdown by condition, or case-level outcomes where there are too few examples, would help interpret the aggregate improvement.
  3. Script execution in complex scenes. The choir and martial-arts examples are useful, but they leave open how consistently the model assigns distinct actions to the correct characters in a larger cast. Could you report outcomes on a small set of such cases, including failures? A paired example that changes only which character performs an action, while keeping the first frame and the rest of the screenplay fixed, would directly complement the speaker reassignment and event-order tests.
Reviewer_ujufOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the task of screenplay-grounded video (w/ audio) generation. The authors propose to first use pre-trained models to curate a screenplay-like document, and then finetune a video generator to take this screenplay as input and generate as output cinematic video. The authors compare the capability of their trained model on two benchmarks, Director-Bench and Verse-Bench, and provide ablation studies on each.

Critical strengths

S1: there is evidently extensive engineering work undertaken to join up all the models for the screenplay curation, and for the generation training. There is plenty of space for inspiring future work on these aspects.

S2: the empirical results are positive and convincing. The resulting system established a new state of the art on most automatic and all human metrics. The ablation study also shows numerically modest but consistent gains (note the authors did not run significance testing, in spite of the small numbers of test case samples; I have not penalized for this given two benchmarks have been used and human evaluation performed).

Critical weaknesses

W1: the paper lacks elegance that might be desired on such a project. The system involves countless different pre-trained all knotted together (I count: Qwen, Gemma, Gemini, Keye, LTX) across textual, auditory and visual modalities. The authors neither provide quantitative evidence for each choice nor in my judgement make any attempt to brings things together in a more coherent way. It's somehow unsurprising that if one adds in all these pre-trained models, better performance is achievable. Beyond the screenplay curation, I don't think the conditioned video generation training is particularly novel.

W2: in my view, the related work section is lacking. I was surprised to see it start with character animation and overall in large it simply lists various works.

W3: in my view, the appendix needs substantial work. It details things like filetypes and so on, which should be in an attached repo not written out in prose.

Questions for authors

Q1: How often did the human raters skip? Is this making differences look bigger than they are?

Q2: training data is listed as publicly downloadable with added internal data. Both concern me as e.g. MovieNet video is not public and not sharing internal data or detailing is an issue for reproducibility. Do you have any comments?

Q3: Gemini by default takes as input 1fps video, have you experimentally validated that it works well at 5fps as specified in your appendix?

7 comments
AuthorsOpenReview ↗

Reply to Reviewer ujuf: (1/3)

W1/Q3. Component choices and annotation settings. Thank you for these suggestions. Our component choices draw on prior famous work in video and joint audio-video generation:

  • LTX and Gemma. LTX is a widely used open foundation model for joint audio-video generation, with existing adapters and inference tools [1,2]. Gemma is the native text encoder for LTX-2.3, much as uMT5 (from the T5 family) is for Wan [3]. We keep this pretrained pairing and freeze Gemma.
  • Qwen conditioning. HunyuanVideo-1.5's public release recommends Qwen2.5-VL-7B-Instruct as an open alternative to its unreleased multimodal encoder [4,5]. MiniMax-H3 uses Qwen3-VL-32B for text and visual conditioning [6]. These precedents support using a pretrained Qwen-VL encoder to ground text in visual content.
  • Gemini annotation. NAVA, one of our comparison methods, uses Gemini for structured audiovisual captions, including multi-speaker data [7]. Veo 3 also uses multiple Gemini models to caption its training data [8].

Our contribution is a grounded screenplay interface with two-stage training for joint audio-video generation. The screenplay assigns dialogue and actions to character descriptions, which the Qwen3-VL Director grounds in the first frame. Its features are computed once and reused by separate audio and video adapters throughout denoising. We first learn these conditioning paths with a frozen Director, then adapt the Director through the joint generation objective.

For training-data annotation and Gemini-based evaluation, we sample video at 5 fps to retain brief lip-motion cues for speaker assignment. In a small pilot on challenging clips, 1 fps often missed mouth movements during short words such as "but" or "okay." Gemini supports the chosen rate through video_metadata=types.VideoMetadata(fps=5) [10]. We will report the pilot and clarify this sampling choice.

We also tested the annotation choices on 50 particularly challenging English clips with multiple characters and dialogue turns, covering lighting changes, profile views and off-screen speech. Each pipeline produced five outputs per clip, randomly assigned to five non-author raters. Binding accuracy was 208/250 (83.2%) for ours versus 149/250 (59.6%) for end-to-end Gemini; a pass requires correct character assignment for all dialogue/actions. This supports separating audio and visual evidence before assigning dialogue and actions to characters.

The ASR replacement experiment (E3) evaluates WER of the final Gemini annotations:

Gemini 3.5 Flash annotation settingMean WER
End-to-end0.14
Qwen3-ASR priors0.06
Whisper large priors [9]0.05

We use manual transcripts from the same clips as references and average corpus-level WER across five runs at temperature 0. The Qwen and Whisper runs differ only in ASR; the Keye priors, Gemini, prompt template, frame rate (5 fps), and decoding settings are the same. With either ASR model, the pipeline uses the same interface and achieves lower WER than end-to-end Gemini.

References

[1] HaCohen Y, Brazowski B, Chiprut N, et al. LTX-2: Efficient joint audio-visual foundation model[J]. arXiv preprint arXiv:2601.03233, 2026.

[2] Lightricks. LTX-2.3 model card[EB/OL].

[3] Wan Team, Wang A, Ai B, et al. Wan: Open and advanced large-scale video generative models[J]. arXiv preprint arXiv:2503.20314, 2025.

[4] Wu B, Zou C, Li C, et al. HunyuanVideo 1.5 technical report[J]. arXiv preprint arXiv:2511.18870, 2025.

[5] Tencent-Hunyuan. HunyuanVideo-1.5: Download the pretrained checkpoints[EB/OL].

[6] MiniMaxAI. MiniMax-H3 model card[EB/OL].

[7] Ji L, Wang G, Wei X, et al. Native audio-visual alignment for generation[J]. arXiv preprint arXiv:2605.30073, 2026.

[8] Google DeepMind. Veo: A text-to-video generation system[R/OL].

[9] Radford A, Kim J W, Xu T, et al. Robust speech recognition via large-scale weak supervision[C]//International Conference on Machine Learning. PMLR, 2023: 28492-28518.

[10] Google. Video understanding: Custom frame rate[EB/OL].

AuthorsOpenReview ↗

Reply to Reviewer ujuf: (2/3)

W2. Related work. Thank you for the suggestion. We have rewritten Section 2 to begin with joint audio-video generation, followed by structured planning and conditioning, then character-audio binding. Each paragraph now explains the connection to our task and compares control interfaces or input assumptions. We distinguish visual layout and motion planning from our joint dialogue and action specification, and animation driven by supplied speech, such as MultiTalk [1], from joint generation of speech and video. The changes are marked in blue. We also welcome any relevant references you recommend and will promptly incorporate them into the Related Work.

W3. Appendix organization. Thank you for the suggestion. We have removed file-format, path-matching, service-deployment and retry/resumption details, together with the full serialized screenplay, from the appendix. These will be provided in the documentation accompanying our code release. We retained the binding example, annotation settings, validation criteria and evaluation protocols, and clarified each ablation's training procedure next to the full results. Added and rewritten passages are marked in blue.

Q1. Skipped ratings. Thank you for asking. Participants could skip a single criterion for a video. For example, participants who were unfamiliar with the language could skip Dialogue while still rating Action and Audio. For each criterion, we divide skips by the sum of valid ratings and skips, leaving out cases where the criterion does not apply.

CriterionValid ratingsSkipsSkip rate
Action2,38660.25%
Dialogue1,8171045.41%
Audio2,38660.25%

The 20 participants skipped 116 of 6,705 applicable ratings (1.73%), with a median of 5 and a mean of 5.8 skips per participant. We hid method names, randomized positions, and gave each case's mean rating equal weight. Skipping was uncommon overall and concentrated in Dialogue. We will clarify the rating unit and report these counts (E4).

References

[1] Kong Z, Gao F, Zhang Y, et al. Let them talk: Audio-driven multi-person conversational video generation[J]. Advances in Neural Information Processing Systems, 2025, 38: 70990-71013.

AuthorsOpenReview ↗

Reply to Reviewer ujuf: (3/3)

Q2. Data provenance. Thank you for asking us to clarify the data sources. Our wording about videos being available through MovieNet was inaccurate. We used MovieNet's title list to search publicly accessible sources for the corresponding videos and obtained them separately. Its annotations informed our annotation design, and we generated screenplay annotations with our pipeline. We have clarified this in the Data Statement and appendix, with revisions marked in blue.

Additional films and TV episodes outside the MovieNet list contribute about 372 hours after filtering and annotation, or 17.1% of the final 2,177-hour corpus. "Internally collected" means that our team gathered and curated these publicly accessible works.

For brevity, we list five examples from each group below, with a video link for every entry. The same examples appear in E5/E6. On Wikimedia Commons, select "Original file" to download the video.

MovieNet-listed works

FilmIMDb ID
Charadett0056923
Close-Uptt0100234
The Exorcisttt0070047
The Tenanttt0074811
Monty Python's Life of Briantt0079470

Additional publicly accessible works we collected

WorkIMDb ID
Tears of Steeltt2285752
Elephants Dreamtt0807840
Big Buck Bunnytt1254207
Sinteltt1727587
Springtt9249278

We will publish the complete source list, including film and episode IDs, access links, and acquisition and preprocessing instructions, with our open-source code and project documentation.

Reviewer_NBxuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses character binding in image-to-video-and-audio (I2VA) generation. The proposed CastDirector has three components: (1) a structured "grounded screenplay", (2) an evidence-separated annotation pipeline, and (3) a Qwen3-VL "Director". Trained on 2,177 hours of data, the model outperforms native LTX-2.3 and other open-source baselines in Speaker Match, WER, and human ratings on the proposed Director-Bench and a re-annotated Verse-Bench, and the design is also shown to transfer to OVI.

Critical strengths
  • Assigning dialogue or actions to the wrong character in multi-character scenes is a common failure of current joint audio-video generators, so the need is genuine and well motivated.
  • Speaker Match, speech accuracy, and human ratings improve consistently on both the LTX-2.3 and OVI backbones, and the frozen-Director control indicates that adapting the Director has an effect of its own.
Critical weaknesses
  • In the ablation, feeding the screenplay through the native text encoder alone ("+Screenplay") already yields most of the WER improvement (Director-Bench: 0.1192 → 0.0826, vs. 0.0769 for the full model), and training steps are not reported for any stage. Meanwhile, the baselines are zero-shot checkpoints given Gemini-rewritten prose prompts, whereas CastDirector is fine-tuned on 2,177 hours of data and evaluated on screenplays produced by the same pipeline as its training data. The margins therefore conflate the training data, an in-distribution input format, and the Director itself.
  • The Director (Qwen3-VL-8B plus about 4.9B parameters of new injection branches) is motivated by the claim that a text-only encoder cannot associate character descriptions with the visible cast. However, the native Gemma3-12B encoder of LTX-2.3 already accepts image input, so feeding the first frame into it is the most direct alternative. Without this control, it is unclear whether the gains come from joint image-text encoding or from the additional model capacity; adding the image to the Director also yields only a small gain (Speaker Match +0.019 on LTX-2.3 and +0.008 on OVI).
  • The pipeline is presented as a contribution, but the only evidence of its quality is that "only a small fraction of the inspected samples exhibit errors", with no sample size, error rate, or comparison to Gemini-only annotation. Since the evaluation screenplays come from the same pipeline, annotation errors affect both training and evaluation.
Questions for authors
  1. Is the "+Screenplay" variant fine-tuned with the same data and number of steps as CastDirector? Please report the training steps of each stage.
  2. Could you report a variant that feeds the first frame into the native Gemma3 encoder (optionally with LoRA), without the additional Qwen3-VL branch?
  3. Could you report LTX-2.3 fine-tuned on the same data with prose-style prompts, to separate the contribution of the data from that of the screenplay format and the Director?
  4. Could you report the error rates of the annotation pipeline on a random sample (e.g., wrong speaker assignment), compared with Gemini-only annotation?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer NBxu

W1/Q1/Q3. Sources of improvement and matched training. Thank you for these suggestions. For + Screenplay, we fine-tune LTX-2.3 with low-rank adaptation (LoRA), using screenplay-formatted prompts as training conditions. This variant has no Director. The original-format control, + Screenplay and Phase 1 share 1,459,594 samples (2,177 h), global batch 128 and LTX LoRA rank/scale 64/64, each for 8k updates.

The full model and frozen-Director continuation start from the same Phase 1 checkpoint and receive the same data and 2k further updates. Both continue training the generator LoRA and conditioning modules; only the full model also trains Director LoRA (32/64). On 32 H200s, Phase 1/2 take about 33/27 s per update: 88.3 h (3.68 days), or 2,827 GPU-hours in total.

Table E1 reports Director-Bench results; the native Gemma image-input variant is discussed under W2/Q2.

SettingStepsFID ↓FVD ↓ID ↑WER ↓Turn Sync-C ↑Match ↑
Native LTX-2.30114.3434.790.49510.11923.40910.7500
Same-data, original LTX format8k110.1533.420.48410.09863.42780.7529
+ Screenplay8k113.2732.550.48600.08263.50910.7672
Frozen Gemma: image + text (Q2)8k135.9248.160.44230.17802.83220.7425
Frozen Director, Phase 18k113.9231.420.49750.07823.65660.7985
Frozen Director, continued10k116.2433.670.48960.10893.46380.7740
CastDirector, full10k108.1928.280.49920.07693.89790.8491

ID is ArcFace similarity; Match is Speaker Match. Turn Sync-C and Speaker Match use the fixed 31-case subset. Fine-tuning on our data in the original format lowers word error rate (WER) from 0.1192 to 0.0986; screenplay-conditioned fine-tuning further lowers it to 0.0826 under the same data and 8k budget. These gains reflect training the generator on our data and screenplay supervision.

WER alone cannot establish character binding: correct words spoken by the wrong character can yield zero WER. At the same 8k budget, adding the frozen Director raises Speaker Match from 0.7672 to 0.7985. At 10k, Director adaptation outperforms frozen continuation on all six metrics, separating the effect of adaptation from extra training.

W2/Q2. Native Gemma image input and Director design. Thank you for the suggestion. We fed the first frame and text to frozen Gemma and trained LTX DiT LoRA for 8k updates, matching the stage-1 ablations. This variant produced visible facial distortion. In the rebuttal table above, it obtains Speaker Match 0.7425 and WER 0.1780, compared with 0.7985 and 0.0782 for our Phase 1 model.

Direct image input to Gemma may alter the conditioning representations expected by LTX. Our design preserves frozen Gemma's native text path and injects Qwen image-text features through zero-initialized residual cross-attention, allowing the added branches to learn visual conditioning.

A separate ablation in Table 4 of the paper compares text-only and multimodal Directors with the same architecture and 8k budget. Adding the first frame raises Speaker Match from 0.7793 to 0.7985, supporting the contribution of the Director's visual input to character binding.

W3/Q4. Quantitative annotation quality. Thank you for the suggestion. We selected 50 challenging English clips with multiple characters and dialogue turns, including lighting changes, profile views and off-screen speech. Each pipeline annotated each clip five times. Five non-author raters each evaluated 50 randomly assigned outputs per pipeline. A pass requires every line and action to be assigned to the intended character (E2).

PipelineBinding correctBinding error rate
End-to-end Gemini149/250 (59.6%)40.4%
Ours208/250 (83.2%)16.8%

The pipeline improves binding accuracy by 23.6 percentage points on this audit. Against manual transcripts, five-run mean WER of the final annotations is 0.14 for Gemini-only, 0.06 with Qwen3-ASR and 0.05 with Whisper large [1]. The ASR runs differ only in ASR: Keye priors, Gemini 3.5 Flash, prompt, 5 fps and temperature 0 match. Each run divides total word edits by total reference words; we average across five runs. Both ASR priors reduce final-annotation WER.

[1] Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision. ICML, 2023.

AuthorsOpenReview ↗

Reply to Reviewer NBxu

Dear Reviewer NBxu, thank you again for your review. We have added the requested controls and annotation audit, including the 10k-update comparison. Could you let us know whether these address your concerns or if further clarification would be helpful?

Comments on the paper

1 comment

We summarize the current status of the rebuttal discussion below.

  • Reviewer t5CX: The matched-update comparison resolves the training-budget mismatch, with the full model leading on all six Director-Bench metrics. The binding-test breakdown and scoring protocol clarify the evidence beyond frontal faces, while the action examples show both successes and failures. The reviewer confirmed that his/her main concerns were addressed and raised his/her recommendation to Weak Accept.

  • Reviewer ujuf: The technical and presentation concerns have been addressed, including component choices, evaluation coverage, data provenance, related work, appendix organization, and quantitative validation of 5 fps sampling. His/her remaining hesitation concerns the contribution's novelty and research value; he/she has left this assessment to the Area Chair.

  • Reviewer NBxu: In response to his/her requests, we provided same-data original-format and screenplay fine-tuning controls, comparisons at matched training budgets, a native Gemma image-input ablation, and a quantitative annotation-quality audit against Gemini-only annotation. His/her response is pending.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.