- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies long-form audio-video generation from a complete narrative script. RaAVGen generates successive segments while retaining the full script, using a short video history and a longer audio history to support visual continuity and speech continuation. For multi-shot generation, it additionally uses keyframes from completed shots and updates shot timestamps relative to the current segment. The model combines several sources of supervision through three-stage training, with pseudo-long script augmentation extending the textual context without requiring longer recordings. Experiments evaluate approximately 40-second narratives, with ablations of history length, sparse memory, and script augmentation.
Critical strengths
- Allowing generation boundaries to fall within utterances and shots is a useful setting. The distinction between visual context for continuity and audio context for locating speech within the script is well motivated. It gives a concrete reason to retain different amounts of history for the two modalities.
- The ablations provide useful evidence for several design choices. Increasing audio history from two to six seconds reduces WER from 4.08% to 2.64%, while sparse visual memory improves cross-shot identity consistency from 0.776 to 0.874. The controlled augmentation experiment in Table 5 is also informative: using the same recordings, the augmented model maintains lower WER in the final evaluation interval.
Critical weaknesses
- The evidence for tracking narrative progress is somewhat indirect. Longer audio history improves WER, and the user study reports better segment continuity and prompt following. However, these aggregate results do not show how often the model repeats or skips unfinished content at generation boundaries. In the multi-shot setting, Eq. (1) supplies shot-level progress from the prescribed timestamps, but the evaluation does not show how reliably history identifies and continues unfinished dialogue and visual actions across generation boundaries.
- Important details needed to interpret and reproduce the results are missing. Appendix A identifies an in-house pretrained backbone, but does not describe its pretraining data or procedure. Appendix B lists the training data categories and sizes, but gives little detail on how scripts, speaker correspondences, and shot annotations were obtained and checked. Table 1 compares complete systems, so it also leaves the contribution of full-script conditioning relative to segment-specific prompting unresolved. Finally, Appendix D excludes invalid face and speech observations from the cross-shot consistency metrics without reporting coverage or explaining how character assignments are obtained. This makes it difficult to assess whether the reported scores reflect comparable sets of observations across methods.
Questions for authors
- What evidence distinguishes improved localization within the full script from improved local speech continuation? In particular, do the reported WER gains correspond to fewer repetitions and omissions at generation boundaries?
- Please clarify the backbone’s pretraining data and procedure and how the training scripts, speaker correspondences, and shot annotations were constructed and checked. Is there any same-backbone evidence separating full-script conditioning from segment-specific prompting?
- How are the fixed character assignments in Appendix D obtained, and what fraction of observations or stories is excluded from X-ID-Sim and X-T-Sim for each method?
4 comments
Response to Reviewer FHBj (Part 2/2)
Re: Backbone pretraining and training data construction
We pretrain the T2AV backbone using a three-stage curriculum on a mixture of paired audio-video, video-only, and audio-only data. The data are curated for usable samples and informative text descriptions. In the first stage, we train on video-only data with progressively increasing spatial resolution, similar to Wan [2]. In the second stage, training focuses on audio-only data while retaining some video-only samples to maintain visual capability. In the third stage, we introduce paired audio-video data for joint generation and cross-modal correspondence alongside video-only and audio-only samples. We then refine the backbone on a curated high-quality subset (Appendix B.1, lines 839–847).
For RaAVGen training, Qwen3-Omni [3] first annotates visual content and dialogue. Multi-shot scripts also include shot boundaries and timestamps. We cross-check these annotations with scene-cut detection, DiariZen [4] speaker turns, and Whisper-large-v3 speech transcripts. For long multi-shot examples, two or three consecutive clips from the same source are combined, and Qwen 3.6 [5] merges their captions into a script (Appendix B.2, lines 860–900 and Fig. 6).
Face tracking provides identity-labeled tracks and reference crops. For multi-speaker clips, DiariZen identifies speaker turns, and SyncNet associates each audio speaker with a face track using the original synchronized audio and video. Selected speech is used to produce voice references with CosyVoice [6]. We also filter for motion, visual quality, text coverage, person and speaker counts, and data validity (Appendix B.2, lines 901–913).
Re: Character assignments and metric coverage
GPT generates each test script together with a corresponding evaluation manifest. The manifest assigns a stable ID to each character and records shot IDs, expected visibility, and speaking intervals. These mappings are checked against the cast, shot descriptions, and dialogue. Visibility and speech intervals are checked for each generated output using the same criteria. Face and speaker annotations are separate, allowing for off-screen speech. Faces and speech turns are assigned according to the shared manifest, independently of the embedding similarities used for scoring (Appendix D.2, lines 1116–1124).
For X-ID-Sim, face observations are excluded when no eligible face is detected, multiple candidates remain, or the same face is assigned to multiple characters. For X-T-Sim, we exclude overlapping speech, turns outside the available media, and turns that are too short or quiet. The minimum audio duration is 1.0s and the minimum RMS level is −50 dBFS (Appendix D.2, lines 1126–1130).
Table 4 reports exclusion percentages for each method. Exclusion rates for face observations, speech turns, and shot pairs are calculated per story and averaged across the 50 stories. A shot pair is excluded if either shot lacks a valid observation. A story is excluded when it has no valid shot pair. The table uses 40-second stories except for the two-minute row (lines 1135–1151 and 1167–1172).
| Method | X-ID face obs. (%) ↓ | X-ID pairs (%) ↓ | X-ID stories (%) ↓ | X-T turns (%) ↓ | X-T pairs (%) ↓ | X-T stories (%) ↓ |
|---|---|---|---|---|---|---|
| DreamID | 7.98 | 5.76 | 4.00 | 3.36 | 7.68 | 6.00 |
| Ovi | 14.50 | 12.80 | 6.00 | 2.69 | 5.28 | 4.00 |
| TCR | 9.20 | 9.18 | 4.00 | 2.30 | 4.99 | 4.00 |
| H3 | 2.40 | 2.88 | 2.00 | 1.44 | 3.88 | 2.00 |
| JoyAI-Echo | 4.76 | 5.61 | 4.00 | 2.02 | 3.36 | 2.00 |
| Backbone (global) | 6.85 | 8.44 | 6.00 | 2.81 | 10.63 | 8.00 |
| Backbone (local) | 3.12 | 6.26 | 4.00 | 1.78 | 4.91 | 2.00 |
| RaAVGen (2s audio) | 2.28 | 4.16 | 2.00 | 2.20 | 4.80 | 4.00 |
| RaAVGen (without memory) | 3.60 | 6.18 | 4.00 | 1.90 | 3.16 | 2.00 |
| RaAVGen (local) | 2.66 | 4.54 | 2.00 | 1.58 | 7.50 | 6.00 |
| RaAVGen (global) | 2.32 | 3.91 | 2.00 | 1.39 | 2.85 | 2.00 |
| RaAVGen (global, 2min) | 2.41 | 4.70 | 4.00 | 1.52 | 4.93 | 4.00 |
With global prompting, RaAVGen excludes an average of 2.32% of face observations and 1.39% of speech turns. The corresponding shot-pair exclusion rates are 3.91% and 2.85%, respectively. Together with the cross-shot consistency scores in Table 1, these low exclusion rates support the stability and effectiveness of RaAVGen.
References
[1] Radford A et al. Robust speech recognition via large-scale weak supervision. ICML, 2023.
[2] Wan T et al. Wan: Open and advanced large-scale video generative models. arXiv:2503.20314, 2025.
[3] Xu J et al. Qwen3-omni technical report. arXiv:2509.17765, 2025.
[4] Han J et al. Leveraging self-supervised learning for speaker diarization. ICASSP, 2025.
[5] Yang A et al. Qwen3 technical report. arXiv:2505.09388, 2025.
[6] Lyu X et al. Build llm-based zero-shot streaming tts system with cosyvoice. ICASSP 2025.
Have our responses addressed your concerns?
Dear Reviewer FHBj,
We thank you for your helpful comments on our paper.
We have taken your concerns into consideration and addressed them carefully. We have also conducted additional experiments to support our claims, and we have included our responses in our rebuttal. Has this response clarified our method and experiments? We kindly ask the reviewer to reassess the paper in light of our response, if it clears things up. If not, we would be happy to provide alternative explanations, or to provide clarifications on any other part of the paper.
Thanks again for your time reviewing our paper.
Official Comment by Authors
Dear Reviewer FHBj,
Thank you again for your detailed and thoughtful feedback. Your questions have helped us strengthen the evidence for script progress tracking and clarify the training and evaluation procedures.
Following your suggestions, we have:
- Added direct boundary evaluations of repetition, omission, and speech continuity, together with a controlled 50-case study in which the next unspoken dialogue belongs to the current, previous, or next visual shot (Appendices E.3–E.4, Tables 2 and 7).
- Provided qualitative examples of unfinished visual actions continuing across generation boundaries, alongside the user study on within-shot visual continuity (Appendix E.6, Fig. 10 and Table 9).
- Compared full-script and segment-specific prompting using the same trained weights, showing improvements in speech accuracy and boundary continuation with the global script (Sec. 4.4, Tables 1–2).
- Documented backbone pretraining and the construction and checking of training scripts, shot annotations, and face–speaker correspondences (Appendices B.1–B.2, Fig. 6).
- Explained how character assignments are obtained and shared across methods, and reported per-method exclusion rates for observations, shot pairs, and stories in the cross-shot metrics (Appendix D.2, Table 4).
We hope these additions provide clearer evidence and address the questions in your review. We would welcome any further comments and would be happy to clarify any remaining concerns.
Thank you again for your time and valuable suggestions.
Response to Reviewer FHBj (Part 1/2)
We thank Reviewer FHBj for highlighting the value of allowing generation boundaries within utterances and shots and for recognizing the controlled augmentation study. We have added boundary evaluations, a controlled shot-relative continuation test, and a same-weight prompting comparison. We have also clarified the training data and cross-shot metrics. All line numbers refer to the revised manuscript.
The relative timestamps indicate the visual shot schedule. The audio history provides cues about the dialogue already spoken, allowing the model to continue unfinished dialogue even when the visual shot changes (Sec. 3.2, lines 258–276).
To evaluate speech continuation across generation boundaries, we measure B-WER on four-second audio windows spanning two seconds before and after each boundary. Whisper large-v3 [1] transcribes each window, and GPT identifies the corresponding script span using the full-sequence transcript and timestamps. We also report Repeat and Skip as percentages of judged boundaries. B-Cont is a GPT-based 1–5 rating of speech continuity when the same speaker continues an utterance (Table 2, lines 433–448 and Appendix E.3, lines 1215–1229).
With video history fixed at four latent frames, increasing audio history from 2s to 6s gives:
The WER reduction is accompanied by fewer repetitions and omissions, as well as smoother speech continuation (Tables 1–2, lines 379–397 and 433–448). Repeat and Skip focus on the spoken content, while B-Cont assesses how naturally the utterance continues. Figure 4(a) illustrates the corresponding continuation behavior (lines 504–506 and 511–514).
To further probe dialogue progress relative to the visual shot, we evaluate 50 cases, each containing three shots with the same speaker visible, under three settings, with one boundary assessed per case in each setting. Each case keeps the audio history and script content fixed while changing the relative timestamps and video history. The next unspoken dialogue belongs to the current, previous, or next shot relative to the current visual shot (Appendix E.4, lines 1241 and 1285–1295, and Table 7, lines 1274–1281).
Together with the comparison of 2s and 6s audio history in Table 2 (lines 433–448), these results support the role of audio history in locating the current dialogue position within the global script and continuing unfinished speech across generation boundaries.
The controlled augmentation study provides complementary evidence. WER in the [16, 24)s interval decreases from 5.85% to 2.53% (Table 10, lines 1362–1378).
Our main contribution is to enable coherent dialogue continuation across segments using a global script. To further examine visual continuation, we consider actions that span generation boundaries. Figure 10 presents two qualitative examples in which an action remains unfinished near the end of the first segment and continues in the next segment. One shows a person lifting a cup to their mouth, and the other shows a person closing a laptop. The next segment is conditioned on video history from the preceding segment and continues the action across the 8-second boundary (lines 1340–1341 and 1350–1360). These examples illustrate RaAVGen’s ability to continue visual actions across generation boundaries.
In addition, we conduct a user study on visual continuity at within-shot generation boundaries. Increasing video history from 2 to 4 latent frames improves the continuity rating from 2.7/5 to 4.3/5 (Table 9, lines 1307–1314 and 1327–1338).
We also compare the two prompting strategies with the same trained RaAVGen checkpoint. For local prompting, the story is divided into 8-second prompts with adjusted timestamps (Sec. 4.4, lines 477–482 and Appendix E.3, lines 1231–1237).
The global script improves overall speech accuracy and boundary continuation under the same model weights, with fewer boundaries involving repeated or skipped dialogue (Tables 1–2, lines 379–397 and 433–448).