acceptodds
Under review as a conference paper at ICLR 2027

RaAVGen: Rollout-Aware Long-Form Audio-Video Narrative Generation from Global Scripts

Abstract

Generating long-form audio-video narratives from a complete script requires tracking what has already been generated and continuing unfinished content across generation boundaries. Segment-specific prompting workflows assign this responsibility to separately prepared local prompts. We introduce a rollout-aware global script framework that jointly generates audio and video segment by segment while retaining the complete script throughout generation. Our key insight is that visual continuity and script localization require different temporal contexts. We therefore pair a short video history that captures the recent visual state with a longer audio history that helps the model locate preceding speech in the script and determine what to generate next. This asymmetric design preserves extended spoken context while limiting the token cost of dense video history. For multi-shot narratives, sparse keyframes from completed shots provide earlier scene and character appearance, while relative shot timestamps guide shot transitions. Together, these conditions allow generation boundaries to fall within shots, utterances, and actions. To address limited long-form training data, we combine complementary supervision with staged training to integrate face and voice conditioning, shot transitions, and long-sequence continuation. We further construct pseudo-long scripts by extending speech text around existing captions while retaining the original audio-video supervision, training script localization without matching full-length recordings. Experiments demonstrate long-form multi-shot generation from a complete script without separately prepared local prompts, alongside support for optional face and voice personalization in single-person and two-person settings.

Primary area:
generative models
Keywords:
Audio-Video Generation, Long-Form Generation, Multi-Shot Narrative Generation, Global Script Conditioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FHBjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-form audio-video generation from a complete narrative script. RaAVGen generates successive segments while retaining the full script, using a short video history and a longer audio history to support visual continuity and speech continuation. For multi-shot generation, it additionally uses keyframes from completed shots and updates shot timestamps relative to the current segment. The model combines several sources of supervision through three-stage training, with pseudo-long script augmentation extending the textual context without requiring longer recordings. Experiments evaluate approximately 40-second narratives, with ablations of history length, sparse memory, and script augmentation.

Critical strengths
  1. Allowing generation boundaries to fall within utterances and shots is a useful setting. The distinction between visual context for continuity and audio context for locating speech within the script is well motivated. It gives a concrete reason to retain different amounts of history for the two modalities.
  2. The ablations provide useful evidence for several design choices. Increasing audio history from two to six seconds reduces WER from 4.08% to 2.64%, while sparse visual memory improves cross-shot identity consistency from 0.776 to 0.874. The controlled augmentation experiment in Table 5 is also informative: using the same recordings, the augmented model maintains lower WER in the final evaluation interval.
Critical weaknesses
  1. The evidence for tracking narrative progress is somewhat indirect. Longer audio history improves WER, and the user study reports better segment continuity and prompt following. However, these aggregate results do not show how often the model repeats or skips unfinished content at generation boundaries. In the multi-shot setting, Eq. (1) supplies shot-level progress from the prescribed timestamps, but the evaluation does not show how reliably history identifies and continues unfinished dialogue and visual actions across generation boundaries.
  2. Important details needed to interpret and reproduce the results are missing. Appendix A identifies an in-house pretrained backbone, but does not describe its pretraining data or procedure. Appendix B lists the training data categories and sizes, but gives little detail on how scripts, speaker correspondences, and shot annotations were obtained and checked. Table 1 compares complete systems, so it also leaves the contribution of full-script conditioning relative to segment-specific prompting unresolved. Finally, Appendix D excludes invalid face and speech observations from the cross-shot consistency metrics without reporting coverage or explaining how character assignments are obtained. This makes it difficult to assess whether the reported scores reflect comparable sets of observations across methods.
Questions for authors
  1. What evidence distinguishes improved localization within the full script from improved local speech continuation? In particular, do the reported WER gains correspond to fewer repetitions and omissions at generation boundaries?
  2. Please clarify the backbone’s pretraining data and procedure and how the training scripts, speaker correspondences, and shot annotations were constructed and checked. Is there any same-backbone evidence separating full-script conditioning from segment-specific prompting?
  3. How are the fixed character assignments in Appendix D obtained, and what fraction of observations or stories is excluded from X-ID-Sim and X-T-Sim for each method?
4 comments
AuthorsOpenReview ↗

Response to Reviewer FHBj (Part 1/2)

We thank Reviewer FHBj for highlighting the value of allowing generation boundaries within utterances and shots and for recognizing the controlled augmentation study. We have added boundary evaluations, a controlled shot-relative continuation test, and a same-weight prompting comparison. We have also clarified the training data and cross-shot metrics. All line numbers refer to the revised manuscript.

Re: Script progress and boundary continuation

The relative timestamps indicate the visual shot schedule. The audio history provides cues about the dialogue already spoken, allowing the model to continue unfinished dialogue even when the visual shot changes (Sec. 3.2, lines 258–276).

To evaluate speech continuation across generation boundaries, we measure B-WER on four-second audio windows spanning two seconds before and after each boundary. Whisper large-v3 [1] transcribes each window, and GPT identifies the corresponding script span using the full-sequence transcript and timestamps. We also report Repeat and Skip as percentages of judged boundaries. B-Cont is a GPT-based 1–5 rating of speech continuity when the same speaker continues an utterance (Table 2, lines 433–448 and Appendix E.3, lines 1215–1229).

With video history fixed at four latent frames, increasing audio history from 2s to 6s gives:

Audio historyWER (%) ↓B-WER (%) ↓B-Cont ↑Repeat (%) ↓Skip (%) ↓
2s4.089.972.946.142.2
6s2.645.743.916.018.4

The WER reduction is accompanied by fewer repetitions and omissions, as well as smoother speech continuation (Tables 1–2, lines 379–397 and 433–448). Repeat and Skip focus on the spoken content, while B-Cont assesses how naturally the utterance continues. Figure 4(a) illustrates the corresponding continuation behavior (lines 504–506 and 511–514).

To further probe dialogue progress relative to the visual shot, we evaluate 50 cases, each containing three shots with the same speaker visible, under three settings, with one boundary assessed per case in each setting. Each case keeps the audio history and script content fixed while changing the relative timestamps and video history. The next unspoken dialogue belongs to the current, previous, or next shot relative to the current visual shot (Appendix E.4, lines 1241 and 1285–1295, and Table 7, lines 1274–1281).

Next dialogueB-WER (%) ↓Repeat (%) ↓Skip (%) ↓
Current shot4.7214.012.0
Previous shot4.9312.014.0
Next shot4.8614.016.0

Together with the comparison of 2s and 6s audio history in Table 2 (lines 433–448), these results support the role of audio history in locating the current dialogue position within the global script and continuing unfinished speech across generation boundaries.

The controlled augmentation study provides complementary evidence. WER in the [16, 24)s interval decreases from 5.85% to 2.53% (Table 10, lines 1362–1378).

Re: Visual action continuation

Our main contribution is to enable coherent dialogue continuation across segments using a global script. To further examine visual continuation, we consider actions that span generation boundaries. Figure 10 presents two qualitative examples in which an action remains unfinished near the end of the first segment and continues in the next segment. One shows a person lifting a cup to their mouth, and the other shows a person closing a laptop. The next segment is conditioned on video history from the preceding segment and continues the action across the 8-second boundary (lines 1340–1341 and 1350–1360). These examples illustrate RaAVGen’s ability to continue visual actions across generation boundaries.

In addition, we conduct a user study on visual continuity at within-shot generation boundaries. Increasing video history from 2 to 4 latent frames improves the continuity rating from 2.7/5 to 4.3/5 (Table 9, lines 1307–1314 and 1327–1338).

Re: Full-script and local prompting

We also compare the two prompting strategies with the same trained RaAVGen checkpoint. For local prompting, the story is divided into 8-second prompts with adjusted timestamps (Sec. 4.4, lines 477–482 and Appendix E.3, lines 1231–1237).

PromptWER (%) ↓X-T ↑B-WER (%) ↓Repeat (%) ↓Skip (%) ↓
Local3.540.9467.3723.821.4
Global2.640.9565.7416.018.4

The global script improves overall speech accuracy and boundary continuation under the same model weights, with fewer boundaries involving repeated or skipped dialogue (Tables 1–2, lines 379–397 and 433–448).

AuthorsOpenReview ↗

Response to Reviewer FHBj (Part 2/2)

Re: Backbone pretraining and training data construction

We pretrain the T2AV backbone using a three-stage curriculum on a mixture of paired audio-video, video-only, and audio-only data. The data are curated for usable samples and informative text descriptions. In the first stage, we train on video-only data with progressively increasing spatial resolution, similar to Wan [2]. In the second stage, training focuses on audio-only data while retaining some video-only samples to maintain visual capability. In the third stage, we introduce paired audio-video data for joint generation and cross-modal correspondence alongside video-only and audio-only samples. We then refine the backbone on a curated high-quality subset (Appendix B.1, lines 839–847).

For RaAVGen training, Qwen3-Omni [3] first annotates visual content and dialogue. Multi-shot scripts also include shot boundaries and timestamps. We cross-check these annotations with scene-cut detection, DiariZen [4] speaker turns, and Whisper-large-v3 speech transcripts. For long multi-shot examples, two or three consecutive clips from the same source are combined, and Qwen 3.6 [5] merges their captions into a script (Appendix B.2, lines 860–900 and Fig. 6).

Face tracking provides identity-labeled tracks and reference crops. For multi-speaker clips, DiariZen identifies speaker turns, and SyncNet associates each audio speaker with a face track using the original synchronized audio and video. Selected speech is used to produce voice references with CosyVoice [6]. We also filter for motion, visual quality, text coverage, person and speaker counts, and data validity (Appendix B.2, lines 901–913).

Re: Character assignments and metric coverage

GPT generates each test script together with a corresponding evaluation manifest. The manifest assigns a stable ID to each character and records shot IDs, expected visibility, and speaking intervals. These mappings are checked against the cast, shot descriptions, and dialogue. Visibility and speech intervals are checked for each generated output using the same criteria. Face and speaker annotations are separate, allowing for off-screen speech. Faces and speech turns are assigned according to the shared manifest, independently of the embedding similarities used for scoring (Appendix D.2, lines 1116–1124).

For X-ID-Sim, face observations are excluded when no eligible face is detected, multiple candidates remain, or the same face is assigned to multiple characters. For X-T-Sim, we exclude overlapping speech, turns outside the available media, and turns that are too short or quiet. The minimum audio duration is 1.0s and the minimum RMS level is −50 dBFS (Appendix D.2, lines 1126–1130).

Table 4 reports exclusion percentages for each method. Exclusion rates for face observations, speech turns, and shot pairs are calculated per story and averaged across the 50 stories. A shot pair is excluded if either shot lacks a valid observation. A story is excluded when it has no valid shot pair. The table uses 40-second stories except for the two-minute row (lines 1135–1151 and 1167–1172).

MethodX-ID face obs. (%) ↓X-ID pairs (%) ↓X-ID stories (%) ↓X-T turns (%) ↓X-T pairs (%) ↓X-T stories (%) ↓
DreamID7.985.764.003.367.686.00
Ovi14.5012.806.002.695.284.00
TCR9.209.184.002.304.994.00
H32.402.882.001.443.882.00
JoyAI-Echo4.765.614.002.023.362.00
Backbone (global)6.858.446.002.8110.638.00
Backbone (local)3.126.264.001.784.912.00
RaAVGen (2s audio)2.284.162.002.204.804.00
RaAVGen (without memory)3.606.184.001.903.162.00
RaAVGen (local)2.664.542.001.587.506.00
RaAVGen (global)2.323.912.001.392.852.00
RaAVGen (global, 2min)2.414.704.001.524.934.00

With global prompting, RaAVGen excludes an average of 2.32% of face observations and 1.39% of speech turns. The corresponding shot-pair exclusion rates are 3.91% and 2.85%, respectively. Together with the cross-shot consistency scores in Table 1, these low exclusion rates support the stability and effectiveness of RaAVGen.

References

[1] Radford A et al. Robust speech recognition via large-scale weak supervision. ICML, 2023.

[2] Wan T et al. Wan: Open and advanced large-scale video generative models. arXiv:2503.20314, 2025.

[3] Xu J et al. Qwen3-omni technical report. arXiv:2509.17765, 2025.

[4] Han J et al. Leveraging self-supervised learning for speaker diarization. ICASSP, 2025.

[5] Yang A et al. Qwen3 technical report. arXiv:2505.09388, 2025.

[6] Lyu X et al. Build llm-based zero-shot streaming tts system with cosyvoice. ICASSP 2025.

AuthorsOpenReview ↗

Have our responses addressed your concerns?

Dear Reviewer FHBj,

We thank you for your helpful comments on our paper.

We have taken your concerns into consideration and addressed them carefully. We have also conducted additional experiments to support our claims, and we have included our responses in our rebuttal. Has this response clarified our method and experiments? We kindly ask the reviewer to reassess the paper in light of our response, if it clears things up. If not, we would be happy to provide alternative explanations, or to provide clarifications on any other part of the paper.

Thanks again for your time reviewing our paper.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer FHBj,

Thank you again for your detailed and thoughtful feedback. Your questions have helped us strengthen the evidence for script progress tracking and clarify the training and evaluation procedures.

Following your suggestions, we have:

  • Added direct boundary evaluations of repetition, omission, and speech continuity, together with a controlled 50-case study in which the next unspoken dialogue belongs to the current, previous, or next visual shot (Appendices E.3–E.4, Tables 2 and 7).
  • Provided qualitative examples of unfinished visual actions continuing across generation boundaries, alongside the user study on within-shot visual continuity (Appendix E.6, Fig. 10 and Table 9).
  • Compared full-script and segment-specific prompting using the same trained weights, showing improvements in speech accuracy and boundary continuation with the global script (Sec. 4.4, Tables 1–2).
  • Documented backbone pretraining and the construction and checking of training scripts, shot annotations, and face–speaker correspondences (Appendices B.1–B.2, Fig. 6).
  • Explained how character assignments are obtained and shared across methods, and reported per-method exclusion rates for observations, shot pairs, and stories in the cross-shot metrics (Appendix D.2, Table 4).

We hope these additions provide clearer evidence and address the questions in your review. We would welcome any further comments and would be happy to clarify any remaining concerns.

Thank you again for your time and valuable suggestions.

Reviewer_vazMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RaAVGen generates long multi-shot audio-video narratives segment by segment from a single complete script, with no hand-split per-segment prompts. At each step the model sees the full script, a short video history, and a longer audio history that helps it locate where it is in the script. For multi-shot scripts it also gets sparse keyframes from finished shots and shot timestamps re-expressed relative to the current segment. It is built on an in-house single-stream text-to-audio-video DiT. Training has three stages (face and voice conditioning, shot transitions, long continuation) and adds pseudo-long scripts, which pad scripts with extra text but keep the original audio-video targets. On 50 GPT-written 40-second prompts, it reports better cross-shot identity and timbre consistency, shot-boundary accuracy, and WER than Ovi, DreamID-Omni, TCR, MiniMax-H3, and JoyAI-Echo.

Critical strengths
  1. Splitting the history this way is simple and well argued: a short video tail for visual state, a long, cheap audio history for script position. The ablation supports it, with WER falling from 4.08% to 2.64% when audio history grows from 2 s to 6 s.
  2. Pseudo-long script augmentation is a cheap way to train script localization without long paired recordings. Appendix Table 5 supports it: without augmentation, WER rises to 5.85% in the [16, 24) s interval, whereas it stays at 2.53% with augmentation. Removing the sparse visual memory clearly hurts cross-shot identity and background consistency, which shows each component has a distinct role.
Critical weaknesses
  1. The headline claim, a global script instead of segment-specific prompts, is never tested on the same model. The baselines get packed 10-second local prompts on different backbones, while RaAVGen runs on an unreleased in-house backbone. Some of the gain may come from the backbone and training data rather than the global-script design. An ablation of RaAVGen itself driven by local prompts is needed.
  2. The evaluation protocol is uneven. The automatic comparison uses 50 GPT-written prompts with no variance reported, although the user study in Appendix E.1 (30 participants, 30 cases) partly complements it. TCR and Ovi are conditioned on frames taken from RaAVGen's own outputs, so the identity-based comparisons are hard to read.
Questions for authors
  1. Using the same RaAVGen weights, how does generation driven by segment-specific local prompts compare with global-script generation on WER, X-T, and boundary metrics?
  2. How large is the in-house T2AV backbone, and how do it and the baselines compare on short single-shot generation? That would separate backbone quality from the method.
  3. How often does script localization fail (skipped or repeated lines) over 40 s, and does it get worse for longer scripts, for example 2 minutes?
3 comments
AuthorsOpenReview ↗

Response to Reviewer vazM (Part 1/2)

We thank Reviewer vazM for recognizing our asymmetric audio-video history, pseudo-long script augmentation, and sparse visual memory. We have added controlled prompting comparisons, a short-sequence backbone evaluation, and boundary analyses. All line numbers refer to the revised manuscript.

Re: Global and local prompting

To assess the effect of script conditioning, we evaluate RaAVGen on 40-second narratives using a global script and segment-specific prompts with the same trained weights. Local prompts are prepared for 8-second segments with adjusted timestamps (Sec. 4.4, lines 477–482 and Appendix E.3, lines 1231–1237).

PromptWER (%) ↓X-T ↑B-WER (%) ↓B-Cont ↑Repeat (%) ↓Skip (%) ↓
Local3.540.9467.373.223.821.4
Global2.640.9565.743.916.018.4

Tables 1–2 (lines 379–397 and 433–448). B-WER measures speech accuracy around generation boundaries, and B-Cont is a GPT-based rating of boundary speech continuity on a 1–5 scale. Repeat and Skip are percentages of assessed boundaries.

RaAVGen achieves more coherent speech continuation across generation boundaries with global prompting than with segment-specific prompts. The lower repeat and skip rates also support the benefit of using the full script. Figure 9 (lines 1242–1270) provides qualitative examples.

Re: Backbone size and short-sequence capability

The pretrained backbone contains 14B DiT parameters, excluding the text encoder and audio-video codecs. Its architecture and pretraining are described in Appendix A.2 (lines 748–755) and Appendix B.1 (lines 839–847). We evaluate it alongside existing methods on 50 prompts for 8-second single-shot videos (Table 6, lines 1189–1201 and Appendix E.2, lines 1206–1211).

MethodWER (%) ↓Sync-C ↑Sync-D ↓
DreamID2.624.54910.155
Ovi2.898.3917.574
TCR0.914.92211.060
H30.978.6727.347
JoyAI-Echo1.362.74213.138
Pretrained backbone1.037.4678.193

The backbone achieves competitive short-sequence performance, while the controlled comparison above directly examines the contribution of global-script conditioning.

Re: Evaluation scale and variability

The 40-second test set contains 50 GPT-generated narratives with 6–14 shots (Sec. 4, lines 359–362). We randomly generated a wide range of scene settings and designed multiple topics per scene. For example, possible scene–topic combinations include travel planning in a café, project discussions in an office, grocery shopping in a supermarket, and book discussions in a library. In future work, we plan to build a more comprehensive benchmark with more narratives and a broader range of scene settings, dialogue topics, and script sources.

Per-script results (mean ± SD) are WER (2.64 ± 1.62)%, X-ID 0.874 ± 0.051, X-T 0.956 ± 0.046, and SB-F1 (74.60 ± 4.62)%. These results suggest generally stable cross-shot identity and timbre consistency across the evaluated stories.

AuthorsOpenReview ↗

Response to Reviewer vazM (Part 2/2)

Re: Identity conditioning for TCR and Ovi

TCR and Ovi do not support direct face-reference conditioning. As a practical compromise, we use frames from our generated videos as first-frame inputs for both methods to enable quantitative comparison (Sec. 4, lines 401–404). To avoid bias in reference-identity comparisons, we omit their ID-Sim scores (Table 1, lines 379–383).

X-ID-Sim compares character identities across generated shots instead of directly with the reference image. It can therefore reflect cross-shot identity consistency to some extent under these input conditions (Appendix D.2, lines 1103–1114). RaAVGen achieves an X-ID-Sim of 0.874, compared with 0.831 for H3, 0.325 for TCR, and 0.373 for Ovi (Table 1, lines 386–397).

Our main focus is maintaining continuity across generation segments. RaAVGen achieves a GPT-rated B-Cont score of 3.9, compared with 3.4 for TCR and 3.1 for H3 (Table 2, lines 433–448). The user study also rates RaAVGen’s segment continuity at 4.1/5, compared with 3.3/5 for TCR and 3.9/5 for H3 (Table 5, lines 1154–1164).

Re: Repetition and omissions over longer sequences

We report repetition and omission as percentages of judged generation boundaries. Repeat denotes restating previously spoken content, while Skip denotes advancing past unspoken dialogue. A boundary can be counted in both (Table 2, lines 433–448 and Appendix E.3, lines 1223–1224).

Generation lengthB-WER (%) ↓B-Cont ↑Repeat (%) ↓Skip (%) ↓
40s5.743.916.018.4
Two minutes5.933.716.720.4

For the two-minute evaluation, we extend the 40-second test scripts (Sec. 4.3, lines 466–473). Repeat rises from 16.0% to 16.7% and Skip from 18.4% to 20.4%, while B-WER changes from 5.74% to 5.93%. The results indicate that boundary speech accuracy and continuation remain close to their 40-second levels over longer sequences.

Thanks for your rebuttal. I raised my review rating by one level.

Reviewer_FkmFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a framework for generating long-form, multi-shot audio-video narratives directly from a complete script, without manually splitting it into segment-specific prompts. Its key innovation is using asymmetric temporal context for audio and video, to improve boundary continuity and script localization. It additionally introduces sparse visual memory to maintain character and scene consistency across distant shots. Experiment results show the proposed method against baselines and ablations further validate the effectiveness of the proposed method.

Critical strengths

+)Intuitive and technically reasonable design. Longer audio context supports script progress tracking, while shorter video history maintains local visual continuity at a lower computational cost.

+) Comprehensive evaluation and ablations. The paper presents multiple aspects for the evaluation and conducted useful ablation studies for asymmetric audio history length, and parse visual memory.

Critical weaknesses

-) Insufficient evidence of long-horizon generalization. Experiments primarily focus on approximately 40-second videos, which is relatively short for long-form narrative generation.

-) Comparison was not fair. The paper uses in-house T2AV backbone models, and compared to existing methods with other backbones. It would be more convincing if all experiments are performed on the same backbone.

Questions for authors

a) The proposed method is built on an in-house pretrained audio-video generation backbone. Could the authors clarify the capabilities of this backbone before the proposed training stages? How much of the improvement over existing methods can be attributed to rollout-aware conditioning rather than differences in backbone capability?

b) How does the method perform on substantially longer sequences (e.g., 2–5 minutes)?

c) Discuss how computational complexity scales with script length and the accumulation of sparse visual memory

4 comments
AuthorsOpenReview ↗

Response to Reviewer FkmF (1/2)

We thank Reviewer FkmF for the constructive feedback and for recognizing our asymmetric audio-video history and ablation studies. We have revised the manuscript to clarify the backbone, add controlled prompting comparisons, evaluate longer sequences, and report computational cost. All line numbers below refer to the revised manuscript. Bold values indicate the best numerical result among the rows shown.

Re: Fairness of cross-backbone comparisons and improvements

To address your concern about backbone differences, we compare global-script and segment-specific local prompting using the same model weights for both the pretrained T2AV backbone and RaAVGen (Sec. 4.4, lines 477–482). Local prompts are constructed for 8-second segments, with timestamps adjusted as described in Appendix E.3 (lines 1231–1237).

CheckpointPromptWER (%) ↓X-T ↑SB-F1 (%) ↑B-WER (%) ↓B-Cont ↑
T2AV backboneGlobal11.410.85345.7718.761.7
T2AV backboneLocal3.110.90264.316.093.3
RaAVGenLocal3.540.94672.687.373.2
RaAVGenGlobal2.640.95674.605.743.9

Tables 1–2 (lines 379–397 and 433–448). B-WER is measured on four-second generation-boundary windows. B-Cont is a GPT-based 1–5 continuity rating (Appendix E.3, lines 1215–1229).

With the same RaAVGen weights, global prompting improves WER from 3.54% to 2.64% and B-WER from 7.37% to 5.74% relative to local prompting. The pretrained backbone instead performs worse with a global script, consistent with its short-sequence pretraining. These results support the contribution of global-script conditioning.

Under global prompting, RaAVGen reduces WER from 11.41% to 2.64% and B-WER from 18.76% to 5.74% compared with the pretrained backbone. Figure 9 (lines 1242–1270) provides qualitative comparisons for both models under global and local prompting.

Controlled component ablations provide further evidence. Extending audio history from 2s to 6s reduces B-WER from 9.97% to 5.74%, and adding sparse visual memory improves X-ID from 0.776 to 0.874 (Tables 1–2). With the same source recordings, pseudo-long script augmentation reduces WER in the [16, 24)s interval from 5.85% to 2.53% (Table 10, lines 1362–1378).

Re: Capabilities of the T2AV backbone before RaAVGen training

The pretrained backbone is a 14B-parameter DiT, excluding the text encoder and audio-video codecs. It does not support voice-reference conditioning (Appendix E.3, line 1235). Before the proposed training stages, it was pretrained for joint text-to-audio-video generation on sequences of at most 10 seconds (Appendix B.1, lines 837–847 and Appendix E.3, lines 1235–1237). To characterize its native short-sequence capability, we evaluate the backbone on 50 prompts for short, 8-second single-shot videos (Appendix E.2, lines 1206–1211 and Table 6, lines 1189–1201):

MethodWER (%) ↓Sync-C ↑Sync-D ↓
Ovi2.898.3917.574
TCR0.914.92211.060
H30.978.6727.347
Pretrained T2AV backbone1.037.4678.193

The pretrained backbone achieves short-sequence performance comparable to existing methods. However, the long-sequence comparison above shows that it struggles to follow a global script and maintain continuity across generation boundaries. These results suggest that RaAVGen’s long-form improvements arise from the proposed global-script conditioning and long-form training, rather than simply from a stronger pretrained backbone.

AuthorsOpenReview ↗

Response to Reviewer FkmF (Part 2/2)

Re: Long-horizon generation beyond 40 seconds

Our work addresses the challenge of maintaining coherent audio-visual generation across multiple segments. In the 40-second evaluation, RaAVGen follows a global script across five 8-second generation segments and achieves a GPT-rated speech continuity score of 3.9/5 at generation boundaries (Table 2, lines 433–448). To further assess longer-horizon generation, we extended the 50 test scripts from 40 seconds to two minutes (Sec. 4.3, lines 466–473 and Appendix E.8, lines 1384–1387, with examples in Fig. 11).

Generation lengthWER (%) ↓X-ID ↑X-T ↑SB-F1 (%) ↑B-WER (%) ↓B-Cont ↑Repeat / Skip (%) ↓
40s2.640.8740.95674.605.743.916.0 / 18.4
Two minutes2.970.8460.93172.765.933.716.7 / 20.4

Results are from Tables 1–2 (lines 379–397 and 433–448). Repeat and Skip are percentages of judged boundaries, and one boundary can count in both. The two-minute results remain close to the 40-second results across the reported metrics, indicating that RaAVGen maintains speech accuracy, cross-shot consistency, and boundary continuity over the longer sequences.

Re: Computational cost

We measured how script length and sparse memory affect inference cost, keeping each target chunk at 8 s (448×896) with fixed audio-video history and identity conditions. All measurements were made on a single H100 80GB GPU using BF16 (Appendix C.3, lines 1064–1069 and Table 3, lines 1027–1039).

VariableConditioning tokensMemory framesModel time / denoising step (s) ↓Peak GPU memory (GiB) ↓
Script length756124.9564.32
Script length1023125.1064.37
Script length1987125.4964.55
Memory size1023125.3264.37
Memory size1023426.9265.27
Memory size1023828.9766.47
Memory size10231231.4367.67

Times average three steps after warm-up and include all classifier-free guidance evaluations. Encoding, decoding, and I/O are excluded. Conditioning-token counts include fixed image references.

Within the tested ranges, increasing conditioning from 756 to 1,987 tokens increases per-step time by approximately 2.2%. Increasing sparse memory from 1 to 12 frames increases time by approximately 24.1% and peak memory by 3.30 GiB. A fixed-budget memory pool with scene-relevant anchor selection is a possible future direction for longer rollouts.

AuthorsOpenReview ↗

Have our responses addressed your concerns?

Dear Reviewer FkmF:

Thanks for your valuable comments. We have carefully considered your concerns and questions, and we have made every effort to address them thoroughly in our rebuttal. As the discussion phase enters its final days, we would like to ask whether our responses have adequately addressed your concerns.

We kindly ask the reviewer to reassess the paper in light of our responses, if they clear things up. If not, we would be happy to provide alternative explanations or clarifications on any part of the paper.

Once again, we thank you for your time and effort in reviewing our paper and appreciate your feedback.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer FkmF,

Thank you again for your thoughtful comments and constructive suggestions. We sincerely appreciate the time and effort you have devoted to reviewing our work.

Following your suggestions, we have:

  • Compared global-script and segment-specific local prompting using the same weights for both the pretrained backbone and RaAVGen, providing controlled evidence for the contribution of global-script conditioning (Sec. 4.4, Tables 1–2 and Fig. 9).
  • Clarified the capabilities of our 14B T2AV backbone and compared it with existing methods on short single-shot generation, showing competitive basic generation capability (Appendices B.1 and E.2, Table 6).
  • Extended the 50 test scripts from 40s to two minutes and evaluated speech accuracy, cross-shot consistency, and boundary continuation, with results remaining close to the 40s setting (Sec. 4.3, Tables 1–2 and Fig. 11).
  • Measured inference time and GPU memory usage as script length and sparse visual memory increase, with the hardware configuration and measurement procedure specified (Appendix C.3, Table 3).

We hope these additional experiments and clarifications address your concerns. We would greatly appreciate any further feedback and would be happy to discuss any remaining questions.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.