acceptodds
Under review as a conference paper at ICLR 2027

Cine-Long: Consistent Long-Horizon Multi-Shot Audio-Visual Generation

Abstract

Long-horizon, multi-shot audio-visual generation remains challenging in both data and modeling. Although modern generators can already produce multiple shots within a single generation chunk, existing datasets are not explicitly organized around such chunks or the cross-chunk dependencies encountered during iterative long-video generation. On the modeling side, sustaining a coherent long video requires selecting the relevant and reliable visual and audio context from an ever-growing generation history. We address both challenges with three contributions. First, we introduce a long-video dataset annotated at the chunk level, where each chunk spans multiple shots, aligning supervision with how modern models generate and targeting cross-chunk coherence. Second, we present Cine-Long, which equips the generator with an audio-visual memory bank as explicit context during training, so it learns to condition on key frames and audio drawn from past chunks rather than on a fixed window. Third, at inference we introduce a memory harness, an agentic controller that steers each target generation by retrieving, from the model's own generated history, the key frames and audio segments most relevant to the current shot—under a fixed memory capacity and without rewriting the script or re-sampling the generator. To evaluate this setting, we build CineLong-Bench, a benchmark for multi-shot audio-visual generation with character re-entry and voice continuation. Cine-Long achieves state-of-the-art coherence and audio-visual consistency on CineLong-Bench.

Primary area:
generative models
Keywords:
Video Generation, Multi-Modal Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bc81OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

この論文は、高い一貫性を持つ長時間の音付き動画の生成のための、学習・評価用データセットと、既存の音・動画同時生成モデルを拡張したモデルを提案している。データセットの構築では、チャンクレベルの局所的な細かいアノテーションを行うだけではなく、チャンク間にわたる大局的な人物・物体のidentity情報を正しく取り扱うための工夫(特にstage 4のcross-chunk revision)がなされている。生成モデルは、最新の同時生成モデル(MiniMax-H3)に長期記憶と短期記憶を扱う機構を追加し、この機構から出力されるトークンを生成時にコンテクストとして利用することで、一貫性の高い生成を実現している。短期記憶としては単純に最も近い過去のチャンクの末尾を選び、長期記憶についてはMLLMを利用して過去のチャンクから最も関連しそうなものを少数選択して、そのチャンクに含まれるキーフレーム画像や音情報、タイミングの情報を長期記憶として利用する。また、長時間にわたる生成で蓄積する誤差を低減するために、error recyclingを導入している。実験では、複数のベースライン手法と比較して、提案するベンチマークデータにおいて提案手法が多くの指標で最も良い性能を示した。

Critical strengths
  • The proposed model clearly outperforms the baselines in the experiments, and the examples on the demo page show much better consistency than the baseline outputs.
    • Generating minutes-long audio-visual videos with high consistency is an active research direction in this field and has clear practical value.
Critical weaknesses
  • The paper provides too little information on how the dataset was built and whether it will be released. This makes it hard to verify the validity and safety of the dataset contribution, and, in addition, the quality of the data is not validated.
    • The dataset is presented as one of the paper's main contributions, yet neither the main text nor the Reproducibility Statement mentions whether the dataset or code will be released. It is therefore unclear whether others will be able to verify or build on this contribution.
    • The training data draw on sources such as Condensed Movies and online platforms like YouTube, and the scripts shown on the demo page appear to be clearly based on TV dramas and films. In addition, the paper does not explain how personal information such as faces was handled, how the licenses and terms of use of each source were respected, or what will be released (see "Details of Ethics Concerns").
    • The filtering stage assesses video and audio independently, with no check on whether the two actually correspond or are synchronized (e.g., narration, dubbing, or audio offsets). The annotation stage assigns each utterance to a registered speaker, but no mechanism is described for off-screen speakers or narration, which could lead to misattributions. In addition, the accuracy of the annotation pipeline is not evaluated (e.g., through human inspection), so the quality of the dataset is hard to judge.
  • The memory mechanism offers limited novelty, and the experiments do not adequately demonstrate the effectiveness of its specific design.
    • Novelty: The core of the proposed model is the memory mechanism added to an existing joint audio-visual generator. However, short-term memory simply reuses the tail of the preceding chunk, and long-term memory delegates the selection of relevant chunks and frames entirely to an external MLLM. Using an MLLM to select past frames and storing them alongside the audio of their source chunks is essentially similar to the VLM Director of JoyAI-Echo1.5, which is included as a baseline. Selecting memory based on a query is likewise a core idea of TetherMem, and AVER is a fairly direct extension of the error recycling in SVI to audio.
    • Effectiveness: The MiniMax-H3 baseline is conditioned only on the last frame of the preceding chunk, so it has no access to characters, locations, or audio from any earlier chunk. Yet the main gains are reported on Subject Recall and Background Recall, which specifically measure the reappearance of characters and locations absent from the preceding chunk. This comparison thus mainly reflects the presence or absence of long-term memory and does not by itself support the claimed "effectiveness of our memory mechanism." Table 2(b) does evaluate the proposed design, but its margins over simple alternative policies are considerably smaller than those in Table 1 (e.g., BGR 3.19→4.07 and SBR 4.21→4.33 relative to Random + HPSv3).
    • Effectiveness: The proposed model is fine-tuned on CineLong-709K and evaluated with prompts in the same structured annotation format, but it is not compared against a model fine-tuned on the same data without the memory mechanism. The gains from adapting to the data and prompt format therefore cannot be separated from those of the memory mechanism.
    • Effectiveness: Because the proposed model is built on MiniMax-H3, a careful comparison with MiniMax-H3 is essential. Table 1 provides a quantitative comparison, but MiniMax-H3 is absent from both the demo page and the user study, so no qualitative comparison is available.
    • Reliability of evaluation: Gemini 3.1 Pro serves as the VLM judge for Subject Recall and Background Recall, and according to Appendix B, GDMA can also use Gemini 3.1 Pro as its selector. The paper does not state which MLLM was used in the main experiments, but if the same model acts as both selector and judge, the main metrics could be biased. Moreover, the evaluation covers only 30 scripts and reports no variance or confidence intervals, making it difficult to tell whether small differences in the ablations (e.g., SBR 4.27 vs 4.33) are significant.

Taken together, these concerns weigh heavily on my assessment. Since the release and rights clearance of the dataset cannot be confirmed, it is hard to evaluate it as a contribution, and the methodological contribution is largely a combination of existing techniques. Despite the strong experimental results, I do not find the contribution sufficient for acceptance in its current form.

Questions for authors
  • Dataset and code release: Do the authors plan to release CineLong-709K, CineLong-Bench, the code, and the trained models? If so, in what form and under which license? Please also explain how the licenses and terms of use of the Condensed Movies and YouTube sources were respected, and how personal information such as faces was handled. In addition, please clarify whether the CineLong-Bench scripts are based on films or TV dramas.
  • Data quality: Did the data construction include any check on audio-visual correspondence or synchronization (e.g., narration, dubbing, or audio offsets)? How are narration and off-screen speech represented in the annotations? If the accuracy of the annotation pipeline has been evaluated (e.g., through human inspection), please report the results.
  • Effectiveness of the model: Please compare against a model fine-tuned on CineLong-709K without the memory mechanism, for example one conditioned only on the last frame of the preceding chunk as in the MiniMax-H3 baseline. This would disentangle how much of the improvement comes from the data and how much from the memory mechanism.
  • Effectiveness of the model: Please compare GDMA with memory schemes from prior work implemented within the proposed model, such as the VLM Director of JoyAI-Echo1.5, which selects each character's first-appearance frame and a representative frame of the previous shot and pairs them with the audio of their source chunks. Table 2(b) includes no scheme of this kind, so the benefit of MLLM-based selection guided by the target prompt remains to be shown.
Reviewer_MEiDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CineLong, a framework for long-horizon multi-shot audio-visual generation. It constructs a large-scale dataset through global entity registration, local captioning, and cross-chunk annotation refinement. The method fine-tunes MiniMax-H3 with long-term memory containing historical keyframes and audio, and short-term memory preserving recent audio-visual context. A training-free Goal-Driven Memory Agent selects relevant historical chunks and per-shot keyframes, while Audio-Visual Error Recycling improves robustness to imperfect history during training. The paper also introduces CineLong-Bench to evaluate long-range consistency and script adherence, reporting improvements in character and background recall and audio continuity over existing baselines.

Critical strengths
  1. Potentially useful data contribution. CineLong-709K provides large-scale annotations covering entities, dialogue, and cross-chunk dependencies. Its value lies primarily in scale and the integrated annotation workflow; character registration builds on established character-bank approaches such as MovieBench.

  2. Practical extension of error recycling. AVER extends SVI’s error-recycling strategy to audio-visual context. Retrained ablations report improvements in background recall and script adherence over video-only recycling, supporting its practical value as an incremental extension, although gains are not consistent across all metrics.

Critical weaknesses

Critical Weaknesses

  1. The benefits of additional training are insufficiently isolated. CineLong fine-tunes MiniMax-H3 on newly annotated data, but a same-data, comparable-budget fine-tuning baseline is not reported. The inference-only memory ablations establish the trained model’s dependence on memory, but do not separate the proposed design’s benefits from those of additional training.

  2. The methodological novelty appears incremental. Character registration has close precedents in character-bank-based annotation, particularly MovieBench, while AVER explicitly builds on SVI’s error recycling. The contribution beyond these established ideas would be clearer with quantitative validation of the proposed annotation refinements and audio-specific training adaptations.

  3. TA-RoPE’s positional extrapolation behavior is insufficiently evaluated. Preserving original timestamps allows memory-to-target temporal offsets to grow even with a fixed memory budget. Prior AR video work identifies positional extrapolation beyond the training regime as a potential source of attention degradation and visual drift. The main benchmark’s approximately 144-second horizon is comparable to the maximum training-video duration of 148 seconds, leaving stability at substantially larger temporal gaps unclear.

  4. Efficiency and conditioning-budget comparability remain unclear. Appendix B specifies candidate reduction and a fixed two-chunk LTM budget. However, the selector still considers candidates across the full history, and two chunks can contain different numbers of images and audio tokens. Actual selection overhead and comparisons under matched conditioning budgets are not reported.

  5. The audio-specific gains are mixed. Compared with video-only recycling, AVER improves audio continuity from 0.480 to 0.486, while voice consistency decreases from 0.467 to 0.448. The absence of uncertainty estimates makes it difficult to assess the reliability of these differences.

Questions for authors

Questions for Authors

  1. Can you provide a MiniMax-H3 baseline fine-tuned on the same data and annotations, using its native conditioning interface and a comparable budget? Please also report the LoRA configuration and training schedule.

  2. Beyond the candidate-reduction rules in Appendix B, what are GDMA’s actual token usage, latency, and cost as history grows? Do its gains persist under matched image and audio-token budgets?

  3. What memory-to-target temporal offsets are encountered during training and inference? Can you evaluate TA-RoPE beyond the training horizon and compare it with bounded or remapped temporal indices using the same retrieved memories, reporting identity and scene drift as temporal gaps increase?

  4. Appendix A.3 describes local grounding and cross-chunk revision. Can you quantify their annotation accuracy and individual contributions, and clarify how entities missed during initial global registration are recovered?

  5. Can you provide uncertainty estimates for the AVER ablations and discuss the decrease in voice consistency? A comparison against strength-matched generic audio perturbations would also help establish the specific value of recycling model-generated audio errors.

Reviewer_XQUWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents CineLong, a framework for consistent long-horizon, multi-shot audio-visual generation. It combines long-term memory for recalling recurring characters, voices, and locations with short-term memory for maintaining motion and speech continuity.

Critical strengths

The paper addresses an problem: preserving recurring characters, voices, and locations while maintaining ongoing motion and speech across chunk boundaries. The separation into long-term retrieval memory and short-term continuation memory is intuitive and well aligned with these distinct requirements. The global-to-local annotation pipeline, particularly its shared entity registry and cross-chunk revision, also provides a coherent data foundation for this formulation.

Critical weaknesses

CineLong-Bench contains only 30 cinematic scripts, each comprising 10 chunks and approximately 144 seconds of generated content (Section 3.2). This is a useful initial benchmark, but it does not establish robustness over substantially longer narratives, more diverse domains, or increasingly long character absences. The reported results also lack confidence intervals or variability across generation seeds, making the reliability of smaller improvements difficult to assess.

The audio continuity metric provides incomplete evidence for the central acoustic claim. Section 5.1 measures Audio Continuity using LAION-CLAP similarity across chunk boundaries. Semantic similarity alone does not directly establish seamless waveform transitions, uninterrupted speech, or stable acoustic conditions. Two segments could remain semantically similar while containing audible discontinuities. The overall human preference study does not isolate these boundary-specific audio properties. Targeted listening evaluations and boundary-sensitive measurements would make the acoustic continuity claims more convincing.

The scalability of goal-driven retrieval remains unverified. Section 4.3 retains the full history and uses an external MLLM to score historical chunks and candidate frames against the target prompt. Bounding the generator’s selected memory does not necessarily bound the cost of searching that history. The main text provide no retrieval latency, total generation cost, history-storage requirements, or scaling measurements as the number of chunks increases. This is an important omission for a method specifically motivated by long-horizon generation.

Questions for authors

Can the authors provide an equal-data, equal-training-budget comparison between CineLong and its backbone equipped with a simpler memory strategy, as well as separately trained no-LTM and no-STM variants? How much of the improvement remains after controlling for CineLong-709K training and the distribution shift introduced by inference-only ablations?

How are the visual and audio prediction residuals in Eq. (4) constructed, stored, scaled, and sampled during training? Can the authors explain why adding audio recycling improves CLAP-based Audio Continuity while reducing Voice Consistency and LSE-C relative to video-only recycling, and verify this trade-off through an audio-focused listening study?

Is the model trained from scratch or from a pretained open source model? Am I missing something?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.