- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
この論文は、高い一貫性を持つ長時間の音付き動画の生成のための、学習・評価用データセットと、既存の音・動画同時生成モデルを拡張したモデルを提案している。データセットの構築では、チャンクレベルの局所的な細かいアノテーションを行うだけではなく、チャンク間にわたる大局的な人物・物体のidentity情報を正しく取り扱うための工夫(特にstage 4のcross-chunk revision)がなされている。生成モデルは、最新の同時生成モデル(MiniMax-H3)に長期記憶と短期記憶を扱う機構を追加し、この機構から出力されるトークンを生成時にコンテクストとして利用することで、一貫性の高い生成を実現している。短期記憶としては単純に最も近い過去のチャンクの末尾を選び、長期記憶についてはMLLMを利用して過去のチャンクから最も関連しそうなものを少数選択して、そのチャンクに含まれるキーフレーム画像や音情報、タイミングの情報を長期記憶として利用する。また、長時間にわたる生成で蓄積する誤差を低減するために、error recyclingを導入している。実験では、複数のベースライン手法と比較して、提案するベンチマークデータにおいて提案手法が多くの指標で最も良い性能を示した。
Critical strengths
- The proposed model clearly outperforms the baselines in the experiments, and the examples on the demo page show much better consistency than the baseline outputs.
- Generating minutes-long audio-visual videos with high consistency is an active research direction in this field and has clear practical value.
Critical weaknesses
- The paper provides too little information on how the dataset was built and whether it will be released. This makes it hard to verify the validity and safety of the dataset contribution, and, in addition, the quality of the data is not validated.
- The dataset is presented as one of the paper's main contributions, yet neither the main text nor the Reproducibility Statement mentions whether the dataset or code will be released. It is therefore unclear whether others will be able to verify or build on this contribution.
- The training data draw on sources such as Condensed Movies and online platforms like YouTube, and the scripts shown on the demo page appear to be clearly based on TV dramas and films. In addition, the paper does not explain how personal information such as faces was handled, how the licenses and terms of use of each source were respected, or what will be released (see "Details of Ethics Concerns").
- The filtering stage assesses video and audio independently, with no check on whether the two actually correspond or are synchronized (e.g., narration, dubbing, or audio offsets). The annotation stage assigns each utterance to a registered speaker, but no mechanism is described for off-screen speakers or narration, which could lead to misattributions. In addition, the accuracy of the annotation pipeline is not evaluated (e.g., through human inspection), so the quality of the dataset is hard to judge.
- The memory mechanism offers limited novelty, and the experiments do not adequately demonstrate the effectiveness of its specific design.
- Novelty: The core of the proposed model is the memory mechanism added to an existing joint audio-visual generator. However, short-term memory simply reuses the tail of the preceding chunk, and long-term memory delegates the selection of relevant chunks and frames entirely to an external MLLM. Using an MLLM to select past frames and storing them alongside the audio of their source chunks is essentially similar to the VLM Director of JoyAI-Echo1.5, which is included as a baseline. Selecting memory based on a query is likewise a core idea of TetherMem, and AVER is a fairly direct extension of the error recycling in SVI to audio.
- Effectiveness: The MiniMax-H3 baseline is conditioned only on the last frame of the preceding chunk, so it has no access to characters, locations, or audio from any earlier chunk. Yet the main gains are reported on Subject Recall and Background Recall, which specifically measure the reappearance of characters and locations absent from the preceding chunk. This comparison thus mainly reflects the presence or absence of long-term memory and does not by itself support the claimed "effectiveness of our memory mechanism." Table 2(b) does evaluate the proposed design, but its margins over simple alternative policies are considerably smaller than those in Table 1 (e.g., BGR 3.19→4.07 and SBR 4.21→4.33 relative to Random + HPSv3).
- Effectiveness: The proposed model is fine-tuned on CineLong-709K and evaluated with prompts in the same structured annotation format, but it is not compared against a model fine-tuned on the same data without the memory mechanism. The gains from adapting to the data and prompt format therefore cannot be separated from those of the memory mechanism.
- Effectiveness: Because the proposed model is built on MiniMax-H3, a careful comparison with MiniMax-H3 is essential. Table 1 provides a quantitative comparison, but MiniMax-H3 is absent from both the demo page and the user study, so no qualitative comparison is available.
- Reliability of evaluation: Gemini 3.1 Pro serves as the VLM judge for Subject Recall and Background Recall, and according to Appendix B, GDMA can also use Gemini 3.1 Pro as its selector. The paper does not state which MLLM was used in the main experiments, but if the same model acts as both selector and judge, the main metrics could be biased. Moreover, the evaluation covers only 30 scripts and reports no variance or confidence intervals, making it difficult to tell whether small differences in the ablations (e.g., SBR 4.27 vs 4.33) are significant.
Taken together, these concerns weigh heavily on my assessment. Since the release and rights clearance of the dataset cannot be confirmed, it is hard to evaluate it as a contribution, and the methodological contribution is largely a combination of existing techniques. Despite the strong experimental results, I do not find the contribution sufficient for acceptance in its current form.
Questions for authors
- Dataset and code release: Do the authors plan to release CineLong-709K, CineLong-Bench, the code, and the trained models? If so, in what form and under which license? Please also explain how the licenses and terms of use of the Condensed Movies and YouTube sources were respected, and how personal information such as faces was handled. In addition, please clarify whether the CineLong-Bench scripts are based on films or TV dramas.
- Data quality: Did the data construction include any check on audio-visual correspondence or synchronization (e.g., narration, dubbing, or audio offsets)? How are narration and off-screen speech represented in the annotations? If the accuracy of the annotation pipeline has been evaluated (e.g., through human inspection), please report the results.
- Effectiveness of the model: Please compare against a model fine-tuned on CineLong-709K without the memory mechanism, for example one conditioned only on the last frame of the preceding chunk as in the MiniMax-H3 baseline. This would disentangle how much of the improvement comes from the data and how much from the memory mechanism.
- Effectiveness of the model: Please compare GDMA with memory schemes from prior work implemented within the proposed model, such as the VLM Director of JoyAI-Echo1.5, which selects each character's first-appearance frame and a representative frame of the previous shot and pairs them with the audio of their source chunks. Table 2(b) includes no scheme of this kind, so the benefit of MLLM-based selection guided by the target prompt remains to be shown.