acceptodds
Under review as a conference paper at ICLR 2027

ARD: Agentic Autoregressive Diffusion for Long Video Consistency

Abstract

While diffusion models synthesize high-fidelity short clips, transforming them into consistent and coherent long videos remains challenging. Existing agentic approaches plan and coordinate segment generation, but often leave each segment unchanged once generated, leading to persistent semantic drift over long horizons. We introduce Agentic Auto-Regressive Diffusion (ARD), a training-free agentic architecture that decouples creative synthesis from consistency enforcement. ARD formulates long video synthesis as a closed-loop autoregressive process that synthesizes and self-improves video segment-by-segment against the evolving video world through a Retrieve–Synthesize–Refine–Update cycle. Its architecture combines (i) Multimodal Video Memory, which tracks visual and narrative states across segments; (ii) Adaptive Segment Generation, which selects extrapolation or interpolation according to the narrative transition; and (iii) Hierarchical Test-Time Self-Improvement, which refines boundary frames and video segments at their respective levels. We further introduce LVBENCH-C, a challenging benchmark with non-linear entity and environment transitions to stress-test long-horizon consistency. Across public and LVBENCH-C benchmarks spanning one- to ten-minute videos, ARD outperforms state-of-the-art baselines by up to 30% in consistency and 20% in narrative coherence. Human evaluations corroborate these gains while also highlighting notable improvements in motion and transition smoothness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long Video Generation, Video Consistency, Diffusion Model, Agentic AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iRQNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes A²RD, a training-free agentic framework that formulates long-video synthesis as a closed-loop, segment-level autoregressive process to improve cross-segment and transition smoothness. It also introduces LVBench-C, a benchmark with non-linear entity and environment transitions and evolving states. Result-wise, it achieves state-of-the-art temporal consistency and narrative coherence.

Critical strengths

S1. The adaptive extrapolation/interpolation mechanism is novel. It automatically selects begin-only or begin-and-end boundary frames per segment, improving consistency while preserving narrative progression.

S2. LVBench-C is a challenging benchmark for long-form T2V story generation, requiring entities to disappear for at least 10 segments and reappear with evolved states.

S3. The evaluation is comprehensive and thorough. Beyond automatic metrics and case analysis, this work also provides human studies with all-round evaluations, which further demonstrate the research gap for state consistency and superior performance of the proposed method.

Critical weaknesses

W1. Several gains are marginal. e.g., In Table 1, the proposed framework is comparable to the best baseline on inter-shot motion (0.9935 vs. 0.9903), intra-shot subject (0.9231 vs. 0.9215), and intra-shot environment (0.9514 vs. 0.9492). In Table 2, the same holds for semantic alignment (0.2215 vs. 0.2116 at 3 min) and inter-shot motion (0.9941 vs. 0.9856 at 3 min).

Questions for authors

Q1. What are the end-to-end latency and cost per complete video? Sec. E.9 gives only per-segment estimates under ideal hardware, and the reported cost (under $0.50 per segment, Sec. C.1) excludes image and video generation.

Q2. How reliable are the MLLM-as-a-judge rubrics for Boundary Frame and Video Segment Self-Improvement (Eqs. 7 and 9)? No human agreement is reported for them.

Q3. In Fig. 2, why does the full model underperform the always-interpolate variant on inter-shot character and environment consistency (0.7353 vs. 0.7400; 0.8368 vs. 0.8499)?

3 comments
AuthorsOpenReview ↗

Official replies by authors

Thank you for your insightful reviews and encouraging comments. We greatly appreciate your careful reading and the time and effort you have put into providing constructive feedback. We would like to address your concerns below with additional results and highlight our updated details in orange in our PDF manuscript.

Several gains are marginal.

Thank you for the careful observation. We would like to respectfully clarify that the automatic intra-shot consistency and inter-shot motion metrics show signs of saturation with state-of-the-art video generation models (e.g., Veo 3.1). Therefore, in our experiments, they are crucial for verifying that high local segment quality is maintained or improved, but we do not intend to use them to demonstrate substantial improvements over the baselines. Instead, our experiments aim to show A²RD’s primary benefits in tackling long-horizon inter-shot consistency and narrative coherence, where we achieve significant gains. In addition, for inter-shot motion, we emphasize more on our human evaluations in Tables 3 and 10, where human evaluators can better distinguish the naturalness and transition smoothness across segments.

Q1. What are the end-to-end latency and cost per complete video?

Thank you for your question. On VBench-Long (8 segments; 1 minute generation), on average, A²RD with our default configuration requires 79 minutes and 50 USD, compared with 26 minutes and 13 USD for VideoMemory and 17.5 minutes and 12 USD for Naive-AR as of Nov, 2026.

Additionally, we can estimate such statistics following our analytical framework in Appendix E.9 (Cost and Latency Analysis) if we always use two refinement iterations. Assume that:

  • Latency: 10s/MLLM call, 30s/image call, 120s/video call.
  • Pricing: 0.067 USD/image (Nano Banana) and 0.96 USD/video (Veo 3.1, 1080p).
  • MLLM cost for each method is 0.5 USD.
  • VideoMemory and A²RD synthesize 5 global references.
  • Maximum of kf=kv=3k_f=k_v=3 iterations.

We obtain the upper-bound end-to-end runtime and generation cost for 1 segment:

MethodRuntime (s)Total Cost│Video CostImage CostMLLM Cost
Naive-AR130$1.46│$0.96$0.00$0.50
VideoMemory170$1.53│$0.96$0.07$0.50
A²RD (Ours)590$6.66│$5.76$0.40$0.50

In practice, we see that the actual latency is quite similar to the upper bound estimate because batch inference incurs slightly more overhead than a single call, whereas A²RD sometimes stops early without using all refinement rounds. These results justify the practicality of our proposed framework, especially compared to the human effort and time needed to manually achieve high-quality long videos.

Finally, we also see opportunities to further improve the e2e efficiency: (i) we can reduce the number of refinement iterations, as Figure 3 and Table 13 show that even one iteration already brings substantial gains, and (ii) we can serve many independent generation requests in parallel to improve throughput.

Following your question, we have added a short summary of empirical end-to-end latency and cost in Appendix E.9.2, L1692-L1696.

Q2. How reliable are the MLLM-as-a-judge rubrics for Boundary Frame and Video Segment Self-Improvement (Eqs. 7 and 9)?

Thank you for your question. Our MLLM-as-a-judge rubrics in Eqs. 7 and 9 were carefully designed and calibrated during development. Following your suggestion, we have conducted a human verification study by randomly sampling 50 frame evaluations and 50 video evaluations from our generation logs, and asking three highly qualified independent human evaluators to check whether they agree with the final better and worse decisions from the MLLM-as-a-judge rubrics.

Metric / EvaluatorBoundary Frame (Eq. 7)Video Segment (Eq. 9)
Ann. 186.0%80.0%
Ann. 282.0%78.0%
Ann. 382.0%74.0%
Avg. Agreement83.3%77.3%
Krippendorff's Alpha0.66120.620

We observe an average agreement rate of 83.3% for Boundary Frame judging and 77.3% for Video Segment judging, with good inter-annotator agreement, confirming the reliability of our rubrics.

Following your question, we have added these results in Appendix E.8.2.

AuthorsOpenReview ↗

Official replies by authors (cont.)

Q3. In Fig. 2, why does the full model underperform the always-interpolate variant on inter-shot character and environment consistency (0.7353 vs. 0.7400; 0.8368 vs. 0.8499)?

Thank you for your insightful question. This reflects the fundamental trade-off between visual consistency and narrative progression in long-video generation. We explained this in Section 5.4 (L474–478).

Specifically, in the always-interpolating setting, we always synthesize the end frame for every segment. This can lead to high consistency since sometimes we see that the newly generated end frame is not very different from the segment’s beginning frame. This happens because the TI2I model struggles to anticipate how the scene should evolve over time, while being heavily conditioned on retrieved past frames and states to maintain visual faithfulness. Coupled with HITS (Video) enforcing intra-shot consistency, the generated segment can be over-constrained with little visual progression, resulting in higher consistency scores but low narrative coherence. Our adaptive mechanism balances this trade-off, achieving comparable consistency while preserving much stronger progression.

We sincerely thank the reviewer for your valuable and constructive feedback again, which has helped us improve our manuscript. We hope our clarifications address your concerns. We thank you again for your encouragement.

Thank you for your detailed response! Most of my concerns have been resolved, and I will maintain my positive rating : )

Reviewer_6yq7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents A²RD, a training-free framework for long-video generation that combines multimodal memory, adaptive extrapolation/interpolation, and hierarchical frame- and video-level refinement. By checking and refineing each segment against historical context before proceeding, it aims to reduce accumulated inconsistencies.

Critical strengths
  1. This problem is important and well-motivated. Maintaining visual consistency while allowing meaningful narrative progression is a central challenge in long-video generation.

  2. The distinction between frame-level and video-level refinement is sensible. Editing end frames obtained through extrapolation is particularly useful for reconciling natural progression with consistency constraints.

  3. The paper conducted substantial empirical evaluation to provide complementary evidence for the framework’s effectiveness.

  4. The qualitative analysis is informative. The writing is good.

Critical weaknesses
  1. The MAPO implementation details could be elaborated. Lines 286–287 define positive/negative examples as all rubric scores improving/worsening, but do not explain how mixed or unchanged scores are handled. The source of the initial examples of the prompt database remains unclear.

  2. It lacks a comprehensive empirical account of end-to-end runtime and total generation cost to show the framework’s practical efficiency.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Official replies by authors

Thank you for your constructive reviews and encouraging comments. We greatly appreciate your careful reading and the time and effort you have put into providing insightful feedback. We would like to address your concerns below and highlight our updated details in orange in our PDF manuscript.

The MAPO implementation details could be elaborated. Lines 286–287 define positive/negative examples as all rubric scores improving/worsening, but do not explain how mixed or unchanged scores are handled. The source of the initial examples of the prompt database remains unclear.

Thank you for your constructive feedback. For mixed (mixing improvements and degradations) or completely unchanged rubric scores, we exclude these cases from the prompt database to avoid ambiguous feedback. MAPO only retains a case as positive when rubric scores improve, and as negative when scores worsen (L286-287).

For the initial examples, we manually initialize the database with 20 positive and 20 negative refinement examples each for frame and video prompts. These examples cover different refinement cases, and were collected from VBench. These seed cases contain the original prompt, refined prompt, rubric scores, and refinement reasoning.

Following your feedback, we have added these details shortly in L286–287.

It lacks a comprehensive empirical account of end-to-end runtime and total generation cost to show the framework’s practical efficiency.

Thank you for your feedback. On VBench-Long (8 segments; 1 minute generation), on average, A²RD with our default configuration requires 79 minutes and 50 USD, compared with 26 minutes and 13 USD for VideoMemory and 17.5 minutes and 12 USD for Naive-AR as of Nov, 2026.

Additionally, we can estimate such statistics following our analytical framework in Appendix E.9 (Cost and Latency Analysis) if we always use two refinement iterations. Assume that:

  • Latency: 10s/MLLM call, 30s/image call, 120s/video call.
  • Pricing: 0.067 USD/image (Nano Banana) and 0.96 USD/video (Veo 3.1, 1080p).
  • MLLM cost for each method is at most 0.5 USD.
  • VideoMemory and A²RD synthesize 5 global references.
  • Maximum of kf=kv=3k_f=k_v=3 iterations.

We obtain the upper-bound end-to-end runtime and generation cost for 1 segment:

MethodRuntime (s)Total Cost│Video CostImage CostMLLM Cost
Naive-AR130$1.46│$0.96$0.00$0.50
VideoMemory170$1.53│$0.96$0.07$0.50
A²RD (Ours)590$6.66│$5.76$0.40$0.50

In practice, we see that the actual latency is quite similar to the upper bound estimate because batch inference incurs slightly more overhead than a single call, whereas A²RD sometimes stops early without using all refinement rounds. These results justify the practicality of our proposed framework, especially compared to the human effort and time needed to manually achieve high-quality long videos.

Finally, we also see opportunities to further improve the e2e efficiency: (i) we can reduce the number of refinement iterations, as Figure 3 and Table 13 show that even one iteration already brings substantial gains, and (ii) we can serve many independent generation requests in parallel to improve throughput.

Following your question, we have added a short summary of empirical end-to-end latency and cost in Appendix E.9.2, L1692-L1696.

We sincerely thank the reviewer for your valuable and constructive feedback again, which has helped us improve our manuscript. We thank you again for your encouragement.

Reviewer_tpuWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes A²RD, a training-free agent-based long video generation framework. It generates segment by segment in an autoregressive manner using the combindaiton of MLLM, image generation/editing models and image-to-video models. Each segment undergoes a closed loop of retrieving memory—synthesizing—self-checking and repairing—writting back to memory. The authors specifically propose strategies such as multimodal Video MEMory and HIerarchical Test-time Self-improvement, as well as a new benchmark LVBench-C, demonstrating the effectiveness of the proposed method and dataset.

Critical strengths
  1. The paper tackles a real problem in segment-by-segment long video generation, where a mistake in one segment gets passed on to every later one, and it builds a complete pipeline that checks and fixes each segment before the next one is generated.

  2. The method consistently beats the baselines on both benchmarks, in the human study, with open-weight models, and under matched compute budgets, which suggests the gains do not come just from sampling more candidates and keeping the best one.

  3. The paper uses ablations to show clearly how keeping segments consistent trades off against moving the story forward, and it adds a long-video benchmark built to test characters and locations that come back after a long absence.

Critical weaknesses
  1. I think it is a very inspiring work overall, but hacking warrants caution. The method revises its videos using Gemini scores during generation, while key metrics such as narrative coherence are scored by another Gemini model on similar criteria with no variance reported, so part of the gain may come from fitting the judge rather than from better videos. I hope this bias can be avoided.

  2. I think LVBench-C still has aspects that could be made more solid. This dataset has 125 scenes, and its overall scale is not large enough. It may not robustly reflect changes in model performance. I hope the scale can be further expanded to demonstrate effectiveness. In addition, at the current scale of several hundred samples, I think reviewing each sample one by one is feasible, and there is no need for spot checks.

  3. I suggest improving the scoring of LVBench-C to prevent it from being hacked. It seems that it currently focuses mainly on whether things stay the same before and after, and it does not check whether what should change has actually changed. Although this seems to have limited impact on the current evaluation, considering general scenarios after its release, I recommend that the authors improve this evaluation dimension.

Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Official replies by authors

Thank you for your constructive reviews and insightful suggestions. We greatly appreciate the time and effort you have put into providing constructive feedback. We would like to address your concerns below with additional results, and highlight our updated details in orange in our PDF manuscript.

Hacking warrants caution. The method revises its videos using Gemini scores during generation, while key metrics such as narrative coherence are scored by another Gemini model on similar criteria with no variance reported, so part of the gain may come from fitting the judge rather than from better videos.

Thank you for your feedback. Indeed, we have carefully avoided this in our work. We would like to respectfully clarify that among our main evaluation metrics, only Narrative Coherence is measured by Gemini 3.1 Pro with Self-Consistency. We did this following Wang et al. (2025a) (L362) because it is the only way to measure narrative coherence automatically. This metric was carefully designed by our experts using incremental scoring and Self-Consistency, and is independent of the metrics used during development. Importantly, it shows highly similar trends with our other independent Narrative Coherence human evaluations in Tables 3 and 10. For example, Naive-AR is the best among baselines on VBench-Long, while A²RD outperforms baselines significantly. Finally, we also want to note that Gemini 3.1 Pro is not used during generation; we use a different model, Gemini 3 Flash, as detailed in L338. This avoids directly optimizing against the same model used for final evaluation.

Cross-model verification. To further address your concern, we additionally cross-check Narrative Coherence using two independent non-Gemini models, Qwen 3.8 Omni and Xiaomi MiMo-V2.5, on VBench-Long generated videos.

MethodGemini 3.1 ProQwen 3.8 OmniXiaomi MiMo-V2.5
Naive-AR0.74660.66250.6793
Naive-Par0.67960.58250.5500
MovieAgent0.60710.52710.5520
ViMax0.69120.63570.6321
VideoMemory0.67170.65920.7033
A²RD-Par (Ours)0.80820.72160.7862
A²RD (Ours)0.89870.80880.9012

We observe that A²RD consistently achieves the highest Narrative Coherence scores across all three evaluators, providing further evidence that its improvements indeed reflect better narrative quality.

Narrative Coherence variance. We also appreciate your concern regarding Narrative Coherence variance. Following your suggestion, we conducted an additional sanity check by repeating the Narrative Coherence evaluation three times, reporting the standard deviation (SD), and performing a one-sided t-test to assess the statistical significance of the improvements:

MethodOriginal Narrative CoherenceNew Run 1New Run 2SDΔ (Ours - Baseline)p-value
Direct Prompting0.38750.41780.35720.0303+0.5112<< 0.001
Naive-AR0.74660.72160.69660.0250+0.1521<< 0.001
Naive-Par0.67960.70950.67970.0172+0.2191<< 0.001
MovieAgent0.60710.55930.58050.0240+0.2916<< 0.001
ViMax0.69120.71100.67880.0162+0.2075<< 0.001
VideoMemory0.67170.66320.65020.0108+0.2270<< 0.001
A²RD-Par (Ours)0.80820.79030.82910.0194+0.09052.66 × 10^(-3)
A²RD (Ours)0.89870.91590.89910.0098——

We see that the standard deviations remain relatively low across all methods, ranging from 0.0098 to 0.0303, with the highest variance observed for the lowest-performing baseline and only 0.0098 for A²RD. These results support the reliability of our evaluation framework, which we attribute to structured incremental scoring rubrics and self-consistency.

Following your suggestion, we have added these results to Appendix D.7 Narrative Coherence Reliability.

AuthorsOpenReview ↗

Official replies by authors (cont.)

This dataset has 125 scenes, and its overall scale is not large enough. I hope the scale can be further expanded to demonstrate effectiveness.

Thank you for your feedback. We would like to respectfully clarify that LVBench-C actually has 5,200 screen descriptions (as detailed in Table 6) and 125 generation scenarios, since each scenario consists of multiple screens: a 3-minute scenario contains 24 screens, a 5-minute contains 40, and a 10-minute contains 80. In total, this amounts to 50 × 24 + 50 × 40 + 25 × 80 = 5200 screens, which is, as you see in Table 6, already more than most existing datasets.

Given the large number of screen descriptions, we conducted spot checks instead of manually reviewing every screen. We agree with you that scaling further is better. However, while our paradigm allows for scaling further, we believe that this requires more systematic quality control to ensure good benchmark quality. We will encourage future studies on it accordingly.

In addition, regarding diversity, our human-in-the-loop process actually involves manually reviewing and modifying story topics in the prompt (we said ``manual review and prompt modification’’ in our paper in L311) across diverse categories, including adventure, crime and mystery, everyday life, sports, and science fiction. Examples of some of these diverse topics can be found in Figures 17–19 and Section I (Full Storyboard Examples).

Following your suggestion, we have added a small detail in L303 that LVBench-C has 5200 screens and diversity topics in Appendix F.3, L1734.

I suggest improving the scoring of LVBench-C to prevent it from being hacked. It seems that it currently focuses mainly on whether things stay the same before and after, and it does not check whether what should change has actually changed. Although this seems to have limited impact on the current evaluation, considering general scenarios after its release, I recommend that the authors improve this evaluation dimension.

Thank you for this thoughtful suggestion. We would like to clarify that we actually implemented this reasoning evaluation dimension in our Appendix D.3. Specifically, our MLLM-based consistency evaluation protocol in Appendix D.3 already considers expected state changes justified by the storyline (i.e., what should change) and distinguishes them from unexplained inconsistencies (i.e., what actually changed).

In addition, we also complement the evaluation dimension you suggested. Our Narrative Coherence metric (see F.4.1 for its prompt) explicitly evaluates story, character, object, and environment progression, including whether state changes are causally justified by the storyline. Finally, we would also like to note that our Inter-Shot Character and Environment Consistency metrics do not simply evaluate that things should stay the same in the full long video. As detailed in Appendix F.4, these metrics compare characters and environments only when they are expected to remain the same.

Following your suggestion, we have added a sentence in L371 to mention this evaluation protocol and direct readers to Appendix D.3 for further details.

We thank the reviewer again for your valuable and constructive feedback, which has helped us improve our manuscript. We will be glad to clarify anything further. We hope our clarifications address your concerns and improve your ratings.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

We hope this message finds you well. Thank you once again for your constructive feedback, time, and effort in reviewing our paper.

We have posted our rebuttal and revised manuscript addressing your concerns and would be happy to provide any additional clarifications. As the discussion period is approaching its end, we invite you to have a look, and we look forward to engaging with you!

Best regards, Authors

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer tpuW,

Thank you again for your time and constructive feedback on our paper. We would like to kindly follow up and ask whether our rebuttal, including new Narrative Coherence verification results, has addressed your comments. We would be happy to discuss any remaining questions you might have.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Paper Updates

Dear Reviewers and Area Chairs,

Thank you for taking the time to review our paper and provide thoughtful and constructive feedback.

In our rebuttal, we have provided responses to each reviewer's comments and revised our manuscript. All changes are highlighted in orange in our manuscript. The updates include:

  • Narrative coherence reliability (Appendix D.7): We add evaluations using two independent non-Gemini judge models and repeated scoring with variance and significance tests. A²RD consistently outperforms all baselines across evaluators, supporting the robustness of its gains.

  • Human validation of boundary frame and video segment self-improvement rubrics (Appendix E.8.2): We conduct a human verification study, finding the rubrics are reliable with average human agreement of 83.3% and 77.3%, respectively.

  • End-to-end runtime and cost (Appendix E.9): We report empirical runtime and total generation costs for complete videos, including comparisons with VideoMemory and Naive-AR, and discuss opportunities to improve efficiency.

  • Method and benchmark clarifications: We clarify MAPO’s initialization and handling of mixed or unchanged feedback (Section 3), LVBench-C’s scale of 5,200 screen descriptions and topic diversity (Section 4), and how our evaluation accounts for expected state changes with consistency (Section 5.2).

We hope that our responses and the additional experiments address the reviewers' comments and clarify our paper's strengths and contributions. We would be happy to discuss if there are any remaining questions during the discussion period.

Thank you once again for your valuable feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.