acceptodds
Under review as a conference paper at ICLR 2027

When the Future Remembers: Past Reconstruction in Autoregressive Long Video Generation

Abstract

As autoregressive video diffusion models become the new paradigm for long-form video generation, a fundamental issue remains: these models tend to forget earlier content once it falls outside the bounded KV cache, resulting in identity drift and scene resets that compromise long-term temporal consistency. We propose Eviction Forcing, a simple yet effective training strategy that teaches the model to recover the past by conditioning on its own future. After evicting a prefix of frames from a standard base teacher rollout, a lightweight LoRA adapter is trained to reconstruct the evicted frames at their original RoPE indices, using only the remaining frames that are later on the temporal axis as context. This approach teaches the model to remember the past beyond the immediate attention window, thereby significantly improving video consistency across long durations. The model is further optimized via periodic supervision from the base teacher model under full-context rollouts, ensuring its stable performance during standard rollouts. Extensive experiments demonstrate that Eviction Forcing achieves competitive overall VBench scores and consistently improves long-range temporal consistency over state-of-the-art autoregressive baselines.

Primary area:
generative models
Keywords:
Computer Vision, Video Generation, Generative AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jLAHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies long horizon consistency in autoregressive video diffusion and argues that existing methods only control which context is retained or indexed in the bounded KV cache, without teaching the model to recover information from evicted frames. The paper proposes Eviction Forcing, which evicts a prefix of frames from a frozen teacher’s rollout while keeping the original RoPE indices of the remaining frames, then trains a LoRA adapter to reconstruct the evicted frames using only the later frames as context, with a periodic preservation loss against the teacher. Built on Self Forcing and combined with Infinity-RoPE at inference, Eviction Forcing achieved competitive overall VBench scores.

Critical strengths
  1. The method is lightweight. It trains a rank-64 LoRA on a single A100 for 24 hours. Once merged, it adds almost no inference overhead, maintaining about 17 FPS on a single H100.
  2. Demonstrated on diverse backbones. The proposed method, without backbone-specific tuning, consistently improves consistency and overall scores across three backbones (Self Forcing, CausVid, and SkyReels-V2-DF-1.3B).
Critical weaknesses
  1. Unclear why past reconstruction improves forward generation. The adapter is trained only to reconstruct evicted frames, while at inference it simply generates forward, and the paper provides no analysis linking the two.
  2. Gains over Infinity-RoPE are marginal and not disentangled. The method uses Infinity-RoPE at inference, yet its overall score matches Infinity-RoPE at 60s and 120s. The baselines in Table 4 also exclude Infinity-RoPE, so it is unclear how much of the improvement comes from the proposed training.
  3. Generated videos appear repetitive. Many long rollouts in the supplementary seem to loop or revert to earlier frames, possibly as a side effect of training the model to reproduce past frames.
  4. Contribution is overstated. The contribution list claims that the method outperforms baselines on overall metrics, but it is best only at 240s.
Questions for authors
  1. Why does past reconstruction improve forward generation? Could the authors provide supporting analysis or a theoretical explanation?
  2. Could the authors report results without Infinity-RoPE, and re-report Table 4 with Base + Infinity-RoPE as the baseline? This is needed to show that the two components contribute independently.
  3. Could the authors report the proportion of generated videos that loop or revert to earlier frames? This would clarify whether the reconstruction training causes the model to generate mostly repetitive videos.
Reviewer_xJMjOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Eviction Forcing, a training strategy for improving long-horizon autoregressive video generation under a bounded KV cache. The method trains a lightweight LoRA adapter to reconstruct evicted past frames using the remaining future frames as context, while periodically applying teacher supervision to preserve standard generation behavior. Experiments on VBench-Long show improvements in subject and background consistency across multiple generation durations, with additional ablations, cross-backbone evaluations, and a user study.

Critical strengths

The paper presents a simple and interesting training strategy for addressing a practical limitation of autoregressive video generation. Instead of increasing the cache size or relying solely on inference-time modifications, it trains the model to reconstruct evicted content from its remaining future context. The idea is technically concrete and could complement existing approaches to long-horizon generation.

The experimental results show consistent improvements in subject and background consistency across 60-, 120-, and 240-second generation settings. The paper also includes several useful ablations, evaluations on additional backbones, and a user study, providing evidence beyond a single quantitative benchmark.

Critical weaknesses

The experiments do not fully isolate the contribution of the central future-conditioned reconstruction objective. The ablations on teacher supervision compare different reconstruction targets and preservation strategies, but do not directly compare the proposed objective with ordinary forward teacher-supervised fine-tuning under otherwise matched conditions. Such a comparison would help establish whether reconstructing the past from future frames is specifically responsible for the observed long-horizon improvements.

The evaluation protocol is not sufficiently detailed to assess the robustness of some reported gains. The paper evaluates more than 100 videos across four durations, but does not clearly specify the number of prompts and generations per duration or report uncertainty for the main benchmark results. This matters particularly when the overall-score differences are small, such as the 0.002 gap over Infinity-RoPE at 120 seconds and over Reward Forcing at 240 seconds. The baseline configurations in the cross-backbone experiments could also be clarified to make the contribution of the training strategy easier to interpret.

The qualitative examples mainly feature a single prominent subject with relatively stable scene layouts. It remains unclear whether Eviction Forcing can maintain consistency in more complex scenarios involving multiple interacting subjects, diverse actions, or substantial scene changes. Evaluating such cases would help clarify how well the approach generalizes beyond the demonstrated examples.

Questions for authors

Can the authors compare Eviction Forcing with ordinary forward teacher-supervised fine-tuning under otherwise matched training conditions?

How many prompts and generations are used for each duration, and are the reported improvements robust across prompts or random seeds?

Does Eviction Forcing remain effective for prompts involving multiple interacting subjects, complex actions, or substantial scene changes?

Reviewer_xG8KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Eviction Forcing, a training strategy for autoregressive long-video generation. A LoRA adapter reconstructs evicted prefix frames from the remaining later KV states, while periodic teacher supervision helps preserve standard forward-generation behavior. At inference, the adapter is merged into the backbone and combined with Infinity-RoPE. Experiments evaluate generation up to 240 seconds, with additional backbone transfers, ablations, and a user study.

Critical strengths
  1. The paper turns cache eviction into an explicit reconstruction task. Improved reconstruction quality and long-horizon subject consistency provide encouraging evidence for the approach, although the specific contribution of backward reconstruction remains to be isolated.

  2. The method trains a small LoRA adapter on short rollouts and requires no online reconstruction during inference. Applying the training recipe to Self Forcing, CausVid, and SkyReels-V2 yields consistent improvements, suggesting applicability beyond a single generator.

Critical weaknesses
  1. The specific benefit of backward reconstruction is not sufficiently isolated. The paper lacks compute-matched comparisons with ordinary LoRA fine-tuning or forward reconstruction using the same teacher targets and perceptual supervision. Consequently, it remains unclear how much improvement comes from the proposed temporal direction, rather than additional adaptation and regularization.

  2. The evaluation does not directly establish retention of information beyond the currently visible context. The reconstruction diagnostic closely matches the training objective, while subject and background consistency can be maintained through continuously visible cues. Tests involving objects leaving and later re-entering the view would better distinguish genuine historical retention from improved appearance stability.

Questions for authors

Does backward reconstruction outperform compute-matched LoRA fine-tuning or forward reconstruction on tasks that require recovering previously observed details no longer visible in the current context, when teacher targets and perceptual supervision are held fixed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.