acceptodds
Withdrawn from ICLR 2027

Are We Caching the Right History for Autoregressive Video Generation?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In autoregressive video generation, the historical key-value (KV) cache is crucial for long-term visual consistency, yet GPU memory and real-time inference constraints require it to remain bounded. A more recent view decomposes historical KV into sink, middle, and recent regions, yet prior studies have centered more on the two boundary regions, leaving the expanding and potentially information-rich middle KV largely unexplored. We systematically investigate middle KV across diverse autoregressive video backbones from two perspectives: (1) its functional role, showing that retaining middle KV improves long-range visual consistency and that it serves as a visual reference rather than an event memory; and (2) its underlying attention mechanism, revealing broadly distributed attention across the historical context rather than concentration on a few frames or prompt-transition boundaries. Based on these findings, we propose Temporal-Weighted Reservoir (TWR), a training-free cache policy that combines reservoir sampling with a time-weighted survival rule. TWR maintains broad coverage of an unknown-length middle history while preferentially preserving earlier visual references, improving long-horizon consistency under matched cache budgets. Extensive evaluations across diverse backbones consistently validate both the effectiveness of TWR and our functional analysis of middle KV.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Video KV Cache, Reservoir Sampling, Exposure Bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dvMrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines which historical KV states support long-term consistency in autoregressive video generation, focusing on the middle region between sink and recent caches. Removal, restoration, identity replacement, and content-corruption interventions test the role of this history, while attention analyses examine its use across frames and prompt transitions. The authors interpret the results as evidence that middle KV primarily provides visual references rather than event memory. Based on this interpretation, they propose Temporal-Weighted Reservoir (TWR), a training-free policy that maintains broad historical coverage while preferentially retaining earlier references under a fixed active-cache budget. Evaluations across four video backbones compare cache policies and budgets, reporting improved identity similarity, reduced identity drift, and better long-term visual consistency.

Critical strengths
  1. Identity gains extend across four backbones. Relative to Uniform Reservoir, FaceSim improves by 7.2, 4.6, 12.8, and 2.6 points on daVinci-MagiHuman, Wan2.1, Wan2.2, and HY-World 1.5, respectively; ID drift decreases on all four.

  2. Direct interventions support the value of historical KV. Restoring middle KV improves identity similarity with paired intervals above zero. Identity replacement and content corruption also affect subsequent frames.

Critical weaknesses
  1. Identity improvements are not jointly validated with action completion. Dynamic, motion smoothness, and prompt-switch results do not directly establish whether actions are completed, repeated, or followed responsively. Dynamic does not uniformly decrease relative to Uniform, so reduced motion is not a demonstrated explanation. Nevertheless, without joint identity-and-action evaluation, the practical significance of improved consistency remains uncertain.

  2. The mechanism interpretation and baseline comparison support narrower conclusions than claimed. The absence of stable attention peaks at matching past events does not exclude distributed event information; content corruption establishes an influence of history, not an absence of event memory. Separately, Section 5.1 describes PC and Figure 7 labels it DF-PC with one fixed sink slot. Unless Deep Sink and positional handling are included, this supports a controlled policy comparison rather than superiority over complete Deep Forcing. These limitations affect both the explanatory and comparative contributions.

Questions for authors
  1. Can existing videos support joint identity-and-action success rates, repetition or stalled-action frequencies, and response delay after prompt switches? A targeted blinded assessment would help distinguish appropriate stillness from failure.

  2. Does removing or replacing relevant-event history affect later related behavior more than an equally sized intervention on unrelated history? Alternatively, please limit the conclusion to the absence of stable matched-event attention under the tested readout.

  3. Does the Deep Forcing baseline include Deep Sink and positional handling? Please clarify its scope and either provide a representative full-method comparison or restrict superiority claims to the tested policies.

  4. How sensitive are gains over Uniform to generation seeds and weighting? Please show paired intervals and representative sensitivity results. Report memory and time alongside budget curves, separating LongLive-RAG’s archive costs from active KV.

Reviewer_p2LfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates KV-cache management in autoregressive video generation, focusing on the importance of different historical regions, sink, middle, and recent range, for long-term visual consistency. Through a systematic analysis, the authors demonstrate the importance of retaining middle-history KV entries and propose a training-free cache-retention strategy that favors earlier frames while maintaining broad historical coverage. The analysis and evaluation focus primarily on single-person video generation, emphasizing facial identity preservation and visual consistency over long generation sequences.

Critical strengths

The question of which historical KV-cache entries should be retained under a fixed memory budget to improve long-horizon video consistency is interesting and has practical relevaance. The authors propose a training-free cache-retention policy based on weighted reservoir sampling, which maintains broad coverage of historical frames while preferentially preserving earlier visual frames. The proposed approach demonstrates improvements in long-term visual consistency on the selected benchmark across several autoregressive video generation backbones. The paper controlled experiments are meaningful and show comparisons with uniform reservoir sampling, and consistent results across different model architectures.

Critical weaknesses
  1. Limited analysis of historical KV-cache content. While investigating the role of historical KV-cache entries is interesting, prior work such as FAST-AR [1] offers a more fine-grained perspective on KV-cache management, examining not only which temporal regions should be retained but also which individual tokens or spatial patches carry important information. In contrast, the current paper focuses primarily on frame-level retention and aggregated temporal attention patterns. The authors should discuss the differences between these approaches and clarify the advantages and limitations of their temporal selection strategy relative to more fine-grained, content-aware cache management, and add comparison to such method.

  2. Limited methodological novelty. A concern is the novelty of the proposed Temporal-Weighted Reservoir (TWR) method. Weighted reservoir sampling is an established technique, and the proposed extension relies on a relatively simple, predefined temporal weighting scheme that favors earlier frames. Although this design is motivated by the authors' empirical analysis, the algorithmic contribution itself appears rather incremental. Consequently, the paper's main contribution may lie more in its empirical investigation of historical KV-cache behavior than in the proposed caching algorithm.

  3. Limited scope of evaluation. A major concersn of mine goes to the experimental evaluation that focuses on 39 single-person image-to-video cases, with facial identity preservation serving as a primary performance measure. While this provides a controlled setting for studying long-term consistency, the small exteremly specific test case significantly limits the scope of the conclusions. It remains unclear whether the proposed observations and retention policy generalize to more complex scenarios involving multiple objects, changing scene layouts, substantial camera motion, or interactions between objects. A broader evaluation on establishe benchmarks such as Vbench abd VisionReward would be necessary to establish the general applicability of the approach.

  4. Potential trade-off between temporal consistency and motion dynamics. Although TWR substantially improves identity preservation, the results also reveal a notable reduction in Dynamic Degree for several backbones. For example, on daVinci-MagiHuman, Dynamic Degree decreases from 20.6% with Sink+FIFO to 5.4% with TWR. This raises the concern that improved identity consistency may come partly at the expense of meaningful motion or scene dynamics, potentially favoring more static video generation. This is also related to the limited scope of evaluation.

Reference [1] D. Samuel et al., FAST-AR: Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention, ICML 2026.

Questions for authors

Although the method uses a fixed cache budget, the manuscript does not provide a detailed wall-clock latency or GPU-memory comparison demonstrating its practical inference overhead.

Reviewer_Jn9SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates which historical KV states should be retained for autoregressive video generation. Through visibility interventions, event probes, and controlled corruption, it finds that middle-history KV provides useful visual references, while evidence for event-specific retrieval is limited. Motivated by these findings, it proposes Temporal-Weighted Reservoir (TWR), a training-free sampling policy that favors earlier frames within a fixed cache budget. Experiments on four backbones show improved long-term identity and appearance consistency over FIFO, score-based compression, and uniform sampling, with rollouts reaching 240 seconds.

Critical strengths

This paper provides useful evidence about how historical KV affects generation. Removing and corrupting middle-history KV shows that it meaningfully influences subsequent identity and spatial structure. These interventions provide stronger support than attention visualization alone and help motivate the proposed retention policy.

Critical weaknesses
  1. The distinction between visual references and event memory is not fully established. Moreover, repeating an action may not require recalling its earlier occurrence, limiting the strength of the mechanistic conclusion.

  2. TWR builds on established weighted reservoir sampling, but comparisons with other training-free history-retention methods remain limited. Broader comparisons under matched cache budgets would better establish its practical advantage.

Questions for authors

Since repeating an action does not necessarily require recalling details of its earlier occurrence, how do the sparse-event probes distinguish a lack of event memory from a lack of need to retrieve it? Should the conclusion be limited to the absence of stable event-aligned attention in these probes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.