acceptodds
Withdrawn from ICLR 2027

Eviction Is Not Forgetting: Read Long, Write Local in Streaming Video Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Block-autoregressive video diffusion models generate minute-long videos one short latent block at a time. Each block is denoised while attending to a key/value (K/V) cache of earlier blocks; the finished block then passes through the network once more to compute the K/V that the cache stores, and a fixed window evicts the oldest block. We find that eviction does not remove an old block’s influence. Because this writing pass attends to the current cache, every new entry stores a copy of older entries, and the copy stays after its source is evicted: for fixed video, a cache difference in an -layer model with a -block window can last up to writes instead of . Writing each block alone removes the copies but also loses how the video entered the block. We therefore read long and write local. ARROW keeps the full cache for denoising and writes each entry from the clean latents of the previous and current blocks, without old K/V, storing only the current block’s K/V. It needs no training and keeps one extra latent block. On Self Forcing (Wan2.1), MAGI-1 and HunyuanVideo-1.5, ARROW raises VBench-Long quality by 3.6–6.2 points and LoCoT2V character consistency by 10.6–29.4 points. Replay and writer ablations trace these gains to removing the copies and adding the boundary frames, and reader masking shows that long-range reading must be kept.

Primary area:
generative models
Keywords:
autoregressive video generation, transformer cache, history writing, video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2kj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Block-autoregressive video diffusion models generate long videos from a fixed K/V cache, and quality drifts over time. This paper traces one cause to the cache-writing pass: writing a new entry while attending to old K/V copies history into that entry, which then survives eviction long after its source is gone. The authors quantify how long such copies linger, and propose ARROW, a training-free fix. The fix is simple: keep reading the full cache while denoising, but write each entry without it, from the current and previous clean blocks only. Across three backbones this improves long-video quality and character consistency substantially (up to 29.4 points) at 5 to 17% extra time, with matched-pair ablations and a controlled intervention supporting the mechanism.

Critical strengths

(1) The paper offers a new explanation for why video quality degrades in long autoregressive generations, and the fix is simple but effective. The write step copies old cache content into new cache entries, so old mistakes survive eviction and continue to affect the video. The proposed method is simple and effective: write each entry without reading the cache. The change is training-free and brings large gains.

(2) The experiments show large and consistent gains on three models (Self Forcing on Wan2.1, MAGI-1, and HunyuanVideo-1.5), up to 29.4 points in character consistency at 5 to 17% overhead. The ablations are careful, and a controlled intervention confirms the mechanism.

Critical weaknesses

(1) The empirical results are strong, but the contribution needs to be distinguished more clearly from prior work. Writing cache entries without old K/V already exists in MAGI-1, and DSCache explicitly discusses residual influence from evicted caches. The novelty therefore lies more narrowly in the adjacent-block overlap, the decision to store only the current block’s K/V, and their analysis. The explanation of the overlap benefit is also less convincing than the results themselves.

(2) The presentation makes a simple method unnecessarily difficult to follow. For example, figure 3 combines the unchanged denoising pipeline, layer-wise cache construction, cache updates, generated examples, and diagnostic heatmaps in one overcrowded diagram. The central modification to the writer is consequently easy to miss. More broadly, the exposition should explain the concrete change and its motivation before introducing the formal analysis, and distinguish experimental findings from proposed interpretations.

Questions for authors

(1) Could you clarify the distinction from MAGI-1 and DSCache, particularly regarding context-independent cache writing and the residual influence of evicted entries? It would help to separate the novelty of the propagation analysis from that of the adjacent-pair writing rule, and explain what new insight ARROW provides beyond these precedents.

(2) Could the authors substantially improve the clarity of the method presentation? Despite the simplicity of the proposed modification, I found the explanation and figures unnecessarily difficult to follow.

Reviewer_xjqFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ARROW which stores the current block’s KV to preserve country context without rewriting earlier entires in the KV cache in order to solve the long-horizon drift problem in block-autoregressive video diffusion models.

Critical strengths

The method is simple and training-free, and works with existing block based autoregressive models without changing the cache size etc.

Controlled experiments and ablations are done in detail, and appendix further supports the claims.

Critical weaknesses

Applicability seems to be limited to the models with a clean writing pass; it is not clear how it can be applied to rolling designs etc.

It is not clear how much of the gain the copy mechanism explains vs a change in the entry distrubiton.

Questions for authors

In table 1 and table 7 (5 sec vs 60 sec results) MAGI’s dynamic degree seems to be exactly 43.3, can authors verify whether this is correct?

What is “detail score” in D.3/Fig 15?

I am not sure what the tentacle example on right bottom side of Fig 3 related to the person who is riding a bike in the same figure.

Reviewer_1EB5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that part of the long-video drift in block-autoregressive video diffusion comes from the cache-writing pass. The native writer attends to old K/V while writing (Eq. 2), so old content is copied into new entries through the second term of Eq. (3) and its influence survives eviction. Theorem 1 bounds that lifetime at LW writes (W when old K/V are excluded; 180 on Wan). ARROW is training-free and writes Ct=Uθ([xt−1,xt],∅)∣xtC_t=U_\theta([x_{t-1},x_t],\emptyset)|_{x_t}: it stores only the current block, never rewrites a stored entry, and denoising still reads full history. On Wan / MAGI-1 / Hunyuan, VBench-Long Quality improves by 3.56–6.17 and LoCoT2V CC by 10.6–29.4, at +5.3–16.6% time.

Critical strengths
  1. Low cost and composable: persistent state grows by one latent block; combined with Deep Forcing it reaches 85.57 Quality / 66.9 CC, above every Wan system.

  2. Transparent, and negative evidence is reported honestly: paired bootstrap and revision hashes are published; five-second differences of 0.01–0.03 with CIs containing zero are not hidden.

  3. The diagnosis is supported by a fixed-input experiment: all 8 replay-intervention cells are positive, with Wan CC +3.46 (95% CI 2.97–3.96).

Critical weaknesses
  1. Table 3 lacks a "pair + old K/V" cell. Native to current-only (+16.1 CC) mixes copying with reading an already-drifted cache, while the replay intervention that truly isolates copying gives only +3.46.

  2. Section 4 compares with BAgger only on cost; there is no BAgger-style long clean-window refresh control on frozen Self Forcing.

  3. All 8 Wan values are identical between Table 4 (t=2…) and Table 6 (t=3…), while MAGI / Hunyuan differ by about 0.01; current-only has higher peak memory than native (15.8 vs. 15.0 GiB, 47.0 vs. 44.4 GiB).

Questions for authors
  1. Add a pair + old K/V writer (Wan, Hunyuan, with paired CIs).

  2. Report prompt-averaged, per-layer decay curves of ∣ΔCℓ∣|\Delta C^\ell| against the number of writes (native vs. ARROW).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.