acceptodds
Withdrawn from ICLR 2027

Memory Outweighs Control: A Causal Audit of Cached History in Autoregressive Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive video generators are sold on the promise that a viewer can steer the story as it unfolds. We audited whether the steering arrives. On two autoregressive video diffusion backbones we measured how far a mid-rollout prompt switch moves the continuation, in two regimes: with no cached history, and with twelve frames of it. With no history the prompt fully determines the video — a switch to an unrelated scene leaves the two continuations near-orthogonal in feature space (1.00 cosine distance). With twelve frames cached, the same switch retains 0.98% of that effect on one backbone and 0.15% on the other. The backbone that keeps less is the one built for prompt switching, whose KV-recache re-encodes the entire prefix under the new prompt on every switch and did so here. Ten seeds each; the no-history regime doubles as the positive control. That measurement reframes the object we set out to formalize. We define the minimal sufficient visual memory at a branch point through a causal information bottleneck, and we can define it — but on current systems the decision it is defined against does not reach the continuation, so the specification is not wrong so much as without anywhere to land. A distance between generations could be blind to a switch that preserves texture, so we checked with a quantity that cannot be: CLIP alignment to the prompt text. On both backbones, with no history the continuation follows the newly chosen prompt in 10/10 seeds and with twelve frames cached in 0/10, scoring 0.369 against the prompt the viewer left and 0.089 against the one they chose. Two backbones and two metrics that share no blind spot put the surviving fraction between 0.15% and 0.41% for that switch. Over a grid of 4 switches — from an alternative action inside the established scene to an unrelated one — at 5 prefix lengths, the median across 128 measurements is 10.1% and the largest single one 44.9%; 0 exceed half. Most of the steering is absorbed, not all of it. Where a switch is lost entirely, one frame accounts for it. Adding the single oldest cached frame — an emergent attention sink — to a nine-frame window reproduces the full-history collapse, while adding two other old frames does not; a ten-frame history containing that slot suppresses the switch more than an eleven-frame one without it. The effect is indifferent to what the frame depicts — one lifted from an unrelated history works as well — but not to whether it is a real frame, since zeroing the slot restores the switch. The audit also finds structure in the memory itself. Pruning by measured relevance matches the full cache down to 50% of it, while recency ordering leaves the noise floor at the first frame removed. The same two frames at opposite ends carry the effect through opposite mechanisms, so a necessity criterion defined by overwriting — ours, and the natural one — certifies a memory that is unusable (14.45× the floor), and the obvious repair (2.47×) still cannot certify sufficiency: the remaining frames are individually dispensable and jointly required. The audit and every run record are released.

Primary area:
generative models
Keywords:
interactive video generation, autoregressive video diffusion, KV cache, causal audit, attention sink, controllability, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EdySOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses an interesting finding: the history cache in autoregressive video generation can considerably reduce the impact of new text input controls.

The authors define minimal sufficient visual memory (MSVM) as the smallest amount of history needed to preserve the future video generation distribution. They also introduce D-IBVM, a framework that uses memory ablation and causal interventions to identify important memory units.

However, the authors find that new text prompts have very little impact when the history cache is present. Therefore, several planned experiments cannot be conducted. Instead, the paper focuses on analyzing the history cache itself. The experiments show that some old frames, especially the first cached frame, can have a large impact on generation, even when their actual visual content is not important.

Critical strengths

The paper studies an interesting and important problem in autoregressive video generation: the relationship between history memory and new input controls. The finding that cached history can significantly reduce the effect of new prompts is interesting and may be useful for future research.

The authors perform several controlled experiments, including memory ablation and frame replacement, to better understand the role of different cached frames.

Critical weaknesses

Personally, the paper is hard to follow and read generally. The abstract is almost a full page, I was overwhelmed with many numbers before getting to the main body. In terms of writing style, the paper has an “We tried to do this, but it did not work; instead, we did that” structure.

I also suspect there are AI-generated contents without verification, e.g. did the author really check Fig 2?

For the math/theory part, it is very unclear and non-rigorous. E.g. In Def 1, what is the clear definition of A,D\mathcal{A}, D? The previous one is the full decision space? For Proposition 1: Is the deficit I(Y;H∣a)−I(Y;M∣a)I(Y;H|a) − I(Y;M|a) really convex? Not concave?

Paragraphs 3 and 4 (MSVM definitions, DIVM, and Algorithm 1) contain formalizations that did not produce any results, and it is in section 5 that the reader finally learns about what actually was tested.

The work is inconsistent with its own framework. The title, the abstract and the Sections 3-4 discuss MSVM, DIVM and the "clamped decision" criterion. Afterwards, the work states that this mode is "currently unrunnable" for both backbones. Proposition 1, Algorithm 1 and experiments E2-E4 yield no result. About half of the work is devoted to defining things. The main contribution is measuring attenuation and finding attention sinks.

Technically, the paper does not clearly distinguish visual consistency from controllability. More clarification that needs to be made is the meaning of prompt-switch attenuation. A less significant change made in the produced continuation does not have to mean that it has less controllability because sometimes it may be good to keep the chosen scene.

Questions for authors

Please refer to the weakness section.

Reviewer_YzxLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how much cached history an autoregressive video model really needs when the user changes the prompt in the middle of a rollout. The authors define a "minimal sufficient visual memory" (MSVM) with an information-bottleneck view. They also propose a tool (DiVM) that ranks cached frames and tests them with interventions. They test it on Self Forcing and LongLive. With 12 cached frames, a prompt switch to an unrelated scene keeps only about 1% of its effect. Because of this, the authors cannot run the main experiments of their framework. The rest of the paper is a frame-ablation study of the KV cache. It shows that the oldest cached frame acts as an attention sink and has the largest effect.

Critical strengths
  • The question is relevant: how much of the cached history do these models actually use?
  • The no-history setting is a good positive control.
  • The CLIP check is a good test of whether the DINOv2 metric is missing the change.
  • The authors are open about what they did not run.
Critical weaknesses
  • The paper is hard to read. The abstract is five paragraphs and one full page, but the ICLR template asks for one paragraph. Many sentences are long and unclear. The main results table is only in the appendix. Figure 2 is too small to read. For a video paper there are very few visual results (one seed in Figure 1), and no videos are given.
  • The paper does not deliver its main contribution. The formalization (Sec. 3) is not tested. The decision-based experiments (E2-E4) are not run. The retention policy is not reported (Sec. 5.4, Sec. 6). What is left is one observation on two models and a definition that cannot be tested.
  • The headline numbers (0.98% and 0.15%) come from the most extreme case: a switch to an unrelated scene. On the authors’ own grid of 4 switch types, the median is 10.1% and the maximum is 44.9%. Also, keeping the same subject and scene after a prompt switch is often what these models are trained to do. The paper does not separate “the control fails” from “the model keeps the video consistent, as intended.”
  • It is already known that a cached history can absorb a new prompt. LongLive’s KV-recache was designed for this exact problem. The paper finds that LongLive keeps less of the switch than Self Forcing, but it does not explain why. Without an explanation, the main finding is weak.
  • The attention-sink finding is not new. Keeping the first frames as a sink plus a recent window is the StreamingLLM idea, and LongLive already uses it. The paper also shows that this simple rule works as well as DiVM’s ranking. So the proposed tool gives no practical gain.
  • The authors show that their own necessity criterion (Def. 2) fails. Their proposed fix also fails. The paper gives no criterion that works.
  • The experiments are too small for general claims: 4 scripted sequences per backbone, 12-frame prefixes, 9-frame continuations, about 10 seeds, and 15.8 GPU-hours in total. Also, LongLive is tested in a setting that it does not ship with (Appendix J).
Questions for authors
  • How do you tell apart a model that ignores the prompt from a model that correctly keeps the scene consistent? Did you test prompt switches that are natural continuations, like the examples in LongLive?
  • LongLive re-encodes the prefix with the new prompt. Why does it keep less of the switch than Self Forcing?
  • Do the results hold with longer prefixes, longer continuations, and more sequences?
Reviewer_BwoXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper discusses how cached history affects prompt switching and memory retention in Self Forcing and LongLive. It reports strong attenuation of prompt-induced changes and different effects of deleting versus replacing cached frames. It also proposes a minimal sufficient visual memory framework, whose identification criterion fails in the reported experiments.

Critical strengths

The authors verify that cache interventions preserve positional encodings and reproduce the original output when no frames are removed. These checks help distinguish genuine memory effects from implementation artifacts. The deletion, replacement, and zeroing experiments also provide a useful diagnostic finding: removing a cached frame can substantially change generation even when replacing it with a frame from another history has little effect (§5.3; Appendix I). This directly demonstrates why a replacement-based necessity test can mistakenly discard an important memory unit.

Critical weaknesses
  1. The headline interpretation exceeds what the measurements establish. In §5.2, the ratio of prompt-induced feature distances with and without history is interpreted as the fraction of control that survives. Removing history also removes the established scene and continuity constraints, so this denominator does not calibrate instruction-following success. Appendix K's within-scene switches and the CLIP check are useful, but neither establishes task-level action compliance or distinguishes failure from delayed response within the nine-latent-frame continuation. The assertion that DINOv2 and CLIP share no blind spot is unsupported. These results establish reduced measured sensitivity; they do not justify treating the decision as causally ineffective and declaring the decision-conditioned experiments unrunnable.

  2. The presentation and writing require major revision and careful polishing. The overlong abstract and repeated discussion of unexecuted experiments obscure the completed findings, while the headline Table 1 appears only in the appendix (p. 20). In §5.2, 0.98% is included among four values described as below 0.5%. Figure 3's set-level-distance description is also unclear relative to Table 3's seed-level summaries, despite Appendix K's explanation of their shared normalization. These issues require a focused rewrite and consistent statistical reporting.

  3. The claimed theoretical guarantee is unsupported. Appendix A's proof of Proposition 1(a) assumes convexity that fails under encoder mixing, invalidating the stated Slater argument. Averaging constraints does not preserve Definition 1's per-decision guarantee, and the one-sided rate–footprint bound does not establish equivalent minimizers. The reported two-sequence comparison shows no advantage over sink-plus-window, and E2–E4 remain unrun (§5.4).

Questions for authors
  1. Validation of the headline claim: Can task-level evaluation of the existing within-scene switches, including response timing, distinguish genuine instruction failure from preserved appearance or delayed compliance?
  2. Conceptual contribution: Which central conclusion requires the proposed causal/MSVM framework, and what generalizable insight does it establish beyond known prompt inertia and attention-sink behavior? Given that decision-conditioned identification remains untested and the proposed necessity criterion fails, what substantive theoretical contribution remains, and which experimental results support it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.