Memory Outweighs Control: A Causal Audit of Cached History in Autoregressive Video Generation

Interactive video generators are sold on the promise that a viewer can steer the story as it unfolds. We audited whether the steering arrives. On two autoregressive video diffusion backbones we measured how far a mid-rollout prompt switch moves the continuation, in two regimes: with no cached history, and with twelve frames of it. With no history the prompt fully determines the video — a switch to an unrelated scene leaves the two continuations near-orthogonal in feature space (1.00 cosine distance). With twelve frames cached, the same switch retains 0.98% of that effect on one backbone and 0.15% on the other. The backbone that keeps less is the one built for prompt switching, whose KV-recache re-encodes the entire prefix under the new prompt on every switch and did so here. Ten seeds each; the no-history regime doubles as the positive control. That measurement reframes the object we set out to formalize. We define the minimal sufficient visual memory at a branch point through a causal information bottleneck, and we can define it — but on current systems the decision it is defined against does not reach the continuation, so the specification is not wrong so much as without anywhere to land. A distance between generations could be blind to a switch that preserves texture, so we checked with a quantity that cannot be: CLIP alignment to the prompt text. On both backbones, with no history the continuation follows the newly chosen prompt in 10/10 seeds and with twelve frames cached in 0/10, scoring 0.369 against the prompt the viewer left and 0.089 against the one they chose. Two backbones and two metrics that share no blind spot put the surviving fraction between 0.15% and 0.41% for that switch. Over a grid of 4 switches — from an alternative action inside the established scene to an unrelated one — at 5 prefix lengths, the median across 128 measurements is 10.1% and the largest single one 44.9%; 0 exceed half. Most of the steering is absorbed, not all of it. Where a switch is lost entirely, one frame accounts for it. Adding the single oldest cached frame — an emergent attention sink — to a nine-frame window reproduces the full-history collapse, while adding two other old frames does not; a ten-frame history containing that slot suppresses the switch more than an eleven-frame one without it. The effect is indifferent to what the frame depicts — one lifted from an unrelated history works as well — but not to whether it is a real frame, since zeroing the slot restores the switch. The audit also finds structure in the memory itself. Pruning by measured relevance matches the full cache down to 50% of it, while recency ordering leaves the noise floor at the first frame removed. The same two frames at opposite ends carry the effect through opposite mechanisms, so a necessity criterion defined by overwriting — ours, and the natural one — certifies a memory that is unusable (14.45× the floor), and the obvious repair (2.47×) still cannot certify sufficiency: the remaining frames are individually dispensable and jointly required. The audit and every run record are released.