Are We Caching the Right History for Autoregressive Video Generation?

In autoregressive video generation, the historical key-value (KV) cache is crucial for long-term visual consistency, yet GPU memory and real-time inference constraints require it to remain bounded. A more recent view decomposes historical KV into sink, middle, and recent regions, yet prior studies have centered more on the two boundary regions, leaving the expanding and potentially information-rich middle KV largely unexplored. We systematically investigate middle KV across diverse autoregressive video backbones from two perspectives: (1) its functional role, showing that retaining middle KV improves long-range visual consistency and that it serves as a visual reference rather than an event memory; and (2) its underlying attention mechanism, revealing broadly distributed attention across the historical context rather than concentration on a few frames or prompt-transition boundaries. Based on these findings, we propose Temporal-Weighted Reservoir (TWR), a training-free cache policy that combines reservoir sampling with a time-weighted survival rule. TWR maintains broad coverage of an unknown-length middle history while preferentially preserving earlier visual references, improving long-horizon consistency under matched cache budgets. Extensive evaluations across diverse backbones consistently validate both the effectiveness of TWR and our functional analysis of middle KV.