acceptodds
Withdrawn from ICLR 2027

Learnable Compressed Memory for Autoregressive Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive video diffusion transformers generate videos chunk-by-chunk with causal attention over a sliding-window KV cache. While the window bounds computation and memory, it discards early context and can lead to long-horizon drift. Frame-sink variants retain a few anchor frames, but often introduce temporal discontinuities and abrupt regressions toward anchor-like appearances. In this paper, we present , a learnable memory system that eliminates the need for frame sinks while preserving access to a compact, evolving summary of long-range history under a fixed token budget. trains a KV Compress Adapter end-to-end with the base model and uses perceiver-style cross-attention with learnable latent queries to select and summarize KV tokens, compressing KV tokens to represent 7 more history while maintaining attention fidelity. Compressed memories are organized in a fixed-capacity memory bank with exponential temporal binning and consolidated through content-aware eviction and merging, yielding a diverse and temporally balanced summary without hard boundaries. Experiments on long-form video generation show that achieves long video generation with better visual quality, mitigates drift, avoids sink-induced artifacts, and preserves dynamic motion with smooth camera movement.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, KV Compression, Hierarchical Memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KReMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KVMEM introduces a learned compressed-memory system for autoregressive video generation. A KV compression adapter combines local spatial summaries with learned global residual corrections, and is trained using attention reconstruction and generation-level DMD supervision. Compressed entries are stored in a fixed-capacity hierarchical bank, with content- and time-aware merging across temporal bins. The intended benefit is broader historical coverage without expanding the active attention budget. Evaluation includes minute-long single-prompt and multi-scene generation, component and memory-allocation ablations, throughput measurements, and a user study.

Critical strengths
  1. The method offers a coherent approach to an important memory trade-off. Learning spatial summaries while progressively merging older history separates spatial detail from temporal coverage. The pre-RoPE representation and explicit integration with the recent context provide a reasonably concrete design for bounded-context generation.

  2. The component results indicate that both learned compression and temporal organization matter. In Table 3(a), imaging quality falls from 72.16 to 66.74 when the adapter is replaced and to 65.49 without the hierarchy. The reported reduction in abrupt changes is also practically relevant. These results make the method promising, although their interpretation depends on resolving the configuration and training-control issues below.

Critical weaknesses
  1. The reported experimental configurations contradict the central equal-budget comparison. Table 1 defines TB as the context token budget excluding the current chunk and reports 23.4k for KVMEM. However, Table 4 specifies 36 memory entries with 390 tokens each and nine recent frames with 1,560 tokens each. These sum to (36\times390+9\times1560=28{,}080) historical tokens, which is 20% greater than 23,400. The throughput discussion likewise describes nine compressed-memory frame-equivalents plus nine preceding frames, consistent with the larger budget. This discrepancy directly affects comparisons against methods reported at 23.4k. Additionally, Table 3(e) uses 64 latent queries as the default, whereas Table 4 specifies 390 queries per head. Since these queries determine the global summaries used by the adapter, the difference is relevant to both the architecture and its ablations. Until the actual configurations are reconciled, the reported quality gains cannot reliably substantiate the claimed equal-budget advantage.

  2. The reconstruction objective does not establish attention fidelity in the context where compressed memory is actually used. Equation 5 compares attention over original versus compressed historical KV states in isolation, whereas Equation 2 jointly normalizes attention over compressed memory, recent context, and the current chunk. Matching the former does not ensure matching the latter. For example, compressing four identical memory tokens into one identical token leaves memory-only attention output unchanged and can yield zero reconstruction error. Yet, with one local token having the same logit, the total memory attention mass changes from (4/5) to (1/2). If the local and memory values differ, the actual attention output changes. This is an analytical limitation of the surrogate objective, not evidence that the trained model necessarily fails. DMD co-adaptation could compensate, but the paper does not isolate this effect or directly measure joint-attention fidelity. The concern becomes more consequential for repeatedly merged memories because the historical merge cascade is detached. The existing whole-adapter ablation does not establish whether gains come from reconstruction supervision, DMD co-adaptation, additional capacity, or the merge policy.

  3. [1–3] are absent from the submitted bibliography and relevant to multi-scale memory, adaptive context compression, and hierarchical long-video memory. Echo-Infinity [4] is an especially important additional omission because it learns an evolving compressed history jointly with the video generator. Please distinguish the proposed local-plus-global KV adapter and explicit merge hierarchy from learned recurrent memory consolidation, and identify which claimed benefits are supported by controlled comparisons.

The method is promising, and the reported component gains deserve further investigation. However, the unresolved token-budget and adapter-configuration contradictions undermine the main comparison. In addition, the claimed attention fidelity is not established for the joint attention operation used during generation. These are central evidential concerns, rather than presentation issues alone.

References

[1] Zhaochong An, Menglin Jia, Haonan Qiu, et al. OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory. CVPR, 2026; preprint first released in 2025.

[2] Hongjun Wang, Lin Liu, Jianguo Li, and Tao Lin. Dual-Granularity Memory for Efficient Video Generation. CVPR, 2026.

[3] Jingyu Lin, Xinyi Shang, Peng Sun, Cunjian Chen, and Zhiqiang Shen. DynaMem: Consistent Long Video Generation via Hierarchical Memory and Motion Priors. ICML, 2026.

[4] Yuxuan Bian, Zeyue Xue, Songchun Zhang, et al. Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation. arXiv:2606.04527, 2026.

Questions for authors
  1. Which exact token budget and latent-query count produced each main table? Please reconcile 23.4k versus 28.08k historical tokens and 64 versus 390 queries. If the reported configurations differ, please provide corrected comparisons at an identical actual attention budget.

  2. Can you measure reconstruction error using actual future queries and the complete attention context, including recent and current tokens? Does the method preserve both the memory attention output and its normalization relative to local context, particularly after repeated temporal merges?

  3. Were all memory-policy and pooling controls trained with matched generator initialization, data, teacher, optimization steps, and generator adapters? Please separate reconstruction-only, DMD-only, and combined supervision. Also specify the evaluation sample counts, random-seed protocol, numerical LPIPS threshold, and participant/trial counts for the user study.

These clarifications and controls are necessary to assess the reported efficiency and compression claims.

Reviewer_FMHfOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

KVmemKV_{mem} targets the loss of long-range context in chunk-wise autoregressive video diffusion. At each transformer layer, a learnable adapter compresses historical key-value (KV) states, combining local 2×2 spatial pooling with learned global residuals. The compressed summaries are stored in a fixed-capacity, four-bin temporal memory; when a bin overflows, the method merges a pair selected using content and temporal distance. The adapter is trained jointly with the generator’s trainable adapters using attention reconstruction and a DMD objective. Experiments on Wan2.1-T2V-1.3B compare KVmemKV_{mem} with cache-compression, memory, and frame-sink baselines, reporting VBench metrics, an LPIPS-based abrupt-change count, longer-video results, ablations, and participant preferences. The results are promising, but ambiguities in the token-budget accounting and evaluation protocol make the headline comparisons difficult to verify.

Critical strengths

The paper addresses an important practical problem: sliding-window KV caches discard early history, while fixed frame sinks can overemphasize the opening state. KVMEM’s evolving memory is a coherent approach to preserving long-range context without growing the cache indefinitely.

The method combines learnable, per-layer KV compression with a bounded multi-resolution memory, and the component ablations provide some evidence that both parts contribute to the reported results. The paper also evaluates more than one setting, including longer videos and multi-scene generation.

Critical weaknesses

The reported budget for the main KVMEM comparison does not match the configuration in Appendix Table 4. The appendix gives 36 × 390 = 14,040 memory tokens and 9 × 1,560 = 14,040 recent-window tokens, totaling 28,080 historical tokens, excluding the current chunk. Table 1 reports 23.4k, also excluding the current chunk. Including the appendix’s 3-frame current chunk gives 32,760 total tokens. This discrepancy makes it unclear whether the comparisons are actually matched to the stated budget.

The empirical evidence is difficult to assess or reproduce from the paper. It does not report evaluation sample counts, prompts/scripts or their selection procedure, seeds, or uncertainty estimates; the LPIPS threshold for the abrupt-change metric is also unspecified. The participant-preference results lack the study size and protocol. In addition, Table 1’s LongLive (SF) Motion value of 49.72 is an isolated outlier relative to the roughly 98–99 values elsewhere and needs explanation. The runtime results also show KVMEM at 20.6 fps, below LongLive at 23.7 fps and MemFlow at 21.4 fps, while the adapter adds 130.2M parameters; this tempers the efficiency framing.

Questions for authors
  1. Can you reconcile Table 1’s KVMEM budget of 23.4k tokens, excluding the current chunk, with Appendix Table 4’s 28,080 historical tokens and 32,760 tokens including the current chunk? What exact attended-token budget was used for each method, and were the reported comparisons run at matched budgets?
  2. How do the 21-latent-frame training chunks, the 3-frame current chunk, and the 3-frame extraction interval map to one another? Please clarify the units used for each, how memory writes occur, and the definition and calculation of HH in Table 1.
  3. What were the numbers of generated videos and prompts, the prompt/script selection procedure, and the random seeds or repeated runs? What exact LPIPS threshold and aggregation protocol define “Abruptness,” and what uncertainty estimates are available?
  4. How many participants took part in the preference study, and what were the recruitment, blinding, consent, and oversight procedures? Also, can you explain or correct the LongLive (SF) Motion value of 49.72?
Reviewer_nrUwOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

KVMEM addresses historical context loss and potential regressions toward fixed opening-frame references in autoregressive long-video generation. It combines a recent uncompressed KV window with an evolving compressed memory. Spatial compression uses local pooling with learned global refinements, while a hierarchical memory bank bounds storage through heuristic temporal merging. The compression module and generator adapters are jointly trained using attention reconstruction and DMD supervision. Experiments report improved long-video consistency and visual quality while maintaining inference throughput close to the baselines.

Critical strengths
  • Presents a coherent design combining learned compression and bounded memory to address long-range consistency.
  • Shows promising multi-scene results, supported by qualitative comparisons and a user study.
Critical weaknesses
  • Comparison settings and token budgets are insufficiently clarified. Table 1 reports 23.4k context tokens for KVMEM, excluding the current chunk, whereas Table 4 specifies 36 × 390 = 14,040 memory tokens plus 9 × 1,560 = 14,040 recent-window tokens, totaling 28,080 tokens—approximately 20% more. This discrepancy requires clarification to substantiate the equal-budget comparisons. The paper also does not clearly identify baseline checkpoints, retraining protocols, or how the CF variants were constructed, leaving the comparability of training data and optimization budgets unclear.
  • The compression design and training objective lack key ablations. The separate contributions of reconstruction and DMD gradients to the compression module are not established. A learned-query-only baseline is missing, leaving the necessity of the local-summary-plus-refinement design unverified. Furthermore, using historical keys as reconstruction probes is not validated against actual future queries.
Questions for authors
  • Can the authors provide a controlled evaluation of long-range information retention where relevant visual information falls outside the recent context window? Such an experiment would help distinguish the contribution of distant memory from that of recent context and substantiate the claim of preserving long-range visual information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.