acceptodds
Under review as a conference paper at ICLR 2027

Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation

Abstract

Standard video generators do not natively compact historical context into reusable memory tokens. As generation continues, the growing history makes it increasingly difficult to retain information from earlier frames due to long-context degradation. Key-frame-based approaches address this challenge by retaining selected past frames, but can discard information needed for future generation. Rather than relying on frame selection alone, we study whether a frozen video generator can supply the supervision needed to learn a compact representation of the history. We propose Prediction-Aligned Context Compaction (PACC), which uses a learned compressor to aggregate information across past frames into compact memory tokens. We train the compressor through on-policy distillation, using the same frozen generator both as a student when conditioned on compressed memory and as a teacher when conditioned on the full history. The student generates continuations, while the teacher provides targets for the same noisy inputs at each denoising step. Only the compressor is updated to align the student's predictions with these targets. We evaluate PACC on MBench, which jointly measures memory-event coverage and consistency. PACC outperforms the strongest baseline by 6.63 points on Causal-rCM and 3.19 points on Causal Forcing. Evaluation on VBench-Long using MovieGen prompts further shows that PACC produces minute-long videos with generation quality competitive with baselines. Together, these results show that learning to compact historical context can improve long-video memory without modifying the underlying generator.

Primary area:
generative models
Keywords:
Long Video Generation, Autoregressive Diffusion, Memory Compaction, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yRT6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PACC learns to compress video history into 10× fewer tokens through on-policy distillation: a frozen generator’s full-history predictions supervise its predictions conditioned on compressed memory. Short-rollout training supports later generation through retrieval of archived memory blocks.

Critical strengths
  • A coherent preservation objective trains memory specifically for the frozen generator that consumes it, without requiring generator fine-tuning or long-video supervision.
  • MBench improvements are consistent across two backbones, with competitive aggregate minute-long quality.
Critical weaknesses
  • Gains are not isolated to learned compression. PACC uses 7,956 active context tokens versus Relax Forcing’s 6,240 and broader retrieval eligibility. Ablations omit matched-budget/retrieval controls and on-policy versus off-policy training.
  • MBench uses one seed per case, without uncertainty estimates or separate coverage/reliability results. Explicit memory challenges last 26 seconds; minute-long tests assess visual quality. Happy to raise score if addressed properly.
Questions for authors

Please cite more video memory papers and discuss on them. Here is a list:

  • Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
  • Mixture of Contexts for Long Video Generation. (ICLR 2026)
  • SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
  • Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
  • Moga: Mixture-of-groups attention for end-to-end long video generation. (ICLR 2026)

And there are much more. Video generation memory is not a new topic and the authors should cite and discuss more related work. Literally no one working in this area will be happy by the depth of related work discussion.

3 comments
AuthorsOpenReview ↗

Thanks! (1/2)

We thank the reviewer for your observant review and recognition. Here, we address your concerns and questions.


W1 – Context budget, retrieval range and on-policy training – We added matched-budget controls and an on-policy vs. off-policy comparison. PACC is ahead in both.

Thank you for highlighting these factors. Our main comparison evaluates complete memory strategies on the same frozen generator, using each strategy’s respective context budget and retrieval range. To isolate these factors, we added matched-budget Relax Forcing controls. Released Relax Forcing retrieves one raw frame from the recent half of the past (6,240 active tokens). Relax-H2 retrieves two frames instead. Relax-H2-all keeps two frames but searches the whole past, as PACC does. Relax-H3-all takes three frames from the whole past, so it uses more tokens than PACC. We evaluate all five policies on the full MBench benchmark (611 cases, three seeds per case) and report macro M as mean ± SD over seeds:

PolicyActive tokensRetrieval candidatesCausal-rCMCausal Forcing
Relax Forcing6,240recent half of past45.15 ± 3.2957.00 ± 0.85
Relax-H27,800recent half of past45.24 ± 1.9654.74 ± 0.74
Relax-H2-all7,800entire past44.58 ± 1.4258.18 ± 0.61
Relax-H3-all9,360entire past45.26 ± 2.3256.82 ± 1.61
PACC7,956entire past49.21 ± 1.4759.62 ± 0.89

A larger budget alone does not help Relax Forcing: Relax-H2 is no better than the released model, and PACC beats it by about 4 points on both backbones. When Relax Forcing can also retrieve from the whole past (Relax-H2-all), PACC is still clearly ahead. Relax-H3-all retrieves three frames from the whole past and uses 18% more tokens than PACC, yet it still trails PACC on both backbones: by 4.0 points on Causal-rCM and 2.8 on Causal Forcing. These controls are now in Appendix C.1 of the revised manuscript (in blue).

Our existing ablations also examine compression capacity and training duration. In particular, the compression-rate sweep shows that fewer memory tokens can achieve higher memory scores, indicating that performance does not simply increase with token count.

We adopt on-policy training because the compressed memory itself influences subsequent generation. Fixed off-policy continuations may not reflect the behavior induced by the current compressor. We therefore generate continuations using the current compressed memory and obtain full-history teacher predictions at the same generated contexts and noisy inputs. This exposes the compressor to states affected by its own compression errors and trains it to preserve the information needed along the trajectories that the current compressed-memory system actually produces.

To address the reviewer's concern, we add the experiment of on-policy trainig v.s. off-policy training. Due to the limit amount of time, we train both of the method to 2k steps and compare their performance with Causal-rCM on MBench. On-policy is better than off-policy training.

Training stepsOff-policyOn-policy (PACC)Gap95% CI
1,00046.3646.96+0.60[−2.36, +3.58]
2,00050.8053.97+3.17[+0.36, +6.03]

AuthorsOpenReview ↗

Thanks! (2/2)


W2 – Memory evaluation – We now report coverage and reliability separately, test minute-long videos, and run three seeds with confidence intervals.

Separating Coverage and Conditional Reliability

MBench's M-score is the harmonic mean of coverage (C), which measures whether the requested memory event occurs, and conditional reliability (R), which measures consistency when the event is triggered. We agree that reporting only the aggregate M-score obscures the source of the improvements.

We have recovered the per-dimension C and R scores from our existing evaluations, without requiring additional generation or evaluation. The following table compares PACC against Relax Forcing, the strongest baseline in Table 1, on both backbones.

Table 1 evaluation: C / R in %

BackboneMethodIdentityAppearanceGeometryTextureStateCorrectnessM-score
Causal-rCMRelax22.5 / 29.420.8 / 46.020.0 / 30.933.0 / 88.961.3 / 67.961.5 / 73.743.01
Causal-rCMPACC38.3 / 35.541.7 / 47.428.0 / 27.340.0 / 84.862.8 / 70.861.0 / 76.949.64
Causal ForcingRelax75.8 / 30.975.0 / 49.722.0 / 26.470.0 / 86.680.3 / 69.680.0 / 65.158.58
Causal ForcingPACC75.8 / 36.775.0 / 52.746.0 / 22.272.0 / 85.981.8 / 72.182.3 / 67.761.77

This decomposition shows that PACC's memory-score improvements arise from both improved event coverage and improved conditional consistency in most dimensions. We have added this decomposition to Appendix B.4 (highlighted in blue).

Memory Evaluation on Minute-Long Videos

To test memory beyond 26 seconds, we extend MBench to 78.6-second videos by tripling the length of each caption segment, so the gap between a first appearance and its recall reaches about a minute. PACC keeps its advantage over Relax Forcing at this length: macro M improves by 3.89 points on Causal-rCM and 3.96 on Causal Forcing (95% paired case-bootstrap CIs [1.61, 6.31] and [2.10, 5.87]), and PACC scores higher in 11 of 12 dimensions.

MBench on 78.6-s videos (seed 0): M in %

BackboneMethodIdentityAppearanceGeometryTextureStateCorrectnessMacro M
Causal-rCMRelax37.047.232.071.368.760.852.84
Causal-rCMPACC42.146.937.982.969.461.256.73
Causal ForcingRelax47.664.226.876.772.259.557.82
Causal ForcingPACC56.369.030.180.473.461.561.79

Multiple Seeds and Uncertainty

We re-ran MBench with three seeds per case on all 611 cases. The table reports per-dimension and macro M as mean ± std over seeds, and the gain of PACC over Relax Forcing in macro M with a 95% paired bootstrap confidence interval over cases. PACC improves over Relax Forcing on both backbones with intervals that exclude zero, and it is higher in all three seeds on Causal-rCM and in two of three on Causal Forcing. We have added these results to Appendix B.4 and noted them in Section 4.2 of the revised manuscript (highlighted in blue).

MBench, 3 seeds × 611 cases: M in %

BackboneMethodIdentityAppearanceGeometryTextureStateCorrectnessMacro MGain (95% CI)
Causal-rCMRelax29.1 ± 3.236.0 ± 5.827.3 ± 3.754.5 ± 7.566.1 ± 0.958.0 ± 1.045.17 ± 3.24–
Causal-rCMPACC39.0 ± 0.944.7 ± 0.132.5 ± 2.459.0 ± 5.963.5 ± 2.657.5 ± 1.249.36 ± 1.84+4.19 [1.98, 6.31]
Causal ForcingRelax45.6 ± 1.961.9 ± 2.820.0 ± 2.177.9 ± 1.074.2 ± 1.962.3 ± 1.456.97 ± 0.85–
Causal ForcingPACC48.3 ± 2.061.8 ± 0.928.5 ± 4.776.9 ± 1.874.5 ± 1.163.5 ± 0.758.92 ± 1.63+1.94 [0.52, 3.32]

Q1 – Related work – We expanded the Related Work section in the revised manuscript.

We thank the reviewer for pointing out these omissions and suggesting relevant references. We have expanded the Related Work section of the revised manuscript (highlighted in blue), organizing video memory methods by how they store and access history.

I appreciate the authors' detailed rebuttal and do not have further problem with this paper and intend to increase my score to a 3. (That being said, it is something of a borderline accept for me.)

Reviewer_5CQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents PACC, which learns to compress video history for a frozen generator. The compressor is trained on-policy to match the same generator's predictions under full-history conditioning. During inference, compressed blocks are archived and retrieved during long-video generation.

Critical strengths
  1. Instead of reconstructing the history, the compressor learns to preserve its effect on future predictions, while the generator remains frozen. This is a direct way to learn memory that the existing generator can actually use.

  2. The memory results are promising on both generators. I also find it interesting that stronger compression can improve memory scores, although this comes with a trade-off in imaging quality.

Critical weaknesses

My main concern is the distinction from TinyHistory (originally titled Pretraining Frame Preservation in Autoregressive Video Memory Compression), which also learns compact history embeddings for autoregressive video generation but is neither cited nor compared against. My understanding is that PACC differs by freezing the generator and using on-policy prediction matching, rather than TinyHistory's two-stage training with generator adaptation. But the general idea are very similar to TinyHistory which makes the novelty of this paper to be limited. Without a comparison or controlled analysis of these choices, the additional contribution over existing learned history compression is unclear to me.

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Thank you!

W1 - Comparison with TinyHistory - Good question! But code, data, and checkpoints are all unavailable.

Thank you for pointing us to TinyHistory. We have added it to the revised Related Work section (highlighted in blue). The key difference is how memory is learned. TinyHistory uses approximately five million real videos for history-encoder pretraining and joint encoder–generator adaptation. PACC instead lets the video generator teach its own memory, using no additional real-video training data and no generator fine-tuning.

Our main checkpoint uses 5,000 compressor updates, corresponding to 40,000 generated rollout-record exposures—less than 1% of a single pass over TinyHistory’s reported training data, by example count. The same frozen generator produces continuations with compressed memory and supervises the compressor with its full-history predictions at the same denoising states. This teaches memory what to preserve from the generator’s own experience, including states affected by earlier compression errors. PACC demonstrates that useful memory can be learned and consumed by an existing generator while its parameters remain unchanged.

The reported systems also differ substantially. TinyHistory adapts Wan 2.1 14B and HunyuanVideo 12.8B, whereas PACC operates on frozen Causal-rCM and Causal Forcing backbones. Comparing the complete systems would therefore mix differences in backbone, generator adaptation, and memory learning. Furthermore, TinyHistory’s official project pages provide no training data, training code, or trained checkpoints. These missing artifacts prevent direct evaluation of its reported models and faithful reproduction of its training setup. A controlled comparison with the reported TinyHistory system is therefore unavailable. Our evaluation holds the backbone fixed to measure the benefit of memory strategies without conflating it with generator adaptation.

Reviewer_SnCBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The subject of the paper is autoregressive video generation with a model diffusion that recursively generates embeddings of a chunk of frames based on embeddings of preceding frames. The paper introduces a module that compresses embedding of preceding frames into a smaller sequence. This module is trained along with the transformer that translates preceding embeddings into the condition vector which is fed to the frozen diffusion model.

Critical strengths
  1. The very idea of introducing compression of previous frames into the autoregressive video generation is interesting and inspiring.
  2. Engaging distillation of the original transformer into the training of the combination of the compressor and the new transformer was a reasonable choice.
  3. The paper is very well written.
Critical weaknesses

Although the paper presents an interesting development, its novelty is still limited. The idea the authors introduce does not change qualitatively the basic mechanics of recursive video generation based on the transformer and diffusion.

Questions for authors

Did the authors try to apply memory compression recursively or any other way to make the memory grow slower than linearly with the duration of the past video? Is there any fundamental reason for such recursion not to be feasible?

1 comment

W1 – Central contribution – PACC lets the video model teach its own memory.

Thank you for the feedback. PACC’s central contribution is to let the video model teach its own memory. We learn a compressor through on-policy distillation: the same frozen generator produces continuations from compressed memory and teaches the compressor using its full-history predictions at the same denoising states. This trains memory on the trajectories it actually induces, including the effects of earlier compression errors. The learned memory aggregates information across historical frames without real-video supervision or generator fine-tuning. Across two frozen backbones, PACC improves long-horizon memory over frame-selection baselines while maintaining competitive minute-long generation quality.


Q1 – Recursive compression – A great extension. We will investigate it in future work.

Thank you for the suggestion. Recursive compression would be a great extension and an even more natural way to maintain memory as a video grows. Our current implementation compresses each block once; recursively updating memory could further reduce storage growth. The extension is nontrivial because (1) the compressor must learn to preserve useful information when its input is already compressed, and (2) end-to-end training introduces a longer recurrent computation graph, making optimization across successive memory updates more challenging. We will investigate this extension and adapt our on-policy training to recursive memory updates in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.