- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PACC learns to compress video history into 10× fewer tokens through on-policy distillation: a frozen generator’s full-history predictions supervise its predictions conditioned on compressed memory. Short-rollout training supports later generation through retrieval of archived memory blocks.
Critical strengths
- A coherent preservation objective trains memory specifically for the frozen generator that consumes it, without requiring generator fine-tuning or long-video supervision.
- MBench improvements are consistent across two backbones, with competitive aggregate minute-long quality.
Critical weaknesses
- Gains are not isolated to learned compression. PACC uses 7,956 active context tokens versus Relax Forcing’s 6,240 and broader retrieval eligibility. Ablations omit matched-budget/retrieval controls and on-policy versus off-policy training.
- MBench uses one seed per case, without uncertainty estimates or separate coverage/reliability results. Explicit memory challenges last 26 seconds; minute-long tests assess visual quality. Happy to raise score if addressed properly.
Questions for authors
Please cite more video memory papers and discuss on them. Here is a list:
- Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
- Mixture of Contexts for Long Video Generation. (ICLR 2026)
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
- Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
- Moga: Mixture-of-groups attention for end-to-end long video generation. (ICLR 2026)
And there are much more. Video generation memory is not a new topic and the authors should cite and discuss more related work. Literally no one working in this area will be happy by the depth of related work discussion.
3 comments
Thanks! (2/2)
W2 – Memory evaluation – We now report coverage and reliability separately, test minute-long videos, and run three seeds with confidence intervals.
Separating Coverage and Conditional Reliability
MBench's M-score is the harmonic mean of coverage (C), which measures whether the requested memory event occurs, and conditional reliability (R), which measures consistency when the event is triggered. We agree that reporting only the aggregate M-score obscures the source of the improvements.
We have recovered the per-dimension C and R scores from our existing evaluations, without requiring additional generation or evaluation. The following table compares PACC against Relax Forcing, the strongest baseline in Table 1, on both backbones.
Table 1 evaluation: C / R in %
| Backbone | Method | Identity | Appearance | Geometry | Texture | State | Correctness | M-score |
|---|---|---|---|---|---|---|---|---|
| Causal-rCM | Relax | 22.5 / 29.4 | 20.8 / 46.0 | 20.0 / 30.9 | 33.0 / 88.9 | 61.3 / 67.9 | 61.5 / 73.7 | 43.01 |
| Causal-rCM | PACC | 38.3 / 35.5 | 41.7 / 47.4 | 28.0 / 27.3 | 40.0 / 84.8 | 62.8 / 70.8 | 61.0 / 76.9 | 49.64 |
| Causal Forcing | Relax | 75.8 / 30.9 | 75.0 / 49.7 | 22.0 / 26.4 | 70.0 / 86.6 | 80.3 / 69.6 | 80.0 / 65.1 | 58.58 |
| Causal Forcing | PACC | 75.8 / 36.7 | 75.0 / 52.7 | 46.0 / 22.2 | 72.0 / 85.9 | 81.8 / 72.1 | 82.3 / 67.7 | 61.77 |
This decomposition shows that PACC's memory-score improvements arise from both improved event coverage and improved conditional consistency in most dimensions. We have added this decomposition to Appendix B.4 (highlighted in blue).
Memory Evaluation on Minute-Long Videos
To test memory beyond 26 seconds, we extend MBench to 78.6-second videos by tripling the length of each caption segment, so the gap between a first appearance and its recall reaches about a minute. PACC keeps its advantage over Relax Forcing at this length: macro M improves by 3.89 points on Causal-rCM and 3.96 on Causal Forcing (95% paired case-bootstrap CIs [1.61, 6.31] and [2.10, 5.87]), and PACC scores higher in 11 of 12 dimensions.
MBench on 78.6-s videos (seed 0): M in %
| Backbone | Method | Identity | Appearance | Geometry | Texture | State | Correctness | Macro M |
|---|---|---|---|---|---|---|---|---|
| Causal-rCM | Relax | 37.0 | 47.2 | 32.0 | 71.3 | 68.7 | 60.8 | 52.84 |
| Causal-rCM | PACC | 42.1 | 46.9 | 37.9 | 82.9 | 69.4 | 61.2 | 56.73 |
| Causal Forcing | Relax | 47.6 | 64.2 | 26.8 | 76.7 | 72.2 | 59.5 | 57.82 |
| Causal Forcing | PACC | 56.3 | 69.0 | 30.1 | 80.4 | 73.4 | 61.5 | 61.79 |
Multiple Seeds and Uncertainty
We re-ran MBench with three seeds per case on all 611 cases. The table reports per-dimension and macro M as mean ± std over seeds, and the gain of PACC over Relax Forcing in macro M with a 95% paired bootstrap confidence interval over cases. PACC improves over Relax Forcing on both backbones with intervals that exclude zero, and it is higher in all three seeds on Causal-rCM and in two of three on Causal Forcing. We have added these results to Appendix B.4 and noted them in Section 4.2 of the revised manuscript (highlighted in blue).
MBench, 3 seeds × 611 cases: M in %
| Backbone | Method | Identity | Appearance | Geometry | Texture | State | Correctness | Macro M | Gain (95% CI) |
|---|---|---|---|---|---|---|---|---|---|
| Causal-rCM | Relax | 29.1 ± 3.2 | 36.0 ± 5.8 | 27.3 ± 3.7 | 54.5 ± 7.5 | 66.1 ± 0.9 | 58.0 ± 1.0 | 45.17 ± 3.24 | – |
| Causal-rCM | PACC | 39.0 ± 0.9 | 44.7 ± 0.1 | 32.5 ± 2.4 | 59.0 ± 5.9 | 63.5 ± 2.6 | 57.5 ± 1.2 | 49.36 ± 1.84 | +4.19 [1.98, 6.31] |
| Causal Forcing | Relax | 45.6 ± 1.9 | 61.9 ± 2.8 | 20.0 ± 2.1 | 77.9 ± 1.0 | 74.2 ± 1.9 | 62.3 ± 1.4 | 56.97 ± 0.85 | – |
| Causal Forcing | PACC | 48.3 ± 2.0 | 61.8 ± 0.9 | 28.5 ± 4.7 | 76.9 ± 1.8 | 74.5 ± 1.1 | 63.5 ± 0.7 | 58.92 ± 1.63 | +1.94 [0.52, 3.32] |
Q1 – Related work – We expanded the Related Work section in the revised manuscript.
We thank the reviewer for pointing out these omissions and suggesting relevant references. We have expanded the Related Work section of the revised manuscript (highlighted in blue), organizing video memory methods by how they store and access history.
I appreciate the authors' detailed rebuttal and do not have further problem with this paper and intend to increase my score to a 3. (That being said, it is something of a borderline accept for me.)
Thanks! (1/2)
We thank the reviewer for your observant review and recognition. Here, we address your concerns and questions.
W1 – Context budget, retrieval range and on-policy training– We added matched-budget controls and an on-policy vs. off-policy comparison. PACC is ahead in both.Thank you for highlighting these factors. Our main comparison evaluates complete memory strategies on the same frozen generator, using each strategy’s respective context budget and retrieval range. To isolate these factors, we added matched-budget Relax Forcing controls. Released Relax Forcing retrieves one raw frame from the recent half of the past (6,240 active tokens). Relax-H2 retrieves two frames instead. Relax-H2-all keeps two frames but searches the whole past, as PACC does. Relax-H3-all takes three frames from the whole past, so it uses more tokens than PACC. We evaluate all five policies on the full MBench benchmark (611 cases, three seeds per case) and report macro M as mean ± SD over seeds:
A larger budget alone does not help Relax Forcing: Relax-H2 is no better than the released model, and PACC beats it by about 4 points on both backbones. When Relax Forcing can also retrieve from the whole past (Relax-H2-all), PACC is still clearly ahead. Relax-H3-all retrieves three frames from the whole past and uses 18% more tokens than PACC, yet it still trails PACC on both backbones: by 4.0 points on Causal-rCM and 2.8 on Causal Forcing. These controls are now in Appendix C.1 of the revised manuscript (in blue).
Our existing ablations also examine compression capacity and training duration. In particular, the compression-rate sweep shows that fewer memory tokens can achieve higher memory scores, indicating that performance does not simply increase with token count.
We adopt on-policy training because the compressed memory itself influences subsequent generation. Fixed off-policy continuations may not reflect the behavior induced by the current compressor. We therefore generate continuations using the current compressed memory and obtain full-history teacher predictions at the same generated contexts and noisy inputs. This exposes the compressor to states affected by its own compression errors and trains it to preserve the information needed along the trajectories that the current compressed-memory system actually produces.
To address the reviewer's concern, we add the experiment of on-policy trainig v.s. off-policy training. Due to the limit amount of time, we train both of the method to 2k steps and compare their performance with Causal-rCM on MBench. On-policy is better than off-policy training.