- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CARE-MEM uses constrained nonnegative sparse coding to select complementary historical video latents, then applies the resulting coefficients as attention priors. The training-free method aims to preserve appearance consistency while reducing motion collapse during long autoregressive rollouts.
Critical strengths
- A practitcal method jointly addresses memory selection and influence without retraining the generator.
- Three backbones and 30/60/120-second rollouts is great effort for evaluation.
- Useful ablations are done, component and sparse-solver comparisons support contributions from both selection and coefficient weighting.
Critical weaknesses
- Coefficients change total memory influence as well as relative weighting. Missing uniform-attenuation and weighted-top-K controls leave this confound unresolved; some alternatives improve dynamics despite lower diversity.
- Proposition 1 concerns optimal fits on prescribed two-atom supports, not the implemented six-slot greedy solver; Proposition 2 bounds conditional memory shares, not generation quality. These theoretical claims seem a bit forced.
Happy to raise score if these are addressed.
Questions for authors
- How sensitive are results to dictionary size and optimization iterations? Does refitting coefficients to convergence on the selected support change the gains?
- Please cite more video memory papers with discussion in the related work. Here is a list:
- Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
- Mixture of Contexts for Long Video Generation. (ICLR 2026)
- SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
- Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
- Moga: Mixture-of-groups attention for end-to-end long video generation. (ICLR 2026)
There are much more video generation memory compression/retrieval papers than I listed here and it would be nice if the related work discussion could be done with more efforts.
6 comments
W2. Proposition 1 concerns optimal fits on prescribed two-atom supports, not the implemented six-slot greedy solver; Proposition 2 bounds conditional memory shares, not generation quality. These theoretical claims seem a bit forced.
We extend Proposition 1 to arbitrary retrieval budgets and the actual finite-iteration coefficients. We also clarify that Proposition 2 establishes coefficient-to-attention alignment: it explains how the contributions estimated during retrieval remain effective as memory-specific priors within the generation process.
Generalized Proposition 1. Let be the -atom support returned by the -slot greedy solver, with terminal coefficients , and let be the Top- support, with unit-normalized descriptors and . Define
where . Write
Define the following quantities directly from the returned and comparator supports:
Here and are the minimum and the mean pairwise cosine within , and is the spectral coherence of . Assume only , , and . Define
where is the binary entropy in nats and is the von Neumann entropy of . Then:
(i) Reconstruction.
(ii) Diversity.
Remark (consistency with the original Proposition 1). In the redundant-comparator regime of the original proposition, i.e. , monotonicity of the two bounds gives
Hence guarantees . Because the entropy lower bound decreases with , the choice maximizes the guaranteed diversity lower bound of the selected memory atoms without increasing the query reconstruction error relative to Top- retrieval, when . This is the conclusion of the original Proposition 1 and its Remark 1, now for any and for the actual terminal coefficients.
Recovery of the original Proposition 1. Define the selected-support fitting gap
In the original two-atom setting, is the cosine of the Top-2 pair (written in the submission), , , , and . The original assumptions and are exactly , so the Remark above applies and gives
\qquad H(S) > h_2\left( \frac{1+\tau}{2} \right) \ge H(T),$$ with the original $\tau_0 = (sb - g^2)/(s^2 + g^2)$. Therefore, $\varepsilon_\Omega = 0$ recovers Eq. (8) exactly. Empirically, $\varepsilon_\Omega$ becomes very small as the number of optimization steps increases. The no-worse reconstruction conclusion remains valid whenever the original certified margin covers this gap.Proof. Part (i). Put , which is positive definite because . Because ,
\qquad \mathbf 1^\top v = \frac{B}{1+(n-1)\rho}.$$ Because $\rho \le \rho_T \le d_i^\top d_j$ for all distinct $i, j \in T$, every $a \ge 0$ satisfies $\lVert D_T a \rVert_2^2 \ge a^\top C a$. Completing the square, $$\begin{aligned} \frac12 \lVert q - D_T a \rVert_2^2 &\ge \frac12 - \mathbf b^\top a + \frac12 a^\top C a \\\\ &= \frac{1 - \mathbf b^\top v}{2} + \frac12 (a-v)^\top C (a-v) \\\\ &\ge \frac12 \left( 1 - \frac{sB}{1+(n-1)\rho} \right). \end{aligned}$$ The last step uses $\mathbf b^\top v \le s \mathbf 1^\top v$, which holds because $v \ge 0$ and $b_i \le s$. Taking the minimum over $a \ge 0$ and subtracting $\widehat E_S = (1 - R_\Omega)/2$ gives (i). Part (ii). The eigenvalues of $G_S/n$ sum to one and each is at least $(1-c_S)/n$. Entropy is concave, so it is minimized by concentrating the remaining mass in one eigenvalue, which gives $H(S) \ge \Phi_n(c_S)$; this holds for every support. For $T$, the all-ones Rayleigh quotient gives $$\lambda_{\max}(G_T/n) \ge \frac{\mathbf 1^\top G_T \mathbf 1}{n^2} = \frac{1+(n-1)\mu_T}{n} \ge \frac1n.$$ For a fixed largest eigenvalue $\lambda \ge 1/n$, uniform remaining eigenvalues maximize entropy, and the resulting value $h_2(\lambda) + (1-\lambda) \log(n-1)$ is non-increasing in $\lambda$; hence $H(T) \le \Phi_n(\mu_T)$. Since $\Phi_n$ is strictly decreasing on $[0,1]$, the condition $c_S < \mu_T$ yields $$H(S) \ge \Phi_n(c_S) > \Phi_n(\mu_T) \ge H(T).$$ $\square$ **Proposition 2: a fidelity guarantee for coefficient-to-attention transfer.** Proposition 2 establishes that **the relative contributions estimated by sparse coding are transferred into relative memory attention with a bounded distortion from native query-key evidence**. Under comparable native evidence, the bound guarantees that attention closely follows the coefficient-derived allocation when generating videos. Its purpose is therefore to justify coefficient calibration *inside the generation process*: the reconstruction coefficients are not merely retrieval scores, but memory-specific priors whose intended relative influence is preserved within the stated bound. The theory establishes the fidelity of coefficient transfer and the experiment below from Table 2 in the paper supports the effectiveness of enabling this calibration during generation. | Methods | AVG ↑ | Subject ↑ | Background ↑ | Motion ↑ | Dynamic ↑ | Aesthetic ↑ | Imaging ↑ | |:---------:|:---------:|:------------:|:--------:|:---------:|:-----------:|:---------:|:-----:| | Constrained Sparse Coding for Memory Selection | 77.55 | **97.73** | 96.68 | **98.62** | 47.97 | 57.77 | 66.54 | | Constrained Sparse Coding for Memory Selection + Coefficient-Calibrated Attention | **78.60** | 97.67 | **96.70** | 98.53 | **53.96** | **57.96** | **66.77** | **Q1. How sensitive are results to dictionary size and optimization iterations? Does refitting coefficients to convergence on the selected support change the gains?** We would like to clarify that dictionary size is not a tunable hyperparameter in CARE-MEM. The dictionary is constructed online from eligible historical latents, so its size grows with the video length. As shown in Table 1 of the paper, CARE-MEM achieves the best AVG at all three durations (30s, 60s, and 120s) on all three backbones. The gains of our method persist as the dictionary grows. As shown in the table below, performance changes little after a moderate number of optimization steps: AVG reaches 78.44 at 15 steps and 78.60 at 20 steps. Further refitting the coefficients to convergence on the selected support yields a nearly identical AVG of 78.55, suggesting limited sensitivity to additional optimization once the coefficients are sufficiently refined. | Optimization steps | AVG ↑ | Subject ↑ | Background ↑ | Motion ↑ | Dynamic ↑ | Aesthetic ↑ | Imaging ↑ | |:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | 5 | 78.09 | 97.75 | 96.77 | 98.49 | 51.15 | 57.57 | 66.82 | | 10 | 78.41 | 97.71 | 96.68 | 98.52 | 52.81 | 58.06 | 66.66 | | 15 | 78.44 | 97.72 | 96.71 | 98.53 | 53.13 | 57.90 | 66.66 | | 20 (Ours) | 78.60 | 97.67 | 96.70 | 98.53 | 53.96 | 57.96 | 66.77 | | 20 + Refit | 78.55 | 97.70 | 96.71 | 98.55 | 53.49 | 58.18 | 66.70 | **Q2. Please cite more video memory papers with discussion in the related work.** Thank you so much for the suggestion and for the concrete list! We have expanded the related work in our revised submission with a dedicated discussion of video generation memory, which now covers all five papers you listed and some other papers.
We thank the reviewer for the constructive suggestion! We provide our response below and have included the discussions in our revised manuscript.
W1. Coefficients change total memory influence as well as relative weighting. Missing uniform-attenuation and weighted-top-K controls leave this confound unresolved; some alternatives improve dynamics despite lower diversity.
We introduce uniform coefficient weighting and weighted Top-K controls to validate the complementary roles of memory selection and coefficient calibration.
First, uniform coefficient weighting applies the same mean coefficient to all selected memories, yielding only a modest Dynamic Degree improvement (47.97 to 49.84), compared with 53.96 using individual sparse coefficients. This supports the importance of preserving the relative contributions of complementary memories, rather than uniformly adjusting memory attention strength.
Second, coefficient-weighted Top-K improves AVG from 76.29 to 77.78, confirming that coefficient calibration is beneficial even with conventional retrieval. However, CARE-MEM further improves AVG to 78.60, demonstrating the additional benefit of jointly selecting complementary memory atoms. Consistently, sparse selection alone improves AVG from 76.29 to 77.55 under unit priors (a unit prior leaves the native attention unchanged).
Overall, these results demonstrate the complementary benefits of sparse coding for selecting diverse, jointly informative memory atoms and coefficient-calibrated attention for preserving their individual contributions during generation.