acceptodds
Under review as a conference paper at ICLR 2027

Coefficient-Calibrated Memory Retrieval via Sparse Coding for Long Video Generation

Abstract

Autoregressive video generation models use fixed context windows to make long rollouts tractable, but progressively lose early information as earlier frames are evicted from the context. Existing retrieval methods retrieve the top- historical latent entries from a memory bank by similarity, improving subject and background consistency. However, independent top- selection may retrieve near-duplicate historical latents whose accumulated evidence causes *memory over-anchoring*, suppressing motion evolution and ultimately leading to motion collapse. We propose a training-free memory interface to address this failure, namely the Coefficient-calibrated Atom REtrieval of MEMory (CARE-MEM). We map historical latent entries in the memory bank to memory atoms that form an online dictionary. Our approach adopts constrained sparse coding to jointly select a fixed-budget set of memory atoms and applies the resulting coefficients to reweight the attention evidence of the corresponding memory entries before normalization. Across three autoregressive backbones and -, -, and -second generation under VBench-Long, the proposed method achieves the best or second-best performance in of metric–setting comparisons. Compared with the top- retrieval baseline, our method improves Dynamic Degree by points on average while maintaining comparable Motion Smoothness, with additional gains in quality and consistency.

Primary area:
generative models
Keywords:
Video Generation, Memory Retrieval, Dictionary Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wstAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE-MEM uses constrained nonnegative sparse coding to select complementary historical video latents, then applies the resulting coefficients as attention priors. The training-free method aims to preserve appearance consistency while reducing motion collapse during long autoregressive rollouts.

Critical strengths
  • A practitcal method jointly addresses memory selection and influence without retraining the generator.
  • Three backbones and 30/60/120-second rollouts is great effort for evaluation.
  • Useful ablations are done, component and sparse-solver comparisons support contributions from both selection and coefficient weighting.
Critical weaknesses
  • Coefficients change total memory influence as well as relative weighting. Missing uniform-attenuation and weighted-top-K controls leave this confound unresolved; some alternatives improve dynamics despite lower diversity.
  • Proposition 1 concerns optimal fits on prescribed two-atom supports, not the implemented six-slot greedy solver; Proposition 2 bounds conditional memory shares, not generation quality. These theoretical claims seem a bit forced.

Happy to raise score if these are addressed.

Questions for authors
  • How sensitive are results to dictionary size and optimization iterations? Does refitting coefficients to convergence on the selected support change the gains?
  • Please cite more video memory papers with discussion in the related work. Here is a list:
    • Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
    • Mixture of Contexts for Long Video Generation. (ICLR 2026)
    • SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
    • Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
    • Moga: Mixture-of-groups attention for end-to-end long video generation. (ICLR 2026)

There are much more video generation memory compression/retrieval papers than I listed here and it would be nice if the related work discussion could be done with more efforts.

6 comments

We thank the reviewer for the constructive suggestion! We provide our response below and have included the discussions in our revised manuscript.

W1. Coefficients change total memory influence as well as relative weighting. Missing uniform-attenuation and weighted-top-K controls leave this confound unresolved; some alternatives improve dynamics despite lower diversity.

We introduce uniform coefficient weighting and weighted Top-K controls to validate the complementary roles of memory selection and coefficient calibration.

First, uniform coefficient weighting applies the same mean coefficient to all selected memories, yielding only a modest Dynamic Degree improvement (47.97 to 49.84), compared with 53.96 using individual sparse coefficients. This supports the importance of preserving the relative contributions of complementary memories, rather than uniformly adjusting memory attention strength.

Second, coefficient-weighted Top-K improves AVG from 76.29 to 77.78, confirming that coefficient calibration is beneficial even with conventional retrieval. However, CARE-MEM further improves AVG to 78.60, demonstrating the additional benefit of jointly selecting complementary memory atoms. Consistently, sparse selection alone improves AVG from 76.29 to 77.55 under unit priors (a unit prior leaves the native attention unchanged).

Overall, these results demonstrate the complementary benefits of sparse coding for selecting diverse, jointly informative memory atoms and coefficient-calibrated attention for preserving their individual contributions during generation.

Memory selectionAttention priorAVG ↑Subject ↑Background ↑Motion ↑Dynamic ↑Aesthetic ↑Imaging ↑
Top-KUnit Prior (No Reweighting)76.2997.5796.5698.7642.2457.1765.43
Top-KFitted reconstruction coefficient prior77.7897.7996.7698.5549.2257.5666.81
Constrained sparse codingUnit Prior (No Reweighting)77.5597.7396.6898.6247.9757.7766.54
Constrained sparse codingUniform (mean) sparse coding coefficient-calibrated prior77.9297.7996.7998.5249.8457.6266.95
Constrained sparse codingSparse coding coefficient-calibrated prior (CARE-MEM)78.6097.6796.7098.5353.9657.9666.77

W2. Proposition 1 concerns optimal fits on prescribed two-atom supports, not the implemented six-slot greedy solver; Proposition 2 bounds conditional memory shares, not generation quality. These theoretical claims seem a bit forced.

We extend Proposition 1 to arbitrary retrieval budgets and the actual finite-iteration coefficients. We also clarify that Proposition 2 establishes coefficient-to-attention alignment: it explains how the contributions estimated during retrieval remain effective as memory-specific priors within the generation process.

Generalized Proposition 1. Let SS be the nn-atom support returned by the nn-slot greedy solver, with terminal coefficients α^S≥0\hat\alpha_S \ge 0, and let TT be the Top-nn support, with unit-normalized descriptors and n≥2n \ge 2. Define

GU=DU⊤DU,EU⋆=min⁡a≥012∥q−DUa∥22,E^S=12∥q−DSα^S∥22,G_U = D_U^\top D_U, \qquad E_U^\star = \min_{a \ge 0} \tfrac12 \lVert q - D_U a \rVert_2^2, \qquad \widehat E_S = \tfrac12 \lVert q - D_S \hat\alpha_S \rVert_2^2,

where U∈{S,T}U \in \lbrace S, T \rbrace. Write

b=DT⊤q,B=1⊤b,s=max⁡ibi,bmin⁡=min⁡ibi,RΩ=1−2E^S.\mathbf b = D_T^\top q, \qquad B = \mathbf 1^\top \mathbf b, \qquad s = \max_i b_i, \qquad b_{\min} = \min_i b_i, \qquad R_\Omega = 1 - 2 \widehat E_S.

Define the following quantities directly from the returned and comparator supports:

ρT=min⁡i≠j∈Tdi⊤dj,μT=1⊤GT1−nn(n−1),ρ=min⁡{ρT,bmin⁡B−(n−1)bmin⁡},cS=1−λmin⁡(GS).\rho_T = \min_{i \ne j \in T} d_i^\top d_j, \qquad \mu_T = \frac{\mathbf 1^\top G_T \mathbf 1 - n}{n(n-1)}, \qquad \rho = \min \Big\lbrace \rho_T, \frac{b_{\min}}{B - (n-1) b_{\min}} \Big\rbrace, \qquad c_S = 1 - \lambda_{\min}(G_S).

Here ρT\rho_T and μT\mu_T are the minimum and the mean pairwise cosine within TT, and cSc_S is the spectral coherence of SS. Assume only bmin⁡>0b_{\min} > 0, 0≤ρT<10 \le \rho_T < 1, and RΩ>0R_\Omega > 0. Define

τ0,Ω(n)=sB−RΩ(n−1)RΩ,Φn(z)=h2(1+(n−1)zn)+(n−1)(1−z)nlog⁡(n−1),\tau_{0,\Omega}^{(n)} = \frac{sB - R_\Omega}{(n-1) R_\Omega}, \qquad \Phi_n(z) = h_2\Big(\frac{1 + (n-1)z}{n}\Big) + \frac{(n-1)(1-z)}{n} \log(n-1),

where h2h_2 is the binary entropy in nats and H(U)H(U) is the von Neumann entropy of GU/nG_U / n. Then:

(i) Reconstruction.

ET⋆−E^S≥(n−1)RΩ(ρ−τ0,Ω(n))2[1+(n−1)ρ].E_T^\star - \widehat E_S \ge \frac{(n-1) R_\Omega \big(\rho - \tau_{0,\Omega}^{(n)}\big)}{2 [1 + (n-1)\rho]}.

(ii) Diversity.

H(S)≥Φn(cS),H(T)≤Φn(μT),hencecS<μT⟹H(S)>H(T).H(S) \ge \Phi_n(c_S), \qquad H(T) \le \Phi_n(\mu_T), \qquad \text{hence} \quad c_S < \mu_T \quad \Longrightarrow \quad H(S) > H(T).

Remark (consistency with the original Proposition 1). In the redundant-comparator regime of the original proposition, i.e. cS<τ≤ρc_S < \tau \le \rho, monotonicity of the two bounds gives

ET⋆−E^S≥(n−1)RΩ(τ−τ0,Ω(n))2[1+(n−1)τ],H(S)>Φn(τ)≥H(T).E_T^\star - \widehat E_S \ge \frac{(n-1) R_\Omega \big(\tau - \tau_{0,\Omega}^{(n)}\big)}{2 [1 + (n-1)\tau]}, \qquad H(S) > \Phi_n(\tau) \ge H(T).

Hence τ≥τ0,Ω(n)\tau \ge \tau_{0,\Omega}^{(n)} guarantees E^S≤ET⋆\widehat E_S \le E_T^\star. Because the entropy lower bound Φn(τ)\Phi_n(\tau) decreases with τ\tau, the choice τ=τ0,Ω(n)\tau = \tau_{0,\Omega}^{(n)} maximizes the guaranteed diversity lower bound of the selected memory atoms without increasing the query reconstruction error relative to Top-nn retrieval, when cS<τ0,Ω(n)≤ρc_S < \tau_{0,\Omega}^{(n)} \le \rho. This is the conclusion of the original Proposition 1 and its Remark 1, now for any nn and for the actual terminal coefficients.

Recovery of the original Proposition 1. Define the selected-support fitting gap

εΩ=E^S−ES⋆≥0.\varepsilon_\Omega = \widehat E_S - E_S^\star \ge 0.

In the original two-atom setting, ρT=μT\rho_T = \mu_T is the cosine of the Top-2 pair (written ρ\rho in the submission), B=s+bB = s+b, ρ=min⁡{ρT,b/s}\rho = \min \lbrace \rho_T, b/s \rbrace, cS=cc_S = c, and RΩ=s2+g2−2εΩR_\Omega = s^2 + g^2 - 2\varepsilon_\Omega. The original assumptions c<τ≤ρTc < \tau \le \rho_T and τ≤b/s\tau \le b/s are exactly cS<τ≤ρc_S < \tau \le \rho, so the Remark above applies and gives

\qquad H(S) > h_2\left( \frac{1+\tau}{2} \right) \ge H(T),$$ with the original $\tau_0 = (sb - g^2)/(s^2 + g^2)$. Therefore, $\varepsilon_\Omega = 0$ recovers Eq. (8) exactly. Empirically, $\varepsilon_\Omega$ becomes very small as the number of optimization steps increases. The no-worse reconstruction conclusion remains valid whenever the original certified margin covers this gap.

Proof. Part (i). Put C=(1−ρ)In+ρ11⊤C = (1-\rho) I_n + \rho \mathbf 1 \mathbf 1^\top, which is positive definite because 0≤ρ<10 \le \rho < 1. Because ρ≤bmin⁡/(B−(n−1)bmin⁡)\rho \le b_{\min} / (B - (n-1) b_{\min}),

\qquad \mathbf 1^\top v = \frac{B}{1+(n-1)\rho}.$$ Because $\rho \le \rho_T \le d_i^\top d_j$ for all distinct $i, j \in T$, every $a \ge 0$ satisfies $\lVert D_T a \rVert_2^2 \ge a^\top C a$. Completing the square, $$\begin{aligned} \frac12 \lVert q - D_T a \rVert_2^2 &\ge \frac12 - \mathbf b^\top a + \frac12 a^\top C a \\\\ &= \frac{1 - \mathbf b^\top v}{2} + \frac12 (a-v)^\top C (a-v) \\\\ &\ge \frac12 \left( 1 - \frac{sB}{1+(n-1)\rho} \right). \end{aligned}$$ The last step uses $\mathbf b^\top v \le s \mathbf 1^\top v$, which holds because $v \ge 0$ and $b_i \le s$. Taking the minimum over $a \ge 0$ and subtracting $\widehat E_S = (1 - R_\Omega)/2$ gives (i). Part (ii). The eigenvalues of $G_S/n$ sum to one and each is at least $(1-c_S)/n$. Entropy is concave, so it is minimized by concentrating the remaining mass in one eigenvalue, which gives $H(S) \ge \Phi_n(c_S)$; this holds for every support. For $T$, the all-ones Rayleigh quotient gives $$\lambda_{\max}(G_T/n) \ge \frac{\mathbf 1^\top G_T \mathbf 1}{n^2} = \frac{1+(n-1)\mu_T}{n} \ge \frac1n.$$ For a fixed largest eigenvalue $\lambda \ge 1/n$, uniform remaining eigenvalues maximize entropy, and the resulting value $h_2(\lambda) + (1-\lambda) \log(n-1)$ is non-increasing in $\lambda$; hence $H(T) \le \Phi_n(\mu_T)$. Since $\Phi_n$ is strictly decreasing on $[0,1]$, the condition $c_S < \mu_T$ yields $$H(S) \ge \Phi_n(c_S) > \Phi_n(\mu_T) \ge H(T).$$ $\square$ **Proposition 2: a fidelity guarantee for coefficient-to-attention transfer.** Proposition 2 establishes that **the relative contributions estimated by sparse coding are transferred into relative memory attention with a bounded distortion from native query-key evidence**. Under comparable native evidence, the bound guarantees that attention closely follows the coefficient-derived allocation when generating videos. Its purpose is therefore to justify coefficient calibration *inside the generation process*: the reconstruction coefficients are not merely retrieval scores, but memory-specific priors whose intended relative influence is preserved within the stated bound. The theory establishes the fidelity of coefficient transfer and the experiment below from Table 2 in the paper supports the effectiveness of enabling this calibration during generation. | Methods | AVG ↑ | Subject ↑ | Background ↑ | Motion ↑ | Dynamic ↑ | Aesthetic ↑ | Imaging ↑ | |:---------:|:---------:|:------------:|:--------:|:---------:|:-----------:|:---------:|:-----:| | Constrained Sparse Coding for Memory Selection | 77.55 | **97.73** | 96.68 | **98.62** | 47.97 | 57.77 | 66.54 | | Constrained Sparse Coding for Memory Selection + Coefficient-Calibrated Attention | **78.60** | 97.67 | **96.70** | 98.53 | **53.96** | **57.96** | **66.77** | **Q1. How sensitive are results to dictionary size and optimization iterations? Does refitting coefficients to convergence on the selected support change the gains?** We would like to clarify that dictionary size is not a tunable hyperparameter in CARE-MEM. The dictionary is constructed online from eligible historical latents, so its size grows with the video length. As shown in Table 1 of the paper, CARE-MEM achieves the best AVG at all three durations (30s, 60s, and 120s) on all three backbones. The gains of our method persist as the dictionary grows. As shown in the table below, performance changes little after a moderate number of optimization steps: AVG reaches 78.44 at 15 steps and 78.60 at 20 steps. Further refitting the coefficients to convergence on the selected support yields a nearly identical AVG of 78.55, suggesting limited sensitivity to additional optimization once the coefficients are sufficiently refined. | Optimization steps | AVG ↑ | Subject ↑ | Background ↑ | Motion ↑ | Dynamic ↑ | Aesthetic ↑ | Imaging ↑ | |:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | 5 | 78.09 | 97.75 | 96.77 | 98.49 | 51.15 | 57.57 | 66.82 | | 10 | 78.41 | 97.71 | 96.68 | 98.52 | 52.81 | 58.06 | 66.66 | | 15 | 78.44 | 97.72 | 96.71 | 98.53 | 53.13 | 57.90 | 66.66 | | 20 (Ours) | 78.60 | 97.67 | 96.70 | 98.53 | 53.96 | 57.96 | 66.77 | | 20 + Refit | 78.55 | 97.70 | 96.71 | 98.55 | 53.49 | 58.18 | 66.70 | **Q2. Please cite more video memory papers with discussion in the related work.** Thank you so much for the suggestion and for the concrete list! We have expanded the related work in our revised submission with a dedicated discussion of video generation memory, which now covers all five papers you listed and some other papers.
Reviewer_yHcFOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates the memory retrieval problem in autoregressive video diffusion models. Existing retrieval methods typically rank historical latent states independently, which may result in the selection of largely redundant memory evidence and lead to what the authors refer to as memory over-anchoring. To address this issue, the authors propose CARE-MEM, a training-free memory interface. CARE-MEM introduces two key components: a constrained sparse-coding optimization method for memory selection and coefficient-calibrated priors for attention control. The method is evaluated on three autoregressive video-generation systems, Self-Forcing, LongLive, and Causal-Forcing, and demonstrates competitive performance across multiple settings. Overall, this work identifies an important limitation of existing autoregressive video diffusion models and proposes an effective approach to mitigate it.

Critical strengths
  1. Through comparative experiments, the authors show that, compared with the conventional Top-K selection method, CARE-MEM selects memory states with lower similarity during generation, thereby reducing the redundant memory selection problem associated with conventional Top-K retrieval.

  2. To address the redundancy of retrieved memory evidence, the authors design two targeted components: constrained sparse coding optimization for memory selection, incorporating an atom-correlation constraint, and coefficient-calibrated priors for attention control. Ablation experiments further validate the effectiveness of both components.

Critical weaknesses

Although the main experiments establish gains at K=6 and Table 4 varies the retrieved/recent allocation, the paper does not show how CARE-MEM’s advantage over Top-K retrieval changes across retrieval budgets. Table 4 reports only CARE-MEM results and changes the amount of recent context alongside the retrieval budget, so it does not establish the relative benefit at each allocation. The controlled toy example and the reconstruction–diversity analysis in Section 3.2 also focus on two memories. Matched comparisons across retrieval budgets would better establish the range of settings in which the proposed selection and attention calibration are beneficial.

Questions for authors

Could the authors clarify how CARE-MEM’s gains over Top-K retrieval vary with the retrieval budget, using matched retrieved/recent allocations for both methods? If available, scenario-specific results for multi-subject interactions, occlusion, or scene transitions would also help clarify where the method is most beneficial and where it may be limited.

3 comments

We thank the reviewer for the constructive suggestion! We provide our response below and have included the discussions in our revised manuscript.

Q1. Could the authors clarify how CARE-MEM's gains over Top-K retrieval vary with the retrieval budget, using matched retrieved/recent allocations for both methods?

We extend Table 4 with matched LongLive-RAG results at four retrieved/recent allocations under the same context budget. As shown in the table below, CARE-MEM achieves a higher six-metric AVG at every allocation, and the gain grows with the retrieval budget (+0.38, +1.18, +2.31, and +2.64 AVG at 2, 4, 6, and 8 retrieved slots).

Budget (retrieved + recent)MethodAVG ↑Subject ↑Background ↑Motion ↑Dynamic ↑Aesthetic ↑Imaging ↑
2+9LongLive-RAG77.1697.7096.7498.6647.8657.3964.62
CARE-MEM77.5497.5496.6498.5251.5656.7264.24
4+7LongLive-RAG76.9897.8296.7998.6545.2157.2566.14
CARE-MEM78.1697.7096.7498.5352.3457.8765.75
6+5LongLive-RAG76.2997.5796.5698.7642.2457.1765.43
CARE-MEM78.6097.6796.7098.5353.9657.9666.77
8+3LongLive-RAG73.5895.4695.4297.6930.4258.4064.10
CARE-MEM76.2295.5195.3397.3446.2058.2364.70

Meanwhile, we have generalized Proposition 1 (the reconstruction-diversity analysis in Section 3.2) from two memories to an arbitrary retrieval budget nn, so the analysis also applies to different retrieval budgets (please see our response to Reviewer wstA, W2).

Q2. If available, scenario-specific results for multi-subject interactions, occlusion, or scene transitions would also help clarify where the method is most beneficial and where it may be limited.

We identify three challenging scenarios from the prompt descriptions: multi-subject interactions, in which two or more subjects act on or respond to one another; occlusion, in which an object partially blocks a subject from view; and scene or camera transitions, in which the shot or setting changes within a video. For each scenario, we select five prompts to evaluate them on VBench-Long.

ScenarioMethodAVG ↑SubjectBackgroundMotionDynamicAestheticImaging
Multi-subjectLongLive-RAG81.8497.8796.3798.5472.0059.3166.96
CARE-MEM84.3997.4896.4598.3086.6760.3567.07
OcclusionLongLive-RAG70.8598.7697.2299.296.6754.7068.47
CARE-MEM73.7698.9697.6099.2518.6756.5971.47
Scene/camera transitionsLongLive-RAG77.2495.4995.8197.5653.3358.4062.83
CARE-MEM82.5394.4095.3396.4689.3355.8163.88

As shown in the table above, CARE-MEM improves AVG in all three scenarios, by +2.55, +2.91, and +5.29 points, respectively. The largest improvement occurs in scene/camera transitions, highlighting the benefits of mitigating memory over-anchoring when substantial motion evolution is required.

Although CARE-MEM improves performance in the occlusion scenario, occlusion remains challenging, potentially due to the underlying pretrained video diffusion foundation model's limited 3D spatial reasoning and object permanence.

Thank you for your response and the additional experiments. My concerns have been largely addressed, and I will maintain my original score.

Thank you so much for your reply and evaluation!

Reviewer_JUVKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARE-MEM, a training-free memory retrieval method for autoregressive long-video generation. It uses constrained sparse coding to select complementary historical memories and further uses the sparse coefficients to calibrate their attention contributions, aiming to reduce memory over-anchoring while preserving long-term consistency and motion dynamics.

Critical strengths

1.Clear motivation. The paper identifies memory over-anchoring as an important failure mode of top-K memory retrieval in long-video generation. 2.Simple and coherent design. Sparse coding is used for both complementary memory selection and coefficient-based attention calibration. 3.Broad evaluation. CARE-MEM is validated on three autoregressive backbones and multiple video lengths, showing consistent improvements in consistency and motion dynamics.

Critical weaknesses
  1. The paper would benefit from comparisons with more recent long-video memory retrieval methods, such as OmniMem and MemRoPE. Since CARE-MEM mainly improves how historical memories are selected and utilized, comparisons with stronger contemporary memory-based baselines would help better demonstrate its relative advantage and clarify its position within recent long-video generation research.
  2. The computational cost of CARE-MEM is not sufficiently analyzed. The method performs iterative sparse optimization over an online memory dictionary that grows with the video length, which may introduce additional inference overhead. Reporting retrieval latency, overall generation speed, GPU memory usage, and the scaling behavior with increasing video length would help clarify its practical efficiency.
  3. Memory over-anchoring is the central motivation of the paper, but the current evidence is still somewhat indirect. The paper mainly shows that CARE-MEM retrieves more diverse memories and improves motion-related metrics. A more direct analysis of the relationship between memory redundancy and subsequent motion degradation would provide stronger support for the proposed failure mode.
  4. The current evaluation is limited to videos up to 120 seconds and mainly relies on VBench-Long metrics. Since the method targets long-horizon video generation, experiments on longer sequences would better demonstrate its scalability. In addition, human evaluation or VLM-based assessment could help verify whether the improved motion dynamics remain natural and semantically meaningful.
  5. The paper directly uses sparse coding coefficients as attention priors, but the motivation for this specific weighting strategy could be further validated. The sparse coefficients are obtained in the retrieval embedding space, while attention operates in the generator feature space. Comparisons with simpler alternatives, such as uniform weighting or similarity-based weighting, would help determine whether the improvement specifically comes from the proposed coefficient calibration.
Questions for authors

Please refer to the Critical Weaknesses above.

3 comments

We thank the reviewer for the constructive suggestion! We provide our response below and have included the discussions in our revised manuscript.

W1. The paper would benefit from comparisons with more recent long-video memory retrieval methods, such as OmniMem and MemRoPE.

As shown in the table below, CARE-MEM is competitive with the additional contemporary baselines. We evaluate OmniMem and MemRoPE on the same 128 VBench-Long prompts using nominal 30s videos (474 frames at 16 fps). MemRoPE and CARE-MEM use the Causal-Forcing backbone; OmniMem uses its official checkpoint and is therefore a system-level comparison.

MethodAVGSubjectBackgroundMotionDynamicAestheticImaging
OmniMem77.9297.8496.5798.8447.8658.4467.95
MemRoPE83.2495.2295.2296.8887.1956.0668.90
CARE-MEM83.5095.4695.2696.7287.9756.7868.81

W2. Reporting retrieval latency, overall generation speed, GPU memory usage, and the scaling behavior with increasing video length would help clarify its practical efficiency.

As shown in the table below, we profiled CARE-MEM on the Self-Forcing backbone at 30s, 60s, and 120s on NVIDIA RTX 3090 (24GB) GPUs. End-to-end generation time grows approximately linearly with video length, while retrieval accounts for only about 0.5% of total time and peak GPU memory grows by only 0.4% from 30s to 120s.

Video lengthEnd-to-end time (s/video)Retrieval time (s/video)Retrieval time proportionPeak GPU memory (GB)
30s219.890.9300.423%19.692
60s480.892.1480.447%19.717
120s1,001.915.1530.514%19.771

These results demonstrate that CARE-MEM introduces minimal computational overhead during video generation inference.

W3. A more direct analysis of the relationship between memory redundancy and subsequent motion degradation would provide stronger support for the proposed failure mode.

We conduct a controlled experiment on 5 randomly sampled videos: we progressively replaced the retrieved Top-K memory atoms with more redundant ones while preserving their memory-query similarity ranking.

Memory Cosine Similarity (memory redundancy)0.630.660.690.720.73
Dynamic Degree42.6741.3341.3340.0034.67

As shown in the table above, Dynamic Degree declines from 42.67 to 34.67 as memory redundancy increases. This trend provides more direct evidence that increasing redundancy among retrieved memories suppresses subsequent motion evolution, supporting our proposed memory over-anchoring failure mode.

W4. Since the method targets long-horizon video generation, experiments on longer sequences would better demonstrate its scalability. In addition, human evaluation or VLM-based assessment could help verify whether the improved motion dynamics remain natural and semantically meaningful.

We extend our evaluation of VBench-Long to 240-second video generation. CARE-MEM remains effective at 240s, twice the longest duration in the submission. As shown in the table below, CARE-MEM outperforms LongLive-RAG in overall performance using Self-Forcing backbone (AVG: 77.38 vs. 75.87), improving Dynamic Degree by +5.52 points while maintaining comparable Motion Smoothness and improving subject consistency, background consistency, and visual quality.

DurationMethodAVGSubjectBackgroundMotionDynamicAestheticImaging
240sLongLive-RAG75.8797.2596.3198.5543.5255.5764.02
240sCARE-MEM77.3897.4696.4898.5249.0456.8865.91

Furthermore, we adapt the VLM evaluation pipeline of LongLive-RAG, using a rubric that explicitly assesses motion naturalness and semantic coherence. As shown in the table below, CARE-MEM receives higher scores at both 120s and 240s, providing evidence that its improved dynamics remain natural and semantically meaningful.

Method120s ↑240s ↑
LongLive-RAG4.45/5.004.40/5.00
CARE-MEM4.70/5.004.70/5.00

W5. Comparisons with simpler alternatives, such as uniform weighting or similarity-based weighting, would help determine whether the improvement specifically comes from the proposed coefficient calibration.

The key advantage of coefficient calibration is not merely adjusting overall memory attention strength, but preserving the relative contributions of complementary memories identified through joint sparse reconstruction.

A uniform coefficient prior adjusts overall memory influence without distinguishing individual memory contributions, yielding only a modest Dynamic Degree improvement (47.97 to 49.84), compared with 53.96 using our coefficients. Similarly, cosine-based priors weight each query–memory pair independently, without explicitly accounting for correlations and complementarity among memories (48.33 Dynamic Degree).

In contrast, sparse coding jointly reconstructs the query, producing coefficients that reflect each memory's contribution conditioned on the other selected memories. Coefficient-calibrated attention transfers these relative contributions into the generator through memory-specific scalar priors, while retaining native query-key compatibility without requiring aligned feature spaces. Proposition 2 provides a theoretical bound on the distortion of this coefficient-to-attention alignment.

Together, these results explain why CARE-MEM achieves the highest AVG (78.60), outperforming uniform coefficient weighting (77.92) and cosine-based weighting (77.54), demonstrating the benefit of preserving joint reconstruction contributions during generation.

Attention priorAVG ↑Subject ↑Background ↑Motion ↑Dynamic ↑Aesthetic ↑Imaging ↑
Unit Prior (No Reweighting)77.5597.7396.6898.6247.9757.7766.54
Cosine similarity (Query-Memory) weighted prior77.5497.7496.7398.6048.3357.2766.54
Uniform coefficient prior77.9297.7996.7998.5249.8457.6266.95
Coefficient-calibrated prior (CARE-MEM)78.6097.6796.7098.5353.9657.9666.77
Area_Chair_2NqNOpenReview ↗

Reminder: Please Follow Up on Authors’ Responses

Dear Reviewer JUVK,

A friendly reminder to check the authors’ responses and follow up with any comments or remaining concerns. Your feedback would be greatly appreciated.

Thank you for your time and effort!

AC

Comments on the paper

1 comment

Dear Reviewers and AC,

As the discussion period approaches its end, we briefly summarize our rebuttal below. Specifically, we have:

  1. Highlighted the significance of CARE-MEM: We propose a practical training-free method to address memory over-anchoring, which reviewers recognized as an important failure mode of Top-K retrieval (JUVK, yHcF). We provided direct evidence that more redundant retrieved memories lead to weaker motion dynamics (JUVK W3).
  2. Validated the contribution of each component: With the proposed sparse coding memory selection, our coefficient calibration outperforms uniform and cosine similarity-based weighting, as it preserves the relative contributions of complementary and diverse memories rather than merely rescaling the overall attention strength (JUVK W5; wstA W1). Proposition 2 further provides a fidelity guarantee for coefficient-to-attention transfer, explaining how these contributions remain effective in attention during generation (wstA W2). Our coefficient calibration also improves Top-K retrieval, but this weighted Top-K still falls behind CARE-MEM, showing the benefit of selecting complementary and diverse memories through our proposed method (wstA W1).
  3. Demonstrated generalization and scalability: CARE-MEM outperforms Top-K retrieval (LongLive-RAG) at every matched retrieval budget, with larger gains as more memories are retrieved (yHcF Q1), and we theoretically extended Proposition 1 to arbitrary budgets and our actual solver (wstA W2). CARE-MEM also achieves a higher overall score than the contemporary memory methods OmniMem and MemRoPE (JUVK W1). In multi-subject, occlusion, and scene-transition scenarios, it consistently improves overall performance over Top-K retrieval (yHcF Q2). On 240s videos, it still outperforms Top-K retrieval (JUVK W4). Moreover, as the video length grows, CARE-MEM adds only negligible computational overhead and latency (JUVK W2).
  4. Verified the stability and robustness of our method: The gains of our method persist as the memory dictionary grows with video length, and performance changes little after a moderate number of optimization steps or with further coefficient refitting (wstA Q1). In a VLM-based evaluation of motion naturalness and semantic coherence, CARE-MEM receives higher scores than LongLive-RAG, providing evidence that its improved dynamics remain natural and semantically meaningful (JUVK W4).

We also expanded the related work to cover all papers suggested by Reviewer wstA (wstA Q2). All new results and discussions have been included in the revised manuscript.

We hope these responses address the reviewers’ concerns and kindly ask the AC to take them into account. We sincerely thank you for your time and thoughtful evaluation!

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.