- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReCam, a memory method for autoregressive long video generation. It is built on Wan2.2-I2V-A14B. The full video history is split into local spatiotemporal blocks. Each block is flattened into a vector and projected by a small MLP into the hidden space of the DiT. The paper also introduces dual-pointer denoising, which gives recent context frames and target frames separate noise schedules to reduce drift.
Critical strengths
- The architecture has some clear advantages. The model learns how to represent and read its memory end to end, and no memory labels are needed.
- The hyperparameter analysis in the appendix is careful and thorough.
- The subject reappearance results are reported separately for subjects that are present and absent in the first frame. This split directly shows the value of accessing distant history.
Critical weaknesses
- The videos in the supplementary material show highly repetitive motion and simple structure. Many models built on Wan2.1-1.3B can already produce this kind of result (such as echo-infinity). The paper uses the much larger Wan2.2-I2V-A14B, yet the method is not real-time and cannot follow complex action instructions. This makes the result feel incremental. If possible, I would like to see more complex videos of at least two minutes, in which a person leaves the scene for a while and then comes back.
- The novelty is limited. The core idea of FramePack is to compress history frames with larger patchify kernels to reduce the token count. ReCam flattens each block and applies an MLP, which is essentially a larger patch embedding. The dual-pointer design is also close to methods that add separate noise to history and target frames, such as Diffusion Forcing [1] and History-guided Video Diffusion [2].
- The ablation study misses a key comparison. The memory mechanism of ReCam should be compared with other designs under the same backbone and the same data. Possible options include a sliding window only, FramePack-style hierarchical compression, the full uncompressed history, and a simple keyframe retrieval baseline. These do not depend on any released code, and they would show whether the gains come from the memory mechanism itself.
[1] Boyuan Chen et al. Diffusion Forcing: Next-Token Prediction Meets Full-Sequence Diffusion. NeurIPS 2024.
[2] Kiwhan Song et al. History-guided Video Diffusion. arXiv 2025.
Questions for authors
- When the history grows to several minutes, how do GPU memory, speed and memory performance change?
8 comments
Response to Reviewer xrfv (Part 2/2): Controlled Ablations and History Scaling (W3, Q1)
Following the methodological distinctions in Part 1/2, we provide the controlled comparisons and scaling measurements.
W3. Missing controlled comparisons of memory mechanisms under the same backbone and training data.
Thank you for suggesting controlled memory ablations. We compare no-memory, sliding-window memory, keyframe retrieval, FramePack-style hierarchical compression, and ReCam on the subject-reappearance benchmark. ReCam achieves the best overall character and clothing consistency, with face distance comparable to keyframe retrieval, without an external retrieval model.
The memory-enabled variants use the same Wan2.2 backbone and training data, at 368p and 16 fps, with 10k training steps and a global batch size of 16. All receive the original video’s first frame as an additional anchor and use the same dual-pointer denoising scheme, differing only in how they represent and access historical context. The configurations are as follows:
- No-memory: We use the trained ReCam model to generate from the continuation’s starting frame, with historical memory disabled.
- Sliding window: We retain the most recent 2 seconds of historical latents using the native patch embedding.
- Keyframe retrieval: During training, we uniformly sample one historical latent frame. At inference, we compute the CLIP similarity between the current prompt and all historical frames, select the top-ranked frame, and use its corresponding latent as the memory input.
- FramePack-style hierarchical compression: Following the hierarchical packing strategy in FramePack, we divide historical latents into four temporal tiers. From newest to oldest, these tiers contain 1, 2, 16, and 32 latent frames, with compression factors , respectively. We cap the compression factor at , as further compression at our evaluation resolution would collapse the spatial token grid.
- ReCam: We retain the full history for the generator.
Note that using the full history without compression causes computation and GPU memory requirements to grow rapidly with history length, making this baseline infeasible with available resources.
Each cell lists All / Present / Absent, with the best displayed value for each metric and group shown in bold, including ties.
| METHOD | FACE ↓ | CHARACTER ↓ | CLOTHES ↑ |
|---|---|---|---|
| No-memory | 0.77 / 0.78 / 0.77 | 8.56 / 8.14 / 8.86 | 1.31 / 1.67 / 1.04 |
| Sliding window | 0.66 / 0.57 / 0.78 | 7.22 / 5.43 / 8.45 | 2.50 / 3.83 / 1.58 |
| Keyframe retrieval | 0.65 / 0.57 / 0.74 | 6.91 / 6.53 / 7.18 | 3.00 / 4.38 / 2.04 |
| FramePack-style | 0.68 / 0.59 / 0.80 | 7.30 / 6.77 / 7.66 | 2.66 / 4.33 / 1.50 |
| ReCam | 0.65 / 0.57 / 0.74 | 5.56 / 5.67 / 5.48 | 3.45 / 4.39 / 2.80 |
The gains are largest in the initially absent split, where identity information must be recovered from intermediate history. Compared with keyframe retrieval, ReCam reduces character distance from 7.18 to 5.48 and improves clothing consistency from 2.04 to 2.80. We also observe that keyframe retrieval sometimes produces frames closely resembling the selected reference, with unnatural motion. A single retrieved frame provides appearance cues but omits the temporal dynamics available in ReCam’s full-history representation.
Q1. When the history grows to several minutes, how do GPU memory, speed and memory performance change?
We extend the efficiency measurements in Fig. 4(a) to two minutes of history, using the same profiling protocol on a single H100 80GB GPU at 448×832 resolution with BF16 and FlashAttention-3.
| History length | Peak GPU memory (GiB) | Time per forward (s) |
|---|---|---|
| 0 s | 31.63 | 1.320 |
| 5 s | 33.45 | 1.503 |
| 10 s | 33.76 | 1.579 |
| 20 s | 34.38 | 1.760 |
| 30 s | 34.99 | 1.898 |
| 45 s | 35.92 | 2.192 |
| 60 s | 36.85 | 2.449 |
| 90 s | 38.70 | 3.065 |
| 120 s | 40.56 | 3.726 |
Doubling the history from one to two minutes increases peak GPU memory by 10.1% and per-forward latency by 52.2%, while keeping both costs within a practical range.
Regarding memory performance, the model exhibits comparable performance for history lengths of 30–45 seconds, with degradation appearing at longer lengths. We attribute this degradation to the distribution shift beyond the maximum training history of 30 seconds. Training on longer videos is expected to help mitigate this issue. Generalization to out-of-distribution memory settings is also acknowledged as a limitation in our paper (Lines 484–486).
Could you report the history coverage, memory-token counts, and computational costs of each variant? The keyframe baseline also uses random sampling during training but CLIP retrieval at inference; could you clarify whether this affects the comparison?
More importantly, could you report subject-reappearance performance at longer history lengths, particularly when the subject has been absent for over a minute? The current results show that two-minute histories are computationally feasible, but do not yet establish effective memory at that length. Please also provide absolute GPU memory usage and end-to-end generation time.
Response to Reviewer xrfv (Part 1/2): Complex Actions and Methodological Distinctions (W1–W2)
Thank you for your positive assessment of our architecture and experimental analysis, and for your constructive feedback.
Thank you for suggesting more challenging demonstrations. The original supplementary videos come from benchmarks designed primarily to assess visual quality and memory, which typically use simple prompts, such as “a panda bear is eating a piece of bamboo” from VBench. Their simple activity structure reflects these evaluation prompts rather than demonstrating an inability to follow complex action instructions.
We evaluate ReCam against strong baselines with comparable or larger parameter counts. These comparisons provide quantitative evidence for the effectiveness of our approach. For more qualitative results, please see the newly added demonstrations in the supplementary material for more complex action sequences and subject reappearance after extended absences.
We appreciate the opportunity to clarify the two contributions and their differences from these prior methods.
1. Learning to use the full history. Our contribution is to identify and demonstrate the importance of providing the full, continuous history to the video generator, rather than the flattening and MLP operations themselves. Reducing token count alone does not determine what historical information remains available to the generator. While FramePack allocates finer representations to recent history, ReCam applies the same block representation across the full history, letting the generator learn which historical information to use. This preserves access to both historical content and its evolution over time, while high-dimensional block representations make reading the full history computationally practical. FramePack’s increasing compression also limits its usable history length: the official F1 implementation retains 19 historical latent frames, corresponding to approximately 4.75 seconds at 16 fps. Our controlled comparison in W3 (Part 2/2) directly tests this distinction: with the same backbone, training data, and training budget, ReCam achieves better overall character and clothing consistency than FramePack-style compression.
2. Dual-pointer denoising. We clarify that the contribution of dual-pointer denoising is to ease the optimization difficulty introduced by adding noise to historical context. The key design is to provide two views of the same historical context with shared noise but different noise levels, rather than simply assign different noise levels to historical context and current content.
This distinction matters because context noising weakens the appearance and motion cues available for predicting subsequent content, increasing prediction uncertainty and task difficulty. Dual-pointer denoising provides additional information through the relationship between its two views. For context O and shared noise ϵ∼N(0,I), we construct
\qquad \tilde O_{\mathrm m}=(1-\sigma_{\mathrm m})O+\sigma_{\mathrm m}\epsilon.$$ At unequal noise levels, their normalized difference directly exposes the context velocity: $$\frac{\tilde O_{\mathrm m}-\tilde O_{\mathrm c}} {\sigma_{\mathrm m}-\sigma_{\mathrm c}} =\epsilon-O =v_{\mathrm{ctx}}^{*}.$$ This relation provides **an input-level shortcut aligned with the velocity-prediction objective**, giving the generator additional information for interpreting noisy context and predicting subsequent content. **Our single-pointer ablation already assigns different noise levels to historical context and current content.** Compared with this setting, dual-pointer denoising achieves lower training loss (Fig. 4(b)) and better results on all four temporal-drift metrics (Table 4(b)). Its final training loss approaches that of clean-context training, supporting the optimization benefit, while the rollout visualization shows less accumulated color drift than clean context (Fig. 5). Part 2/2 presents the controlled memory ablations (W3) and the history-length scaling results (Q1).