- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Attn-Forcing, a framework for preserving historical attention contributions in streaming video generation under bounded memory and computational budgets. Existing autoregressive video diffusion models typically employ fixed-length KV caches, which discard distant historical information and may degrade long-horizon generation quality. The authors identify a limitation of existing KV selection and compression methods: the lack of explicit supervision for preserving full-context attention contributions.
To address this limitation, the authors introduce learnable Attention Proxies that continuously incorporate evicted historical visual latents through a cross-attention-based update module. In addition, they propose an Attention Matching Loss that leverages random feature approximations and online accumulated statistics to construct approximate full-context attention responses as supervision targets with bounded storage requirements, without explicitly retaining the entire historical KV cache. The framework is jointly trained with Distribution Matching Distillation (DMD).
The authors evaluate Attn-Forcing on 5s, 30s, and 60s video generation tasks. The results show that Attn-Forcing outperforms Self-Forcing, Causal-Forcing, and LongLive on several aggregate quality metrics. Ablation studies indicate that Attention Matching Loss contributes substantially to the improvements in generation quality, particularly in motion dynamics, while introducing approximately 13% additional inference time compared with LongLive.
Critical strengths
1. Explicit supervision for preserving historical attention contributions. Beyond compressing historical visual information into a fixed number of memory tokens, the paper introduces an Attention Matching Loss that explicitly encourages learnable proxies to approximate the attention responses induced by evicted historical KV states. Compared with approaches that optimize memory representations indirectly through generation objectives, this provides a more direct supervision signal for preserving historical attention contributions.
2. Rich empirical results across multiple video durations. The method achieves meaningful improvements on video generation tasks of different durations, including 5s, 30s, and 60s videos. These results provide rich evidence for the effectiveness of the proposed framework.
Critical weaknesses
1. The methodological novelty remains debatable, and the proposed architecture is not sufficiently compared with existing memory architectures. The learnable Attention Proxies and their cross-attention-based gated update mechanism are conceptually similar to existing evolving memory approaches, such as Echo-Infinity. Therefore, the main contribution appears to lie in the Attention Matching Loss rather than in a fundamentally new memory architecture. Although Table 3 demonstrates that introducing Attention Matching Loss improves performance compared with the proxy-only model, it does not establish that matching historical attention is more effective than other memory supervision objectives. Alternatives such as Feature Matching and Latent Reconstruction could also provide additional supervision signals for the Proxy Module. The authors should compare these objectives using the same proxy architecture and training data to determine whether the improvements arise specifically from Attention Response Matching or merely from introducing auxiliary supervision.
2. The current experiments do not establish that a fixed number of proxies can reliably approximate increasingly long and complex historical contexts. The method uses only three Proxy Latents to represent the attention contributions of all evicted historical states. Figure 4 shows improved cosine similarity between Proxy Attention and exact historical Attention over a 60s generation horizon, suggesting that the mechanism is effective under the evaluated conditions. However, the paper does not include controlled comparisons with different numbers of proxies or independently investigate the effects of historical content complexity. Moreover, although the paper emphasizes long-horizon video generation under bounded memory, the main experiments are limited to videos of up to 60 seconds. Rather than simply extending the existing video quality evaluations to longer durations, a more informative experiment would jointly examine proxy capacity, sequence length, and the number of distinct historical events. The current results do not establish how the approximation quality changes when the amount of relevant historical information exceeds the representational capacity of the fixed number of proxies.
3. The sensitivity to the random feature dimension raises questions about the interpretation and robustness of the reported quality improvements. Table 3 shows that reducing the random feature dimension from 128 to 64 decreases the 60s total score from 82.37 to 79.50, which is even lower than the 80.16 achieved by the proxy-only model without Attention Matching Loss. The Dynamic Degree score also drops substantially, from 72.14 to 47.08. This observation does not necessarily indicate that the degradation is caused by random feature approximation errors, since changes in the supervision target may also affect the optimization process. However, the paper does not distinguish between these possible explanations. Furthermore, although Attn-Forcing achieves the highest overall score on 60s video generation in Table 2, it underperforms LongLive in Subject Consistency, Background Consistency, and Motion Smoothness. Its Aesthetic Score also improves only marginally, from 55.99 for LongLive to 56.03. Without uncertainty estimates or confidence intervals, these small differences are insufficient to establish consistent improvements across all quality dimensions.
4. The baseline comparisons are potentially confounded by factors that have not been adequately controlled. In Table 4, the authors increase the attention window of LongLive from 12 to 120 only during inference, resulting in a decrease in the total score from 79.26 to 75.01. The authors themselves acknowledge that this degradation may result from a mismatch between the context lengths used during training and inference. Therefore, this experiment does not establish that Attn-Forcing outperforms a baseline appropriately trained for longer attention contexts. Furthermore, while the paper provides a detailed description of its own long-streaming tuning procedure, it does not report the long-horizon training budgets, model checkpoints, and inference configurations of all competing methods with comparable detail. These differences may influence the reported performance gains and should be clarified to ensure a fair comparison.
Questions for authors
1. What are the main differences between the proposed method and existing memory architectures such as Echo-Infinity? Have the authors compared Attention Matching Loss with simpler supervision objectives, such as Feature Matching?
2. Why are only three Proxy Latents used? How does the method perform with different numbers of proxies, longer videos, or more complex historical content?
3. Why does reducing the random feature dimension from 128 to 64 lead to a substantial performance degradation in Table 3? Is this primarily caused by approximation errors or optimization difficulties?
4. Why does the baseline in Table 3 outperform LongLive in Table 2? Are their training and inference configurations comparable?
4 comments
W2 & Q2: Number of Proxy Latents.
We choose K=3 proxy latents as an empirical setting to balance historical representation capacity with bounded memory and computation. We have already discussed the limitations of fixed proxy capacity in Appendix A.5 (Limitations). Specifically, while a fixed number of proxies keeps the additional memory and computational overhead bounded, it also limits the capacity to represent accumulated history. As the generation horizon grows longer or the visual content becomes more diverse, a limited number of proxies may become insufficient to preserve all relevant historical information. To further evaluate the effectiveness of three proxy latents over longer generation horizons, we compare Attn-Forcing with the baselines on 120s video generation, as shown in the table below:
| Model | Total | Subject | Background | Smoothness | Dynamic | Aesthetic | Imaging Quality |
|---|---|---|---|---|---|---|---|
| Self-Forcing | 76.33 | 96.56 | 96.19 | 98.57 | 29.19 | 50.08 | 63.83 |
| Causal-Forcing | 76.82 | 94.97 | 95.85 | 96.76 | 52.36 | 45.95 | 62.80 |
| LongLive | 79.66 | 97.59 | 96.37 | 98.76 | 42.49 | 55.83 | 68.33 |
| Attn-Forcing | 82.57 | 96.68 | 95.60 | 98.51 | 78.54 | 54.91 | 69.18 |
Attn-Forcing consistently achieves better overall generation quality than the baselines, demonstrating its effectiveness beyond the 60s training duration with only three proxy latents.
For more complex historical content, we design prompts describing a static environment and a subject that leaves the frame and returns after a long interval beyond the local attention window, requiring the model to maintain environmental consistency and recall the previously observed subject. As shown in Figure R3, LongLive fails to generate a clear subject exit, with the panda abruptly changing its orientation, multiple penguins appearing, and inconsistent subject behavior during reappearance. In contrast, Attn-Forcing better follows the intended exit-and-return sequence while maintaining a consistent environment and avoiding subject duplication. These results suggest that Attn-Forcing can better retain historical information in more complex long-horizon scenarios, even with only three proxy latents.
The primary focus of this work is to introduce and validate the effectiveness of historical attention approximation through attention proxies and Attention Matching Loss. In future work, we will systematically investigate how proxy capacity, sequence length, and historical complexity affect historical attention approximation quality.
W3 & Q3: Random Feature Dimension & Performance in Table 2.
Random Feature Dimension. Reducing the random feature dimension increases the expected softmax-kernel approximation error, leading to less accurate supervision targets for the Attention Matching Loss. Since these supervision targets directly guide proxy optimization, such errors can further affect the training process. Therefore, when the random feature dimension is small, inaccurate attention supervision may offset or even outweigh the benefits of the Attention Matching Loss, resulting in performance below the proxy-only baseline.
Performance in Table 2. As discussed in Section 5 (``Trade-off across Ability Dimensions'') of VBench[2], there is a practical trade-off between Dynamic Degree and temporal consistency (Subject/Background Consistency and Smoothness), since relatively static scenes can “cheat” to get high temporal consistency scores. However, Attn-Forcing achieves substantially higher motion dynamics (35.55 to 72.14 compared with LongLive) while maintaining comparable temporal consistency (second-best Subject Consistency and Motion Smoothness, and third-best Background Consistency, only slightly behind the best results). It also achieves the highest Aesthetic and Imaging Quality scores, resulting in the highest total score of 82.37 in Table 2. The qualitative comparisons in Figure 3 and Figure 6 further demonstrate the stronger motion dynamics and generation quality of Attn-Forcing compared with the baselines over 60s video generation.
[2] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR, 2024.
W4 & Q4: Baselines Trained with Longer Attention Context & Implementation Details.
Baselines Trained with Longer Attention Context. Training with substantially longer attention contexts requires significantly higher GPU memory and computational costs. To our knowledge, there are no publicly available checkpoints fully trained with longer attention windows in streaming generation. For example, Self-Forcing and Causal-Forcing are trained with an attention context length of 21, while LongLive and Echo-Infinity use a local attention window size of 12. Therefore, we propose Attn-Forcing to approximate historical attention contributions under bounded memory and computation, without explicitly maintaining increasingly long KV contexts. Under these circumstances, we are limited to extending LongLive's attention window from 12 to 120 at inference time in Table 4 to provide an approximate comparison with longer attention contexts.
Implementation Details. For the main comparison in Table 1 and Table 2, we evaluate all baselines using their officially released checkpoints and corresponding inference configurations, including Self-Forcing, Causal-Forcing, and LongLive. We further summarize their main training configurations and inference settings, together with those of Attn-Forcing, in the table below:
| Method | Training GPUs / Budget | Batch Size | Short DMD Tuning Steps | Long DMD Tuning Steps | Training Context | Inference Steps |
|---|---|---|---|---|---|---|
| Self-Forcing | 64 × H100 | 64 | 600 | N/A | 21 | 4 |
| Causal-Forcing | 64 GPUs | 64 | 750 | N/A | 21 | 4 |
| LongLive | 32 H100 GPU-days | 64 | 700 | 2,300 | 12 | 4 |
| Attn-Forcing (Ours) | 16 × H20 | 32 | 700 | 2,300 | 12 | 4 |
We train the baseline in Table 3 following the same training procedure as LongLive but with some different hardware and training configurations. Specifically, we use 16 H20 GPUs with a batch size of 32, while LongLive reports a training budget of 32 H100 GPU-days with a batch size of 64. Differences in hardware, runtime environments, and batch sizes may lead to minor variations in the final results compared with the publicly released LongLive checkpoint.
Seeking Further Feedback
Thank you again for your thoughtful review and constructive feedback. As the discussion phase is coming to a close, we would greatly appreciate any further thoughts on our rebuttal. If any concerns remain or additional clarification would be helpful, please feel free to let us know, and we would be happy to address them. If your main concerns have been sufficiently addressed, we would sincerely appreciate your consideration of updating your assessment accordingly.
Thank you again for your time and consideration.
W1 & Q1: Difference between Attn-Forcing and Memory Architectures.
We compare Attn-Forcing with Echo-Infinity [1] as follows. Echo-Infinity employs layer-shared evolving Memory Queries that compress historical KV states into a shared memory representation and provide them to different DiT layers. However, such a layer-shared memory representation may not naturally align with the different feature spaces and KV distributions across DiT layers.
In contrast, Attn-Forcing aggregates historical information into proxy latents in the visual latent space, maintaining native visual representations. The proxy latents are then propagated through the standard DiT forward pass to produce native KV states for each layer. More importantly, Attn-Forcing introduces an Attention Matching Loss to directly align the attention responses obtained with the proxy latents to those of the historical attention. Thus, the core contribution of Attn-Forcing lies in approximating historical attention through explicit attention-level supervision to improve long-horizon video generation under bounded memory and computation, rather than simply introducing a new memory architecture. We also present the quantitative comparison between Attn-Forcing and Echo-Infinity on 60s and 120s videos generated from MovieGen prompts in the following table:
Attn-Forcing achieves higher total scores than Echo-Infinity on both 60s (82.37 vs. 80.89) and 120s (82.57 vs. 80.85) videos.
Furthermore, we compare the cosine similarity of the proxy attention responses from Attn-Forcing and the memory attention responses from Echo-Infinity against the exact historical attention responses. As shown in Figure R4, Attn-Forcing achieves substantially higher cosine similarity throughout long-horizon generation, indicating a more accurate approximation of historical attention responses.
Other memory supervision objectives, such as Feature Matching and Latent Reconstruction, do not directly provide naturally aligned historical supervision targets for the proxies and may introduce mismatches with historical information. For example, Feature Matching relies on mean/EMA/pooling to aggregate historical features, while Latent Reconstruction requires an additional mapping from proxies to the growing historical sequence. In contrast, Attention Matching Loss utilizes historical attention responses to current queries as more natural supervision targets.
To verify this, we replace the Attention Matching Loss with Feature Matching (using MSE to match the proxy Value features to the mean historical Value features at corresponding temporal slots and spatial positions) under the same training settings and report the qualitative comparison in Figure R5. Feature Matching introduces increasingly severe visual artifacts as generation progresses, resulting in the loss of scene structure and subject appearance. In contrast, Attention Matching better preserves visual quality and scene consistency throughout long-horizon generation, suggesting that the improvements arise from Attention Response Matching rather than merely introducing auxiliary supervision.
[1] Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation, arxiv, 2026.