acceptodds
Under review as a conference paper at ICLR 2027

AttnForcing: Learning Full-Context Attention Proxies for Streaming Video Generation

Abstract

Increasing the attention context length significantly improves streaming video generation quality, while incurring prohibitive memory and computational costs as the context grows. Existing methods attempt to capture this contextual information through attention selection or compression, but remain limited by bounded representations and lack explicit attention-level supervision to retain historical contributions to current queries. To address this, we introduce Attn-Forcing, a novel framework that preserves full-context attention contributions for streaming video generation under bounded memory and computation. We achieve this by introducing attention proxies that continuously approximate the attention responses induced by the historical context. Specifically, we implement the proxies as learnable visual latents and employ a Proxy Update Module to incorporate visual context from historical latents. Moreover, we propose an Attention Matching Loss to explicitly constrain the attention behavior of the learned proxies using full-context attention supervision. Extensive qualitative and quantitative experiments demonstrate that Attn-Forcing consistently improves streaming video generation quality and achieves state-of-the-art performance over existing baseline methods. Code and models will be publicly available.

Primary area:
generative models
Keywords:
diffusion model, streaming video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A6oDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Attn-Forcing, a framework for preserving historical attention contributions in streaming video generation under bounded memory and computational budgets. Existing autoregressive video diffusion models typically employ fixed-length KV caches, which discard distant historical information and may degrade long-horizon generation quality. The authors identify a limitation of existing KV selection and compression methods: the lack of explicit supervision for preserving full-context attention contributions.

To address this limitation, the authors introduce learnable Attention Proxies that continuously incorporate evicted historical visual latents through a cross-attention-based update module. In addition, they propose an Attention Matching Loss that leverages random feature approximations and online accumulated statistics to construct approximate full-context attention responses as supervision targets with bounded storage requirements, without explicitly retaining the entire historical KV cache. The framework is jointly trained with Distribution Matching Distillation (DMD).

The authors evaluate Attn-Forcing on 5s, 30s, and 60s video generation tasks. The results show that Attn-Forcing outperforms Self-Forcing, Causal-Forcing, and LongLive on several aggregate quality metrics. Ablation studies indicate that Attention Matching Loss contributes substantially to the improvements in generation quality, particularly in motion dynamics, while introducing approximately 13% additional inference time compared with LongLive.

Critical strengths

1. Explicit supervision for preserving historical attention contributions. Beyond compressing historical visual information into a fixed number of memory tokens, the paper introduces an Attention Matching Loss that explicitly encourages learnable proxies to approximate the attention responses induced by evicted historical KV states. Compared with approaches that optimize memory representations indirectly through generation objectives, this provides a more direct supervision signal for preserving historical attention contributions.

2. Rich empirical results across multiple video durations. The method achieves meaningful improvements on video generation tasks of different durations, including 5s, 30s, and 60s videos. These results provide rich evidence for the effectiveness of the proposed framework.

Critical weaknesses

1. The methodological novelty remains debatable, and the proposed architecture is not sufficiently compared with existing memory architectures. The learnable Attention Proxies and their cross-attention-based gated update mechanism are conceptually similar to existing evolving memory approaches, such as Echo-Infinity. Therefore, the main contribution appears to lie in the Attention Matching Loss rather than in a fundamentally new memory architecture. Although Table 3 demonstrates that introducing Attention Matching Loss improves performance compared with the proxy-only model, it does not establish that matching historical attention is more effective than other memory supervision objectives. Alternatives such as Feature Matching and Latent Reconstruction could also provide additional supervision signals for the Proxy Module. The authors should compare these objectives using the same proxy architecture and training data to determine whether the improvements arise specifically from Attention Response Matching or merely from introducing auxiliary supervision.

2. The current experiments do not establish that a fixed number of proxies can reliably approximate increasingly long and complex historical contexts. The method uses only three Proxy Latents to represent the attention contributions of all evicted historical states. Figure 4 shows improved cosine similarity between Proxy Attention and exact historical Attention over a 60s generation horizon, suggesting that the mechanism is effective under the evaluated conditions. However, the paper does not include controlled comparisons with different numbers of proxies or independently investigate the effects of historical content complexity. Moreover, although the paper emphasizes long-horizon video generation under bounded memory, the main experiments are limited to videos of up to 60 seconds. Rather than simply extending the existing video quality evaluations to longer durations, a more informative experiment would jointly examine proxy capacity, sequence length, and the number of distinct historical events. The current results do not establish how the approximation quality changes when the amount of relevant historical information exceeds the representational capacity of the fixed number of proxies.

3. The sensitivity to the random feature dimension raises questions about the interpretation and robustness of the reported quality improvements. Table 3 shows that reducing the random feature dimension from 128 to 64 decreases the 60s total score from 82.37 to 79.50, which is even lower than the 80.16 achieved by the proxy-only model without Attention Matching Loss. The Dynamic Degree score also drops substantially, from 72.14 to 47.08. This observation does not necessarily indicate that the degradation is caused by random feature approximation errors, since changes in the supervision target may also affect the optimization process. However, the paper does not distinguish between these possible explanations. Furthermore, although Attn-Forcing achieves the highest overall score on 60s video generation in Table 2, it underperforms LongLive in Subject Consistency, Background Consistency, and Motion Smoothness. Its Aesthetic Score also improves only marginally, from 55.99 for LongLive to 56.03. Without uncertainty estimates or confidence intervals, these small differences are insufficient to establish consistent improvements across all quality dimensions.

4. The baseline comparisons are potentially confounded by factors that have not been adequately controlled. In Table 4, the authors increase the attention window of LongLive from 12 to 120 only during inference, resulting in a decrease in the total score from 79.26 to 75.01. The authors themselves acknowledge that this degradation may result from a mismatch between the context lengths used during training and inference. Therefore, this experiment does not establish that Attn-Forcing outperforms a baseline appropriately trained for longer attention contexts. Furthermore, while the paper provides a detailed description of its own long-streaming tuning procedure, it does not report the long-horizon training budgets, model checkpoints, and inference configurations of all competing methods with comparable detail. These differences may influence the reported performance gains and should be clarified to ensure a fair comparison.

Questions for authors

1. What are the main differences between the proposed method and existing memory architectures such as Echo-Infinity? Have the authors compared Attention Matching Loss with simpler supervision objectives, such as Feature Matching?

2. Why are only three Proxy Latents used? How does the method perform with different numbers of proxies, longer videos, or more complex historical content?

3. Why does reducing the random feature dimension from 128 to 64 lead to a substantial performance degradation in Table 3? Is this primarily caused by approximation errors or optimization difficulties?

4. Why does the baseline in Table 3 outperform LongLive in Table 2? Are their training and inference configurations comparable?

4 comments

W1 & Q1: Difference between Attn-Forcing and Memory Architectures.

We compare Attn-Forcing with Echo-Infinity [1] as follows. Echo-Infinity employs layer-shared evolving Memory Queries that compress historical KV states into a shared memory representation and provide them to different DiT layers. However, such a layer-shared memory representation may not naturally align with the different feature spaces and KV distributions across DiT layers.

In contrast, Attn-Forcing aggregates historical information into proxy latents in the visual latent space, maintaining native visual representations. The proxy latents are then propagated through the standard DiT forward pass to produce native KV states for each layer. More importantly, Attn-Forcing introduces an Attention Matching Loss to directly align the attention responses obtained with the proxy latents to those of the historical attention. Thus, the core contribution of Attn-Forcing lies in approximating historical attention through explicit attention-level supervision to improve long-horizon video generation under bounded memory and computation, rather than simply introducing a new memory architecture. We also present the quantitative comparison between Attn-Forcing and Echo-Infinity on 60s and 120s videos generated from MovieGen prompts in the following table:

ModelTotalSubjectBackgroundSmoothnessDynamicAestheticImaging Quality
60s
Echo-Infinity80.8997.6196.1998.4453.4955.2170.68
Attn-Forcing82.3797.1195.8498.5672.1456.0369.42
120s
Echo-Infinity80.8597.4196.0498.4355.9654.1070.72
Attn-Forcing82.5796.6895.6098.5178.5454.9169.18

Attn-Forcing achieves higher total scores than Echo-Infinity on both 60s (82.37 vs. 80.89) and 120s (82.57 vs. 80.85) videos.

Furthermore, we compare the cosine similarity of the proxy attention responses from Attn-Forcing and the memory attention responses from Echo-Infinity against the exact historical attention responses. As shown in Figure R4, Attn-Forcing achieves substantially higher cosine similarity throughout long-horizon generation, indicating a more accurate approximation of historical attention responses.

Other memory supervision objectives, such as Feature Matching and Latent Reconstruction, do not directly provide naturally aligned historical supervision targets for the proxies and may introduce mismatches with historical information. For example, Feature Matching relies on mean/EMA/pooling to aggregate historical features, while Latent Reconstruction requires an additional mapping from proxies to the growing historical sequence. In contrast, Attention Matching Loss utilizes historical attention responses to current queries as more natural supervision targets.

To verify this, we replace the Attention Matching Loss with Feature Matching (using MSE to match the proxy Value features to the mean historical Value features at corresponding temporal slots and spatial positions) under the same training settings and report the qualitative comparison in Figure R5. Feature Matching introduces increasingly severe visual artifacts as generation progresses, resulting in the loss of scene structure and subject appearance. In contrast, Attention Matching better preserves visual quality and scene consistency throughout long-horizon generation, suggesting that the improvements arise from Attention Response Matching rather than merely introducing auxiliary supervision.

[1] Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation, arxiv, 2026.

W2 & Q2: Number of Proxy Latents.

We choose K=3 proxy latents as an empirical setting to balance historical representation capacity with bounded memory and computation. We have already discussed the limitations of fixed proxy capacity in Appendix A.5 (Limitations). Specifically, while a fixed number of proxies keeps the additional memory and computational overhead bounded, it also limits the capacity to represent accumulated history. As the generation horizon grows longer or the visual content becomes more diverse, a limited number of proxies may become insufficient to preserve all relevant historical information. To further evaluate the effectiveness of three proxy latents over longer generation horizons, we compare Attn-Forcing with the baselines on 120s video generation, as shown in the table below:

ModelTotalSubjectBackgroundSmoothnessDynamicAestheticImaging Quality
Self-Forcing76.3396.5696.1998.5729.1950.0863.83
Causal-Forcing76.8294.9795.8596.7652.3645.9562.80
LongLive79.6697.5996.3798.7642.4955.8368.33
Attn-Forcing82.5796.6895.6098.5178.5454.9169.18

Attn-Forcing consistently achieves better overall generation quality than the baselines, demonstrating its effectiveness beyond the 60s training duration with only three proxy latents.

For more complex historical content, we design prompts describing a static environment and a subject that leaves the frame and returns after a long interval beyond the local attention window, requiring the model to maintain environmental consistency and recall the previously observed subject. As shown in Figure R3, LongLive fails to generate a clear subject exit, with the panda abruptly changing its orientation, multiple penguins appearing, and inconsistent subject behavior during reappearance. In contrast, Attn-Forcing better follows the intended exit-and-return sequence while maintaining a consistent environment and avoiding subject duplication. These results suggest that Attn-Forcing can better retain historical information in more complex long-horizon scenarios, even with only three proxy latents.

The primary focus of this work is to introduce and validate the effectiveness of historical attention approximation through attention proxies and Attention Matching Loss. In future work, we will systematically investigate how proxy capacity, sequence length, and historical complexity affect historical attention approximation quality.

W3 & Q3: Random Feature Dimension & Performance in Table 2.

Random Feature Dimension. Reducing the random feature dimension increases the expected softmax-kernel approximation error, leading to less accurate supervision targets for the Attention Matching Loss. Since these supervision targets directly guide proxy optimization, such errors can further affect the training process. Therefore, when the random feature dimension is small, inaccurate attention supervision may offset or even outweigh the benefits of the Attention Matching Loss, resulting in performance below the proxy-only baseline.

Performance in Table 2. As discussed in Section 5 (``Trade-off across Ability Dimensions'') of VBench[2], there is a practical trade-off between Dynamic Degree and temporal consistency (Subject/Background Consistency and Smoothness), since relatively static scenes can “cheat” to get high temporal consistency scores. However, Attn-Forcing achieves substantially higher motion dynamics (35.55 to 72.14 compared with LongLive) while maintaining comparable temporal consistency (second-best Subject Consistency and Motion Smoothness, and third-best Background Consistency, only slightly behind the best results). It also achieves the highest Aesthetic and Imaging Quality scores, resulting in the highest total score of 82.37 in Table 2. The qualitative comparisons in Figure 3 and Figure 6 further demonstrate the stronger motion dynamics and generation quality of Attn-Forcing compared with the baselines over 60s video generation.

[2] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR, 2024.

W4 & Q4: Baselines Trained with Longer Attention Context & Implementation Details.

Baselines Trained with Longer Attention Context. Training with substantially longer attention contexts requires significantly higher GPU memory and computational costs. To our knowledge, there are no publicly available checkpoints fully trained with longer attention windows in streaming generation. For example, Self-Forcing and Causal-Forcing are trained with an attention context length of 21, while LongLive and Echo-Infinity use a local attention window size of 12. Therefore, we propose Attn-Forcing to approximate historical attention contributions under bounded memory and computation, without explicitly maintaining increasingly long KV contexts. Under these circumstances, we are limited to extending LongLive's attention window from 12 to 120 at inference time in Table 4 to provide an approximate comparison with longer attention contexts.

Implementation Details. For the main comparison in Table 1 and Table 2, we evaluate all baselines using their officially released checkpoints and corresponding inference configurations, including Self-Forcing, Causal-Forcing, and LongLive. We further summarize their main training configurations and inference settings, together with those of Attn-Forcing, in the table below:

MethodTraining GPUs / BudgetBatch SizeShort DMD Tuning StepsLong DMD Tuning StepsTraining ContextInference Steps
Self-Forcing64 × H10064600N/A214
Causal-Forcing64 GPUs64750N/A214
LongLive32 H100 GPU-days647002,300124
Attn-Forcing (Ours)16 × H20327002,300124

We train the baseline in Table 3 following the same training procedure as LongLive but with some different hardware and training configurations. Specifically, we use 16 H20 GPUs with a batch size of 32, while LongLive reports a training budget of 32 H100 GPU-days with a batch size of 64. Differences in hardware, runtime environments, and batch sizes may lead to minor variations in the final results compared with the publicly released LongLive checkpoint.

AuthorsOpenReview ↗

Seeking Further Feedback

Thank you again for your thoughtful review and constructive feedback. As the discussion phase is coming to a close, we would greatly appreciate any further thoughts on our rebuttal. If any concerns remain or additional clarification would be helpful, please feel free to let us know, and we would be happy to address them. If your main concerns have been sufficiently addressed, we would sincerely appreciate your consideration of updating your assessment accordingly.

Thank you again for your time and consideration.

Reviewer_8DP2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses how to preserve historical context in streaming video diffusion while keeping the KV cache size fixed. The proposed method, Attn-Forcing, uses three learnable visual proxy latents that are updated as older visual latents are evicted from the cache. These proxies are then passed through the DiT to produce replacement KV states. To train the proxies, the authors introduce an attention-matching objective that uses accumulated random-feature statistics to approximate historical attention responses. These statistics have a fixed size and are only used during DMD-based long tuning, adding no overhead at inference time. The method is evaluated on Wan2.1-T2V-1.3B using VBench/VBench-Long for 5- and 30-second videos, as well as 128 MovieGen prompts for 60-second videos. Compared with LongLive, the proposed method improves the score from 83.15 to 83.85 at 30 seconds and from 79.26 to 82.37 at 60 seconds.

Critical strengths
  1. Equations (8)–(16) show how historical attention can be approximated using fixed-size random-feature statistics, avoiding the need to store all past KV states. These statistics are used for training, while standard softmax attention is retained at inference. As clarified in Appendix A.2, neither the statistics nor the matching loss is needed at inference. Section 4.4 reports an inference time increase of around 13%, which seems reasonable given the additional memory mechanism.

  2. Table 3 shows that adding proxies alone barely changes the 60-second score (80.03 to 80.16), while adding the matching loss improves it to 82.37. Figure 4 further shows that the learned proxies better approximate historical attention. These results suggest that the improvement comes mainly from the attention-matching supervision rather than simply increasing memory capacity.

Critical weaknesses
  1. Equation (16) matches historical and proxy attention separately, while Equation (7) uses a joint softmax over sink, proxy, and local keys. The loss does not account for the relative attention weight assigned to history, so matching historical responses does not guarantee preservation of the full attention output. Figure 4 also shows improved similarity rather than exact recovery.

  2. The longest evaluation is 60 seconds, matching the training duration. In Table 3, the matching loss mainly improves the dynamic degree (46.82 to 72.14), with little change in imaging quality. This does not directly show whether the model can recall earlier objects or events after a long absence. Longer evaluations and explicit recall tests would help verify the memory benefit.

Questions for authors
  1. How does the method account for the relative attention weight of historical states when three proxies replace a growing number of KV states? Could the authors report the error in the full attention output (including sink, proxy, and local keys) rather than only the similarity between historical responses?

  2. Do the improvements hold for videos longer than 60 seconds, especially when an earlier object or attribute reappears after a long absence? A comparison with memory-based methods under the same cache budget would help clarify whether the gains come from better historical recall or improved motion dynamics.

4 comments

W1 & Q1: Relative Attention Mass and Full Attention Approximation.

We use the Attention Matching Loss (Eq. 16) to match historical and proxy attention responses separately during training, focusing the explicit supervision on approximating the historical response. Matching the full joint-attention output involves both the historical response and its relative attention mass, making it increasingly challenging with fixed-capacity proxies as historical KV states accumulate. Although the relative attention mass is not explicitly constrained by the Attention Matching Loss, optimizing the proxy attention responses alongside the DMD objective may indirectly influence their attention mass under the joint softmax.

To further evaluate this, we additionally report the relative attention mass and joint-output error under Eq. 7 in Figure R1 and Figure R2. Specifically, as shown in Figure R1, the historical and proxy attention masses are relatively close at the beginning of long-horizon generation. As generation proceeds, the historical attention mass gradually increases as more accumulated historical KV states contribute to the joint softmax. In contrast, the proxy attention mass remains relatively stable. Nevertheless, incorporating attention matching loss LAM consistently increases the proxy attention mass compared with the model without LAM. Furthermore, Figure R2 shows that LAM consistently reduces the joint-output relative error throughout long-horizon generation.

We will also clarify that our method targets historical attention response approximation rather than exact full-context attention preservation.

W2 & Q2: Longer Video Evaluation and Long-range Memory.

First, we present the evaluation of Attn-Forcing and the baselines on 120s videos generated from MovieGen prompts in the following table (the best result is marked in bold and the second-best result is highlighted in quote):

ModelTotalSubjectBackgroundSmoothnessDynamicAestheticImaging Quality
Self-Forcing76.3396.5696.1998.5729.1950.0863.83
Causal-Forcing76.8294.9795.8596.7652.3645.9562.80
LongLive79.6697.5996.3798.7642.4955.8368.33
Attn-Forcing82.5796.6895.6098.5178.5454.9169.18

Attn-Forcing consistently achieves better overall generation quality than the baselines, demonstrating its effectiveness beyond the 60s training duration.

We also conduct an ablation study on 120s videos to further evaluate the effectiveness of the Attention Matching Loss in longer-horizon generation, as shown below (the best result is marked in bold):

ModelTotalSubjectBackgroundSmoothnessDynamicAestheticImaging Quality
baseline + proxies80.4397.4095.9898.1953.7854.6669.27
baseline + proxies + LAM82.5796.6895.6098.5178.5454.9169.18

The results further demonstrate that the Attention Matching Loss remains effective for longer video generation.

Second, we conduct a controlled delayed-recall experiment to investigate whether the model can recall earlier objects or events after a long absence. Specifically, we design prompts describing a static environment and a subject that leaves the frame and returns after a long interval beyond the local attention window, requiring the model to maintain environmental consistency and recall the previously observed subject.

As shown in Figure R3, LongLive fails to generate a clear subject exit, with the panda abruptly changing its orientation, multiple penguins appearing, and inconsistent subject behavior during reappearance. In contrast, Attn-Forcing better follows the intended exit-and-return sequence while maintaining a consistent environment and avoiding subject duplication. These observations provide qualitative evidence of improved long-range information retention beyond motion quality alone.

Finally, we compare Attn-Forcing with the memory-based method Echo-Infinity [1] on 60s and 120s videos under the same cache, as shown below (the best result is marked in bold):

ModelTotalSubjectBackgroundSmoothnessDynamicAestheticImaging Quality
60s
Echo-Infinity80.8997.6196.1998.4453.4955.2170.68
Attn-Forcing82.3797.1195.8498.5672.1456.0369.42
120s
Echo-Infinity80.8597.4196.0498.4355.9654.1070.72
Attn-Forcing82.5796.6895.6098.5178.5454.9169.18

Attn-Forcing achieves higher total scores than Echo-Infinity on both 60s (82.37 vs. 80.89) and 120s (82.57 vs. 80.85) videos.

Furthermore, we compare the cosine similarity of the proxy attention responses from Attn-Forcing and the memory attention responses from Echo-Infinity against the exact historical attention responses. As shown in Figure R4, Attn-Forcing achieves substantially higher cosine similarity throughout long-horizon generation, indicating a more accurate approximation of historical attention responses.

[1] Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation, arxiv, 2026.

Thanks to the authors for their clarifications in the rebuttal. They addressed most of my concerns. I will maintain my original positive score.

We sincerely thank the reviewer for acknowledging that our rebuttal has addressed most of the concerns. We greatly appreciate your constructive feedback and continued support for our work.

Reviewer_5Ak1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces Attn-Forcing, a streaming video generation framework that augments a bounded sink/local KV cache with recurrently updated attention proxies. Three learnable visual latents absorb evicted visual context through a gated update module and are forwarded through the DiT to produce layer-wise proxy KV states. In addition to distribution matching distillation, an Attention Matching Loss supervises proxy-only attention outputs using a positive-random-feature approximation of historical attention. Constant-size online statistics construct this target during training and are not required at inference. The main contribution is a practical attention-supervised memory mechanism, rather than an established guarantee of exact full-context equivalence.

Critical strengths
  1. The method offers a concrete and reasonably well-specified connection between recurrent visual memory and attention-level supervision. Eqs. 12–14 construct history-dependent targets without retaining an ever-growing KV cache, while inference uses standard DiT-derived proxy KV states instead of the random-feature approximation itself. Appendices A.1–A.3 provide the derivation, training/inference ordering, update equations, positional convention, and gradient-flow restrictions.
  2. The experimental evidence supports a useful integrated improvement. Tables 1–2 report higher total scores at all three evaluated durations, with 946 prompts and five seeds per prompt specified for the 5s/30s evaluations. Table 3 provides a streaming-long-tuning baseline: proxies alone increase its total score from 80.03 to 80.16, and attention matching further increases it to 82.37.
Critical weaknesses
  1. Historical-subset attention matching does not guarantee preservation of the complete attention contribution. Eqs. 14–16 match separately normalized history/proxy responses, whereas Eq. 7 jointly normalizes sink, proxy, and local keys. Equal subset outputs can still assign different relative attention mass. Figure 4 does not measure this discrepancy; DMD may compensate indirectly, but the full-context-preservation claim remains insufficiently established.
  2. The results demonstrate improved generation quality more clearly than long-range information retention. At 60s, dynamic degree increases from 35.55 to 72.14 over LongLive, while subject/background consistency slightly decreases (Table 2). These motion gains are valuable, but do not isolate retention of evicted history.
Questions for authors
  1. How is history-to-local attention mass calibrated when proxies replace historical KV states? Beyond Figure 4’s subset-response cosine similarity, can you report relative attention mass and joint-output error under Eq. 7? If calibration is not enforced, please qualify the full-context-preservation claim.
  2. Can a controlled delayed-recall or history-perturbation test in the existing 60s setting distinguish retention of evicted information from improved motion? Figure 4 already provides attention-level evidence; this question concerns whether that improvement translates into long-range memory.
3 comments

W1 & Q1: Historical-subset Matching

We chose historical-subset attention matching during training to focus the explicit supervision on approximating the historical response. Matching the full joint-attention output involves both the historical response and its relative attention mass, making it increasingly challenging with fixed-capacity proxies as historical KV states accumulate. Although the relative attention mass is not explicitly constrained by the Attention Matching Loss, optimizing the proxy attention responses alongside the DMD objective may indirectly influence their attention mass under the joint softmax.

To further evaluate this, we additionally report the relative attention mass (the sum of attention weights assigned to the historical or proxy KV states under the joint softmax) and joint-output error (the relative Frobenius error between the proxy-based and full-history attention outputs) under Eq. 7 in Figure R1 and Figure R2. Specifically, as shown in Figure R1, the historical and proxy attention masses are relatively close at the beginning of long-horizon generation. As generation proceeds, the historical attention mass gradually increases as more accumulated historical KV states contribute to the joint softmax. In contrast, the proxy attention mass remains relatively stable. Nevertheless, incorporating attention matching loss LAM consistently increases the proxy attention mass compared with the model without LAM. Furthermore, Figure R2 shows that LAM consistently reduces the joint-output relative error throughout long-horizon generation.

We will also revise the corresponding claims in the paper from full-context attention preservation to historical attention response approximation to better reflect our optimization objective.

W2 & Q2: Long-range Memory.

As discussed in Section 5 (``Trade-off across Ability Dimensions'') of VBench [1], there is a practical trade-off between Dynamic Degree and temporal consistency (Subject/Background Consistency and Smoothness), since relatively static scenes can “cheat” to get high temporal consistency scores. However, Attn-Forcing achieves substantially higher motion dynamics (35.55 to 72.14 compared with LongLive) while maintaining comparable temporal consistency (second-best Subject Consistency and Motion Smoothness, and third-best Background Consistency, only slightly behind the best results), achieving the highest total score of 82.37 in Table 2.

Moreover, we conduct a controlled delayed-recall experiment to evaluate historical information retention beyond these generation quality metrics. Specifically, we design prompts describing a static environment and a subject that leaves the frame and returns after a long interval beyond the local attention window, requiring the model to maintain environmental consistency and recall the previously observed subject.

As shown in Figure R3, LongLive fails to generate a clear subject exit, with the panda abruptly changing its orientation, multiple penguins appearing, and inconsistent subject behavior during reappearance. In contrast, Attn-Forcing better follows the intended exit-and-return sequence while maintaining a consistent environment and avoiding subject duplication. These observations provide qualitative evidence of improved long-range information retention beyond motion quality alone.

[1] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR, 2024.

The authors have addressed some of my concerns, and I have therefore decided to maintain my original score.

We sincerely thank the reviewer for the follow-up and for acknowledging our efforts to address the concerns. We greatly appreciate your constructive feedback and continued positive assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.