acceptodds
Withdrawn from ICLR 2027

Trace Forcing: Anchor Tokens in the KV Cache for Training-Free Long Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive video diffusion models generate videos of arbitrary length by conditioning each frame on the preceding frames stored in a KV cache, yet their quality degrades as generation proceeds beyond the training horizon (the video length seen during training). Existing training-free methods attribute this degradation to RoPE temporal positions beyond the trained range and re-index the temporal positions of the cached tokens at inference, but the degradation persists. We reveal that the cached tokens themselves deviate from the tokens the model was trained on, and that the deviated tokens entering the cache induce artifacts in the regions that attend to these tokens. To address this, we introduce Trace Forcing, a training-free method that replaces the raw tokens in the KV cache with anchor tokens, which aggregate the corresponding tokens across frames by an exponential moving average along traces obtained from the similarity between adjacent frames. Each anchor token thus incorporates earlier representations that are less affected by the deviation while remaining consistent with the corresponding content, so that the model conditions on representations closer to the tokens observed during training. We further propose adaptive updating, which assigns each anchor an update rate according to how its content evolves, suppressing the accumulation of deviation on static content while allowing dynamic content to follow recent information. Experiments show that Trace Forcing exhibits less quality degradation than existing training-free methods when generating far beyond the training horizon.

Primary area:
generative models
Keywords:
autoregressive video generation, long video generation, KV cache, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_rSANOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Trace Forcing addresses long-horizon degradation in autoregressive video diffusion by modifying the inference-time KV cache. It diagnoses deviation of cached features from training-horizon reference statistics and maintains anchor representations through exponential moving averages along cross-frame key-similarity correspondences. Content-adaptive update rates balance preservation of earlier representations against adaptation to recent content. The generator requires no additional training. Experiments on Self-Forcing and LongLive evaluate 240- and 480-second generation against existing training-free methods, using VBench-Long and quality-drift metrics.

Critical strengths
  1. The method addresses a meaningful limitation beyond positional extrapolation. The feature-deviation analysis provides a useful diagnostic of the cache feedback process, while traced anchors and adaptive updates offer a concrete inference-time intervention. The existing adaptive-update ablation matches the effective average update rate, which is a useful control for the quality-versus-motion tradeoff.

  2. The reported long-horizon gains on two backbones are practically relevant. On Self-Forcing at 240 seconds, the average score improves from 78.97 for MemRoPE to 79.75; on LongLive at 480 seconds, it improves from 77.29 to 79.30, with Dynamic Degree increasing from 44.94 to 54.90. At 480 seconds, imaging-quality drift decreases from 5.25 to 2.63 on Self-Forcing and from 3.52 to 2.04 on LongLive. These results support the usefulness of the cache intervention without retraining.

Critical weaknesses
  1. Table 3 supports adaptive updating but does not isolate content tracing from EMA smoothing: the missing control is position-wise EMA with the same cache structure, RoPE scheme, and update settings. Comparisons with MemRoPE change other design choices. The reduced Mahalanobis distance and its correlation with artifacts also establish an association, without demonstrating the stronger causal claim about feature deviation.

  2. On Self-Forcing at 480 seconds, the average gain over MemRoPE is only 0.22 on 20 prompts, with slightly lower imaging quality and Dynamic Degree. Different prompt sets at 240 and 480 seconds prevent a direct comparison over time. Appendix J also identifies difficulty with content emerging after warm-up, and the paper lacks end-to-end latency and peak-memory measurements for matching. These gaps limit the evidence for a reliable practical advantage.

  3. The novelty appears incremental. MemRoPE already combines EMA-based KV memory with position-free caching and online RoPE; TokenFlow and FLATTEN provide precedents for correspondence-based feature propagation and flow-guided attention. Trace Forcing's distinction is its causal, training-free, per-head KV tracing and warm-up-based update rule. The paper should make clearer what methodological advance this adaptation and combination provides.

Questions for authors
  1. Could you compare position-wise and traced EMA, each with fixed and adaptive updating, in the existing Self-Forcing 240-second setting? Match cache allocations, RoPE, prompts, seeds, sampling, and update-rate settings, and recompute warm-up statistics consistently. Report quality, drift, Dynamic Degree, and effective update rates.

  2. Please compare MemRoPE, Trace Forcing, and the position-wise control on common prompts and seeds using 120-, 240-, and 480-second prefixes where applicable. Report quality and motion curves, per-prompt differences, and uncertainty. Examples of objects entering or reappearing after warm-up would also help assess the limitation in Appendix J.

  3. What are generation time, throughput, and peak GPU memory for Trace Forcing and MemRoPE on the same hardware at matched total cache-token budgets? Include each cache allocation, matching/update time, and temporary buffers.

4 comments

We sincerely thank Reviewer rSAN for the insightful comments, which pointed out several aspects that we had overlooked. We believe that addressing them will considerably improve our paper. Along with our responses, we have uploaded the revised paper, in which the revised parts are marked in blue. We respectfully ask the reviewers to please refer to it together with our responses.

[@rSAN-1] Difference from MemRoPE and Comparison with Position-wise EMA (W1, Q1)

Beyond correspondence matching and the adaptive update rule, Trace Forcing differs from MemRoPE in what the EMA is used for. In MemRoPE, the EMA serves as an additional memory: the four most recent frames are kept as raw tokens, and two EMA frames, a long-term and a short-term memory that aggregate the same evicted frames at different rates, are added to retain the frames evicted from the window. The model therefore still relies mainly on raw tokens, in which the deviation accumulates. In Trace Forcing, the EMA instead replaces the raw tokens: under the same 12-frame budget, only a single frame is kept raw (R=1R = 1), and all the other past frames are held as anchor tokens (A=5A = 5), which are less deviated. Each raw token leaving the recent slot passes through the anchor slots one at a time and updates each of them, so that the anchors keep the temporal order of the history, for which a queue of raw tokens is used. The five anchor frames thus form a temporally ordered history of the past frames, rather than two memories of different time scales, so that the context preserves how the content has moved across the past frames. This design suppresses the deviation in the context, which induces artifacts, as much as possible. Consistent with this design, the advantage of Trace Forcing over MemRoPE in aesthetic quality grows with the video length, with a lower σ\sigma in both imaging and aesthetic quality at every length ([@rSAN-3]). Nevertheless, we fully agree that the contribution of correspondence matching should be clarified against position-wise EMA under an identical cache allocation.

In our view, correspondence matching makes two contributions over position-wise EMA: (1) it aggregates more similar tokens into each anchor than position-wise EMA (cosine similarity of 0.790 vs. 0.899 for the tokens whose content moves to a different position), and (2) it makes adaptive updating content-aware, since with position-wise EMA the update rates follow fixed regions of the frame rather than the contents (please refer to Fig. 15, Appendix J). The detailed analysis is provided in our response to Reviewer pyCt ([@pyCt-w1]).

In addition, we compared the two variants on the quality metrics under adaptive updating, with the same cache composition, RoPE, prompts, seeds, sampling, and update settings, recomputing the warm-up statistics under each correspondence (Self-Forcing, 20 prompts), following the setting of our ablation study.

EMAImaging Quality↑Imaging Quality σ\sigma↓Aesthetic Quality↑Aesthetic Quality σ\sigma↓Dynamic Degree↑Subject Consistency↑Background Consistency↑Motion Smoothness↑
Position-wise69.592.7458.653.4752.2297.6296.3698.72
Traced (Ours)69.612.8058.903.2753.7997.7496.4198.73

We consider the main role of tracing to be making the adaptive update rule work as intended. Adaptive updating assigns each anchor a rate that determines how much of the change of its content the anchor accepts, and without tracing, the change of each content cannot be measured properly.

  • For example, with position-wise EMA instead of tracing (Fig. 15, Appendix J), a part of the static background beside the person receives a high rate, whereas the rate decreases sharply where the person moves into a previously static position. With tracing (Fig. 13), the rates are assigned according to the content.

We therefore regard the contribution of tracing as closely tied to adaptive updating, and conducted the ablation under adaptive updating. As requested by the reviewer, we also conducted it with a fixed update rate, in which the two variants showed no considerable difference. Under adaptive updating, tracing achieves slightly higher scores than position-wise EMA, and we would further like to emphasize the contribution of tracing through the qualitative results in Fig. 13 and Fig. 15.

[@rSAN-2] Causal Role of Feature Deviation (W1)

To examine the causal role of the feature deviation, we intervened on the deviation itself: injecting deviation into the cached tokens within the training horizon degrades the visual quality in proportion to its magnitude, whereas removing it beyond the horizon recovers the quality lost over the rollout (Imaging 64.97 →\rightarrow 68.90 and Aesthetic 50.86 →\rightarrow 53.88). For the setting and the full results, we respectfully refer the reviewer to our response to Reviewer vQhd ([@vQhd-w2]).

[@rSAN-3] Common-prompt Comparison with MemRoPE over 120/240/480 s Prefixes (W2, Q2)

To compare over time on common prompts, we evaluated the 120-, 240-, and 480-second prefixes, using the same 20 prompts and seeds as the 480-second evaluation in Table 1.

PrefixMethodImaging Quality↑Imaging Quality σ\sigma↓Aesthetic Quality↑Aesthetic Quality σ\sigma↓Background Consistency↑Motion Smoothness↑Subject Consistency↑Dynamic Degree↑
120 sMemRoPE70.373.4857.563.9396.2998.6797.3047.50
Trace Forcing69.912.7858.583.1196.2998.7397.3451.00
240 sMemRoPE70.333.7856.044.4696.2498.6297.2547.29
Trace Forcing69.793.1157.903.4096.3398.7197.3249.25
480 sMemRoPE69.994.3354.994.7096.2298.5797.2047.43
Trace Forcing69.783.4357.523.6196.3598.6797.3246.07

The per-prompt differences (Trace Forcing − MemRoPE) are summarized below as mean ± standard error over the 20 prompts.

PrefixΔ\Delta Imaging QualityΔ\Delta Imaging Quality σ\sigmaΔ\Delta Aesthetic QualityΔ\Delta Aesthetic Quality σ\sigmaΔ\Delta Dynamic Degree
120 s−0.46 ± 0.56−0.71 ± 0.39*+1.01 ± 0.71−0.82 ± 0.48*+3.50 ± 4.03
240 s−0.54 ± 0.82−0.67 ± 0.52+1.85 ± 0.77*−1.06 ± 0.46*+1.96 ± 3.45
480 s−0.21 ± 0.81−0.90 ± 0.35*+2.53 ± 0.82**−1.09 ± 0.37**−1.36 ± 3.07

* p<0.05p < 0.05, ** p<0.01p < 0.01 (Wilcoxon).

The aesthetic advantage of Trace Forcing grows with the length and is significant at 240 and 480 seconds, with a lower σ\sigma in both metrics at every prefix, which is significant in all cases except the σ\sigma of imaging quality at 240 seconds. As the prefix grows from 120 to 480 seconds, the aesthetic quality of MemRoPE drops from 57.56 to 54.99, whereas that of Trace Forcing remains nearly constant (58.58 to 57.52). Imaging quality and Dynamic Degree show no significant difference at any prefix. The advantage over MemRoPE therefore holds on a common prompt set, rather than resulting from the different prompt sets used at 240 and 480 seconds.

[@rSAN-4] Content Introduced after Warm-up (W2, Q2)

Under LongLive’s interactive prompt switching, in which a new action or object is introduced every 80 seconds, Trace Forcing achieves the highest CLIP score in every segment (25.15 on average vs. 24.17 for LongLive and 24.41 for ∞\infty-RoPE); the setting and the full results are provided in our response to Reviewer vQhd ([@vQhd-q2]).

Additionally,

  • Examples of content entering after the warm-up are added to Appendix N ([@pyCt-w2]).
  • The remaining limitation is left to future work, for which we consider recording, for each anchor token, the cross-attention weights assigned to the individual prompt words, and re-estimating δˉc\bar{\delta}_c when the prompt is switched or the distribution of these weights changes, as described in our response to Reviewer pyCt ([@pyCt-w3]).

[@rSAN-5] End-to-end Latency and Peak Memory at a Matched Cache Budget (W2, Q3)

We measured the overhead under a matched cache budget of 18,720 tokens per layer (Trace Forcing: 4,680 sink + 7,800 anchor + 1,560 raw + 4,680 new; MemRoPE: 4,680 sink + 3,120 EMA + 6,240 recent + 4,680 new).

Latency. When generating 120 s videos on a single RTX A6000, Trace Forcing is only 0.7% (Self-Forcing) and 0.5% (LongLive) slower than MemRoPE. Per chunk, matching takes 17.3 ms and the anchor update takes 14.4 ms (0.7% and 0.6% of the chunk latency, respectively).

Memory. Trace Forcing requires an additional 3.5 GB of peak memory, which has a fixed size that does not grow with the video length. The temporary buffers of matching (0.88 GB) are freed before the forward pass.

For the generation time, throughput, and peak memory of other methods, we respectfully ask the reviewer to refer to the table in our response to Reviewer pyCt ([@pyCt-q1]) and to Appendix L (Table 6).

[@rSAN-6] Methodological Advance (W3)

Although EMA-based memory and correspondence-based propagation have precedents, Trace Forcing employs them for a different purpose, namely suppressing the deviation of the cached tokens during a causal rollout, and this purpose determines its design. The differences from MemRoPE in the cache composition and in correspondence matching are discussed in [@rSAN-1]. Here, we clarify how Trace Forcing differs from TokenFlow and FLATTEN in the use of correspondences, and why the warm-up-based update rule is required.

Correspondence in a causal rollout. TokenFlow (Geyer et al., 2024) and FLATTEN (Cong et al., 2024) also employ correspondences. However, these methods edit a source video, so that the correspondences are obtained from the given video in advance and are used to keep the edited frames temporally consistent. In contrast, Trace Forcing has no source video: the correspondences are computed online between the generated tokens during a causal rollout, and are used to aggregate the cached tokens into anchors that suppress the deviation.

Warm-up-based update rule. This difference also makes the update rule necessary. In video editing, the motion is already given by the source video, whereas in Trace Forcing the model must generate the motion while being conditioned on the anchors. Since an anchor that is updated slowly retains its past content, a small update rate suppresses the deviation but also hinders the motion: with a uniform rate, the dynamic degree declines as the rate decreases. Our update rule resolves this trade-off by setting the rate of each anchor according to how much its content changes during the warm-up, so that the deviation is suppressed in static content while the motion of dynamic content is preserved.

Following the reviewer’s suggestion, we will revise the paper to emphasize the novelty of each component.

References

  • Geyer, Michal, et al. “TokenFlow: Consistent Diffusion Features for Consistent Video Editing.” ICLR 2024.
  • Cong, Yuren, et al. “FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing.” ICLR 2024.

Thank you for the additional experiments and clarifications. The common-prompt comparisons and cost measurements address several of my questions. However, the contribution of tracing remains insufficiently supported. With adaptive updating, the gains over position-wise EMA are small. With a fixed update rate, the authors report no considerable difference. These results do not provide strong evidence that tracing adds substantial value beyond EMA aggregation.

The authors also acknowledge cases in which the update rule suppresses legitimate new content or changes in motion. The prompt-switching results provide useful evidence, but higher average scores do not resolve these failures. The proposed changes to the update rule have not yet been validated.

The authors could consider defining a narrower scope and evaluating the method in a concrete application domain. Videos with limited motion and relatively stable content may be one direction to explore. However, simpler position-wise EMA may also work well in such settings, so narrowing the scope would still require evidence of a meaningful advantage over that baseline.

The paper reports useful practical improvements, and the additional experiments clarify both its strengths and limitations. Nevertheless, they do not sufficiently establish the value of the distinctive tracing mechanism. Together with the unresolved difficulty in handling evolving content, this leaves the contribution insufficiently strong for acceptance at ICLR. I am therefore lowering my rating from 3 to 2 (weak rejection).

Reviewer_vQhdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trace Forcing, a training-free method for long video generation beyond the training horizon of autoregressive video diffusion models. The authors observe that cached tokens increasingly deviate from those generated within the training horizon, and that this deviation correlates with artifacts. To address this, raw tokens evicted from the recent window are aggregated into anchor tokens by an EMA along traces found by matching tokens of adjacent frames. The update rate of each anchor adapts to how much its content changes.

Critical strengths

S1. The paper is generally well organized, with clear motivation and an intuitive explanation of the proposed method.

S2. Trace Forcing achieves strong long-horizon generation results, generally improving VBench-Long scores and reducing quality drift across two base models and generation horizons of up to eight minutes.

Critical weaknesses

W1. Limited evidence for the contribution of correspondence-based aggregation. Like MemRoPE, Trace Forcing aggregates evicted tokens via EMA, and the paper frames correspondence-based relocation as a key difference from MemRoPE’s position-wise EMA. However, Table 3 goes directly from the base model to “+ Anchor Token”, which already includes relocation, so its contribution is never isolated. A fixed-position variant under the same cache configuration would clarify whether relocation is necessary.

W2. The causal role of token deviation is not sufficiently established. The method assumes that the Mahalanobis deviation of cached tokens indicates corrupted representations, but the evidence is correlational, and deviation may also grow with legitimate content changes. Moreover, the reduced deviation is expected by construction, since EMA averaging lowers variance regardless of whether tokens are corrupted.

Questions for authors

Q1. How does Trace Forcing perform with identity correspondence (fixed-position EMA) instead of the proposed matching, under the same cache configuration and update rules? This would isolate the contribution of tracing content across frames.

Q2. Could the reduced deviation partly come from suppressing novel content rather than correcting corrupted tokens? In particular, the deviation rejection in Eq. 6 suppresses updates whose change far exceeds δ̄_c, which may also apply to scene transitions or newly appearing objects. Since Dynamic Degree measures motion magnitude rather than responsiveness to new content, could the authors evaluate settings with substantial content changes, e.g., prompt switching on LongLive?

Q3. Could the authors perform a controlled intervention on the deviation itself? For example, injecting deviation into cached tokens within the training horizon and checking whether artifacts emerge would directly test the causal claim.

2 comments

Along with our responses, we have uploaded the revised paper, in which the revised parts are marked in blue. We respectfully ask the reviewers to please refer to it together with our responses.

[@vQhd-w1] Contribution of Correspondence-based Relocation

We fully agree that the contribution of relocation should be isolated. We therefore evaluated a fixed-position variant, in which the correspondence map is replaced with the identity under the same cache configuration and update rules. From this comparison, we identified two contributions of relocation: (1) it aggregates more similar tokens into each anchor than the fixed-position variant, and (2) it makes adaptive updating content-aware, since the update rates otherwise follow fixed regions of the frame rather than the contents (please refer to Fig. 15 and the revised text marked in blue in Appendix J). For the details, we respectfully refer the reviewer to our response to Reviewer pyCt ([@pyCt-w1]), and for the comparison on the quality metrics, to our response below ([@vQhd-q1]).

[@vQhd-w2, q3] Causal Role of Token Deviation

We sincerely thank Reviewer vQhd for suggesting this insightful experiment, which helped us strengthen our analysis. To test whether the deviation causally degrades the visual quality, we rescale the cached key/value tokens around the mean μ\mu of the training-horizon distribution, x′=μ+s(x−μ)x' = \mu + s(x - \mu), so that d(x′)=s⋅d(x)d(x') = s \cdot d(x). This changes only the deviation and keeps each token’s direction, i.e., its content, and is applied to each chunk right after it is written to the KV cache.

(1) Injection: from latent frame 9 (about 2 s, within the training horizon) we amplify the deviation of every new chunk by s∈{1.1,1.2}s \in \{1.1, 1.2\}.

(2) Removal: from latent frame 21 (about 5 s, the end of the training horizon) we pull it back toward the video’s in-horizon level dˉ\bar{d} (latent frames 12–20), d←d−λ(d−dˉ)d \leftarrow d - \lambda(d - \bar{d}) whenever d>dˉd > \bar{d}, with λ∈{0.5,1}\lambda \in \{0.5, 1\}.

We use Self-Forcing, 20 held-out MiraBench prompts (Ju et al., 2024) and the same seeds for all arms, which match before the intervention (late segment, mean ± s.e.).

InterventionArmImaging ↑Aesthetic ↑
Injection (15–30 s of 30 s)Clean69.51 ± 1.7757.45 ± 1.47
s=1.1s = 1.163.89 ± 2.5951.25 ± 1.67
s=1.2s = 1.251.37 ± 1.8544.92 ± 1.51
Removal (80–120 s of 120 s)Clean64.97 ± 2.3350.86 ± 1.91
λ=0.5\lambda = 0.568.04 ± 1.7151.71 ± 1.81
λ=1\lambda = 168.90 ± 1.6053.88 ± 1.44

Injecting deviation degrades both metrics in proportion to ss, while removing it recovers the quality lost over the long horizon. This experiment therefore confirms that deviation alone is sufficient to degrade visual quality and that removing it restores the quality, which supports a causal role of the deviation. The qualitative results of the deviation intervention with injection and removal are provided in Appendix K (Fig. 16). In the injection, amplifying the deviation with s=1.3s = 1.3 produces artifacts even within the training horizon, such as excessive color contrast and a deformed hand. In the removal, the flickering and the motion stagnation that emerge far beyond the training horizon are mitigated with λ=0.5\lambda = 0.5 and disappear with λ=1\lambda = 1.

[@vQhd-q1] Performance with Identity Correspondence

Under the same cache configuration and update rules, the fixed-position variant yields the following results on the 20 prompts of the ablation study:

Adaptive updatingImaging Quality↑Imaging Quality σ\sigma↓Aesthetic Quality↑Aesthetic Quality σ\sigma↓Dynamic Degree↑Subject Consistency↑Background Consistency↑Motion Smoothness↑
Fixed-position (identity)69.592.7458.653.4752.2297.6296.3698.72
Matching (Ours)69.612.8058.903.2753.7997.7496.4198.73

We consider the main role of matching to be making the adaptive update rule work as intended. Adaptive updating assigns each anchor a rate that determines how much of the change of its content the anchor accepts, and without matching, the change of each content cannot be measured properly.

  • For example, with the fixed-position variant (Fig. 15, Appendix J), a part of the static background beside the person receives a high rate, whereas the rate decreases sharply where the person moves into a previously static position. With matching (Fig. 13), the rates are assigned according to the content.

We therefore regard the contribution of matching as closely tied to adaptive updating, and conducted this comparison under adaptive updating. Matching achieves slightly higher scores than the fixed-position variant, and we would further like to emphasize the contribution of matching through the qualitative results in Fig. 13 and Fig. 15.

[@vQhd-q2] Novel Content and Interactive Prompt Switching on LongLive

We agree with the reviewer that the deviation rejection may also suppress the update from newly appearing content, as a change that far exceeds δˉc\bar{\delta}_c may reflect either genuinely new content or the deviation. Indeed, in the examples added to Appendix N, we observed cases in which an inappropriate update rate is assigned to newly entering content, so that its update is suppressed. We leave its resolution to future work. Since a text-to-video model generates a video conditioned on a given text prompt, an abrupt change in the motion or appearance of content can be interpreted as a change in the condition under which that content is generated. Building on this observation, we consider recording, for each anchor token, the cross-attention weights assigned to the individual prompt words, and re-estimating δˉc\bar{\delta}_c when the prompt is switched or the distribution of these weights changes. For further details, we respectfully refer the reviewer to our response to Reviewer pyCt ([@pyCt-w3]).

In addition, as suggested by the reviewer, we evaluated Trace Forcing under LongLive’s interactive prompt switching protocol:

  • Prompts: Each of the 20 prompts used in our ablation study is extended into six prompts following the protocol of LongLive (with Qwen2-72B-Instruct), each adding a new action or object.
  • Metrics: CLIP score (ViT-L/14) of each segment and its average, and VBench-Long Aesthetic and Imaging Quality.

In our setting, each prompt is used for 80 seconds (480 seconds in total), and Trace Forcing estimates δˉc\bar{\delta}_c only once during the warm-up and keeps it fixed across all prompt switches.

MethodSegment 1Segment 2Segment 3Segment 4Segment 5Segment 6CLIP Score (avg.)↑Aesthetic Quality↑Imaging Quality↑
LongLive28.2123.4223.7423.5024.6921.4524.1756.6668.04
∞\infty-RoPE28.2823.6723.8624.2124.4122.0424.4157.5070.30
Trace Forcing29.2324.4324.2924.8925.6122.4325.1557.8870.65

As shown in the table above, Trace Forcing achieves the highest CLIP score in every segment, together with the highest Aesthetic and Imaging Quality.

Reviewer_CD6gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows experimentally that the quality degradation of autoregressive video diffusion models in long video generation stems not only from RoPE positions beyond the trained range, but also from the cached tokens themselves deviating from the distribution the model was trained on. To address this, it introduces Trace Forcing that uses anchor tokens to average matching tokens across frames and reduce drift in the KV cache, with adaptive updating allowing slower updates for static content and faster updates for dynamic content. Experiments show that Trace Forcing exhibits less quality degradation than existing training-free methods when generating far beyond the training horizon.

Critical strengths
  1. The paper provides a quantitative analysis of token deviation with RoPE effects excluded, and the attention-weighted deviation maps line up with where artifacts appear.

  2. The method is simple, training-free, and applies directly to different base models.

  3. The improvement on drift metrics (Δ\Delta, σ\sigma) is consistent, and the qualitative results show noticeably less color shift and structural collapse.

Critical weaknesses
  1. The “deviation” finding largely restates the well-known error-accumulation premise, which the paper itself acknowledges (L47–49). On LongLive (App. B, Fig. 6a), the value deviation rises by only about 2% in the first frames and then stays flat, and Trace Forcing barely reduces the key deviation. This does not match the premise for using an EMA (L69–71).

  2. The tolerance δˉ\bar{\delta} is estimated only once, during warm-up. Genuine large changes, such as new objects or scene transitions, are therefore likely to be suppressed as deviation. As a result, the method may not be suitable for LongLive’s interactive prompt switching, and no such experiments are reported.

  3. Trace Forcing builds closely on MemRoPE’s EMA memory and online RoPE indexing. While it adds cross-frame token matching and adaptive updates, the ablations do not isolate the benefit of matching from EMA aggregation. Comparing traced and position-wise EMA under the same cache budget and update rates would help clarify the contribution beyond MemRoPE.

  4. Long-range identity is not fully preserved. In the second example of Fig. 10, the woman’s earrings repeatedly change shape over the rollout, which contradicts the claim in App. G that Trace Forcing “preserves the subject identity throughout the generation”.

Questions for authors
  1. How are anchors initialized, and how are empty anchor slots handled before all slots are filled?

  2. Why is matching performed on post-RoPE keys? The rotary embedding biases the similarity toward spatially nearby positions.

  3. How does Trace Forcing compare with the base model and MemRoPE in inference FPS and peak GPU memory under the same hardware and generation settings? What additional overhead comes from correspondence matching and maintaining the queue of raw KV states?

  4. The paper uses different update rates and anchor thresholds for Self-Forcing and LongLive, but does not explain how they were chosen. Were these values tuned on prompts separate from the evaluation set, and how would one select them for a new base model?

2 comments

Along with our responses, we have uploaded the revised paper, in which the revised parts are marked in blue. We respectfully ask the reviewers to please refer to it together with our responses.

[@CD6g-w1] Deviation on LongLive

We sincerely thank Reviewer CD6g for the careful examination of our deviation analysis on LongLive. Accordingly, we examined why the deviation of LongLive appears small in Fig. 6a. Inspecting individual prompts, we found that the deviation of LongLive does not persist once it emerges: it arises together with an artifact for a limited interval and subsides as the artifact transforms into another object or moves out of the frame. Since these intervals occur at different times across prompts, they cancel out when averaged, so that the averaged deviation appears small. Examples of such intervals are provided in Appendix C (Fig. 9), to which we respectfully refer the reviewer. We hope that this addresses the reviewer’s concern.

In addition, we repeated the same experiment on LongLive with a stronger suppression of the deviation, using α=0.05\alpha = 0.05 instead of 0.10.1. The averaged key deviation, which the reviewer noted was barely reduced, decreases from +1.7% to +0.9% (base: +1.9%), and the value deviation from +1.8% to +0.8%. The updated Fig. 6 with α=0.05\alpha = 0.05 is also provided in Appendix C (Fig. 8). For further details and explanations, we respectfully ask the reviewer to refer to Appendix C.

[@CD6g-w2] Interactive Prompt Switching on LongLive

We evaluated Trace Forcing under LongLive’s interactive prompt switching protocol, in which a new action or object is introduced every 80 seconds over 480 seconds while δˉc\bar{\delta}_c is estimated only once during the warm-up. Trace Forcing achieves the highest CLIP score in every segment (25.15 on average vs. 24.17 for LongLive and 24.41 for ∞\infty-RoPE), together with the highest Aesthetic and Imaging Quality. For the setting and the full results, we respectfully refer the reviewer to our response to Reviewer vQhd ([@vQhd-q2]).

Despite these results, we agree with the reviewer that, in principle, a change of content introduced by a new prompt can be suppressed as a deviation in Trace Forcing, since δˉc\bar{\delta}_c remains fixed after the warm-up. We also observed such a case, which is presented in Appendix N (Fig. 18), and leave its resolution to future work. Since a text-to-video model generates a video conditioned on a given text prompt, an abrupt change in the motion or appearance of content can be interpreted as a change in the condition under which that content is generated. Building on this observation, we consider recording, for each anchor token, the cross-attention weights assigned to the individual prompt words, and re-estimating δˉc\bar{\delta}_c when the prompt is switched or the distribution of these weights changes.

[@CD6g-w3] Isolating Matching from EMA Aggregation

We sincerely thank Reviewer CD6g for raising this point, which we had not examined. We fully agree that the benefit of matching should be isolated from EMA aggregation. We therefore compared matching with position-wise EMA under the same cache budget and update rates. From this comparison, we identified two contributions of matching: (1) it aggregates more similar tokens into each anchor than position-wise EMA, and (2) it makes adaptive updating content-aware, since with position-wise EMA the update rates follow fixed regions of the frame rather than the contents (please refer to Fig. 15 and the revised text marked in blue in Appendix J). For the details, we respectfully refer the reviewer to our response to Reviewer pyCt ([@pyCt-w1]), and for the comparison on the quality metrics, to our response to Reviewer rSAN ([@rSAN-1]).

[@CD6g-q3] Inference FPS, Peak Memory, and the Cost of Matching and the Raw KV Queue

We added a detailed comparison of the inference cost with the base model and MemRoPE to Appendix L, and the full results are also provided in the table of our response to Reviewer pyCt ([@pyCt-q1]). In summary, under the same hardware and generation settings, Trace Forcing is only 1.5% slower than the base model and 0.7% slower than MemRoPE on Self-Forcing for 120-second videos (5.09 FPS), where correspondence matching and the maintenance of the raw KV queue together take only 1.1–1.3% of the per-chunk latency. The additional 3.5 GB of peak GPU memory, which holds the queue of raw tokens evicted from the recent slot, has a fixed size that does not grow with the video length.

[@CD6g-q4] Choice of Update Rates and Anchor Thresholds

Both values that differ between the base models (α0\alpha_0 and δ0,j\delta_{0,j}) were tuned for Self-Forcing with a small sweep on MiraBench prompts (Ju et al., 2024), which do not overlap with the evaluation set (MovieGenBench). We set α0\alpha_0 inversely proportional to the training horizon NtrainN_\text{train}, so that even at the maximum rate 99% of an anchor’s content comes from the most recent NtrainN_\text{train} frames. Since the horizon of LongLive (241 latent frames) is about 11×11\times that of Self-Forcing (21), its α0\alpha_0 is an order of magnitude smaller (0.02 vs. 0.2). As shown in the graph in Appendix M, the gains are not sensitive to the value of α0\alpha_0: Trace Forcing outperforms both baselines on aesthetic quality and both drift metrics for every α0\alpha_0 from 0.01 to 0.5 (Self-Forcing, 120 s, 20 prompts).

For a new base model, one can therefore set α0≈4.5/Ntrain\alpha_0 \approx 4.5 / N_\text{train} without tuning. The constant step by which the anchor thresholds δ0,j\delta_{0,j} decrease from the oldest anchor to the newest one can then be chosen such that the average update rate is about 40% of α0\alpha_0, as is the case for both base models.

[@CD6g-etc] Other Comments and Questions

W4. We agree that the statement in App. G (App. H in the revision) is not an appropriate description of this example. We have revised it accordingly.

Q1. An anchor is initialized with the tokens of the first frame that reaches its slot, and anchors that have not been initialized do not participate in attention. We have revised Algorithm 1 accordingly.

Q2. We match on the rotary-embedded keys since this is the space in which the attention itself compares tokens. This is also the practice of training-free correspondence estimation in RoPE-based video DiTs; e.g., DiffTrack (Nam et al., 2025) computes its matching cost from the rotary-embedded queries and keys. Between adjacent frames, where the content moves only slightly, the preference for nearby positions serves as a locality prior, while a substantial portion of the tokens is still matched to a different position in each frame (35.1% on average, measured on 120-second videos generated by Self-Forcing with the 20 prompts of our ablation study).

References

  • Nam, Jisu, et al. “Emergent Temporal Correspondences from Video Diffusion Transformers.” NeurIPS 2025.
Reviewer_pyCtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Trace Forcing, a training-free KV-cache modification for long-horizon autoregressive video generation. It maintains anchor tokens through exponential moving averages along cross-frame correspondences obtained by nearest-neighbor matching. An adaptive update rule uses content-change statistics estimated within the training horizon to control subsequent updates and suppress tokens whose differences exceed the historical reference. Experiments apply the method to Self-Forcing and LongLive for 240- and 480-second generation, reporting VBench-Long scores, quality-drift metrics, and qualitative comparisons.

Critical strengths
  1. The paper addresses a practically relevant problem: quality degradation during autoregressive generation beyond the training horizon. The method requires no additional generator training, and the qualitative examples illustrate reductions in repetitive textures and color artifacts.
  2. The evaluation covers two base models with different training horizons and supplements video-level average scores with measurements of quality variation over time.
Critical weaknesses
  1. The methodological novelty relative to existing EMA-based cache methods is limited. MemRoPE already combines evolving EMA memory tokens, unrotated key storage, and online RoPE indexing. Trace Forcing primarily adds nearest-neighbor correspondence and a hand-designed adaptive update rule. While these modifications are plausible, the paper does not sufficiently establish their independent contribution or the broader insight they provide. In particular, the ablations do not compare traced aggregation against position-wise EMA under identical cache allocation, positional indexing, and update rates, leaving the significance of the central tracing mechanism unclear.
  2. The adaptive update mechanism relies on a strong assumption that early content-change statistics remain informative throughout a substantially longer rollout. These statistics are estimated during warm-up and subsequently propagated with the anchors, although later content may involve different motion patterns, newly appearing objects, occlusions, or camera movements. Appendix J acknowledges that newly emerging content inherits the update statistics of the anchor it enters. This limitation directly affects the proposed adaptation mechanism, yet the experiments do not establish its consequences or adequately characterize the method’s applicability to evolving scenes.
  3. The deviation-rejection rule assumes that changes substantially exceeding the historical reference indicate erroneous representation drift. However, large differences can also arise from legitimate changes in action, appearance, or scene content. The paper does not establish that its dissimilarity-based rule can distinguish these cases. Consequently, the mechanism may favor preserving historical content at the expense of incorporating valid new information. The reported quality and dynamic-degree metrics do not directly resolve this concern, and targeted evaluations of action transitions, new-content introduction, and prompt changes are missing.
Questions for authors

What is the inference overhead of per-layer, per-head correspondence matching and the auxiliary raw-token queue? Please report generation throughput, peak GPU memory, and matching time relative to MemRoPE under the same hardware, resolution, and generation settings, including auxiliary storage in the memory accounting.

2 comments

Along with our responses, we have uploaded the revised paper, in which the revised parts are marked in blue. We respectfully ask the reviewers to please refer to it together with our responses.

[@pyCt-w1] Novelty over MemRoPE and Contribution of Correspondence Matching


We sincerely thank Reviewer pyCt for pointing out an aspect that we had overlooked.

Novelty over MemRoPE

Beyond nearest-neighbor correspondence and the adaptive update rule, Trace Forcing differs from MemRoPE in what the EMA is used for. In MemRoPE, the EMA serves as an additional memory: the four most recent frames are kept as raw tokens, and two EMA frames, a long-term and a short-term memory that aggregate the same evicted frames at different rates, are added to retain the frames evicted from the window. The model therefore still relies mainly on raw tokens, in which the deviation accumulates. In Trace Forcing, the EMA instead replaces the raw tokens: under the same 12-frame budget, only a single frame is kept raw (R=1R = 1), and all the other past frames are held as anchor tokens (A=5A = 5), which are less deviated. Each raw token leaving the recent slot passes through the anchor slots one at a time and updates each of them, so that the anchors keep the temporal order of the history, for which a queue of raw tokens is used. The five anchor frames thus form a temporally ordered history of the past frames, rather than two memories of different time scales, so that the context preserves how the content has moved across the past frames. This design suppresses the deviation in the context, which induces artifacts, as much as possible. Consistent with this design, the advantage of Trace Forcing over MemRoPE in aesthetic quality grows with the video length, with a lower σ\sigma in both imaging and aesthetic quality at every length ([@rSAN-3]). Nevertheless, we fully agree that the contribution of correspondence matching should be clarified against position-wise EMA under an identical cache allocation.

Contribution of Correspondence Matching

(1) It aggregates more similar tokens into each anchor. As stated in Sec. 5.1, “updating the anchor at a fixed position would blend tokens of different contents into one anchor and compromise the representation.” To verify this, we measured the cosine similarity between each token entering an anchor and the tokens previously aggregated into it, along the matched trace and at the fixed position of the same generated tokens (20 prompts of Table 3, all layers and heads; mean ± standard error over prompts). The table reports the tokens that matching assigns to a different position (35.1% of all tokens).

Cosine similarity of the entering token to ↑Position-wiseMatching (Ours)
the token aggregated 1 frame earlier0.790 ± 0.0110.899 ± 0.003
the token aggregated 8 frames earlier0.725 ± 0.0170.769 ± 0.011

Position-wise EMA aggregates less similar tokens into each anchor, whereas matching aggregates tokens of higher similarity. The same trend holds over all tokens (0.871 vs. 0.922 for the token aggregated one frame earlier).

(2) It makes adaptive updating content-aware. Adaptive updating sets the rate of each anchor from how much its content changes during the warm-up. This design reflects how the deviation accumulates across anchors: static content persists in the same anchor the longest, where the deviation accumulates the most, and thus requires a small rate, whereas dynamically evolving content changes faster than the deviation accumulates and thus admits a large rate.

The rate must therefore be determined per content. Without matching, however, the content at a fixed position changes as objects move, so the rate follows fixed regions of the frame rather than the contents. To examine this, we visualized the update rates of our method with position-wise EMA (Fig. 15 in Appendix J, under the same setting as Fig. 13). With position-wise EMA, the rates are no longer assigned according to the content. For example, a part of the static background receives a high rate, since the content at that position changed dynamically during the warm-up and the content-adaptive gate thus assigns it a large rate (third row, j=3j = 3, red box). Conversely, the rate drops sharply where the person moves into a previously static position, since this change far exceeds the tolerance δˉc\bar{\delta}_c of that position and is suppressed by the deviation rejection (second row, j=4j = 4, red boxes). Correspondence matching is therefore required for adaptive updating to operate on the contents.

Additionally, we present the ablation that compares traced aggregation against position-wise EMA on the quality metrics in our response to Reviewer rSAN ([@rSAN-1]).

[@pyCt-w2] Applicability to Evolving Scenes

We agree that our paper did not concretely characterize this limitation. In the revision, we added examples to Appendix N in which content with a new motion enters or the camera movement changes after the warm-up. In these examples, the update rates do not properly reflect how the new content evolves, since the tolerance δˉc\bar{\delta}_c of each anchor token is estimated from the content observed during the warm-up and is not re-estimated for the new content. We respectfully refer the reviewer to Appendix N, where the revised parts are marked in blue.

[@pyCt-w3] Distinguishing Legitimate Changes from Deviation

The limitation discussed above ([@pyCt-w2]) and the concern raised here stem from the same difficulty inherent to generating beyond the training horizon: a token that departs from its anchor may reflect either a genuine change of content or the deviation, and the cached representations alone cannot distinguish the two.

We acknowledge this limitation and leave its resolution to future work. A text-to-video model generates a video conditioned on a given text prompt. Accordingly, an abrupt change in the motion or appearance of content can be interpreted as a change in the condition under which that content is generated. Building on this observation, we consider recording, for each anchor token, the cross-attention weights assigned to the individual prompt words, which serve as a representation of the past state of the content. When the prompt is switched, or when the distribution of these weights changes for an anchor token, the state of the content can be regarded as changed. For such cases, we are considering an approach in which the tolerance δˉc\bar{\delta}_c used in the deviation rejection is re-estimated to reflect the new state. Since the cross-attention weights reflect which part of the prompt each anchor token depicts, they are expected to provide a cue for a change in the state of the content that is independent of the token deviation itself. We leave the validation of this approach to future work.

In addition, in the prompt-change experiment, Trace Forcing achieves the highest CLIP score in every segment (25.15 on average vs. 24.17 for LongLive and 24.41 for ∞\infty-RoPE). We kindly invite the reviewer to refer to our response to Reviewer vQhd ([@vQhd-q2]) for the details.

[@pyCt-q1] Inference Overhead of Matching and the Raw-Token Queue

To verify whether correspondence matching and the raw-token queue add meaningful cost over the base model and MemRoPE, we measured throughput, generation time, peak memory, and the time of each component. All methods run on one RTX A6000 at 480×832480 \times 832 with the same checkpoints and sampler, generate 120-second videos, and use the same 12-frame cache (18,720 tokens per layer). We report the mean ± std over three runs.

BackboneMethodThroughput (FPS) ↑Generation time (s) ↓Peak memory (GB) ↓
Self-ForcingBase5.17 ± 0.01369.3 ± 0.521.96
MemRoPE5.13 ± 0.01372.2 ± 0.622.04
Ours5.09 ± 0.01374.8 ± 0.525.56
LongLiveBase4.41 ± 0.01432.7 ± 0.722.66
MemRoPE4.37 ± 0.01435.8 ± 0.722.74
Ours4.35 ± 0.00437.6 ± 0.426.26
  • Matching. It takes 17.3 ms per chunk (0.7% of the chunk latency) and adds 0.48% FLOPs.
  • Raw-token queue. Updating the anchors with the queued tokens takes 14.4 ms per chunk (0.6%). The additional 3.5 GB of peak memory, which holds the queue of raw tokens evicted from the recent slot, has a fixed size that does not grow with the video length.

Therefore, Trace Forcing is only 0.7% (Self-Forcing) and 0.5% (LongLive) slower than MemRoPE, and 1.5% and 1.2% slower than the base model, and its fixed-size auxiliary storage accounts for the entire increase in peak memory. We included these measurements in Appendix L.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.