acceptodds
Under review as a conference paper at ICLR 2027

In-Distribution Forcing for Long Video Generation at Test-Time

Abstract

While modern autoregressive (AR) video diffusion models have demonstrated remarkable success in short-horizon video generation, generating long videos remains challenging due to a phenomenon known as drifting, where colors and textures shift, and motion dynamics decay. Existing approaches primarily rely on KV conditioning, which selects or modifies cached key-value (KV) entries to mitigate drifting. However, we observe that KV conditioning alone is insufficient because it assumes cached KV entries remain in-distribution. This assumption fails beyond the training horizon because no mechanism constrains the construction of cached KV entries during rollout, giving rise to the KV-provenance problem where cached entries themselves become out-of-distribution. To address this limitation, we propose In-Distribution Forcing (ID-Forcing), a test-time framework that aligns both KV caching and KV conditioning with training configurations. Its key mechanism, self-caching, prevents out-of-distribution KV entries at their source: each chunk is cached without attending to any prior KV entry, keeping the rolling window exactly in-distribution without retraining. Consequently, ID-Forcing seamlessly extends short-horizon models to minute-scale long video generation. Extensive evaluations show that our method remains competitive on standard video generation benchmarks while substantially outperforming prior work in mitigating drifting, as validated by both our drift metrics and a user study.

Primary area:
generative models
Keywords:
Diffusion based test-time autoregressive long video generation, autoregressive video generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GebSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces In-Distribution Forcing (ID-Forcing), a training-free method for extending autoregressive video generation beyond the training horizon. The method self-caches historical chunks without attending to preceding entries, retains the first chunk as a persistent condition, and re-rotates temporal positions within a bounded window. Experiments on Self-Forcing and LongLive report improved long-video quality and dynamics at 120 and 240 seconds, alongside color- and motion-drift scores and human preference results.

Critical strengths
  1. A simple training-free intervention. The method requires no model retraining or additional learned parameters and reports improved long-horizon generation quality and dynamics on Self-Forcing and LongLive.
Critical weaknesses
  1. The claim of exact training-distribution alignment is not justified. For noninitial chunks, Self-Forcing constructs cached representations while attending to previously generated history. ID-Forcing changes this policy by encoding selected chunks without preceding context. Although context-free caching occurs for the initial chunk during training, applying the same operation to arbitrary later chunks does not establish that their inputs, cached features, or joint distribution match those encountered during training. Matching cache-operation configurations and positional ranges is therefore insufficient to justify the stronger claim that inference remains exactly in-distribution. Training with the same caching policy would also provide a useful control for evaluating the effect of this change.

  2. The conditioning components substantially overlap with prior work. LongLive already retains the first frame chunk as a persistent attention sink alongside a short rolling window. Rolling Forcing also combines initial-frame sinks and recent-history caching with dynamic RoPE adjustment to keep relative positional offsets within the trained range. Consequently, the proposed First Chunk as a Persistent Condition and Bounded Window and Exact Rolling mechanisms appear closely related to established techniques. The paper should clarify the precise technical differences and distinguish the contribution of self-caching from these existing conditioning strategies.

Questions for authors
  1. Does the exact in-distribution claim concern only cache-operation configurations, or the actual joint distribution of cached representations and generated content? What assumptions or empirical measurements justify extending context-free caching from initial to later chunks? Could the authors compare against a model trained with the same caching policy to assess the effect of changing that policy only at inference?

  2. What are the precise differences between the proposed persistent first chunk and bounded position mechanisms and LongLive's frame sink and Rolling Forcing's sink and dynamic-RoPE design? What additional technical contribution remains beyond the self-caching?

5 comments

We thank the reviewer for the careful reading and constructive comments, in particular on the scope of the in-distribution claim and the relation to LongLive and Rolling Forcing. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with GebS in purple.

[W1, Q1]-1. Scope of the in-distribution claim

We thank the reviewer for raising this point. We state our in-distribution claim precisely in terms of the structure of KV operations, and add KV feature statistics that check whether the resulting KV entries stay close to those within the training horizon. Our argument proceeds in three steps: (a) every KV operation in ID-Forcing is one that training executes, which holds by construction; (b) as a result, drift is largely alleviated; and (c) in turn, the KV entries that the model reads stay closer to the in-horizon KV than with autoregressive caching. Step (a) is the design principle of ID-Forcing, and steps (b) and (c) are measured outcomes.

(a) In-distribution KV operations. ID-Forcing ensures that (i) every KV entry is written by an operation executed in training, either context-free (self-caching), or autoregressively on top of the chunks right before it, starting from a context-free entry, and (ii) every window contains the sink and keeps all relative positions within the trained range. As the reviewer points out, this does not imply that the inputs, cached features, or their joint distribution match those in training, and we do not claim so. We therefore remove "exactly in-distribution" and rephrase content-level expressions such as "OOD KV entries" as statements on caching operations, e.g., "KV entries written by out-of-distribution caching operations" (Abstract; Sec. 1; Sec. 3; Sec. 5; Sec. 6).

(b) Drift alleviation. With KV operations kept in-distribution, ID-Forcing largely alleviates drift both quantitatively and qualitatively (Tab. 1, 3, 4; Fig. 4, 5).

(c) KV feature statistics. We then check whether the alleviated drift is also reflected in the KV entries that the model reads: with ID-Forcing, they stay closer to the in-horizon KV, and their gap stays flat up to 240 s (Tab. A, B). Specifically, we measure the gap between the KV feature statistics of 7-chunk intervals ending at 30, 60, 120, and 240 s and those of the first 7 chunks of the same video (the training horizon): Δμ\Delta\mu, the L1 distance between the channel-wise mean vectors (averaged over layers), and the ratio σref/σ\sigma_{\text{ref}}/\sigma of the channel-wise standard deviations. The statistics are computed for each layer over the tokens of pre-RoPE keys and values, on one 240 s video for each of the 128 prompts. For ID-Forcing, we report the context-free (self-cached) entry, the tail entry re-cached autoregressively on top of it, and their average.

Tab. A. Mean gap Δμ\Delta\mu (K / V) to the first 7 chunks. Lower is better. (Fig. 6, Sec. 4.4)

Caching25–30 s55–60 s115–120 s235–240 s
AR (Self-Forcing)0.225 / 0.1870.299 / 0.2710.396 / 0.3740.503 / 0.488
AR + sink + re-rotation0.187 / 0.1550.196 / 0.1630.210 / 0.1770.210 / 0.175
Self-caching: context-free entry0.183 / 0.1460.178 / 0.1470.180 / 0.1460.185 / 0.151
Self-caching: tail entry0.155 / 0.1220.149 / 0.1190.152 / 0.1220.158 / 0.127
Self-caching: non-sink average0.169 / 0.1340.164 / 0.1330.166 / 0.1340.172 / 0.139

Tab. B. Standard deviation ratio σref/σ\sigma_{\text{ref}}/\sigma (K / V). 1 is identical. (Fig. 6, Sec. 4.4)

Caching25–30 s55–60 s115–120 s235–240 s
AR (Self-Forcing)1.021 / 0.9431.050 / 0.9191.125 / 0.9581.242 / 1.066
AR + sink + re-rotation0.997 / 0.9811.012 / 0.9831.005 / 0.9621.008 / 0.972
Self-caching: context-free entry1.006 / 0.9741.002 / 0.9720.989 / 0.9560.997 / 0.968
Self-caching: tail entry1.002 / 0.9931.009 / 0.9920.995 / 0.9741.005 / 0.980
Self-caching: non-sink average1.004 / 0.9841.006 / 0.9820.992 / 0.9651.001 / 0.974

With ID-Forcing, the KV feature statistics stay closer to the in-horizon KV compared to other baselines. Moreover, the gap stays flat up to 240 s, whereas it more than doubles for AR (Self-Forcing) and grows by 12–14% for AR + sink + re-rotation until 120 s, staying over 20% above ours at 240 s. This confirms step (c). The standard deviation stays close to the reference except for AR (Self-Forcing) (Sec. 4.4).

[W1, Q1]-2. Extending context-free caching to later chunks

Extending context-free caching is based on step (a), which holds by construction: context-free caching is an operation that training executes for the first chunk, so applying it to later chunks keeps every caching operation within those executed in training. Its empirical justification is steps (b) and (c) above: drift is largely alleviated (Tab. 1), and the KV gap stays flat up to 240 s (Tab. A).

[W1, Q1]-3. Control trained with the same caching policy

Training with self-caching brings no consistent gain over applying it only at inference (Tab. C). We compare three setups, denoted (Training, Inference): (SF, SF), (SF, IDF), and (SF + SC, IDF), where SF, IDF, and SC denote Self-Forcing, ID-Forcing, and self-caching, and SF + SC applies self-caching during SF training. Due to the rebuttal period, SF and SF + SC are trained for 400 iterations with batch size 8 (default: 600 iterations, batch size 64), otherwise following the original recipe. All are evaluated on 120 s videos.

Tab. C. Effect of applying the self-caching policy only at inference (120 s). (Tab. 5, App. A)

TrainingInferenceAestheticBackgroundImagingMotionSubjectDynamicColor driftMotion drift
SFSF48.7796.2661.8797.8896.1932.0121.5751.56
SFIDF56.0195.7467.6797.3996.4062.4570.9988.28
SF + SCIDF55.7196.4467.5897.4497.4757.5472.0785.16

(SF, IDF) and (SF + SC, IDF) both improve substantially over (SF, SF), and (SF + SC, IDF) brings no consistent gain over (SF, IDF) at this training budget, so ID-Forcing retains its benefit without retraining (Sec. 4.4; App. A).

W2, Q2. Relation to LongLive and Rolling Forcing, and the remaining contribution

Contribution. To our knowledge, we are the first to (1) identify KV caching as a source of drift separate from KV conditioning, formalizing it as the KV-provenance problem, and (2) show that KV caching must also be addressed for test-time extrapolation, proposing self-caching to resolve it. Self-caching substantially alleviates drift (Tab. 3), and ID-Forcing outperforms existing test-time extrapolation methods, with the best Color Drift and Dynamic Degree in every setting (Tab. 1) and a consistent preference in the user study (Tab. 2). We would greatly appreciate it if the reviewer could take these two contributions into account.

Conditioning. We agree with the reviewer that our conditioning rule, although derived from the training–test gap, coincides in practice with the frame sink of LongLive and the temporal RoPE adjustment of Rolling Forcing; we now acknowledge both explicitly in the revised paper (Sec. 3.2).

Isolating self-caching. Tab. 3 adds self-caching while keeping our conditioning (sink + re-rotation) fixed: it improves every metric, most significantly Color Drift, from 60.66 to 72.37. We have moved this ablation from the appendix to the main paper (Tab. 3, Sec. 4.3).

We hope these clarifications and additional experiments address the reviewer's concerns.

Dear Reviewer GebS,

Thank you again for your time and constructive review. As the author–reviewer discussion period ends on November 18 (AoE), we would be grateful if you could let us know whether our responses have addressed your concerns.

In particular, we clarified the scope of our in-distribution claim, added KV feature statistics and a control trained with self-caching, and clarified the relation to LongLive and Rolling Forcing; all of these are incorporated into the revised paper. If they address your concerns, we would appreciate it if you could take them into account in your evaluation. We would be happy to clarify any remaining questions.

Reviewer_H31XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on drifting in autoregressive video diffusion models, where colors and textures shift and motion decays once generation extends beyond the training horizon. The authors argue that existing test-time methods only control which cached key-value (KV) entries are read and where they are positioned (KV conditioning). They do not control how each entry was written (KV caching). Beyond the horizon, every chunk is cached while attending to a context never seen in training, so the cached entries themselves become out-of-distribution, which the authors call the KV-provenance problem. Building on this, they propose In-Distribution Forcing (ID-Forcing), a training-free method. Its key mechanism, self-caching, caches the earliest entries of the rolling window without attending to any prior entry and re-caches the remaining ones autoregressively on top of them. It also keeps the first chunk as a permanent sink and re-rotates positions into the trained range. On Self-Forcing and LongLive, ID-Forcing improves color and motion drift scores and dynamic degree on 30- to 240-second videos, and is preferred in a user study.

Critical strengths
  1. The method is simple, training-free, and consistently effective. It is a small change to how a rolling KV window is cached. Yet across two base models and four durations, it achieves the best color and motion drift scores and the highest dynamic degree, while also improving aesthetic and imaging quality. Since it requires no training, it can be applied directly to existing autoregressive video models.

  2. Separating KV caching from KV conditioning is a useful lens on long-horizon drift. Fig. 2 isolates the effect with a controlled example where the same conditioning window gives different results depending only on how the last entry was cached. The ablation (Table 4) shows that self-caching adds clear gains on top of the sink and re-rotation, raising color drift from 60.66 to 72.37 and aesthetic quality from 54.14 to 58.23. This suggests that how cache entries are written deserves attention in future work.

Critical weaknesses
  1. The central claim that self-caching keeps the rolling window "exactly in-distribution" is asserted rather than established, and it sits uneasily with the paper's own argument. During training, only the first chunk is ever cached without context. The paper itself argues that this first chunk is distributionally unique, because the VAE encodes it from a single frame. A later, multi-frame chunk cached without context is therefore also a configuration that training never executed. Observations 1–2 are stated without measurement. Without such evidence, self-caching may be better understood as an effective anchoring heuristic than as a guarantee of in-distribution KV states.

  2. The evaluation favors anchoring and does not measure its costs. The drift metrics compare the first and last frames (hue histogram) or the first and last five seconds (dynamic degree). These metrics reward staying close to the opening scene, which the permanent first-chunk sink and the short window (L=3L=3) encourage. They also penalize legitimate scene evolution. The paper does not measure content progression, repetition, or long-range consistency. It also does not report the compute overhead of re-caching at every step, and it does not compare with Pathwise Test-Time Correction [1], which it cites only for its metric. This makes it difficult to judge the trade-offs of the method.

[1] Xiang et al., Pathwise Test-Time Correction for Autoregressive Long Video Generation, Arxiv 2026.

Questions for authors
  1. Can the authors provide direct evidence that self-cached entries are closer to the training-time KV distribution than autoregressively cached ones? For example, they could compare KV feature statistics or measure distances to KV entries collected within the training horizon. Given the argument that the first chunk is distributionally unique, how do they reconcile it with caching later chunks without context?

  2. What is the additional compute of re-caching the window tail at every step, in latency and throughput, compared with the baselines? How sensitive are the results to LL and ℓ\ell? For LongLive, ℓ=L=3\ell = L = 3, which seems to mean that no entry is cached autoregressively. Is that correct?

  3. Does anchoring to the first chunk limit content progression? Could the authors evaluate prompts that require scene changes, LongLive's interactive prompt switching, or a repetition/looping metric?

5 comments

We thank the reviewer for the detailed and constructive review, in particular on direct evidence that self-cached entries stay closer to the in-horizon KV, the concern about the anchoring heuristic, and the sensitivity to LL and ℓ\ell. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with H31X in green.

W1, Q1. Scope of the claim, KV feature statistics, and the uniqueness of the first chunk

(1) Scope of the claim. We appreciate the reviewer for raising this point. We agree that the scope of our in-distribution claim should be stated more precisely. Our claim concerns the structure of KV operations: every entry is written by an operation executed in training, and every conditioning window contains the sink and keeps all relative positions within the trained range. We therefore remove "exactly in-distribution" and revise the text accordingly (Abstract; Sec. 1; Sec. 3; Sec. 5; Sec. 6). Obs. 1 follows directly from the generation process (each chunk starts from noise and passes through frozen projections), and we restate Obs. 2 as a statement on operations: reading the KV entries when generating chunk xix_i is in-distribution only if each of them was cached under a provenance that training executed (Sec. 3.1). Empirically, this alignment alleviates drift (Tab. 1, 3, 4), and in turn, the KV entries that the model reads stay closer to the in-horizon KV (below).

(2) KV feature statistics. Following the reviewer's suggestion, we compare KV feature statistics as direct evidence. We generate one 240 s video for each of the 128 prompts and, for each layer, compute the channel-wise mean and standard deviation of pre-RoPE keys and values over tokens. We compare 7-chunk intervals ending at 30, 60, 120, and 240 s with the first 7 chunks of the same video (the training horizon), and report Δμ\Delta\mu, the L1 distance between the channel-wise mean vectors (averaged over layers), and the ratio σref/σ\sigma_{\text{ref}}/\sigma of the channel-wise standard deviations. For ID-Forcing, we report the context-free entry, the tail entry re-cached autoregressively on top of it, and their average, i.e., the non-sink entries that the model reads.

Tab. A. Mean gap Δμ\Delta\mu (K / V) to the first 7 chunks. Lower is better. (Fig. 6, Sec. 4.4)

Caching25–30 s55–60 s115–120 s235–240 s
AR (Self-Forcing)0.225 / 0.1870.299 / 0.2710.396 / 0.3740.503 / 0.488
AR + sink + re-rotation0.187 / 0.1550.196 / 0.1630.210 / 0.1770.210 / 0.175
Self-caching: context-free entry0.183 / 0.1460.178 / 0.1470.180 / 0.1460.185 / 0.151
Self-caching: tail entry0.155 / 0.1220.149 / 0.1190.152 / 0.1220.158 / 0.127
Self-caching: average0.169 / 0.1340.164 / 0.1330.166 / 0.1340.172 / 0.139

Tab. B. Standard deviation ratio σref/σ\sigma_{\text{ref}}/\sigma (K / V). 1 is identical. (Fig. 6, Sec. 4.4)

Caching25–30 s55–60 s115–120 s235–240 s
AR (Self-Forcing)1.021 / 0.9431.050 / 0.9191.125 / 0.9581.242 / 1.066
AR + sink + re-rotation0.997 / 0.9811.012 / 0.9831.005 / 0.9621.008 / 0.972
Self-caching: context-free entry1.006 / 0.9741.002 / 0.9720.989 / 0.9560.997 / 0.968
Self-caching: tail entry1.002 / 0.9931.009 / 0.9920.995 / 0.9741.005 / 0.980
Self-caching: average1.004 / 0.9841.006 / 0.9820.992 / 0.9651.001 / 0.974

With ID-Forcing, the KV feature statistics stay closer to the in-horizon KV compared to other baselines. Moreover, the gap stays flat up to 240 s, whereas it more than doubles for AR (Self-Forcing) and grows by 12–14% for AR + sink + re-rotation until 120 s, staying over 20% above ours at 240 s. The standard deviation stays close to the reference except for AR (Self-Forcing) (Sec. 4.4).

(3) Reconciling with the first chunk. We would like to clarify that the uniqueness of the first chunk lies in its content (encoded from fewer frames), which is why we keep it as the sink. Self-caching, on the other hand, concerns the operation: a later chunk is cached without past context, as the first chunk is in training. In practice, self-caching alleviates drift (Tab. 1), and the gap of the resulting KV entries to the in-horizon KV stays flat over time (Tab. A).

(4) Anchoring heuristic. To verify that self-caching does not anchor generation to the opening scene, we conducted the suggested prompt-switching evaluation: both ID-Forcing variants stay less similar to the first prompt than LongLive while following each new prompt (Q3 below). This is consistent with the qualitative results, where ID-Forcing sustains motion until the end of the video (Fig. 4), and with the user study, where participants preferred ID-Forcing in Dynamic Degree over every compared method (Tab. 2).

W2-(1), Q3. Content progression, repetition, and long-range consistency

As the reviewer suggested, we use LongLive's standard multi-prompt test (100 prompt sets; 60 s, six prompts of 10 s each) and report VBench (Tab. C) and per-segment ViCLIP similarity to the current prompt for progression (Tab. D) and to the first prompt for repetition, i.e., returning to the opening scene (Tab. E). At each switch, LongLive + ID-Forcing keeps LongLive's own prompt-recache rule, and Self-Forcing + ID-Forcing only re-caches the last chunk under the new prompt, a minimal form of it; all other entries are self-cached.

Tab. C. VBench under prompt switching (60 s). (Tab. 6, App. A)

MethodSubjectBackgroundMotionDynamicAestheticImaging
LongLive97.7196.6199.1727.4359.7271.32
LongLive + ID-Forcing97.3296.3399.0041.8760.0871.81
Self-Forcing + ID-Forcing97.5996.4798.8951.6760.1672.33

Tab. D. ViCLIP similarity to the current prompt. Higher is better. (Tab. 7, App. A)

Method0–10 s10–20 s20–30 s30–40 s40–50 s50–60 sMean
LongLive26.2425.1124.4824.1824.8023.7224.76
LongLive + ID-Forcing26.0425.4224.5624.6625.1524.2425.01
Self-Forcing + ID-Forcing26.2225.0524.6324.3524.6224.2724.86

Tab. E. ViCLIP similarity to the first prompt after switching. Lower is better. (Tab. 8, App. A)

Method10–20 s20–30 s30–40 s40–50 s50–60 sMean
LongLive23.7622.5923.7322.2823.7823.23
LongLive + ID-Forcing23.3521.1620.4820.6720.6921.27
Self-Forcing + ID-Forcing23.6521.7421.3120.7620.6521.62

We believe these results address the reviewer’s question. First, ID-Forcing follows new prompts better. With ID-Forcing, LongLive follows the new prompt better in all five post-switch segments, and even Self-Forcing + ID-Forcing, which is not trained for prompt switching, achieves a higher mean similarity than LongLive, which is explicitly trained for it (Tab. D). Second, ID-Forcing is less anchored to the opening scene. Both ID-Forcing variants are less similar to the first prompt than LongLive in every post-switch segment, by 1.6–2.0 points on average and by over 3 points in the last segment (Tab. E). This indicates that self-caching does not limit content progression but rather improves it (Sec. 4.2; App. A).

Long-range consistency is measured by Subject and Background Consistency in VBench (Tab. 1). Since both metrics also give high scores to static videos, we read them together with Dynamic Degree. ID-Forcing achieves the highest Dynamic Degree in every setting while staying within 0.5 points of the best on both consistency metrics, i.e., it maintains long-range consistency without freezing the video.

W2-(2), Q2-1. Compute overhead

We measure the latency and throughput of generating one chunk on Self-Forcing (Tab. F).

Tab. F. Chunk generation latency and throughput on Self-Forcing. (Tab. 9, App. A)

MethodLatency / chunk (ms)FPS (DiT)FPS (w/ VAE)
Self-Forcing1634 (1.00×)7.35.4
Deep Forcing1554 (0.95×)7.75.6
∞-RoPE1007 (0.62×)11.97.6
MemRoPE1476 (0.90×)8.15.8
ID-Forcing1368 (0.84×)8.86.2

In ID-Forcing, the autoregressive re-caching of the tail occurs at every chunk generation and accounts for 14.5% of its latency. Even so, ID-Forcing has 16% lower latency and higher throughput than Self-Forcing (App. A).

W2-(3). Comparison with Pathwise Test-Time Correction

On Self-Forcing, ID-Forcing is better than or equal to Pathwise TTC on every metric at both 120 s and 240 s, with large margins in Dynamic Degree, Color Drift, and Motion Drift (Tab. G).

Tab. G. Comparison with Pathwise TTC (Self-Forcing, 120 s / 240 s). (Tab. 1)

MethodAestheticBackgroundImagingMotionSubjectDynamicColor driftMotion drift
Self-Forcing (120 s)49.9596.1761.6098.2696.2030.0425.5446.88
Pathwise TTC (120 s)56.2496.1268.1597.4496.7637.4746.9467.19
ID-Forcing (120 s)58.2396.1368.2097.7796.7665.6572.3789.06
Self-Forcing (240 s)44.9096.2158.3897.9795.5929.6120.9856.25
Pathwise TTC (240 s)54.2996.0067.3397.2896.3835.9642.6759.38
ID-Forcing (240 s)57.7296.1067.8897.6296.7064.7670.3492.19

Q2-2. Sensitivity to LL and ℓ\ell

We vary (L,ℓ)(L, \ell) on Self-Forcing (120 s); ℓ\ell includes the sink, i.e., the KV of the first chunk, and L−ℓL-\ell entries are re-cached autoregressively.

Tab. H. Ablation on (L,ℓ)(L, \ell) (Self-Forcing, 120 s). (Tab. 3, Sec. 4.3)

(L,ℓ)(L, \ell)L−ℓL-\ellAestheticBackgroundImagingMotionSubjectDynamicColor driftMotion drift
Self-Forcing-49.9596.1761.6098.2696.2030.0425.5446.88
(3, 2) default158.2396.1368.2097.7796.7665.6572.3789.06
(3, 3)058.0996.1267.8797.6296.6968.2271.1889.06
(2, 2)058.5596.1566.9997.7996.8973.4578.7692.97
(4, 2)256.8496.0466.6897.7896.6362.7566.6884.38
(3, 1) w/o self-caching251.3395.5664.4797.5795.9459.2352.4378.12

Except for (3,1)(3, 1), all settings remain robust to drift, with Color Drift of 66.68–78.76, above every baseline on Self-Forcing at 120 s in Tab. 1 (at most 59.65); the default was fixed in advance without tuning. (3,1)(3, 1) keeps only the sink context-free, i.e., has no self-caching, and caches the other entries on top of the sink; it becomes unstable (Color Drift 52.43), showing that self-caching plays a role that the sink cannot replace and is the key to stability (Sec. 4.3).

Q2-3. LongLive with ℓ=L=3\ell = L = 3

Correct. With ℓ=L\ell = L, every entry is self-cached and none is cached autoregressively, consistent with Eq. 4 (Sec. 4.1). Such settings also work well on Self-Forcing ((3,3)(3, 3) and (2,2)(2, 2) in Tab. H).

All of these results are now included in the revised paper. We hope these additional experiments address the reviewer's concerns.

Reviewer_6HaZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors address the topic of drifting in autoregressive video diffusion. They show that drifting causes where colors and textures shift, and motion dynamics decay, thus requiring modification to the KV cache. The authors observe that the an in-distribution issue for KV conditioning, thus they propose In-Distribution Forcing aligns both KV caching and KV conditioning with training configurations.

Critical strengths
  • Well motivated and well articulated problem
  • Clean insight which is shown explained well from the beginning
  • Strong qualitative evaluation section
Critical weaknesses
  • Evaluation: How do the authors explain the marginal gains? Does the work improve on latency? Why are the results so largely varying between tasks (background, motion etc.). The current numbers are not very convincing
  • Novelty: From my understanding this is just the common attention sink problem (well-known in the KV cache space: StreamingLLM, SnapKV) applied at write time to find the most relevant tokens in the KV cache for the next decode. For me this strongly limits the main novelty claim of the main work. This needs to be addressed what the main differences are and what new insight this additionally brings.

Otherwise I think its a good work but it requires to answer these two points currently. I am generally open to changing my ranking if valid points are made by the authors.

Questions for authors

Have been asked under weakness section.

3 comments

We thank the reviewer for the constructive feedback and for being open to updating the score. We address the evaluation and novelty concerns below. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with 6HaZ in orange.

W1-(1). Marginal gains

VBench quality dimensions are short-clip metrics close to saturation, whereas ID-Forcing targets long-horizon drift. On the metrics that measure drift, the gains are large (Tab. 1). (a) Color Drift: best in all four settings, by +7.8 to +12.7 over the second-best on Self-Forcing and +5.5 to +7.5 on LongLive. (b) Motion Drift: best on Self-Forcing (+3.9), and within 0.8 of the best (MemRoPE) on LongLive. (c) Dynamic Degree: highest in all four settings, e.g., 30.04 → 65.65 on Self-Forcing (120 s). On all other metrics, ID-Forcing is comparable to or better than the baselines, within 0.5 of the best when it is not the best. In the user study, participants preferred ID-Forcing over every baseline on every criterion (Overall Preference 86–100%, Tab. 2).

W1-(2). Latency

Yes. For generating one chunk, ID-Forcing has 16% lower latency and higher throughput than Self-Forcing (Tab. A), even though this includes the autoregressive re-caching of the tail, which accounts for 14.5% of its latency (App. A).

Tab. A. Chunk generation latency and throughput on Self-Forcing. (Tab. 9, App. A)

MethodLatency / chunk (ms)FPS (DiT)FPS (w/ VAE)
Self-Forcing1634 (1.00×)7.35.4
Deep Forcing1554 (0.95×)7.75.6
∞-RoPE1007 (0.62×)11.97.6
MemRoPE1476 (0.90×)8.15.8
ID-Forcing1368 (0.84×)8.86.2

W1-(3). Variation across tasks

The only metrics on which ID-Forcing is below the best in every setting are Background Consistency and Motion Smoothness, by at most 0.5. These two metrics assign high scores to static or collapsed videos, which is why VBench pairs them with Dynamic Degree. Accordingly, the base models without drift mitigation (Self-Forcing and LongLive) score best on them in most (7 of 8) cases in Tab. 1, while having the lowest Dynamic Degree. Even on LongLive, which is trained on minute-long videos and thus drifts less, ID-Forcing achieves the best Color Drift and Dynamic Degree in every setting (Tab. 1).

W2. Novelty

We would like to clarify that self-caching is not a form of KV selection. KV selection methods such as StreamingLLM and SnapKV decide which KV entries are kept for later reading, based on attention sinks or attention scores, i.e., KV conditioning. Self-caching does not decide which entries are kept or read; it decides how each entry is written. The context a chunk attends to while being cached is not chosen by relevance but fixed to match training.

Our novelty is as follows. To our knowledge, we are the first to (1) identify KV caching as a source of drift separate from KV conditioning, formalizing it as the KV-provenance problem, and (2) show that KV caching must also be addressed for test-time extrapolation, proposing self-caching to resolve it. In training, the first chunk is cached without past context, and each later chunk is cached on top of the chunks that directly precede it; self-caching keeps this structure at test time, so every entry other than the sink is either cached without past context or cached on top of the chunks right before it. ID-Forcing outperforms existing test-time extrapolation methods, with the best Color Drift and Dynamic Degree in every setting (Tab. 1) and a consistent preference in the user study (Tab. 2).

If self-caching were an attention sink applied at write time, starting the caching from the sink would suffice. However, starting the autoregressive caching from the sink instead of a self-cached entry, i.e., (L,ℓ)=(3,1)(L, \ell) = (3, 1), drops Color Drift from 72.37 to 52.43 (Tab. 3, Sec. 4.3). Attaching the sink at write time therefore cannot replace self-caching; the sink helps when it is kept separate and read during generation, as in our default setting.

We would greatly appreciate it if the reviewer could take our novelty into account, and we hope these clarifications address the reviewer's concerns.

Dear Reviewer 6HaZ,

Thank you again for your constructive review and for being open to updating your evaluation. As the author–reviewer discussion period ends on November 18 (AoE), we would be grateful if you could let us know whether our responses have addressed your two concerns.

On the evaluation, we showed that the gains are large on the metrics that measure drift, and that ID-Forcing has 16% lower latency than Self-Forcing. On the novelty, we clarified that self-caching is not an attention sink applied at write time: it decides how each entry is written, not which entries are kept or read, and starting the caching from the sink instead drops Color Drift from 72.37 to 52.43. We have also revised the paper accordingly. We would be happy to clarify any remaining questions.

Dear Reviewer 6HaZ,

Thank you again for your time and for your openness to updating your evaluation. As only about a day remains in the discussion (it closes at the end of November 18, AoE), we would be grateful to know whether our responses on the evaluation and the novelty of self-caching have addressed your concerns. We would be happy to clarify any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their constructive feedback. All revisions are shown in blue in the revised PDF and tagged with the reviewer ID (GebS, H31X, 6HaZ).

1. Scope of the in-distribution claim [GebS, H31X]

  • Stated as a claim on KV operations; "exactly in-distribution" removed: Abstract, Sec. 1, Sec. 3 (incl. Obs. 2), Sec. 5, Sec. 6

2. Relation to prior conditioning [GebS]

  • Frame sink of LongLive and RoPE adjustment of Rolling Forcing acknowledged: Sec. 3.2

3. New experiments

  • KV feature statistics [GebS, H31X]: Sec. 4.4, Fig. 6
  • Training with self-caching [GebS]: Sec. 4.4, App. A, Tab. 5
  • Interactive prompt switching [H31X]: Sec. 4.2, App. A, Tab. 6–8
  • Latency and throughput [H31X, 6HaZ]: Sec. 4.2, App. A, Tab. 9
  • Pathwise TTC baseline [H31X]: Sec. 4.1, Tab. 1
  • Window configuration (L,ℓ)(L, \ell) [H31X]: Sec. 4.3, Tab. 3 (bottom)

4. Moved from the appendix to the main paper

  • Self-caching ablation under the same conditioning [GebS, 6HaZ]: Sec. 4.3, Tab. 3 (top)

5. Clarification

  • LongLive setting ℓ=L=3\ell = L = 3 [H31X]: Sec. 4.1
AuthorsOpenReview ↗

Thank you to the AC and reviewers

We sincerely thank the Area Chair for handling our submission and all reviewers for their time and constructive feedback throughout the discussion. The reviewers' suggestions led to new experiments and clarifications that have substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.