- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces In-Distribution Forcing (ID-Forcing), a training-free method for extending autoregressive video generation beyond the training horizon. The method self-caches historical chunks without attending to preceding entries, retains the first chunk as a persistent condition, and re-rotates temporal positions within a bounded window. Experiments on Self-Forcing and LongLive report improved long-video quality and dynamics at 120 and 240 seconds, alongside color- and motion-drift scores and human preference results.
Critical strengths
- A simple training-free intervention. The method requires no model retraining or additional learned parameters and reports improved long-horizon generation quality and dynamics on Self-Forcing and LongLive.
Critical weaknesses
-
The claim of exact training-distribution alignment is not justified. For noninitial chunks, Self-Forcing constructs cached representations while attending to previously generated history. ID-Forcing changes this policy by encoding selected chunks without preceding context. Although context-free caching occurs for the initial chunk during training, applying the same operation to arbitrary later chunks does not establish that their inputs, cached features, or joint distribution match those encountered during training. Matching cache-operation configurations and positional ranges is therefore insufficient to justify the stronger claim that inference remains exactly in-distribution. Training with the same caching policy would also provide a useful control for evaluating the effect of this change.
-
The conditioning components substantially overlap with prior work. LongLive already retains the first frame chunk as a persistent attention sink alongside a short rolling window. Rolling Forcing also combines initial-frame sinks and recent-history caching with dynamic RoPE adjustment to keep relative positional offsets within the trained range. Consequently, the proposed First Chunk as a Persistent Condition and Bounded Window and Exact Rolling mechanisms appear closely related to established techniques. The paper should clarify the precise technical differences and distinguish the contribution of self-caching from these existing conditioning strategies.
Questions for authors
-
Does the exact in-distribution claim concern only cache-operation configurations, or the actual joint distribution of cached representations and generated content? What assumptions or empirical measurements justify extending context-free caching from initial to later chunks? Could the authors compare against a model trained with the same caching policy to assess the effect of changing that policy only at inference?
-
What are the precise differences between the proposed persistent first chunk and bounded position mechanisms and LongLive's frame sink and Rolling Forcing's sink and dynamic-RoPE design? What additional technical contribution remains beyond the self-caching?
5 comments
[W1, Q1]-2. Extending context-free caching to later chunks
Extending context-free caching is based on step (a), which holds by construction: context-free caching is an operation that training executes for the first chunk, so applying it to later chunks keeps every caching operation within those executed in training. Its empirical justification is steps (b) and (c) above: drift is largely alleviated (Tab. 1), and the KV gap stays flat up to 240 s (Tab. A).
[W1, Q1]-3. Control trained with the same caching policy
Training with self-caching brings no consistent gain over applying it only at inference (Tab. C). We compare three setups, denoted (Training, Inference): (SF, SF), (SF, IDF), and (SF + SC, IDF), where SF, IDF, and SC denote Self-Forcing, ID-Forcing, and self-caching, and SF + SC applies self-caching during SF training. Due to the rebuttal period, SF and SF + SC are trained for 400 iterations with batch size 8 (default: 600 iterations, batch size 64), otherwise following the original recipe. All are evaluated on 120 s videos.
Tab. C. Effect of applying the self-caching policy only at inference (120 s). (Tab. 5, App. A)
| Training | Inference | Aesthetic | Background | Imaging | Motion | Subject | Dynamic | Color drift | Motion drift |
|---|---|---|---|---|---|---|---|---|---|
| SF | SF | 48.77 | 96.26 | 61.87 | 97.88 | 96.19 | 32.01 | 21.57 | 51.56 |
| SF | IDF | 56.01 | 95.74 | 67.67 | 97.39 | 96.40 | 62.45 | 70.99 | 88.28 |
| SF + SC | IDF | 55.71 | 96.44 | 67.58 | 97.44 | 97.47 | 57.54 | 72.07 | 85.16 |
(SF, IDF) and (SF + SC, IDF) both improve substantially over (SF, SF), and (SF + SC, IDF) brings no consistent gain over (SF, IDF) at this training budget, so ID-Forcing retains its benefit without retraining (Sec. 4.4; App. A).
W2, Q2. Relation to LongLive and Rolling Forcing, and the remaining contribution
Contribution. To our knowledge, we are the first to (1) identify KV caching as a source of drift separate from KV conditioning, formalizing it as the KV-provenance problem, and (2) show that KV caching must also be addressed for test-time extrapolation, proposing self-caching to resolve it. Self-caching substantially alleviates drift (Tab. 3), and ID-Forcing outperforms existing test-time extrapolation methods, with the best Color Drift and Dynamic Degree in every setting (Tab. 1) and a consistent preference in the user study (Tab. 2). We would greatly appreciate it if the reviewer could take these two contributions into account.
Conditioning. We agree with the reviewer that our conditioning rule, although derived from the training–test gap, coincides in practice with the frame sink of LongLive and the temporal RoPE adjustment of Rolling Forcing; we now acknowledge both explicitly in the revised paper (Sec. 3.2).
Isolating self-caching. Tab. 3 adds self-caching while keeping our conditioning (sink + re-rotation) fixed: it improves every metric, most significantly Color Drift, from 60.66 to 72.37. We have moved this ablation from the appendix to the main paper (Tab. 3, Sec. 4.3).
We hope these clarifications and additional experiments address the reviewer's concerns.
Dear Reviewer GebS,
Thank you again for your time and constructive review. As the author–reviewer discussion period ends on November 18 (AoE), we would be grateful if you could let us know whether our responses have addressed your concerns.
In particular, we clarified the scope of our in-distribution claim, added KV feature statistics and a control trained with self-caching, and clarified the relation to LongLive and Rolling Forcing; all of these are incorporated into the revised paper. If they address your concerns, we would appreciate it if you could take them into account in your evaluation. We would be happy to clarify any remaining questions.
We thank the reviewer for the careful reading and constructive comments, in particular on the scope of the in-distribution claim and the relation to LongLive and Rolling Forcing. All revisions are shown in blue in the revised PDF, and those addressing your comments are marked with GebS in purple.
[W1, Q1]-1. Scope of the in-distribution claim
We thank the reviewer for raising this point. We state our in-distribution claim precisely in terms of the structure of KV operations, and add KV feature statistics that check whether the resulting KV entries stay close to those within the training horizon. Our argument proceeds in three steps: (a) every KV operation in ID-Forcing is one that training executes, which holds by construction; (b) as a result, drift is largely alleviated; and (c) in turn, the KV entries that the model reads stay closer to the in-horizon KV than with autoregressive caching. Step (a) is the design principle of ID-Forcing, and steps (b) and (c) are measured outcomes.
(a) In-distribution KV operations. ID-Forcing ensures that (i) every KV entry is written by an operation executed in training, either context-free (self-caching), or autoregressively on top of the chunks right before it, starting from a context-free entry, and (ii) every window contains the sink and keeps all relative positions within the trained range. As the reviewer points out, this does not imply that the inputs, cached features, or their joint distribution match those in training, and we do not claim so. We therefore remove "exactly in-distribution" and rephrase content-level expressions such as "OOD KV entries" as statements on caching operations, e.g., "KV entries written by out-of-distribution caching operations" (Abstract; Sec. 1; Sec. 3; Sec. 5; Sec. 6).
(b) Drift alleviation. With KV operations kept in-distribution, ID-Forcing largely alleviates drift both quantitatively and qualitatively (Tab. 1, 3, 4; Fig. 4, 5).
(c) KV feature statistics. We then check whether the alleviated drift is also reflected in the KV entries that the model reads: with ID-Forcing, they stay closer to the in-horizon KV, and their gap stays flat up to 240 s (Tab. A, B). Specifically, we measure the gap between the KV feature statistics of 7-chunk intervals ending at 30, 60, 120, and 240 s and those of the first 7 chunks of the same video (the training horizon): Δμ, the L1 distance between the channel-wise mean vectors (averaged over layers), and the ratio σref/σ of the channel-wise standard deviations. The statistics are computed for each layer over the tokens of pre-RoPE keys and values, on one 240 s video for each of the 128 prompts. For ID-Forcing, we report the context-free (self-cached) entry, the tail entry re-cached autoregressively on top of it, and their average.
Tab. A. Mean gap Δμ (K / V) to the first 7 chunks. Lower is better. (Fig. 6, Sec. 4.4)
Tab. B. Standard deviation ratio σref/σ (K / V). 1 is identical. (Fig. 6, Sec. 4.4)
With ID-Forcing, the KV feature statistics stay closer to the in-horizon KV compared to other baselines. Moreover, the gap stays flat up to 240 s, whereas it more than doubles for AR (Self-Forcing) and grows by 12–14% for AR + sink + re-rotation until 120 s, staying over 20% above ours at 240 s. This confirms step (c). The standard deviation stays close to the reference except for AR (Self-Forcing) (Sec. 4.4).