acceptodds
Under review as a conference paper at ICLR 2027

Local Forcing: Local KV-Cache Construction for Long Video Generation

Abstract

One of the central challenges in causal video generation is error accumulation. We find that attending to history during key–value (KV) cache construction may exacerbate error accumulation, even though the same context is essential for denoising. This observation motivates Local Forcing (LF), which follows a simple principle: generate with context; encode locally. This principle is implemented through an attention-mask modification applied consistently across teacher forcing, consistency distillation, self-rollout distillation, and inference. Surprisingly, LF improves robustness to drift, enabling stable minute-level rollouts without specialized attention-sink strategies, additional memory modules, or long-horizon distribution matching distillation. The simplicity of LF facilitates integration with both training-aware and training-free approaches to long-video generation, yielding consistent improvements in overall video quality across the evaluated methods. These findings position local cache construction as a foundational design principle for scaling causal video generation far beyond its training horizon.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Long Video Generation, KV Cache Writing, Temporal Consistency, Error Accumulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2De6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies KV-cache construction in autoregressive video diffusion. It proposes Local Forcing, which retains historical attention during denoising but restricts cache construction to the current completed block. The same attention rule is incorporated into teacher forcing, consistency distillation, self-rollout distillation, and inference. Experiments evaluate its effect on long-video quality, its compatibility with existing training frameworks and inference methods, and the influence of cache-construction locality through controlled ablations.

Critical strengths
  1. The distinction between using history to generate content and using history to encode completed content is well motivated. The intervention is simple, adds no parameters or losses, and preserves the existing cache interface. Importantly, the three-seed locality ablation holds the denoising context fixed and shows that block-local encoding improves over both full-context and sink-isolated encoding (Table 4). This provides useful evidence that the benefit goes beyond excluding sink frames.
  2. The empirical benefits extend across several settings rather than relying on a single configuration. For Causal Forcing, LF improves Quality Score by 1.53 points at 60 seconds and 2.67 points at 240 seconds, with all seven evaluated dimensions improving at the latter horizon (Tables 2–3). Its compatibility with Deep Forcing, Infinity-RoPE, and MemRoPE further supports its practical value as a component of existing long-video generation systems.
Critical weaknesses
  1. The methodological novelty is modest. Relative to the sink-exclusion strategy discussed for Rolling Forcing, the main additional change is removing recent-history attention during cache construction and applying this rule consistently across training stages. This is a useful refinement of existing autoregressive diffusion pipelines, but it does not introduce a substantially new modeling or optimization framework. The contribution therefore rests primarily on the insight and the empirical evidence for its usefulness.
  2. The proposed error-propagation explanation is plausible but less firmly established than the quality improvements. Appendix S10 shows that local cache construction removes direct dependence on historical K/V when the current block is fixed. This establishes locality by construction, but does not show how strongly the removed pathway contributes to drift in the baseline. Since the generated block still depends on history, the current evidence does not clearly distinguish reduced recursive cache contamination from other effects of restricting context during training. This limits the mechanistic conclusion, rather than invalidating the observed gains.
Questions for authors
  1. Is there a controlled diagnostic showing that perturbations in historical K/V propagate through cache construction and materially affect downstream drift? This would help connect the conditional-locality analysis in Appendix S10 to the proposed explanation for the empirical improvements.
  2. Could you report quantitative results for the training-rule × inference-rule crossover mentioned in Appendix S11, particularly for appearance drift and block-boundary continuity? This would clarify the respective contributions of local encoding and adaptation to the changed encoding rule.
3 comments

W1: Contribution relative to Rolling Forcing.

We consider Rolling Forcing as an important relevant work. Its Appendix A describes excluding global context during cache updates while retaining recent context. However, the implications of this choice for long-rollout stability remain underexplored. Representative open-source methods—including Causal Forcing, Reward Forcing, Self Gradient Forcing and training-free methods such as Deep Forcing, Infinity-RoPE, and MemRoPE—continue to use historical context during cache construction without a separate restriction for this pass.

Our contribution is to identify and systematically study the different context requirements of denoising and cache construction. LF preserves historical attention during denoising but restricts cache construction to the completed block. Our controlled diagnostic in Appendix S10 supports historical error transmission through cache construction, providing a mechanistic basis for this separation. The same rule applies across training stages and inference without adding parameters, losses, or forward passes.

The ablations also go beyond sink exclusion: the matched three-seed comparison shows additional Quality gains of 0.41 and 1.19 points at 10 and 20 seconds (Appendix S5.1). The new history-length ablation further shows that zero-history construction achieves the highest Quality at both durations, outperforming one, two, and all available preceding blocks (Appendix S5.2).

W2 / Q1: Historical K/V propagation and downstream drift.

We have added quantitative perturbation experiments to Appendix S10. Using the same Causal Forcing checkpoint and fixed video blocks across six preselected VBench prompts, we perturb only historical K/V with repeated random bias calibrated to VAE round-trip variation. At the final measured block, the newly constructed cache inherits a deviation averaging 0.357 times the injection RMS under native construction (range: 0.329–0.381), versus zero under LF, measured before the current block’s injection. A separate single-injection probe shows that the perturbation persists through subsequent cache updates after the affected block is evicted and influences denoising predictions at identical noisy inputs.

For downstream drift, the Base training / Base inference versus Base training / LF inference comparison in Q2 changes only the cache-construction rule, keeping model weights and denoising attention unchanged. Saturation drift decreases from 0.05140 to 0.02019 (−60.7%). Together, these experiments demonstrate historical perturbation transmission through cache construction and support the benefit of removing this direct dependence during generation, independently of LF training.

Q2: Training-rule × inference-rule crossover.

We use Causal Forcing as the base model and evaluate three combinations (Appendix S11). Appearance drift is the absolute saturation change between the first and last 10 seconds of 60-second videos generated from official VBench Long prompts. Boundary continuity uses VBench’s AMT-based smoothness score at the 13 block boundaries within the first 10 seconds. We focus on this early interval to reduce coupling with accumulated appearance drift.

TrainingInferenceSaturation drift ↓Boundary smoothness score ↑
BaseBase0.0514095.2670
BaseLF0.0201994.9518
LFLF0.0199295.2918

Inference-only LF substantially reduces appearance drift. Using LF in both training and inference preserves this reduction (61.2% versus the baseline) and alleviates the decrease in boundary smoothness observed with inference-only LF, bringing boundary smoothness back to the baseline level.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns.

The new ablations help further clarify the contribution of Local Forcing beyond existing approaches such as Rolling Forcing. In particular, the historical KV perturbation experiments provide more direct evidence for error propagation through cache construction, while the training/inference crossover experiments help distinguish the benefits of local encoding during inference from those of adapting the model during training. These results strengthen the empirical support for the proposed method and its underlying motivation.

I have also reviewed the authors' responses to the other reviewers and found no additional concerns that would change my overall assessment. While the methodological novelty remains relatively modest, I believe the rebuttal has adequately addressed my main questions and strengthened the paper's experimental justification. Therefore, I will maintain my original score of 3 (Weak Acceptance).

Thank you for recognizing the value of our approach and for taking the time to read both our new results and our responses to the other reviewers. Your feedback helped us clarify how cache construction affects drift, and we’ll make that connection clearer in the next version of the paper.

Reviewer_GA1qOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Local Forcing separates contextual denoising from local KV-cache construction to reduce drift in autoregressive video generation. It changes the attention mask without adding parameters, losses, or forward passes.

Critical strengths

The design is simple and supported by matched teacher-forcing experiments across three training seeds. Results show higher aggregate quality across several training and inference pipelines, including 240-second rollouts, alongside a practical speedup in teacher-forcing training.

Critical weaknesses

The method is practical and the empirical results are encouraging, though the smaller gains need uncertainty estimates and the long-horizon behavior deserves more targeted evaluation. The inference-method comparisons demonstrate compatibility, while reusing the same CF checkpoint pair limits their independence.

Questions for authors
  1. How stable are the smaller Quality Score gains across seeds? Paired confidence intervals and additional seeds on a representative subset would help establish their reliability.
  2. Beyond the conditional locality established in S10, how much does LF reduce end-to-end error amplification when historical perturbations also affect subsequent generated blocks? A controlled experiment would strengthen the proposed explanation for reduced drift.
  3. Could quality and temporal smoothness be evaluated separately over late rollout windows? Cumulative scores and sparse-frame Astra preferences may obscure late-stage degradation or block-boundary jitter.
2 comments

W1 / Q1: Stability of Quality Score gains and paired confidence intervals.

Thank you for raising this concern. Our teacher-forcing experiments show consistent Quality improvements across three training seeds (Appendix S5.1), different context windows (S6), frame-wise generation (S7), and the Wan2.2-5B backbone (S8). At the DMD stage, improvements also hold across Causal Forcing and Self Forcing initializations (the latter in S3) and when combined with training-free inference methods (Figure 1 and Tables 2–3). These results support a consistent direction of improvement across the evaluated settings.

To quantify uncertainty in the DMD-stage comparisons, we additionally report Quality improvements (LF − Base) with paired 95% percentile confidence intervals from 10,000 prompt-level bootstrap resamples:

Method30s: ΔQuality [95% CI]60s: ΔQuality [95% CI]
Causal Forcing+1.176 [0.662, 1.689]+1.530 [0.978, 2.063]
Self Gradient Forcing+0.709 [0.212, 1.219]+1.308 [0.798, 1.845]
Deep Forcing+0.476 [−0.029, 0.983]+1.301 [0.751, 1.841]
Infinity-RoPE+0.168 [−0.292, 0.606]+0.349 [−0.099, 0.782]
MemRoPE+0.210 [−0.347, 0.746]+0.669 [0.087, 1.208]
Self Forcing+0.529 [−0.075, 1.108]+0.448 [−0.112, 0.992]

The paired analysis shows positive Quality gains in all 12 comparisons. The 95% confidence intervals lie entirely above zero for Causal Forcing and Self Gradient Forcing at both durations, and for Deep Forcing and MemRoPE at 60 seconds. Together with the three-seed teacher-forcing results and evaluations across model and context settings, these findings support the consistency of LF’s improvements, with the clearest statistical evidence in the comparisons above.

Q2: End-to-end error amplification.

Historical errors can propagate through both generated content and subsequent cache construction. LF removes the direct history-to-cache pathway while preserving historical conditioning during denoising; errors already present in generated content are not eliminated by this restriction.

We have supplemented the theoretical analysis in Appendix S10 with a controlled diagnostic using the same Causal Forcing checkpoint and fixed video blocks across VBench prompts. With repeated K/V perturbations calibrated to VAE round-trip variation, the final newly constructed cache inherits a deviation averaging 0.357 times the injection RMS under native construction (range: 0.329–0.381), versus zero under LF, measured before the current block’s injection. A separate single-injection probe shows that perturbations persist after the affected block is evicted and influence denoising predictions at identical noisy inputs.

These findings identify a route for historical error transmission through cache construction. Complementing this fixed-content diagnostic, the generation experiment in Appendix S11 shows that switching only the inference-time cache-construction rule reduces saturation drift by 60.7%, with model weights and the denoising attention rule unchanged.

Q3: Late-window quality and block-boundary jitter.

We evaluate the 50–60-second window using the official VBench Long protocol and prompts to assess late-rollout quality and temporal continuity:

Metric ↑Base training / Base inferenceLF training / LF inference
Quality81.42083.681
Aesthetic quality61.40861.992
Imaging quality68.28167.731
Subject consistency95.09197.452
Background consistency95.73096.425
Motion smoothness85.75496.195
Temporal flickering98.72899.229
Dynamic degree57.50055.833
Boundary smoothness87.70594.965

LF achieves higher overall Quality and substantially higher motion and boundary smoothness. The baseline exhibits pronounced late-window degradation in motion smoothness, contributing to the large smoothness gap. Imaging quality and dynamic degree are slightly lower with LF. In the first 10 seconds, inference-only LF slightly reduces boundary smoothness, while using LF in both training and inference brings it back to the baseline level. The full training–inference comparison is provided in our response to Reviewer 2De6, Q2.

Dear Reviewer GA1q,

As the rebuttal period draws to a close, we wanted to follow up and ask whether our response addressed your concerns. We would be happy to clarify anything further.

Thank you for your time!

Reviewer_sX8gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Local Forcing (LF), a modification to KV-cache construction for autoregressive video diffusion. Existing pipelines typically use historical attention both when denoising a new block and when re-encoding the completed block into cached keys and values. LF separates these operations: denoising retains historical context, while cache construction uses block-local video self-attention. The same rule can be applied during teacher forcing, consistency distillation, self-rollout distillation, and inference without introducing additional parameters, losses, or forward passes.

The paper evaluates LF through controlled teacher-forcing experiments, integration with Causal Forcing and Self Gradient Forcing, and combinations with three long-video inference methods. Supplementary experiments cover Self Forcing, the complete TF–CD–DMD pipeline, alternative context windows, frame-wise generation, and a larger backbone. Results generally support local cache construction as a useful design choice, although the proposed mechanism and the broader claims about long-horizon stability require further validation.

Critical strengths

The idea is very clean, clear and effective. The paper identifies a concrete architectural choice: historical context useful for denoising may not be equally useful when encoding completed content for future use. Separating these roles is conceptually meaningful, and the proposed intervention is simple enough to integrate into existing pipelines.

Critical weaknesses
  1. Limited ablation of cache-construction history length. The locality ablation removes all recent history at once, leaving intermediate context lengths unexplored. Please compare zero, one, two, and all available preceding blocks while fixing denoising context and other settings, with matched training and inference rules. This would clarify whether zero history is preferable to a short history for balancing drift reduction and temporal continuity.

  2. Potential loss of useful historical information. Local cache construction may suppress error propagation while also discarding useful temporal state. Tests involving prolonged occlusion, object reappearance, and scene revisits would clarify this trade-off.

  3. Incomplete related-work discussion. The discussion would benefit from including Stream-R1 and Reward Forcing on reward-guided streaming video distillation, LongLive on streaming training and KV-cache refreshing and long-term consistency. Please clarify how LF’s local cache construction differs from and potentially complements these approaches.

Questions for authors

Please refer to the weakness part.

3 comments

W1: Limited ablation of cache-construction history length.

Thank you for this suggestion. We compare zero, one, two, and all available preceding blocks during cache construction, excluding the current block, using the same experimental settings as in the main paper.

Cache-construction historyQuality @10s ↑Quality @20s ↑
All available preceding blocks (Base)78.85175.198
2 preceding blocks79.47976.390
1 preceding block79.56376.781
0 preceding blocks (LF)80.05477.320

Quality improves monotonically as cache-construction history decreases at both durations. LF also achieves the highest aesthetic quality, imaging quality, subject consistency, and background consistency. These results support block-local cache construction as our default. Full per-dimension results are provided in Appendix S5.2.

W2: Potential loss of useful historical information.

We agree that restricting historical attention during cache construction may omit information useful for occlusion, reappearance, and scene revisits. LF preserves access to historical K/V during denoising, so historical context has already influenced the generated block before it is encoded into the cache. Nevertheless, compared with cache construction over the full retained context window, local construction may omit historical information not represented in the completed block.

Maintaining such information beyond the retained context window is also a challenge for standard autoregressive models such as Self Forcing and Causal Forcing and may require additional memory modules. Since LF modifies only the attention mask used for cache construction, it could be combined with persistent memory that supplies additional context during denoising. We have added this limitation to the revised manuscript (Appendix S13).

W3: Relationship to Stream-R1, Reward Forcing, and LongLive.

Thank you for pointing out these missing references. We have added them to Related Work:

  • LongLive is discussed under Forcing and Learning from Generated Histories for streaming long tuning that improves stability over extended self-generated rollouts.
  • Stream-R1 is cited in the same subsection for reward-guided distillation that improves streaming video quality by weighting rollouts and spatiotemporal regions.
  • Reward Forcing is discussed under Long-Video Generation and Context Design for EMA-Sink, which preserves historical context by aggregating evicted K/V into a fixed-size sink.

Thanks for the author's rebuttal. My concerns have beed addressed well, and I have also read other reviewer's comments and find no major important concerns regarding to this paper. Overall, I find this paper is practical meaningful and useful, therefore I remain my positve score.

Thank you for recognizing the practical value of Local Forcing and for taking the time to read our rebuttal. Your suggestions helped us explain the choice of block-local encoding and its limitations more clearly, and we’ll carry those revisions into the next version of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the reviewer discussion and manuscript revisions

Dear Area Chair and Reviewers,

We are encouraged that all reviewers recognized the insight behind Local Forcing: a model can use history to generate each block, then encode it locally to limit error propagation. sX8g highlighted the simplicity and practical value of this distinction, 2De6 noted that the gains extend beyond sink exclusion, and GA1q highlighted improvements across settings, including 240-second rollouts.

All three reviewers initially recommended Weak Acceptance (3). After the rebuttal, sX8g and 2De6 confirmed that their main questions had been addressed and maintained their scores. GA1q’s posted score remains 3.

In the revised paper, we:

  1. Compared four cache-construction history lengths and found that shorter histories consistently improved quality (Appendix S5.2, sX8g).
  2. Traced how perturbations to historical K/V propagate through cache updates and affect later denoising predictions (Appendix S10, 2De6 and GA1q).
  3. Showed that local encoding at inference reduces saturation drift and that training with the same rule restores early block-boundary smoothness (Appendix S11, 2De6).
  4. Expanded the discussion of LongLive, Stream-R1, and Reward Forcing (Section 2, sX8g).
  5. Discussed limitations of local encoding for prolonged occlusions, object reappearance, and scene revisits (Appendix S13, sX8g).

In our reply to GA1q, we also reported paired confidence intervals for quality gains and found that LF improved overall quality and boundary smoothness late in generation.

We thank the reviewers again for their thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.