acceptodds
Under review as a conference paper at ICLR 2027

Connected Self Forcing: Beyond Local Learning for Autoregressive Video Generation

Abstract

To stream long videos while maintaining visual quality and temporal consistency, Self Forcing mitigates exposure bias through self-rollout training on self-generated histories with key-value (KV) caching. To keep memory manageable, it detaches historical caches, preserving forward dependencies between chunks but severing the backward gradient paths. We introduce Connected Self Forcing, a training framework that reconnects gradient paths across autoregressive chunks, allowing feedback from later predictions to guide how earlier context is generated. These connections go beyond historical KV-writing: gradients pass through generated latents into the computations that produced them, linking the generation of earlier context to its use in later predictions. To make this connected training memory-efficient, we develop shortcut gradient replay, which recovers cross-chunk gradients without retaining the full rollout computation graph. Integrated with distribution matching distillation, Connected Self Forcing trains historical chunks according to both their direct supervision and their contribution to subsequent generation. Experiments on autoregressive video generation show improvements in long-horizon visual quality and temporal consistency, without changing the inference procedure.

Primary area:
generative models
Keywords:
autoregressive video generation, long video generation, self-forcing, gradient replay

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_LRP8OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Connected Self Forcing (CSF), a training framework for improving long-horizon autoregressive video generation. It propagates feedback from later chunks through historical KV writing and into the computations that generated earlier chunks. Shortcut Gradient Replay recovers selected cross-chunk gradient paths through local recomputation without retaining the full rollout graph. Experiments under multiple initializations report improvements in visual quality and temporal consistency for 60- and 240-second videos, while the inference procedure remains unchanged.

Critical strengths
  1. The paper is well written and easy to follow, with clear motivation and an accessible explanation of the proposed training framework.
  2. The motivation is closely aligned with the proposed method. Reconnecting future-to-history gradients directly addresses the missing supervision of historical generation caused by detached KV caches.
  3. The experiments cover Causal CD, TF, and Causal ODE initializations. The consistently higher Avg.(6) scores across these settings support the effectiveness of the method beyond a single initialization (Tables 1 and 2).
Critical weaknesses
  1. The novelty appears relatively limited. Overall, the method seems to be an incremental extension of SGF, mainly propagating future feedback beyond KV writing into historical latent generation (Section 3.3).
  2. The evaluation is not sufficiently comprehensive. While Appendix D.2 provides full VBench results for short videos, the long-video evaluation reports only a subset of dimensions. Additional assessments, such as VisionReward [1] scores or human preference evaluations, are also missing.
  3. The main experimental comparisons are limited to SF and SGF, leaving the method's competitiveness against a broader range of autoregressive video generation approaches unclear.
  4. The paper has a strong related work section (History modeling), but it could benefit from a brief discussion of Dummy Forcing [2], Light Forcing [3], and HeadCast [4], particularly their approaches to historical context and attention.
  5. The experiments primarily focus on single-shot video generation. Could the authors also evaluate the method on multi-shot video generation tasks to assess its generalizability?

[1] Jiazheng Xu et al. VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(13): 11269–11277, 2026.

[2] Hang Guo et al. Efficient Autoregressive Video Diffusion with Dummy Head. arXiv preprint arXiv:2601.20499, 2026.

[3] Chengtao Lv et al. Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention. Proceedings of the 43rd International Conference on Machine Learning, 2026.

[4] Jinliang Shen et al. HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation. arXiv preprint arXiv:2607.20125, 2026.

Questions for authors

See Weaknesses.

Reviewer_gL8ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses a limitation of Self Forcing: during training, historical KV-cache states are detached to reduce memory consumption. Consequently, although earlier generated chunks influence later chunks during the generation. In previous methods, gradients from later predictions cannot generally propagate backward to improve the generation of those earlier chunks. The authors propose Connected Self Forcing (an extension for Self-Forcing method), which restores selected cross-chunk gradient paths through a technique called Shortcut Gradient Replay.

Rather than performing full backpropagation through the entire autoregressive sequence, CSF replays selected computations, allowing gradients from later predictions to flow through historical KV states and into the generator computations responsible for earlier chunks.

The method introduces no additional model parameters and does not change inference.

Critical strengths

The paper identifies a meaningful limitation of Self Forcing, detaching their KV-cache representations prevents gradients from later predictions from inflencing the generation of earlier chunks. The proposed CSF addresses this limitation by restoring selected cross-chunk gradient paths, allowing earlier chunks to be optimized both for their own quality and for their contribution to subsequent generation.

Although the method introduces no additional model parameters or inference-time computational overhead (NFE), these benefits come at the cost of increased training time—approximately 27% for chunk-wise and 41% for frame-wise generation.

Evaluation: Their main quantitative comparisons are against Self Forcing (SF) and Self Gradient Forcing (SGF). They retrain both baselines using matched backbones, initializations, training prompts, update budgets, and inference configurations. This is a useful controlled comparison designed to isolate differences in training.

Critical weaknesses
  1. Limited novelty and scientific insight. While propagating gradients into earlier generated chunks is interesting, the approach is closely related to Self Gradient Forcing. Beyond incremental benchmark improvements, the paper should better clarify its methodological novelty and I doubt what new general understanding it provides about autoregressive video generation.
  2. Limited generality. Experiments are restricted to Wan2.1-1.3B. Evaluations on at least two additional backbones outside the Wan family would strengthen the claim and the generalazability of the approach, specifically showing that the method is not overfitted to WAN architecture.
  3. Consistency–dynamics trade-off. Improved consistency often coincides with substantially reduced Dynamic Degree, raising concerns that the method favors more static generation. Moreover, the qualitative results still exhibit noticeable artifacts. Video results in the suppl. Material shows a series of short clip examples in a video. The examples are not really long. Presenting 60-120 sec videos is needed to asses the validate the paper’s claim on motion quality and temporal consistency beyond the short clips.
  4. Presentation quality. The paper would benefit from clearer writing and organization. Some passages are unnecessarily cumbersome (e.g., lines 71–75), several acronyms are insufficiently introduced, and initialization schemes such as Causal CD and Causal ODE should be explained in the main text before appearing in Table 1.
Questions for authors

Does CSF’s improvement in long-horizon quality remain when inference uses compressed historical context and sparse attention?

Reviewer_pJ54OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Connected Self Forcing, which lets later chunk update earlier context generation through historical KV states. This is enabled by using Shortcut Gradient Replay which recovers selected cross-chunk gradient paths. Experiments show improved long-video quality and consistency over Self Forcing

Critical strengths
  1. I think the main idea makes sense. An earlier chunk is both an output and context for later generation, so it should receive feedback from how well later chunks use it. The replay mechanism addresses this directly.

  2. The experiments are solid, with controlled SF and SGF comparisons. I also like the comparisons with Full BPTT which shows the benefit of Connected Self Forcing.

Critical weaknesses

My main concern is the trade-off between consistency and motion. I looked at the supporting material videos; it looks like the motion is minimal, and I still see drifting around the 50-60s area, especially in the two-dog neon city case.

Questions for authors

Could you show comparisons with baselines on prompts requiring substantial motion?

Comments on the paper

2 comments

Hi Reviewers,

The authors have posted their rebuttal along with new results. Please review their responses, engage with the authors to discuss any remaining concerns.

Best, Area Chair

Reviewer_gL8ROpenReview ↗

Further Assesement

I appreciate the authors’ detailed response, which addresses several of my original concerns. In particular, the additional experiments on Wan2.2-5B and LTX-Video 2B strengthen the evidence for the method’s generalizability, although the reported improvements remain relatively modest.

The additional qualitative results provide useful supporting evidence. Showing some of the compared methods have noticeable artifacts and degradation after approximately 30 seconds of generation, highlighting the challenges of maintaining long-term visual consistency.

Overall, the rebuttal meaningfully strengthens the paper and resolves several of my concerns. While I still have some reservations regarding the magnitude of the improvements in long-horizon generation, I believe the additional evidence is sufficient to warrant raising my score to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.