acceptodds
Under review as a conference paper at ICLR 2027

Self-Aligned Forcing: Streaming Video Diffusion with Differentiable Noisy History

Abstract

Autoregressive video diffusion enables interactive streaming generation, but suffers from error accumulation over long rollouts. Self-rollout training reduces exposure bias, yet finite rollouts leave long-range drift unresolved. We observe that the noise level of the history key-value (K/V) representations trades visual quality against motion, and that restoring gradients through the history aligns causal train- ing far more closely with bidirectional training. Motivated by these observations, we introduce Self-Aligned Forcing (SAF), a training scheme that aligns the history of each block with the noise level of the block being denoised. Specifically, the history is the K/V produced by preceding blocks at the same denoising stage, so all blocks at a stage can be denoised in a single forward pass under a causal mask. This keeps the noisy history differentiable, allowing future losses to optimize how it is encoded. SAF therefore avoids a separate no-gradient rollout and per-block timestep-zero recaching, training up to 1.8× faster than prior methods with lower memory. At inference, SAF achieves the highest single-GPU throughput among existing methods and keeps one history bank per stage for a multi-GPU pipeline, reaching 49.1 FPS on 4 GPUs. Experiments show superior long-horizon generation with a better balance between visual quality and motion.

Primary area:
generative models
Keywords:
Streaming Video Distillation, Autoregressive Video Diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jSpnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

From my understanding, the paper tries to target long-horizon drift in few-step autoregressive video diffusion distilled with DMD, and uses their two observations (Fig2): 1)at inference a noisier history K/V raises dynamic degree and lowers imaging quality, 2), gradients flow through the history makes the parameter gradient closer to a bidirectional reference. Then they propose SAF which takes each block’s history at the same denoising stage as the block itself. All blocks at a stage can then be denoised in one forward pass. The history K/V is produced inside that pass, so it stays in the graph, and later blocks’ DMD losses reach how earlier blocks write their K/V. Self-rollout runs block-parallel within a stage and sequentially only across the 4 stages, with Self-Forcing-style random exit-stage truncation until a clean sink is recached at the beginning t=0. But at inference, SAF^M keeps one K/V bank per stage. This removes the extra t=0 recache pass. Validation uses VBench, a Interactive benchmark, and MovieGen-100s, against Self-Forcing, LongLive, Causal-Forcing, HiAR and SGF using released weights. And also against their ablations.

Critical strengths
  1. This is "almost for free" differentiable history, since stage alignment removes both the separate no-grad rollout that SGF needs and the per-block t=0 recache. The training-cost numbers in Table 3 are the most convincing result in the paper: 137s vs. 249s per cycle framewise against SGF, with lower memory.
  2. Training sees the same history that SAF^M sees at inference, matching exactly. The text claims, and Alg. 3 supports, that the pipelined and single-GPU multi-bank versions produce identical outputs. Per-stage banks then give stage pipelining for free. This is a cleaner train–test story than HiAR’s or SGF’s, and it is the reason the efficiency gains don’t come with a train–test mismatch (unlike SAF^S, which the authors honestly flag).
Critical weaknesses
  1. the results are very thin and marginal. Chunkwise VBench Total is 85.04 vs. 84.88 (Causal-Forcing) and 84.76 (SGF), from ONE training run and ONE seed, with NO variance reported. Also in table 5, in the framewise setting SAF loses to SGF on Total, Interactive Quality and MovieGen Quality. How does this justify “superior long-horizon generation” in the abstract or “best training and inference efficiency in both settings”?
  2. Seems like the method is still suffering from "motion-fidelity" trade-off, as The “w/o K/V grad” ablations in table 9 show the drop in dynamic degree, while imaging quility and subject consistency is up. IMproving dynamic degree might be a little misleading, as DD is a coarse, flow-thresholded metric that rewards instability as readily as meaningful motion
  3. Novelty is narrower than presented, and related work is missing on the observation and inference side. Observation 1 (partially noised history trades fidelity for motion) closely matches what History-Guided Video Diffusion / DFoT already exploit. That work treats noise as partial masking of history frames and uses guidance over noisy history to improve motion dynamics. It should be cited and discussed. Some related works are FIFO-diffusion (its diagonal denoising), StreamDiffusion (streambatch), MAGI-1 (chunkwiose pipelined denoising), RollingDiffusion
Questions for authors
  1. Can you compare SAF and SGF under the same initialization, data, step budget and checkpoint-selection rule? Table 3 suggests you have an SGF training setup. Without this, It is hard to attribute the Table 1 differences to the method.
  2. Please report the Quality aggregate with dynamic degree removed, or some motion-quality measure that is less gameable, and ideally a small pairwise human study on the 100s videos
  3. For Fig. 2b, what exactly is the “matched-weight bidirectional reference”? Is it the same causal weights run without the mask? Restoring the history path seems almost guaranteed to raise cosine similarity to a gradient that also contains cross-block terms. Why is matching the bidirectional gradient the right target for a model that is deployed causally? How many samples are behind the error bars?
7 comments
AuthorsOpenReview ↗

Response to Reviewer jSpn (1/4)

We thank the reviewer for the detailed and constructive critique, and for recognizing SAF's training efficiency and the train–test consistency of SAF$^{\rm M}$. We address every point; changes in the revised PDF are marked in blue.

W1(a)/Q1. Variance, and a controlled comparison with SGF.

The training settings of the officially released SGF checkpoint are not documented; SGF's official code uses batch size 8 and 1200 iterations (chunkwise), whereas we use 32 and 600. To remove these two possible confounds, we re-trained SGF (chunkwise) under our recipe, using its official code, which we also used for the SGF measurements in Table 3: total batch 32 on 8 GPUs, the same seed and critic warm-up, and the checkpoint rule used for SAF in the paper (EMA at step 600). The VBench results in the paper already average N=5 samples for each of the 944 prompts; we re-ran this evaluation for all three models and report mean ± std over the 5 sample indices (DD: dynamic degree).

Controlled comparison. VBench, chunkwise, 944 prompts × 5 samples.

MethodStepTotalQualityQuality w/o DDSemanticDD
SGF, released checkpoint (Table 1: 84.76 / 85.74 / 80.83)–84.76 ± 0.1685.67 ± 0.1487.24 ± 0.1181.09 ± 0.8766.9 ± 1.2
SGF, re-trained with our recipe60084.76 ± 0.1685.78 ± 0.1787.79 ± 0.0680.66 ± 0.3861.7 ± 1.9
SAF$^{\rm M}$ (Table 1: 85.04 / 86.30 / 80.00)60085.06 ± 0.2586.31 ± 0.1587.09 ± 0.0380.06 ± 0.9176.9 ± 2.1
  • The recipe has no notable effect on SGF's result. Re-trained with our recipe, SGF scores the same Total as the released checkpoint (84.76 for both). The re-run of SAF reproduces Table 1 within 0.02.
  • On VBench (5 s, the training horizon), SAF is slightly higher than SGF in Total, with more motion. The std of Total is 0.15–0.25, comparable to SAF's margin over Causal-Forcing (0.16 in Table 1) and SGF (0.30 in the table above). In a paired t-test over the 5 sample indices (for each index, both models use the same prompts and seed; p < 0.05 is the conventional threshold for significance), SAF's Total is +0.30 over both SGF checkpoints, which is significant against both (p ≈ 0.005 released; p ≈ 0.03 re-trained). This supports the paper's statement that, in the chunkwise setting, SAF$^{\rm M}$ leads the aggregate metrics on all three benchmarks, by a small margin on VBench. The gain comes with higher Quality (+0.64 / +0.53 over the released / re-trained SGF; p ≈ 0.005 / 0.01) and a 10–15 points higher DD (76.9 vs. 66.9 / 61.7), while the other six Quality dimensions together stay within 0.7 (Quality w/o DD: 87.09 vs. 87.24 / 87.79).
  • At long horizons the gains are significant across prompts. Each method produces one video per prompt there, so we use a paired bootstrap over the 100 Interactive-60s prompts (10,000 resamples, 95% CI) and a Wilcoxon signed-rank test on the per-prompt differences. "Prompts won" is the share of prompts on which SAF$^{\rm M}$ scores higher.
SAF$^{\rm M}$ −QualityPrompts won (Quality)Wilcoxon p
SGF+1.86 [+1.41, +2.30]73%5e−9
Causal-Forcing+3.91 [+3.47, +4.36]93%3e−17
Self-Forcing+4.94 [+4.45, +5.43]93%5e−17
HiAR+4.67 [+4.18, +5.13]94%5e−17

The gain is significant against all four baselines: every CI excludes 0, and SAF$^{\rm M}$ wins on 73–94% of the prompts.

W1(b). Claims.

  • Quality. The Introduction already states that SAF "performs best in the chunkwise setting and competitively in the framewise setting". We have aligned the abstract with it. (old): "Experiments show superior long-horizon generation with a better balance between visual quality and motion." → (new): "In the chunkwise setting, SAF achieves the best long-horizon quality with a better balance between visual quality and motion, and it remains competitive in the framewise setting."
  • Efficiency. This statement holds in both settings (Tables 1, 3 and 5). On one GPU, SAF's throughput is slightly above the fastest baseline (22.8–22.9 vs. 20.7 FPS chunkwise; 9.08 vs. 8.9 FPS framewise), and the 4-GPU pipeline raises it to 49.1 and 21.6 FPS. In training, SAF is 1.4× (chunkwise) and 1.8× (framewise) faster than SGF and uses less memory.
AuthorsOpenReview ↗

Response to Reviewer jSpn (2/4)

W2/Q2. Is the gain a motion–fidelity trade-off, and is DD gameable?

In short: against the detached-history baselines, SAF has higher fidelity at matched motion, i.e., on the identical clips in which both models move (defined below), so its gain is not a shift along the trade-off. Against SGF, neither method dominates: SGF has higher fidelity on moving content, while SAF$^{\rm M}$ has 1.6× the motion and better prompt following at 1.4–1.8× lower training cost. We agree that dynamic degree (DD) is coarse and report the measures requested in Q2. In a blind pairwise human study (d), raters prefer SAF$^{\rm M}$ overall against both SGF and Causal-Forcing.

Definition of matched motion. Static clips score higher on consistency and smoothness, so a model with less motion can score higher without having higher fidelity. To remove this effect, (a) and (c) compare two models only on the identical 2-s clips (same prompt, same time) that VBench's per-clip dynamic flag marks as dynamic in both.

(a) The w/o-K/V-grad ablation (Table 9). We compare full SAF and w/o K/V grad. at matched motion. Q5 is the mean of the five dimensions VBench scores on every clip (subject and background consistency, motion smoothness, aesthetic and imaging quality; official normalization).

Interactive, full SAF / w/o K/V grad.Dynamic clipsIdentical clips, both dynamicQ5 on these clipsImaging on these clips
First 20 s (1,000 clips; setting of Table 9)641 / 51348484.57 / 84.3272.23 / 72.46
60 s (3,000 clips)2,219 / 1,5841,32684.18 / 83.6372.11 / 72.04

Where both models move, full SAF has higher Q5 and imaging is on par, so the differentiable history adds motion without lowering fidelity at matched motion. The higher subject consistency of w/o K/V grad. in Table 9 follows from its larger share of static clips, which score higher on consistency within every method (e.g., 97.90 vs. 96.91 for SAF$^{\rm M}$; Table 11b). In our new 60-s evaluation of the ablations, ViCLIP also drifts more without the history gradient: from segment 1 to 6 it changes by −2.38 for full SAF (27.06 → 24.68) and by −3.31 for w/o K/V grad. (26.69 → 23.38).

(b) Quality without DD. We recompute the VBench Quality score with the DD dimension dropped, i.e., the weighted mean of the other six Quality dimensions over all videos (official weights and normalization; chunkwise):

Self-ForcingLongLiveCausal-ForcingHiARSGFSAF$^{\rm S}$SAF$^{\rm M}$
Interactive Quality w/o DD85.5386.7383.7985.5286.7987.0886.56
MovieGen-100s Quality w/o DD82.8986.0381.1685.3486.8386.8886.62
(Interactive DD)22.7327.4756.2026.4348.2350.8773.97
(MovieGen DD)28.9144.4465.6939.8058.1553.9565.61
  • SAF is on the Pareto front at 60 s and 100 s. SAF$^{\rm S},thesingle−bankinferencemodeofthesamecheckpoint,isthehighestwithoutDDonbothbenchmarks,andSAF, the single-bank inference mode of the same checkpoint, is the highest without DD on both benchmarks, and SAF^{\rm M}$ is within 0.23 of the best baseline (SGF) with 1.5× and 1.1× its DD. Every other baseline is lower than a SAF variant in both rows of each benchmark, except for a tie in DD with Causal-Forcing on MovieGen-100s (65.69 vs. 65.61), where SAF$^{\rm M}$ is 5.46 higher in Quality w/o DD.
  • Without DD, the score may favor static videos. VBench itself notes that "a completely static video can score well" on its temporal dimensions [1, Sec. 3.1.1], which trade off against DD across models [1, Sec. 5]. Consistent with this, on Interactive-60s our w/o-sink-recache ablation (newly evaluated at 60 s; full results in our response to Reviewer 7ng6, Q3) is nearly static (DD 5.26 vs. 73.97 for full SAF$^{\rm M}$) and stops following new prompts (ViCLIP 22.73 vs. 25.34), yet its Quality w/o DD is the highest of all models (87.89 vs. 86.56). Thus, while Quality with DD can reward instability, as the reviewer notes, Quality without DD can reward stillness.

(c) A less gameable measure: fidelity at matched motion. We therefore compare SAF$^{\rm M}$ with each baseline on the identical Interactive-60s clips that are dynamic in both, or static in both (3,000 clips per method; flag and Q5 as in (a)).

Interactive-60sDynamic / static clipsQ5 gap, both dynamic (clips)Q5 gap, both static (clips)Q5 gap, both dynamic: first / last 30 sMotion smooth., dynamic clips
SAF$^{\rm M}$2,219 / 781–––98.68
SGF1,357 / 1,643−0.86 (1,197)−0.20 (621)−0.83 / −0.8998.77
HiAR793 / 2,207+0.96 (670)+1.63 (658)+0.93 / +0.9898.17
Causal-Forcing1,686 / 1,314+3.20 (1,396)+2.54 (491)+2.42 / +3.8896.96
Self-Forcing682 / 2,318+1.44 (575)+1.44 (674)+0.51 / +2.7998.42
AuthorsOpenReview ↗

Response to Reviewer jSpn (3/4)

Q5 gap = SAF$^{\rm M}$ minus the method, on the identical clips that are dynamic (or static) in both; the number of such clips is in parentheses. Motion smoothness uses each method's own dynamic clips. All numbers come from the per-clip scores behind Table 11 (Appendix D).

  • Against the detached-history methods, SAF$^{\rm M}$ is higher at matched motion, on moving and static content alike. This includes HiAR, whose history is noisy, and the lead over Self-Forcing and Causal-Forcing grows over time.
  • SAF and SGF: neither dominates. SGF is 0.86 higher in Q5 where both move and 0.20 higher where both are static, and the gap does not grow over time, so SAF$^{\rm M}$ accumulates about as little error as SGF. SAF$^{\rm M}$ in turn has 1.6× the motion((2,219 vs. 1,357 dynamic clips)) and better prompt following (60-s ViCLIP 25.34 vs. 24.63; Table 1), at 1.4–1.8× lower training cost (Table 3).
  • SAF's motion is not jitter. On dynamic clips, its motion smoothness is on par with SGF and above the other three baselines; unstable motion would lower it.

(d) Blind pairwise human study. As suggested in Q2, we ran a two-alternative forced choice (2AFC) study on 8 randomly sampled MovieGen-100s prompts, comparing SAF$^{\rm M}$ with SGF (the strongest baseline) and Causal-Forcing (the same DD as SAF$^{\rm M}$). 15 non-author raters each watched 4 of the 8 prompts against both baselines, side by side with method names hidden and order randomized (60 votes per baseline). For each pair they chose the better video or "no preference" on three criteria: motion quality (natural, meaningful motion that is neither jittery nor nearly static), visual quality (sharpness and freedom from artifacts over the full 100 s) and overall preference.

SAF$^{\rm M}$ vs. (win / tie / loss, %)Motion qualityVisual qualityOverall
SGF46.7 / 21.7 / 31.641.7 / 28.3 / 30.056.7 / 18.3 / 25.0
Causal-Forcing76.7 / 15.0 / 8.385.0 / 10.0 / 5.091.7 / 5.0 / 3.3

Raters prefer SAF$^{\rm M}$ overall against both baselines (56.7% vs. 25.0% against SGF; 91.7% vs. 3.3% against Causal-Forcing). Causal-Forcing has the same DD as SAF$^{\rm M},yetSAF, yet SAF^{\rm M}$ wins 76.7% of the votes in motion quality and 85.0% in visual quality, so its motion is not the instability that DD can reward. Against SGF, SAF$^{\rm M}$ also receives more votes in motion quality (46.7% vs. 31.6%) and in visual quality (41.7% vs. 30.0%).

[1] Z. Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

Reviewer_7ng6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Aligned Forcing for AR video generation. The core idea is Stage-Aligned Differentiable History, which is simple yet effective. By using the past frames' K/V caches at the same noise level as the current block, the method enables more efficient training and allows gradients to propagate through the historical K/V representations. However, using noisy history may lead to visual instability. To mitigate this issue, Self-Algined Forcing incorporates a clean attention sink to preserve appearance and scene consistency. Experiments are conducted across several settings, including standard video generation benchmarks, long-horizon generation, and interactive generation with prompt switching.

Critical strengths

S1. The core idea of the proposed method is simple yet effective. By aligning the noise level of historical KV representations with that of the current block, Differentiable history and block-parallel training can be implemented within a single causal forward pass. This allows gradients from future blocks to optimize historical K/V representations without requiring an additional history reconstruction rollout.

S2. The paper provides extensive experiments across standard video generation benchmarks, long-horizon generation, and interactive generation with prompt switching. The proposed method achieves competitive or superior performance while significantly improving training efficiency. The ablation studies also demonstrate the importance of differentiable KV history and the clean attention sink.

Critical weaknesses

W1. While the proposed method achieves promising results on long-horizon video generation, I am not fully convinced that the improvements directly demonstrate reduced error accumulation. As the authors discussed, using noisy history introduces a trade-off between motion dynamics and visual fidelity. So, I wonder if the method fundamentally improves long-horizon stability or mainly shifts the trade-off between motion and visual quality.

W2. To my understanding, the method requires maintaining separate KV caches for different denoising stages, which introduces additional memory overhead compared to maintaining a single clean KV cache. Although the authors also propose some strategies to reduce memory consumption, it may introduce a mismatch between training and inference. It is thus difficult toassess the practical efficiency of the proposed method in terms of the trade-off between memory consumption, inference throughput, and generation quality.

Questions for authors

Q1. Regarding W1, could the authors provide more direct evidence that the proposed method reduces error accumulation during long-horizon generation? For example, it would be useful to report temporal consistency or generation quality across different time intervals

Q2. Regarding W2, could the authors provide a more detailed analysis of the memory overhead introduced by multi-bank KV caching? In particular, I wonder how peak GPU memory, inference throughput, and latency change with different numbers of denoising steps and attention window sizes. It would also be helpful to compare SAFM and SAFS under same settings to better understand the trade-off between efficiency and quality.

Q3. The ablation study (table2) shows that removing KV gradients mainly reduces dynamic degree and removing sink recaching degrades performance. Could the authors provide similar ablation results on longer generation horizons like 60s? This would help clarify the relative contributions of differentiable history and the clean attention sink to long-horizon stability.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7ng6 (1/3)

We thank the reviewer for finding the core idea simple yet effective and the experiments extensive. Our responses follow; changes in the revised PDF are marked in blue.

W1/Q1. Direct evidence of reduced error accumulation.

Per-interval results show that SAF$^{\rm M}$ accumulates far less error than Self-Forcing and Causal-Forcing, and about as little as SGF and HiAR while moving 1.6× and 2.8× as much. Like SAF, these methods are trained on 5-s single-prompt clips, so the comparison reflects how each treats the history rather than its training data. As suggested, we split each Interactive-60s video into six 10-s segments and score each on its five 2-s clips (VBench, official normalization). Q5 is the mean of the five dimensions scored on every clip: subject and background consistency, motion smoothness, aesthetic and imaging quality. It thus covers both temporal consistency and frame quality; every dimension is reported separately in the new Table 11 (Appendix D), and a summary is below. Dynamic degree (DD, which measures motion) and temporal flickering (scored only on static clips) are excluded.

Q5, Interactive-60sSeg. 1Seg. 2Seg. 3Seg. 4Seg. 5Seg. 6Δ 6−1Δ 6−1, dynamic clips
SAF$^{\rm M}$85.0884.3684.2684.1984.1483.97−1.11−0.95
SGF85.5185.2585.2884.9184.7884.60−0.91−0.80
HiAR84.3483.5283.4083.2283.1483.05−1.28−1.11
Causal-Forcing84.5682.4080.2780.5379.7480.70−3.86−4.10
Self-Forcing84.7884.2383.5882.9282.0381.18−3.60−4.81

Imaging quality, motion and prompt following, first → last segment:

Seg. 1 → 6ImagingDDViCLIP (60-s mean)
SAF$^{\rm M}$71.80 → 71.7350.80 → 80.8027.06 → 24.68 (25.34)
SGF72.45 → 72.9240.40 → 44.6027.14 → 24.00 (24.63)
HiAR69.76 → 72.6419.20 → 26.4027.25 → 23.64 (24.16)
Causal-Forcing72.39 → 69.3242.60 → 62.4027.51 → 22.74 (24.21)
Self-Forcing71.57 → 62.8828.20 → 18.8027.12 → 17.38 (21.88)
  • Far less drift than Self-Forcing and Causal-Forcing. Over 60 s, SAF$^{\rm M}$ loses 1.1 Q5; they lose 3.6–3.9 (4.1–4.8 on dynamic clips). Self-Forcing's imaging falls by 8.7 as its motion fades, whereas SAF$^{\rm M}$'s imaging stays flat and its DD stays at 77–81 from segment 2 on. Prompt following also degrades least: ViCLIP drops by 2.38 from segment 1 to 6, vs. 3.14–9.74 for the other four methods (per-segment values in Table 7).

  • As little drift as SGF and HiAR, with 1.6× and 2.8× the motion. SGF back-propagates through the history like SAF; HiAR conditions on a noisy history. Both drift about as much as SAF$^{\rm M}$ but move far less (mean DD over the six segments in Table 11: 45.2 for SGF and 26.4 for HiAR vs. 74.0 for SAF$^{\rm M}$).

  • Error accumulation at matched motion. To separate error accumulation from the amount of motion, we compare SAF$^{\rm M}$ with each baseline on the identical clips (same prompt, same time) that are dynamic in both methods, in the first and in the last 30 s:

    BaselineDynamic clips (of 3,000)Dynamic in both, first / last 30 sQ5 of SAF$^{\rm M}$, first / last 30 sQ5 of baseline, first / last 30 sQ5 gap, first / last 30 s
    SGF1,357563 / 63484.26 / 83.7985.09 / 84.68−0.83 / −0.89
    HiAR793322 / 34884.13 / 83.9883.19 / 82.99+0.93 / +0.98
    Causal-Forcing1,686647 / 74984.19 / 83.8481.77 / 79.96+2.42 / +3.88
    Self-Forcing682341 / 23483.86 / 83.0983.35 / 80.29+0.51 / +2.79

    SAF$^{\rm M}$ has 2,219 dynamic clips. Q5 is averaged over the clips that are dynamic in both methods; Q5 gap = SAF$^{\rm M}$ minus baseline (computed before rounding).

    SAF$^{\rm M}$ is higher than HiAR, Causal-Forcing and Self-Forcing at matched motion, so its gain is not a shift along the trade-off, and its lead over Causal-Forcing and Self-Forcing grows from 2.4 to 3.9 and from 0.5 to 2.8. Against SGF, from the first to the last 30 s, Q5 falls by 0.47 for SAF$^{\rm M}$ (84.26 → 83.79) and by 0.41 for SGF (85.09 → 84.68): SAF$^{\rm M}$ accumulates about as little error as SGF while producing 1.6× as many dynamic clips (2,219 vs. 1,357).

AuthorsOpenReview ↗

Response to Reviewer 7ng6 (2/3)

W2/Q2. Memory, throughput and latency; SAF$^{\rm M}$ vs. SAF$^{\rm S}$.

Cost model. SS is the number of denoising steps and LL the attention window in latent frames (chunkwise defaults: L=12L=12 with a 3-frame sink, S=4S=4). One K/V bank costs 0.268 GiB per latent frame (30 layers × 1560 tokens × 1536 channels × {K, V} × bf16). On one GPU, SAF$^{\rm M}$ keeps SS banks instead of one. The sink is clean and identical in every bank, so it is stored once and only the other L−3L-3 frames are stored per bank: the extra memory is (S−1)⋅(L−3)⋅0.268(S-1)\cdot(L-3)\cdot 0.268 GiB, i.e., +7.2 GiB at the default setting.

Setup. All methods run on one H200, our current hardware, with the protocol of Table 1 (SAF$^{\rm M}$: 24.4 / 51.5 FPS here vs. 22.8 / 49.1 FPS on the H100 of Table 1). LongLive is re-timed with its official inference code on the same GPU, whereas Table 1 quoted its reported throughput.

LL / SSMethodGPUsFPSFirst-block latency (s)Peak memory (GiB, max over GPUs)
12 / 4LongLive / SAF$^{\rm S}$ / SAF$^{\rm M}$124.1 / 24.7 / 24.40.29 / 0.31 / 0.3120.9 / 20.9 / 28.2
12 / 4SAF$^{\rm M}$ pipeline451.50.3220.9
6 / 4LongLive / SAF$^{\rm S}$ / SAF$^{\rm M}$128.8 / 30.1 / 30.00.28 / 0.31 / 0.3119.3 / 19.3 / 21.7
6 / 4SAF$^{\rm M}$ pipeline461.00.3219.3
21 / 4LongLive / SAF$^{\rm S}$ / SAF$^{\rm M}$121.7 / 21.8 / 21.80.29 / 0.31 / 0.3123.3 / 23.3 / 37.8
21 / 4SAF$^{\rm M}$ pipeline444.10.3223.3
12 / 3LongLive / SAF$^{\rm S}$ / SAF$^{\rm M}$130.1 / 32.7 / 32.20.22 / 0.23 / 0.2320.9 / 20.9 / 25.8
12 / 3SAF$^{\rm M}$ pipeline359.80.2420.9
12 / 2LongLive / SAF$^{\rm S}$ / SAF$^{\rm M}$140.1 / 48.2 / 47.80.14 / 0.16 / 0.1620.9 / 20.9 / 23.3
12 / 2SAF$^{\rm M}$ pipeline271.60.1620.9
  • Memory grows linearly in SS and LL, as the cost model predicts. On one GPU, SAF$^{\rm M}$ uses 2.4 / 7.2 / 14.5 GiB more than single-bank methods at LL = 6 / 12 / 21 (SS = 4), and 2.4 / 4.8 GiB more at SS = 2 / 3 (LL = 12). The default setting needs 28.2 GiB and the largest 37.8 GiB, so both fit on one 48-GB GPU.
  • Throughput is on par on one GPU and doubles with the pipeline. At SS = 4, single-GPU SAF$^{\rm M}$ matches LongLive (24.4 vs. 24.1 FPS). With fewer steps, skipping the per-block recache pass gives 1.07× at SS = 3 and 1.19× at SS = 2 (47.8 vs. 40.1 FPS). The 4-GPU pipeline reaches 2.0–2.1× (51.5 vs. 24.4 FPS at LL = 12). The pipeline leaves first-block latency unchanged (0.32 vs. 0.31 s), since the first block still passes through all stages.
  • SAF$^{\rm M}$ vs. SAF$^{\rm S}$ at the same setting (LL = 12, SS = 4; quality from Table 1). On one GPU, the two run at similar speed (24.4 vs. 24.7 FPS), and SAF$^{\rm M}$ uses 7.2 GiB more for its SS banks. SAF$^{\rm S}$ saves this memory by letting all stages read one bank; this is the train–inference mismatch the reviewer notes, and it lowers long-horizon quality (Interactive Quality 84.29 vs. 85.59, MovieGen-100s Quality 84.34 vs. 85.00). With multiple GPUs, the pipeline places each stage's bank on its own GPU: at the cost of SS GPUs, the per-GPU peak equals that of a single-bank method (20.9 GiB) and throughput doubles. We therefore use SAF$^{\rm M}$ by default and keep SAF$^{\rm S}$ for a single memory-limited GPU.
AuthorsOpenReview ↗

Response to Reviewer 7ng6 (3/3)

Q3. Ablations at 60 s. We have now evaluated them on the full 60-s Interactive benchmark, and the longer evaluation strengthens the conclusions:

Interactive-60sViCLIPQualityDDSmooth.Img.Q5 at matched motion$^\dagger$
SAF$^{\rm M}$ (full)25.3485.5973.9798.8371.9084.18
w/o K/V grad.24.5884.3852.8099.1472.5683.63
w/o sink recache22.7381.535.2699.6170.82–

†^\dagger Q5 (defined in W1/Q1) on the identical 2-s clips that are dynamic in both full SAF and w/o K/V grad. (1,326 of 3,000; in the first 20 s: 84.57 vs. 84.32 on 484 of 1,000). It does not apply to w/o sink recache, which is nearly static.

Prompt following per 10-s segment (the prompt switches at each segment boundary):

ViCLIPSeg. 1Seg. 2Seg. 3Seg. 4Seg. 5Seg. 6Δ 6−1
SAF$^{\rm M}$ (full)27.0626.0125.2424.2724.7824.68−2.38
w/o K/V grad.26.6925.6124.6723.5223.6123.38−3.31
w/o sink recache26.5922.2522.0721.8821.9721.62−4.97
  • Differentiable history matters more over longer horizons. In the first 20 s (Table 2), full SAF vs. w/o K/V grad. is 85.18 vs. 84.64 in Quality and 64.10 vs. 51.30 in DD, i.e., gaps of 0.54 and 12.8. Over 60 s, the gaps grow to 1.21 and 21.2, and ViCLIP drifts less with it from segment 1 to 6 (−2.38 vs. −3.31). At matched motion (last column), full SAF is also 0.55 higher in Q5, so the gain is not only more motion. The higher motion smoothness and imaging quality of w/o K/V grad. (99.14 vs. 98.83; 72.56 vs. 71.90) come with its lower motion.
  • The clean sink is a necessary anchor. Without it the videos are nearly static (DD 5.26, hence the highest motion smoothness) and stop following new prompts (ViCLIP falls from 26.59 to 22.25 at the first switch and does not recover). The two components are thus complementary: the sink keeps generation from collapsing, and, given this anchor, differentiable history reduces drift, increasingly so over longer horizons.

We hope these additions resolve the reviewer's concerns, and we are happy to provide further details.

Thank you for the rebuttal and the additional results. My concerns are addressed and I would like to maintain my original rating.

Reviewer_p4C7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Aligned Forcing (SAF), a training method for AR video diffusion that enables future-block losses to backpropagate through historical K/V representations using stage-aligned and differentiable history. By combining block-parallel self-rollout with persistent clean sinks, SAF aims to improve video quality and motion while reducing training costs. Experiments demonstrate improvements in long-horizon video generation quality and training efficiency.

Critical strengths

S1. Clear motivation and well-aligned method. The paper is clearly motivated by two empirical observations: noisy history trades visual quality for motion, and detaching the history K/V prevents the model from learning how to encode it. The proposed method directly addresses both. By aligning the history noise level with the current block, SAF keeps the noisy history differentiable within a single forward pass, so the design follows naturally from the motivation.

S2. Strong long-horizon results. SAF performs well on long video generation. It achieves the best Quality on both 60-second Interactive and MovieGen-100s in the chunkwise setting, and the qualitative results show stable scene structure and consistent subjects over the full horizon, where baselines suffer from drift, duplicated subjects, or artifacts.

Critical weaknesses

W1. Limited evidence for the contribution of differentiable history. The ablation that detaches history K/V substantially reduces Dynamic Degree but has only a marginal effect on Interactive Quality. In contrast, removing clean sink recaching leads to substantially larger degradations in both metrics. This raises questions about the extent to which differentiable history contributes to the overall improvements, as opposed to clean sink recaching. Moreover, these component ablations are conducted only on 20-second Interactive sequences, leaving it unclear whether the observed effects generalize to longer-horizon generation.

W2. Potential train–inference mismatch in clean sink construction. During training, the clean sink is constructed from the estimate at a randomly sampled exit stage, whereas inference uses the final-stage estimate. Given the substantial performance degradation observed when clean sink recaching is removed, it is important to understand whether this difference introduces a mismatch in the conditioning context between training and inference. However, the paper does not investigate the impact of using sink estimates from different denoising stages. An ablation examining this difference would help clarify whether it affects generation quality.

Questions for authors

Q1. The ablation that detaches history K/V substantially reduces Dynamic Degree but has only a marginal effect on Interactive Quality. Could the authors provide further analysis of the role of differentiable history and explain why its impact on motion dynamics does not translate into a similarly significant change in overall quality? Does this effect persist in longer-horizon generation?

Q2. During training, the clean sink is constructed from the estimate at a randomly sampled exit stage, whereas inference uses the final-stage estimate. Could the authors provide an ablation comparing sink estimates from different denoising stages to clarify whether this difference affects generation quality?

Q3. In the supplementary videos, SAF appears to mitigate oversaturation, but some generated motions seem repetitive, with similar movements recurring over time. Could the authors explain this behavior and clarify whether the method trades motion diversity for visual stability?

5 comments
AuthorsOpenReview ↗

Response to Reviewer p4C7 (1/3)

We thank the reviewer for the careful reading and for recognizing the clear motivation and the strong long-horizon results. We address each point below; changes in the revised PDF are marked in blue.

W1/Q1. Role of differentiable history, and why its effect on Quality is smaller than on motion.

(a) Why Quality moves little. Dynamic degree (DD) has weight 0.5 out of 6.5 in VBench Quality, so the 12.8-point DD gain adds only 0.98 points. The other six dimensions are correlated with the amount of motion: VBench notes that "a completely static video can score well" on the temporal dimensions [1, Sec. 3.1.1], which trade off against DD across models [1, Sec. 5]. Within every method on Interactive-60s, dynamic clips likewise score lower than static ones on subject and background consistency and motion smoothness (Table 11).

Decomposition of Table 9 (SAF$^{\rm M}$; contributions to Quality in points; positive = full SAF higher):

DDSubj.Bg.FlickerSmooth.Aes.Img.Total
Full − w/o K/V grad.+0.98−0.09−0.09−0.05−0.07−0.02−0.12+0.54
Full − w/o sink recache+4.40−0.30−0.31−0.03−0.29−0.07+0.09+3.48

Full SAF moves more, and all six other dimensions are slightly lower, offsetting 0.44 of the 0.98 DD gain. Conversely, without sink recaching DD collapses from 64.10 to 6.90 and these six dimensions rise by 0.91 in total (second row) as the videos become nearly static. We examine the trade-off directly in (c), at matched motion, and in (d), at 60 s.

(b) The two components play complementary roles.

  • The clean sink is a necessary condition. Without a high-SNR anchor, the model collapses toward static video even though its history is still differentiable, so differentiable history alone cannot compensate.
  • Differentiable history determines how well the model exploits the noisy local window once the anchor exists. Beyond DD, it improves prompt following (ViCLIP averaged over the two segments: 26.15 → 26.54; Table 2). Fig. 2b is consistent with this role: under detachment, the key projection WkW_k receives gradients with cosine ≈ 0.2 to the reference, i.e., the model learns little about how to write retrievable history.

(c) Not a shift along the trade-off. We compare fidelity at matched motion with VBench's per-clip dynamic flag on 2-s clips (10 per prompt in the first 20 s, 30 per prompt over 60 s): we take the identical clips (same prompt, same time) that are dynamic in both models and score them by Q5, the mean of the five dimensions VBench scores on every clip (subject and background consistency, motion smoothness, aesthetic and imaging quality; official normalization).

InteractiveDynamic clips, full / w/o K/V grad.Identical clips, both dynamicQ5 on these clips, full / w/oImg. on these clips, full / w/oQ5 on own dynamic clips, full / w/o
First 20 s (1,000 clips)641 / 51348484.57 / 84.32 (+0.25)72.23 / 72.46 (−0.23)84.35 / 84.28
60 s (3,000 clips)2,219 / 1,5841,32684.18 / 83.63 (+0.55)72.11 / 72.04 (+0.07)84.05 / 83.91

On these clips, full SAF scores higher, not lower: Q5 is +0.25 in the first 20 s and +0.55 over 60 s, with imaging on par, while full SAF produces 1.25× (20 s) and 1.40× (60 s) as many dynamic clips. A model that only traded fidelity for motion would score lower here. The slightly lower consistency and imaging scores of full SAF in (a) therefore come from its additional dynamic clips, not from lower fidelity at matched motion. The Q5 gap also widens from 20 s to 60 s, consistent with the smaller drift in (d).

AuthorsOpenReview ↗

Response to Reviewer p4C7 (2/3)

(d) Longer horizons. We have now evaluated them on the full 60-s Interactive benchmark, and the longer evaluation strengthens the conclusions: full SAF drifts least, and its advantage over w/o K/V grad. grows with the horizon.

Interactive-60sViCLIPQualitySmooth.DDImg.Q5 Δ seg. 6−1ViCLIP seg. 1Seg. 2Seg. 3Seg. 4Seg. 5Seg. 6
SAF$^{\rm M}$ (full)25.3485.5998.8373.9771.90−1.1127.0626.0125.2424.2724.7824.68
w/o K/V grad.24.5884.3899.1452.8072.56−1.7226.6925.6124.6723.5223.6123.38
w/o sink recache22.7381.5399.615.2670.82−2.6126.5922.2522.0721.8821.9721.62
  • Differentiable history matters more at 60 s than at 20 s. Without it, the Quality gap grows from 0.54 in the first 20 s (85.18 vs. 84.64; Table 2) to 1.21 over 60 s, the DD gap from 12.8 (64.10 vs. 51.30) to 21.2, and Q5 drifts 55% more (−1.72 vs. −1.11). The per-segment ViCLIP gap also grows, from +0.37 and +0.40 in segments 1–2 to +0.57, +0.75, +1.17 and +1.30 in segments 3–6, i.e., the detached model follows later prompts increasingly poorly. Together with (c), this indicates reduced error accumulation, not only more motion.
  • The clean sink remains necessary. Without it, the videos are nearly static (DD 5.26), ViCLIP falls from 26.59 to 22.25 at the first prompt switch (3.8 below full SAF) and does not recover (21.6–22.1 in segments 3–6, vs. 24.3–25.2), and Q5 drifts most (−2.61) despite the lack of motion.
  • The ablations are higher in motion smoothness (99.14 and 99.61 vs. 98.83), and w/o K/V grad. also in imaging quality (72.56 vs. 71.90). Both follow their lower motion (see (a) and (c)). W/o sink recache is lower in imaging (70.82) although it is nearly static, because it drifts most.

W2/Q2. Sink estimate stage: training vs. inference.

The sink is always encoded at t=0t=0, so its K/V is clean; the settings differ only in which stage's estimate it is built from:

  • Training. Each sample draws one exit stage dd, uniform over {1,…,S}\{1,\dots,S\} and shared by the sink and all blocks. The rollout stops at dd, so the sink is recached from X^Ω(d)\widehat X_\Omega^{(d)}, the latest estimate available without extra denoising steps, and every stage s≤ds\le d reads this one sink. The loss is applied at stage dd.
  • Inference (default). Every stage reads the sink recached from the final estimate X^Ω(S)\widehat X_\Omega^{(S)}. This is exactly the training rollout with d=Sd=S (1/4 of the samples).
  • Per-step (the requested ablation). Stage ss reads a sink recached from its own estimate X^Ω(s)\widehat X_\Omega^{(s)}, i.e., the sink that stage ss reads whenever it is the supervised stage (d=sd=s).

We compared the two inference settings on the trained SAF$^{\rm M}$ model. VBench, 944 prompts × 5 samples; the default row is from Table 1:

Sink at inferenceTotalQualitySemanticSubj. cons.Bg. cons.FlickerSmooth.DDAes.Img.
Final-stage (default, Table 1)85.0486.3080.0097.0596.6299.5498.3976.3967.8368.88
Per-step84.4285.4680.2895.0293.9198.8497.9589.1766.4667.85
Per-step − default, contribution to Quality (points)−0.37−0.56−0.29−0.23+0.98−0.21−0.16
  • The final-stage sink is better on every Quality dimension except DD. The per-step sink lowers Total by 0.62 and Quality by 0.84, while Semantic changes little (+0.28). All six non-DD dimensions drop, most of all background and subject consistency (−2.71 and −2.03).
  • Why. The estimates of the early, high-noise stages are blurry, because these stages mainly determine the coarse layout and motion. A sink built from them is therefore a weaker anchor than one built from the final-stage estimate. With the per-step sink, the generated videos move more (DD +12.8), but the six non-DD dimensions lose about 1.8 Quality points, roughly twice the 0.98 gained from DD. The final-stage estimate is the least noisy and gives the strongest anchor, so the default setting yields higher overall quality at the cost of a moderate amount of DD.
  • It is also cheaper. The default encodes the sink once per video; the per-step sink needs one t=0t=0 encoding per stage, i.e., SS in total.

The train–inference difference the reviewer identified is therefore real, but benign: reading the sharper final-stage sink at every stage improves quality, and this configuration also occurs in training (d=Sd=S).

AuthorsOpenReview ↗

Response to Reviewer p4C7 (3/3)

Q3. Repetitive motion.

We agree that this is visible in some long videos and have added it to the Limitations (Appendix G). It appears when a single prompt describing one sustained action is held for 100 s (e.g., wiping a table, Fig. 6), for two reasons:

  1. Finite window. Apart from the 3-frame sink, the model attends only to the 9 most recent latent frames (≈2 s). The sink anchors appearance but carries no motion history, so the model cannot tell which motions it has already produced and may re-enact similar cycles.
  2. Mode collapse in DMD. The DMD objective is prone to mode collapse [2, Sec. 3.3], and DMD2 reports slightly lower diversity than the teacher [3, Sec. 6], narrowing the motion distribution per prompt.

Is motion diversity traded for visual stability? We believe it is not. The repetition comes from the finite window (9 latent frames, ≈2 s), not from stability: SAF adheres to the prompted action, so one action held for 100 s recurs. LongLive and SGF share the sink-plus-window design; in Figs. 5 and 6, their motion changes when content the prompt does not describe appears (the duplicated cat and the duplicated person), which is not a desirable form of diversity. SAF's stability costs neither motion nor prompt following (Interactive-60s; Tables 7 and 11):

  • Motion persists. DD stays at 77–81 in segments 2–6 with flat imaging (71.7–72.1), whereas Self-Forcing's DD decays from 31 to 19 over the same segments.
  • Prompts are followed. ViCLIP declines least from segment 1 to 6 (−2.38, vs. −2.59 to −9.74). After the first switch, SAF is highest in every segment except against LongLive, which is trained on prompt switches and long videos (SAF: 5-s single-prompt clips).

We hope these results address the reviewer's concerns, and we would be glad to clarify anything further.

References

[1] Z. Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.

[2] T. Yin et al. One-step Diffusion with Distribution Matching Distillation. In CVPR, 2024.

[3] T. Yin et al. Improved Distribution Matching Distillation for Fast Image Synthesis. In NeurIPS, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.