- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
From my understanding, the paper tries to target long-horizon drift in few-step autoregressive video diffusion distilled with DMD, and uses their two observations (Fig2): 1)at inference a noisier history K/V raises dynamic degree and lowers imaging quality, 2), gradients flow through the history makes the parameter gradient closer to a bidirectional reference. Then they propose SAF which takes each block’s history at the same denoising stage as the block itself. All blocks at a stage can then be denoised in one forward pass. The history K/V is produced inside that pass, so it stays in the graph, and later blocks’ DMD losses reach how earlier blocks write their K/V. Self-rollout runs block-parallel within a stage and sequentially only across the 4 stages, with Self-Forcing-style random exit-stage truncation until a clean sink is recached at the beginning t=0. But at inference, SAF^M keeps one K/V bank per stage. This removes the extra t=0 recache pass. Validation uses VBench, a Interactive benchmark, and MovieGen-100s, against Self-Forcing, LongLive, Causal-Forcing, HiAR and SGF using released weights. And also against their ablations.
Critical strengths
- This is "almost for free" differentiable history, since stage alignment removes both the separate no-grad rollout that SGF needs and the per-block t=0 recache. The training-cost numbers in Table 3 are the most convincing result in the paper: 137s vs. 249s per cycle framewise against SGF, with lower memory.
- Training sees the same history that SAF^M sees at inference, matching exactly. The text claims, and Alg. 3 supports, that the pipelined and single-GPU multi-bank versions produce identical outputs. Per-stage banks then give stage pipelining for free. This is a cleaner train–test story than HiAR’s or SGF’s, and it is the reason the efficiency gains don’t come with a train–test mismatch (unlike SAF^S, which the authors honestly flag).
Critical weaknesses
- the results are very thin and marginal. Chunkwise VBench Total is 85.04 vs. 84.88 (Causal-Forcing) and 84.76 (SGF), from ONE training run and ONE seed, with NO variance reported. Also in table 5, in the framewise setting SAF loses to SGF on Total, Interactive Quality and MovieGen Quality. How does this justify “superior long-horizon generation” in the abstract or “best training and inference efficiency in both settings”?
- Seems like the method is still suffering from "motion-fidelity" trade-off, as The “w/o K/V grad” ablations in table 9 show the drop in dynamic degree, while imaging quility and subject consistency is up. IMproving dynamic degree might be a little misleading, as DD is a coarse, flow-thresholded metric that rewards instability as readily as meaningful motion
- Novelty is narrower than presented, and related work is missing on the observation and inference side. Observation 1 (partially noised history trades fidelity for motion) closely matches what History-Guided Video Diffusion / DFoT already exploit. That work treats noise as partial masking of history frames and uses guidance over noisy history to improve motion dynamics. It should be cited and discussed. Some related works are FIFO-diffusion (its diagonal denoising), StreamDiffusion (streambatch), MAGI-1 (chunkwiose pipelined denoising), RollingDiffusion
Questions for authors
- Can you compare SAF and SGF under the same initialization, data, step budget and checkpoint-selection rule? Table 3 suggests you have an SGF training setup. Without this, It is hard to attribute the Table 1 differences to the method.
- Please report the Quality aggregate with dynamic degree removed, or some motion-quality measure that is less gameable, and ideally a small pairwise human study on the 100s videos
- For Fig. 2b, what exactly is the “matched-weight bidirectional reference”? Is it the same causal weights run without the mask? Restoring the history path seems almost guaranteed to raise cosine similarity to a gradient that also contains cross-block terms. Why is matching the bidirectional gradient the right target for a model that is deployed causally? How many samples are behind the error bars?
7 comments
Response to Reviewer jSpn (2/4)
W2/Q2. Is the gain a motion–fidelity trade-off, and is DD gameable?
In short: against the detached-history baselines, SAF has higher fidelity at matched motion, i.e., on the identical clips in which both models move (defined below), so its gain is not a shift along the trade-off. Against SGF, neither method dominates: SGF has higher fidelity on moving content, while SAF$^{\rm M}$ has 1.6× the motion and better prompt following at 1.4–1.8× lower training cost. We agree that dynamic degree (DD) is coarse and report the measures requested in Q2. In a blind pairwise human study (d), raters prefer SAF$^{\rm M}$ overall against both SGF and Causal-Forcing.
Definition of matched motion. Static clips score higher on consistency and smoothness, so a model with less motion can score higher without having higher fidelity. To remove this effect, (a) and (c) compare two models only on the identical 2-s clips (same prompt, same time) that VBench's per-clip dynamic flag marks as dynamic in both.
(a) The w/o-K/V-grad ablation (Table 9). We compare full SAF and w/o K/V grad. at matched motion. Q5 is the mean of the five dimensions VBench scores on every clip (subject and background consistency, motion smoothness, aesthetic and imaging quality; official normalization).
| Interactive, full SAF / w/o K/V grad. | Dynamic clips | Identical clips, both dynamic | Q5 on these clips | Imaging on these clips |
|---|---|---|---|---|
| First 20 s (1,000 clips; setting of Table 9) | 641 / 513 | 484 | 84.57 / 84.32 | 72.23 / 72.46 |
| 60 s (3,000 clips) | 2,219 / 1,584 | 1,326 | 84.18 / 83.63 | 72.11 / 72.04 |
Where both models move, full SAF has higher Q5 and imaging is on par, so the differentiable history adds motion without lowering fidelity at matched motion. The higher subject consistency of w/o K/V grad. in Table 9 follows from its larger share of static clips, which score higher on consistency within every method (e.g., 97.90 vs. 96.91 for SAF$^{\rm M}$; Table 11b). In our new 60-s evaluation of the ablations, ViCLIP also drifts more without the history gradient: from segment 1 to 6 it changes by −2.38 for full SAF (27.06 → 24.68) and by −3.31 for w/o K/V grad. (26.69 → 23.38).
(b) Quality without DD. We recompute the VBench Quality score with the DD dimension dropped, i.e., the weighted mean of the other six Quality dimensions over all videos (official weights and normalization; chunkwise):
| Self-Forcing | LongLive | Causal-Forcing | HiAR | SGF | SAF$^{\rm S}$ | SAF$^{\rm M}$ | |
|---|---|---|---|---|---|---|---|
| Interactive Quality w/o DD | 85.53 | 86.73 | 83.79 | 85.52 | 86.79 | 87.08 | 86.56 |
| MovieGen-100s Quality w/o DD | 82.89 | 86.03 | 81.16 | 85.34 | 86.83 | 86.88 | 86.62 |
| (Interactive DD) | 22.73 | 27.47 | 56.20 | 26.43 | 48.23 | 50.87 | 73.97 |
| (MovieGen DD) | 28.91 | 44.44 | 65.69 | 39.80 | 58.15 | 53.95 | 65.61 |
- SAF is on the Pareto front at 60 s and 100 s. SAF$^{\rm S}^{\rm M}$ is within 0.23 of the best baseline (SGF) with 1.5× and 1.1× its DD. Every other baseline is lower than a SAF variant in both rows of each benchmark, except for a tie in DD with Causal-Forcing on MovieGen-100s (65.69 vs. 65.61), where SAF$^{\rm M}$ is 5.46 higher in Quality w/o DD.
- Without DD, the score may favor static videos. VBench itself notes that "a completely static video can score well" on its temporal dimensions [1, Sec. 3.1.1], which trade off against DD across models [1, Sec. 5]. Consistent with this, on Interactive-60s our w/o-sink-recache ablation (newly evaluated at 60 s; full results in our response to Reviewer 7ng6, Q3) is nearly static (DD 5.26 vs. 73.97 for full SAF$^{\rm M}$) and stops following new prompts (ViCLIP 22.73 vs. 25.34), yet its Quality w/o DD is the highest of all models (87.89 vs. 86.56). Thus, while Quality with DD can reward instability, as the reviewer notes, Quality without DD can reward stillness.
(c) A less gameable measure: fidelity at matched motion. We therefore compare SAF$^{\rm M}$ with each baseline on the identical Interactive-60s clips that are dynamic in both, or static in both (3,000 clips per method; flag and Q5 as in (a)).
| Interactive-60s | Dynamic / static clips | Q5 gap, both dynamic (clips) | Q5 gap, both static (clips) | Q5 gap, both dynamic: first / last 30 s | Motion smooth., dynamic clips |
|---|---|---|---|---|---|
| SAF$^{\rm M}$ | 2,219 / 781 | – | – | – | 98.68 |
| SGF | 1,357 / 1,643 | −0.86 (1,197) | −0.20 (621) | −0.83 / −0.89 | 98.77 |
| HiAR | 793 / 2,207 | +0.96 (670) | +1.63 (658) | +0.93 / +0.98 | 98.17 |
| Causal-Forcing | 1,686 / 1,314 | +3.20 (1,396) | +2.54 (491) | +2.42 / +3.88 | 96.96 |
| Self-Forcing | 682 / 2,318 | +1.44 (575) | +1.44 (674) | +0.51 / +2.79 | 98.42 |
Response to Reviewer jSpn (3/4)
Q5 gap = SAF$^{\rm M}$ minus the method, on the identical clips that are dynamic (or static) in both; the number of such clips is in parentheses. Motion smoothness uses each method's own dynamic clips. All numbers come from the per-clip scores behind Table 11 (Appendix D).
- Against the detached-history methods, SAF$^{\rm M}$ is higher at matched motion, on moving and static content alike. This includes HiAR, whose history is noisy, and the lead over Self-Forcing and Causal-Forcing grows over time.
- SAF and SGF: neither dominates. SGF is 0.86 higher in Q5 where both move and 0.20 higher where both are static, and the gap does not grow over time, so SAF$^{\rm M}$ accumulates about as little error as SGF. SAF$^{\rm M}$ in turn has 1.6× the motion((2,219 vs. 1,357 dynamic clips)) and better prompt following (60-s ViCLIP 25.34 vs. 24.63; Table 1), at 1.4–1.8× lower training cost (Table 3).
- SAF's motion is not jitter. On dynamic clips, its motion smoothness is on par with SGF and above the other three baselines; unstable motion would lower it.
(d) Blind pairwise human study. As suggested in Q2, we ran a two-alternative forced choice (2AFC) study on 8 randomly sampled MovieGen-100s prompts, comparing SAF$^{\rm M}$ with SGF (the strongest baseline) and Causal-Forcing (the same DD as SAF$^{\rm M}$). 15 non-author raters each watched 4 of the 8 prompts against both baselines, side by side with method names hidden and order randomized (60 votes per baseline). For each pair they chose the better video or "no preference" on three criteria: motion quality (natural, meaningful motion that is neither jittery nor nearly static), visual quality (sharpness and freedom from artifacts over the full 100 s) and overall preference.
| SAF$^{\rm M}$ vs. (win / tie / loss, %) | Motion quality | Visual quality | Overall |
|---|---|---|---|
| SGF | 46.7 / 21.7 / 31.6 | 41.7 / 28.3 / 30.0 | 56.7 / 18.3 / 25.0 |
| Causal-Forcing | 76.7 / 15.0 / 8.3 | 85.0 / 10.0 / 5.0 | 91.7 / 5.0 / 3.3 |
Raters prefer SAF$^{\rm M}$ overall against both baselines (56.7% vs. 25.0% against SGF; 91.7% vs. 3.3% against Causal-Forcing). Causal-Forcing has the same DD as SAF$^{\rm M}^{\rm M}$ wins 76.7% of the votes in motion quality and 85.0% in visual quality, so its motion is not the instability that DD can reward. Against SGF, SAF$^{\rm M}$ also receives more votes in motion quality (46.7% vs. 31.6%) and in visual quality (41.7% vs. 30.0%).
[1] Z. Huang et al. VBench: Comprehensive Benchmark Suite for Video Generative Models. In CVPR, 2024.
Response to Reviewer jSpn (1/4)
We thank the reviewer for the detailed and constructive critique, and for recognizing SAF's training efficiency and the train–test consistency of SAF$^{\rm M}$. We address every point; changes in the revised PDF are marked in blue.
W1(a)/Q1. Variance, and a controlled comparison with SGF.
The training settings of the officially released SGF checkpoint are not documented; SGF's official code uses batch size 8 and 1200 iterations (chunkwise), whereas we use 32 and 600. To remove these two possible confounds, we re-trained SGF (chunkwise) under our recipe, using its official code, which we also used for the SGF measurements in Table 3: total batch 32 on 8 GPUs, the same seed and critic warm-up, and the checkpoint rule used for SAF in the paper (EMA at step 600). The VBench results in the paper already average N=5 samples for each of the 944 prompts; we re-ran this evaluation for all three models and report mean ± std over the 5 sample indices (DD: dynamic degree).
Controlled comparison. VBench, chunkwise, 944 prompts × 5 samples.
The gain is significant against all four baselines: every CI excludes 0, and SAF$^{\rm M}$ wins on 73–94% of the prompts.
W1(b). Claims.