acceptodds
Under review as a conference paper at ICLR 2027

WHEN TO SYNCHRONIZE, HOW MUCH TO REPAIR: CACHE CONTROL FOR AUTOREGRESSIVE VIDEO DIFFUSION

Abstract

Autoregressive (AR) video generation makes long-horizon visual synthesis possible, but every generated chunk requires repeated forward passes through a large diffusion transformer (DiT). Training-free caching reduces this cost by reusing intermediate computation. However, existing cache controllers often couple two decisions: when the cache is synchronized exactly and how much approximate computation is spent in between. Our characterization suggests that these decisions should be separated. For reactive staleness signals, exact synchronization should be scheduled before measured drift accumulates, while measured drift should decide only how much repair to spend between synchronizations. Based on this separation, we introduce Frequency-Step Cache (FSC), a training-free controller that performs exact synchronization at full-computation anchors on a dense-to-sparse schedule and adaptive repair between them. A lightweight latent-space risk combining frequency-resolved drift with token age routes each non-anchor step from whole-step replay to selective token repair or full repair without changing the synchronization schedule. We further show on SkyReels-V2 that whole-step reuse provides the main acceleration, while selective token computation is most effective as fidelity repair. In a controlled ablation, using one reactive signal for both synchronization and repair reduces fidelity at matched latency, with the separation advantage also observed on FramePack and Matrix-Game. Across multiple AR video backbones, FSC provides favorable speed–quality trade-offs. On SkyReels-V2, FSC-Speed reaches a 2.36× speedup, above MotionCache’s 2.275×, while improving SSIM by 0.019 and PSNR by 0.5 dB. At MotionCache’s speed, FSC-Quality improves SSIM by 0.031 and PSNR by 1.0 dB, while lowering LPIPS by 0.024. Without retuning, FSC-Quality transfers to the 14B SkyReels-V2 model at 2.12× speedup, compared with 1.79× for MotionCache.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Video Diffusion Models, Inference Acceleration, Caching, Efficient Generative Models, Training Free Inference, Long Video Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JsspOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author propos Frequency-Step Cache (FSC), a training-free caching controller for autoregressive (AR) video diffusion that separates cache control into two decisions: (1) scheduled exact synchronization ("anchors") that fully recompute the cache at predetermined steps, and (2) measurement-driven adaptive repair that decides how much computation to spend between anchors. The core claim is that reactive staleness signals should not govern when to synchronize, only how much to repair, because such signals report drift only after it occurs.

Critical strengths
  • Well-motivated separation principle with direct empirical support.
  • Thorough characterization motivating the design.
  • Strong and consistent empirical results across different foundation models.
Critical weaknesses
  • Could the proposed method also be applied to more world models that follow a similar autoregressive diffusion paradigm? Inlcuding an experiemnt about this can improve the generalization of this method.
  • “Accelerating Autoregressive Video Diffusion via History-Guided Cache and Residual Correction” (CVPR 2026) is a strong and highly relevant baseline, but it is not included in the current comparison. The authors should compare the proposed method with it.
  • Methods discussed in Section 1 should be cited, for example, SkyReels-V2.
Questions for authors

see weakness

2 comments

Response to Reviewer Jssp

We thank the reviewer for the positive assessment and for pointing us to the ARCache baseline, additional world models and missing citations.

W1: Generalization to AR World Models

FSC is evaluated on Matrix-Game-3.0, an interactive autoregressive world model (Sec. 5.1, 5.3; App. E.3).

FSC ConfigurationSpeedup vs. VanillaSame-Seed SSIM
Calibrated2.445×0.8323
Transferred2.62×0.8229

Transferred uses the SkyReels-V2 configuration unchanged; Calibrated is tuned on Matrix-Game development prompts. Both deliver effective acceleration with preserved trajectory fidelity on an interactive AR world model. Results on other AR backbones are in our reply to Reviewer 6n53 (W2, W4).

W2: Comparison with ARCache (CVPR 2026)

ARCache's HGC schedules reuse from history-token changes and ERC corrects segment-level errors; FSC instead separates scheduled exact synchronization from reactive token-selective repair. With no verifiable author-released code as of November 9, 2026, we implemented HGC+ERC independently. Our port did not reproduce the published fidelity, so we post its results for reference only; they do not represent official ARCache performance (App. E.8).

MethodRuntimeSSIM
FSC-Quality161.49 s0.829
Independent HGC+ERC160.35 s0.567
Independent HGC alone160.61 s0.575

ARCache is now discussed in related work (Sec. 2).

W3: Citations in Section 1

The revision cites SkyReels-V2 and the other named methods at their first mentions in Sec. 1.

Thank you for your rebuttal. My concerns are well addressed.

Reviewer_6n53OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper presents FSC, a training-free caching method for AR video generation acceleration. It separates cache synchronization from adaptive computation repair, estimating the error through the frequency variance and token age to trigger refreshing computation. The authors adapt their method to various AR video generation models to demonstrate acceleration and video quality.

Critical strengths
  1. Discusses the importance of separating the synchronization and repair of cache through abundant experiments, which provides insights for the design of similar methods.
  2. The methods narrative is well organized and easy to understand.
Critical weaknesses
  1. These designs don’t seem to fit the specific scenario of autoregressive generation; most features could be transferred to general multi-step bidirectional attention video/image generation. It doesn’t consider specific modules in autoregressive video generation, such as AR-specific KV Cache management, cross-chunk dependencies, and long-horizon video drifting.
  2. The generalizability of the method is unclear. In the experimental settings provided in the paper, it only generates 2 chunks (177 frames) of video on SkyReels-V2 and 7 chunks (168 frames) of video on MAGI-1. It is unclear whether the method will still be competitive when applied to long-horizon generation, which is vital for autoregressive video generation. Also, MAGI-1 seems not to apply complete FSC, which weakens the claims.
  3. The improvement in acceleration is limited. As shown in Tables 1 and 2, the proposed method does not seem to gain a promising advantage compared with the baselines in SkyReels-V2 1.3B (lower VBench score, 3% acceleration).
  4. Lacking results on other autoregressive generation models. The experiments heavily rely on SkyReels-V2 and MAGI-1, which are both old approaches from about 1 and a half year ago. Evaluations on the up-to-date autoregressive generation models such as Causal Forcing [1], LongLive 2.0 [2], and LingBot-World-Infinity [3] will be helpful. These works provide weights for multi-step denoising, fitting the situation of the proposed method. At the same time, they produce longer, higher-quality results, and this may help demonstrate the advantage in long-horizon generation.

[1] Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation, ICML 2026

[2] LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation, arXiv 2026

[3] Infinite Worlds with Versatile Interactions, arXiv 2026

Questions for authors

No additional questions.

5 comments

Response to Reviewer 6n53 (1/2)

We thank the reviewer for the detailed and constructive comments, which helped us strengthen the evaluation substantially.

W1: AR specificity, K/V handling and long-horizon drift

Thank you for the reminder; below is how we position our contribution.

Scope. FSC is a caching method for accelerating AR video diffusion, not a K/V-compression method for the growing AR context or a method for extending video length; our finding is that separating when to synchronize from how much to repair makes caching more faithful and faster.

AR handling. Scheduled exact synchronization limits approximation error within each denoising trajectory before it enters the conditioning history of later chunks. FSC keeps the backbone's cross-chunk K/V cache: recomputed tokens update only their current-block entries, leaving generated history unchanged.

Long-horizon evaluation. Our main benchmark keeps window 2 (177 frames), the SkyReels-V2 setting used by FlowCache and MotionCache. We extend SkyReels-V2 to window 10 (817 frames) and LongLive-2.0 to 381 frames at 2.449× speedup (W4).

SkyReels-V2, cumulative SSIM (separate long-rollout run on a prompt subset):

MethodWindow 2 (177 f)Window 4 (337 f)Window 6 (497 f)Window 8 (657 f)Window 10 (817 f)
MotionCache0.89060.81010.74470.70800.7007
FSC0.89280.81880.74860.72310.7176

FSC stays above MotionCache at every evaluated window (+0.0022 at window 2, +0.0087 at window 4, +0.0169 at window 10).

W2: Complete MAGI-1 controller

Thank you for this correction. We now run the complete FSC controller on MAGI-1 (App. E.13).

MethodMean SSIMRuntime
FlowCache0.7782460 s
MotionCache0.8468504 s
FSC, full controller0.8795469 s

FSC improves SSIM by +0.101 over FlowCache at 2% higher latency and by +0.033 over MotionCache at 7% lower latency. At equal latency, the complete controller slightly exceeds the submitted native-interface FSC-Speed (0.8795 vs. 0.8782).

Response to Reviewer 6n53 (2/2)

W3: Practical acceleration and VBench

Thank you for raising this; our paper did not make the scale of the speedup clear enough. Published speedups (e.g., 6–7× for FlowCache and MotionCache) were measured on different hardware and runtime stacks; we re-measure all methods on the same hardware and runtime stack, so absolute speedups are lower but directly comparable within each table. A few-percent speed gain is a normal generational step: MotionCache's published gains over FlowCache are +0.32% (slow) and +0.97% (fast) speedup, with +4.1% and +0.3% SSIM. Our submitted paper already reports:

SkyReels-V2Speedup (MotionCache → FSC)SSIM (MotionCache → FSC)
FSC-Speed, 1.3B2.275× → 2.356× (+3.6%)0.7899 → 0.8088 (+2.4%)
FSC, 14B1.792× → 2.121× (+18.4%)0.7600 → 0.7745 (+1.9%)

Faster and more faithful: FSC-Speed is both faster and more faithful, and at 14B without retuning the speed advantage grows to 18.4%. At matched speed (2.275×), FSC-Quality gains +0.031 SSIM (+3.9%); submitted operating points reach 2.445× (Table 6). A faster point (App. E.1) at 2.845× gains +0.033 SSIM over FlowCache while running 14% faster, and +0.0452 SSIM over retuned MotionCache.

Why report same-seed fidelity alongside VBench? In the submitted Table 1, FSC-Quality's VBench is not lower than MotionCache's (0.7900 vs. 0.7898 on SkyReels-V2; 0.7954 vs. 0.7950 on MAGI-1), while its fidelity is substantially higher at equal or lower latency (SSIM 0.8209 vs. 0.7899; 0.8883 vs. 0.8468). VBench scores perceptual and aesthetic quality without a ground truth; same-seed SSIM measures consistency with the unaccelerated Vanilla output.

ObservationFinding
MAGI-1FlowCache exceeds its own Vanilla VBench despite the lowest same-seed SSIM
SkyReels-V2, 275-video comparisonPer-video VBench vs. fidelity differences: Spearman ≈ 0.00

Interpretation: Whether acceleration preserves the original trajectory requires same-seed metrics such as SSIM; we therefore treat same-seed fidelity as a primary criterion for caching methods, complementary to VBench.

W4: Generalization to additional AR backbones

Thank you for pointing out that the submission lacked results on recent AR backbones. FSC transfers to two additional multistep AR backbones without retuning (details in App. E.3).

1. Causal Forcing: same-seed fidelity at matched speed

MethodSSIMPSNRLPIPS
MotionCache port0.756317.090.3174
FlowCache port0.677714.720.3997
FSC0.785518.330.2966

FSC improves SSIM by +0.029 over the MotionCache port (PSNR +1.24 dB, LPIPS −0.021) and by +0.108 over the FlowCache port.

2. Acceleration and fidelity relative to Vanilla

Backbone / SettingFSC SpeedupFSC SSIM
Causal Forcing2.19×0.7855
LongLive-2.0 (multi-step base), 125 frames2.34×0.8619
LongLive-2.0 (multi-step base), 381 frames (12 AR blocks)2.449×0.6992

LingBot-World-Infinity: its released checkpoints are a short distilled 1.3B model or a 14B model documented with eight GPUs, so we prioritized Causal Forcing and LongLive-2.0; Matrix-Game-3.0 adds an interactive AR world model (Jssp W1).

Reviewer_6n53OpenReview ↗

Response to Authers of Submission26837

Thanks to the authors for their detailed rebuttal. The response has addressed most of my major concerns, and I have updated my score from Reject to Weak Accept.

I still have two questions regarding the quality evaluation:

  1. Do the results on LongLive-2.0 suggest that the quality degradation introduced by acceleration becomes more severe as the number of generated frames increases? I wonder whether this could become a more significant issue for longer video generation.

  2. Unlike bidirectional-attention video generation, acceleration methods for AR video generation often report PSNR values below 20 dB, suggesting substantial divergence from the original generation trajectory. In this setting, to what extent can reference-based metrics such as PSNR and SSIM reflect the actual perceptual quality of the generated videos, rather than simply their similarity to the unaccelerated outputs?

Reviewer_LyvNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Frequency-Step Cache (FSC), a training-free controller for accelerating autoregressive video diffusion. FSC places exact full-computation anchors on a dense-to-sparse schedule and uses a latent-space signal combining low-frequency drift, high-frequency drift, and token age to choose whole-step replay, selective token repair, or full repair between anchors. A distinguishing state-management choice is that scheduled anchors replace the latent reference used to measure drift, whereas risk-triggered full repairs retain it. The paper motivates these choices through profiling and evaluates speed and fidelity on SkyReels-V2, a native-cache adaptation on MAGI-1, and the separation principle on FramePack and Matrix-Game. At MotionCache-matched latency on SkyReels-V2 1.3B, FSC-Quality improves same-seed SSIM by 0.031, PSNR by 1.04 dB, and LPIPS by 0.024; the VBench difference is statistically unresolved. Additional experiments examine baseline operating points, seeds, hardware, model scale, motion, schedules, and hyperparameter sensitivity.

Critical strengths
  1. Better fidelity at matched latency. FSC-Quality improves SSIM by 0.031 and reduces LPIPS by 0.024 over MotionCache on SkyReels-V2, with fidelity gains reproduced across multiple seeds (Table 1; Appendix E.4).

  2. Useful evidence for the combined design. Ablations support both scheduling and repair, and the advantage over the tested reactive comparator extends to FramePack and Matrix-Game (Tables 3, 8, and 16).

Critical weaknesses
  1. The synchronization claim is not cleanly isolated. The reactive comparator changes both synchronization timing and when the latent reference is reset (Appendices B.4 and E.2). Its lower fidelity therefore cannot be attributed to reactive synchronization alone, weakening the broader claim that synchronization should be scheduled independently of repair.

  2. The practical value of frequency-based routing remains unclear. Elapsed steps alone strongly predict reuse error (Spearman 0.865 versus 0.893 for the full risk), but there is no matched-latency comparison against this simpler routing rule. The diagnostic also modifies the K/V cache (Appendix C), so the reported correlations do not establish the signal’s added benefit during normal generation.

Questions for authors
  1. How much of FSC’s gain comes from scheduled anchors versus its latent-reference reset rule? Can these effects be separated through a matched-latency comparison?

  2. Does the full risk outperform elapsed-step-only routing at matched latency, and do its diagnostic correlations hold when oracle probes leave the K/V cache unchanged?

1 comment

Response to Reviewer LyvN

We thank the reviewer for identifying that our reactive comparator confounds synchronization timing with reference reset, and for asking whether a simpler routing rule suffices. Both points led to the new controlled experiments below.

W1 + Q1: Scheduled synchronization versus reference reset

We isolated scheduled synchronization from reference reset at matched latency (Table 23; 21 evaluation prompts scored with the fidelity harness of Table 1, so absolute values differ from Table 3; the no-anchor row is Table 3's matched-latency purely reactive comparator).

ConfigurationWall time (s)SSIM
Scheduled anchors + reset159.90.8294
Scheduled anchors + no reset (FSC-Quality)161.20.8272
No scheduled anchors + reset (Table 3 comparator)160.30.6849
  • Scheduled synchronization: Holding reset fixed, scheduled anchors improve SSIM by +0.1445.
  • Reference reset: Holding anchors fixed, reset changes SSIM by +0.0022 and latency by −0.8%.
  • Interpretation: Scheduled exact synchronization drives the fidelity gain. Our contribution, scheduling exact synchronization independently of adaptive repair, does not depend on the reset setting.

W2 + Q2: Frequency routing versus elapsed steps; K/V oracle

Short answer: Yes, the full risk outperforms elapsed-step-only routing at matched latency, and its diagnostic correlations still hold with the K/V cache left unchanged.

Only the routing signal is replaced; repair levels and token selection are kept (Table 24).

MethodMean SSIM
Binary routing0.8367
Graded routing0.8390
FSC (FSC-Quality operating point)0.8418

At matched latency, FSC improves SSIM by +0.0051 over binary and +0.0028 over graded routing; the +0.0028 is about 9% of FSC's margin over MotionCache.

K/V-restoring oracle (Table 25, App. E.16; oracle methodology in App. C):

Risk signalPartial Spearman (controlling for the other signal)ROC-AUC for reuse error
Elapsed-step-only risk0.0620.875
FSC frequency-aware risk0.4580.903

Key finding: Frequency-aware risk carries predictive information beyond elapsed steps. The oracle restores K/V states exactly (612/612), and its routing agreement with normal runs (422/432) matches the agreement between two normal runs (425/432).

Comments on the paper

1 comment

Response to All Reviewers

We thank the reviewers. In response, we ran the following new experiments:

  • Separation of synchronization and repair. A controlled experiment that changes only the synchronization schedule shows that scheduled exact synchronization drives the fidelity gain (+0.1445 SSIM).
  • Frequency-aware routing. With everything else fixed, FSC's frequency-aware risk still outperforms elapsed-step routing at matched latency and carries predictive information beyond elapsed steps.
  • Longer horizons. We extended SkyReels-V2 from 177 to 817 frames (window 10) and LongLive-2.0 to 381 frames. FSC's cumulative SSIM stays above MotionCache's at every evaluated window up to 817 frames (+0.0169 at window 10), and FSC keeps a 2.449× speedup at 381 frames on LongLive-2.0.
  • Complete controller on MAGI-1. Re-run with the complete controller on MAGI-1, FSC remains more faithful than MotionCache (+0.033 SSIM) while running 7% faster.
  • New AR backbones. FSC transfers without retuning to Causal Forcing (+0.029 SSIM over a MotionCache port) and LongLive-2.0 (2.34×). On the interactive world model Matrix-Game-3.0, already in the submission, the unchanged SkyReels-V2 configuration reaches 2.62×.

All baseline comparisons are latency-matched unless a speed difference is stated. FSC-Quality uses one fixed configuration per backbone wherever it is reported; settings, prompt sets and baseline ports are documented in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.