acceptodds
Under review as a conference paper at ICLR 2027

UnStep: Training-Free Acceleration of Causal Video Diffusion with Fewer Steps Than Distillation

Abstract

Distilling bidirectional multi-step video diffusion transformers into few-step causal models has become a common approach for streaming video generation. While these few-step students are significantly faster than the teachers they are distilled from, they remain slow for real-time generation. In this work we present UnStep, a training-free wrapper that accelerates few-step causal video models at inference by running them with fewer diffusion transformer (DiT) steps than during distillation and limiting the temporal window retained in the attention KV cache. We propose two inference-only mechanisms to recover quality lost by step reduction and attention windowing: renoising the generated latent frames to a near-clean level and reusing the existing clean-cache pass to refine them, and applying truncated SVD to the DiT attention value and output projections. We also accelerate inference with a quality-preserving runtime stack for the DiT and VAE decoder, including more efficient attention calls and KV indexing, fused Triton RoPE with cached coefficients, and VAE decoding with optimized memory layout, precision, and convolution kernels. By reducing computation and optimizing the runtime stack, UnStep sets a new throughput regime for causal video diffusion, by running substantially faster than current methods, reaching 50 FPS on a single H100 without quality loss, and 77 FPS on GB200, all without retraining.

Primary area:
generative models
Keywords:
diffusion, video generation, distillation, few-step, causal, training-free, wrapper, inference, acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_otjnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UnStep is a training-free inference wrapper for distilled causal video diffusion models. It combines DiT and VAE runtime optimizations with fewer denoising steps and a restricted temporal attention window. Two mechanisms recover quality: reusing the existing clean-cache forward pass to refine the generated latent, and applying truncated SVD to selected attention value/output projections. On Self Forcing at 832 × 480, the paper reports 49.8 FPS versus 17.0 FPS for the original implementation, with better VBench score. The runtime-optimized baseline reaches 32.2 FPS before the schedule and context reductions. The same settings are also evaluated on other distilled checkpoints, GB200, image-to-video generation, and longer videos.

Critical strengths
  1. The clean-cache refinement makes productive use of an existing model evaluation. After step and context reduction, this operation raises VBench total from 83.87 to 84.33 without adding a DiT call or reducing reported throughput (Table 1). This is the most compelling contribution. It addresses the quality loss from shortening an already-distilled trajectory without further training.
  2. The system improvement is substantial and clearly broken down. Table 1 separates compilation, DiT/VAE runtime changes, compute reductions, and quality recovery. The fixed configuration also transfers to other distilled checkpoints and evaluation settings within the same backbone family. These results support the practical value of the approach, even if the generality of the quality claim requires more evidence.
Critical weaknesses
  1. Quality preservation is not sufficiently established. Aggregate gains hide tradeoffs, including a 4.04-point decrease in Self Forcing's dynamic degree. More importantly, the long-video evaluation disables cross-clip consistency, leaving long-range identity preservation under the restricted cache untested.
  2. Average FPS does not establish the streaming benefit. Cache refinement adds no DiT call, but makes the current output wait for the cache pass, unlike the original method. This may affect first-chunk latency, since both methods retain four initial denoising steps. Matched-runtime measurements of first-chunk latency and inter-chunk output intervals are needed.
Questions for authors
  1. Why is cross-clip consistency disabled in the long-video evaluation? Can you evaluate subject/background consistency across distant segments of the same video? This would directly affect my assessment of quality preservation under the restricted context.
  2. In the baseline and UnStep implementations, when do VAE decoding and RGB emission occur relative to the clean-cache pass? Please report first-chunk latency and inter-chunk output intervals under matched runtime optimizations, including a with/without-refinement comparison at the same denoising schedule and attention window.
  3. How was the V/O truncation range of layers 8–29 selected: fixed in advance, or selected through validation? What candidate ranges were considered, on which data, and how sensitive is the result to the starting layer? The LASER motivation supports avoiding early layers, but does not establish this particular boundary for the video DiT. What if we change the selected layers to 8-20 for example?
5 comments
AuthorsOpenReview ↗

Response to Reviewer otjn

Many thanks for your thorough review, for appreciating the contribution of our work, for noting that the improvements are substantial and clearly broken down, and for recognizing the practical value of our approach. We address your concerns below.

 

1. Quality preservation

Please see point 2 of our common response where we explained that the 4 point decrease in Self Forcing’s dynamic degree was due to run-to-run variance and that on average the max decrease is 1.52 points in dynamic degree and max increase is 2.87 in spatial relationship. We also created an anonymous project page with videos generated with and without UnStep for qualitative analysis.

Regarding long video evaluation, we did not selectively disable cross-clip consistency, we followed the official VBench-Long defaults which exclude this component. Following your request, we evaluated cross-clip consistency (subject and background), averaged across all segments of each video and also measured separately between distant segments: first vs last, first vs middle, and middle vs last. Scores are reported in the table below, and show similar or improved cross clip consistency also between distant frames.

     Consistency        SF      SF + UnStep   
Subject across clips0.8700.890
Background across clips0.9010.929
Subject: first vs last0.6700.799
Background: first vs last0.7400.874
Subject: first vs middle0.8240.868
Background: first vs middle0.8700.917
Subject: middle vs last0.8010.844
Background: middle vs last0.8380.907

 

2. Streaming latency

Please see point 3 of our common response, where we report first-chunk RGB latency and inter-chunk output intervals for every cumulative step of UnStep. In these measurements, VAE decoding and RGB emission occur before the cache pass without refinement (C0 - C4) and after it with refinement (C5, C6). Compared with original Self Forcing, UnStep reduces first chunk latency to about half and inter-chunk intervals to about a third, which enables earlier first frame emission and overall faster streaming.

 

3. V/O truncation range

Like all other UnStep design choices, the V/O truncation layer range of 8-29 was selected empirically on Self Forcing evaluated with VBench 5s text to video, then fixed for all other experiments. Following your request, we ablated the layer range, while keeping all other settings fixed. We tested the requested 8-20 range in addition to earlier or later starting layers, and reported the results in the table below. The results show limited sensitivity for small changes in the layer range.

     V/O layers        VBench total   
8–29, default84.56
8–2084.50
12–2984.47
2–2984.40

 

We hope your quality preservation and streaming latency concerns have been correctly addressed, we are happy to clarify further!

Thanks authors for the answers. I have a further question here. Appendix A.7 shows that clean-cache refinement reduces VBench total from 84.31 to 83.88 under the original four-step Self Forcing setup, whereas it improves quality under the shortened schedule. Could you clarify why its effect reverses? This raises the concern that the benefit may be specific to particular configurations, rather than reflecting a generally effective refinement mechanism across models and inference settings.

Thanks for your question and your further engagement.

The clean-cache refinement and truncated SVD are designed to reduce errors introduced by the shortened schedule. Their parameters were tuned for the shortened schedule, so the same settings can be too aggressive when applied to the full schedule. Using less aggressive settings (SVD retention factor of 0.99 and a clean-cache noise level of 0.001), we obtained an increase of 0.1 points in VBench total over the baseline with the full schedule.

However, since UnStep is designed to recover quality after step reduction, we believe that its transfer using the same parameters and design choices across backbones (Wan, Cosmos, SANA) and generation tasks (text-to-video, image-to-video, long-video) provides more general evidence of effectiveness across models and inference settings than applying the same refinement parameters to the original full schedule.

Reviewer_3ELCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is describes an engineering pipeline to dramatically speed up autoregressive video generation methods, using a set of inference-only techniques. The method speeds up the method with runtime engineering (KV position indices and optimized kernels and datatypes), reducing the context, and reducing the number of DiT steps, and makes up for the lost quality by truncating SVD and re-caching with a small noise to refine the image. The result is about 2-3x inference speedup with no reduction in quality, as measured by VBench on Causal Forcing, Self-Forcing, and Long Live baselines. There is also a cumulative ablation table with all of the components.

Critical strengths

The paper is well-organized and the method is easy to follow. Table 1 adds each component cumulatively (C0-6), showing FPS speed and quality through each step. It shows substantive speedups (3x) over related methods (Self-Forcing, Causal Forcing, and Long Live), while maintaining quality according to metrics.

Critical weaknesses

My primary concern is that this is primarily a systems engineering paper, with a large bag of techniques put together, with limited ML insight.

  • The runtime stack (C0-C2) of kernel fusion and avoiding CPU-GPU synchronization is heavily engineering-based. In the writing in Section 2.1, there is no engagement with previous work, leaving the reader unsure if these are newly proposed techniques or previously-used methods being applied to this specific setting. There is some work relegated to Appendix D, but the main text does not reflect or inform the reader on the positioning of this paper's contribution with respect to previous work.
  • The two new ideas in Section 2.2 of clean-cache and truncated SVD are interesting and cause recovery of performance. Curiously, when applied to the full model (Table 5), they reduce performance. Is there a reason for this?
  • Overall, the paper reads like a well-executed systems recipe or technical report, but does not advance the understanding of causal video diffusion.

Since it's an engineering-oriented paper, the thoroughness of the results and evaluation is of greater importance. Given this context, I found the results section rather thin.

  • Though it shows impressive results on 3 checkpoints (Self-Forcing, Causal Forcing, Long Live), these all share the same backbone, so leave the question of broader applicability in the air.
  • A much stronger contribution would be a system study of how many of these methods apply to a larger catalogue of other models and distillation families, which of the many components transfer to those models, and if the same hyperparameters could be used. The current results limit how much a reader can conclude about the method's generality to future methods.
  • Specifically, in Figure 1, Self-Forcing and Causal Forcing are missing from the chart. Cross referencing to the table, they would be the slower methods relative to the other baselines, and relatively high performing. Could the method/engineering stack from this paper be applied to these other baselines that are presented?
Questions for authors

Please see the weaknesses section for the relevant questions.

A point in the communication of the paper is the title is rather awkward and does not convey the meaning of the work. In particular, "Fewer steps than distillation" does not make grammatical sense, as it is comparing a count (number of steps) to a process (distillation). "Used in distillation" would be more accurate phrasing (though wordier). I recommend some improvement in the title.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3ELC

Thanks for mentioning that the paper is well organized, the method is easy to follow, and that our method shows substantive speedups. We address your concerns below.

  • Models beyond Wan

    Following your suggestion, we extended the study to two new backbones beyond Wan: SANA-Video and Cosmos-Predict. Both maintain benchmark quality at substantially higher throughput using the same UnStep settings. Please see point 1 of our common response for the more details.

  • Contribution to causal video diffusion

    Reducing denoising steps while preserving quality is a major challenge in causal video generation and a central motivation for recent distillation methods. After a model has been distilled to a few steps, shortening its schedule further degrades quality.

    rCM [ICLR 26] and AnyFlow [ECCV 26] learn flexible step counts through additional distillation. Surprise Forcing [ECCV 26] reduces steps without training, but shows a quality drop.

    UnStep is the first to jointly reduce the denoising-step and KV attention window of an existing distilled causal model while recovering quality lost entirely at inference, using the existing clean-cache pass and truncated SVD of attention V/O projections. It achieves generation quality competitive with models explicitly trained or distilled for one or two steps, without any additional training or redistillation.

    Combining these compute reductions with our optimized runtime stack substantially accelerates existing distilled causal models. Our new SANA-Video and Cosmos experiments further show that the approach transfers beyond Wan.

  • Runtime stack attribution

    To position our runtime stack relative to existing work, we explain which components we use from standard libraries, which techniques we apply from prior video and LLM inference work, and what we change in Self Forcing. We are the first to combine and jointly optimize these components in a unified runtime stack for few-step causal video generation.

    • Standard libraries: We use PyTorch compilation, FP16 VAE decoding, channels-last 3D layout, cuDNN convolution autotuning and output preallocation.

    • Existing video techniques: We use the same RoPE precomputation and Triton-fusion ideas as Helios [PAMI 2026], but we additionally index the coefficient cache by each chunk’s starting latent frame and grid size, which reduces repeated computation and GPU kernel launches.

    • Adapted from LLM inference: We use CPU cache-position counters as in Hugging Face’s Transformers implementation, but adapt them to repeated video-chunk denoising by maintaining the same cache-write range across the denoising steps.

    • Changes to original Self Forcing: We cache converted scheduler tables instead of converting them on every DiT call and obtain attention lengths from query/key tensor shapes instead of constructing GPU length tensors.

    These runtime optimizations bring Self Forcing to 32 FPS (1.9x speedup) without changing what the model computes. For comparison, Light Forcing [ICML 26] reports 1.28x when combining sparse-attention training and runtime optimizations. Helios [PAMI 26] reports 1.17x for its optimized runtime. Our runtime stack alone also exceeds DiagDistill’s [ICLR 26] 31 FPS, the fastest method in our main H100 comparison, which uses additional distillation, fewer steps and a smaller decoder. We achieve this purely through improved runtime optimization.

  • Applicability to baselines in Figure 1

    Figure 1 shows methods that accelerate inference relative to original Self/Causal Forcing. Most of these methods already reduce denoising or attention work, which overlaps with UnStep, leading to limited additional speedup when combined. However, the new component orthogonal to UnStep is the smaller decoder TAEW2.1 [GitHub 25] distilled from Wan’s decoder and used in DiagDistill [ICLR 26]. Motivated by your question, we replaced Wan’s decoder in UnStep with TAEW2.1, without any training or finetuning. This raises end-to-end throughput from 50 to 73 FPS on H100, with nearly unchanged VBench total (84.54 versus 84.56). For reference, the DiT-only throughput (without decoder) is 81 FPS, showing how closely this variant approaches the DiT-only speed. We thank you for suggesting this direction and view it as an important complement to UnStep.

  • Clean-cache and truncated SVD on full schedule

    Clean-cache refinement and truncated SVD are designed to reduce errors introduced by the shortened schedule, not to guarantee improved performance under any setup. Their parameters were tuned for the shortened schedule, so the same settings can be too aggressive when applied to the full schedule, as shown in Table 5. Using less aggressive settings: an SVD retention factor of 0.99 and a clean-cache noise level of 0.001, we obtained an increase of 0.1 points in VBench total over the baseline with the full schedule.

We hope we addressed all your concerns and are happy to clarify further!

Dear Reviewer 3ELC,

As the discussion period approaches its end, we wanted to check whether you have had the chance to look at our rebuttal and additional experiments. Your suggestions motivated experiments showing that UnStep transfers to other architectures and can be combined with other baselines to accelerate generation even further.

If our responses have addressed your concerns, we would appreciate it if you would reconsider your initial assessment. We are happy to address any remaining questions.

Thank you!

Reviewer_scZcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UNSTEP is a training-free inference wrapper that accelerates already-distilled causal video diffusion models through runtime optimizations, fewer denoising steps, and limited KV caching, while largely preserving overall quality via clean-cache refinement and V/O spectral truncation.

Critical strengths
  1. The idea is interesting.

  2. The paper is well written.

Critical weaknesses
  1. Experiments focus on Wan2.1-1.3B-based causal checkpoints, leaving transfer to other architectures, model sizes, and resolutions unverified.

  2. Aggregate VBench scores mask regressions in individual dimensions, such as dynamic degree, while the one-step variant introduces a clearer quality drop.

  3. The sampling schedule, KV window, refinement noise level, and SVD retention are selected using VBench 5s, without sufficiently establishing their robustness beyond this setting.

  4. Many competing methods’ results are taken from their original papers rather than reproduced under a common software stack, making it difficult to separate methodological gains from implementation advantages.

  5. Reported FPS measures warmed-up generation throughput, excluding initialization, compilation, and text encoding; first-frame latency and interactive response latency are not reported.

  6. For reproduction, do you have a plan to release code, model, and dataset?

Questions for authors

see weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer scZc

Thank you for noting that the idea is interesting and the paper is well written. We address your concerns below.

 

1. Models beyond Wan.

Following your suggestion, we tested UnStep on SANA-Video and Cosmos-Predict. Both maintain their benchmark quality while running much faster with UnStep. Please see point 1 of our common response for the results and details.

 

2. Individual VBench dimensions and the one-step variant.

Please see point 2 of our common response for the individual dimensions. Regarding the one-step variant, we agree that it introduces a quality drop, and clearly present it as a speed-quality trade-off in Section 3.4. It offers comparable quality to methods explicitly trained for one-step generation, but at significantly higher throughput. Our quality-preservation claim concerns the default two-step configuration.

 

3. Hyperparameter selection and transfer.

As explained in the Introduction, Section 3, and Section 4 (Limitations), all UnStep parameters were selected in one setting: Self Forcing, a single H100, and VBench 5s T2V. We then held them fixed when testing Causal Forcing and LongLive (§3.1), GB200 (§3.3), image-to-video generation (§3.5), and long video generation (§3.6). These experiments evaluate transfer beyond the tuning setting. The new SANA and Cosmos results further extend this validation to new architectures using the same parameters and design choices.

 

4. Measured and reported comparisons.

For every model we tested (Self Forcing, Causal Forcing, Long Live), we measured quality and throughput ourselves, both before and after applying UnStep. This also now includes our new comparisons on SANA-Video and Cosmos-Predict. Also, table 1 shows the cumulative effects of UnStep’s components on throughput and quality, making each component’s contribution explicit. All these measurements were done under the same evaluation setup and timing protocol, isolating any implementation advantages.

 

5. Streaming latency.

Please see point 3 of our common response for measurements of first-chunk RGB latency and inter-chunk output intervals. These latency measurements include text encoding and VAE decoding.

 

6. Reproducibility.

As stated in the Reproducibility Statement on page 10, we will release our code and reproduction instructions on our GitHub repository. UnStep is training-free and introduces no newly trained model or dataset. It uses publicly available models and benchmarks.

 

We hope this addresses all your concerns. We are happy to clarify further!

Dear Reviewer scZc,

Thanks once again for your review and your positive assessment of our work. Your suggestions to verify UnStep's transfer to other backbones and report streaming latency have helped strengthen our contribution.

As the discussion period approaches its end, we wanted to check if you have any more questions. We are happy to clarify any remaining concerns.

Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common response to all reviewers

We thank the reviewers for their thoughtful reviews and constructive feedback. We appreciate that all reviewers acknowledge UnStep’s significant speedups and value for causal video generation.

We address the common reviewer concerns here, and respond to each reviewer individually below.

 

1. Reviewers 3ELC and scZc: Models beyond Wan

While Wan is the most widely used model for causal distillation, we fully agree with the reviewers that the transfer of UnStep to other architectures would significantly strengthen its contribution. To this end, we tested UnStep with its default parameters on two new models: SANA-Video-2B [a] and Cosmos-Predict2.5-2B [b] distilled with CMD [c]. UnStep is fully applicable to Cosmos, whereas for SANA, we omit the attention windowing as SANA uses linear attention with fixed size state.

As before, we report end-to-end FPS on H100 at 832x480 resolution, and follow the evaluation pipelines in the respective papers [a,c]. We report the results in the table below. Despite being tuned on Wan, UnStep transfers well to the new architectures: it maintains their quality and significantly boosts their speed, all at inference only.

[a] Chen et al., SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer, ICLR 2026
[b] NVIDIA, World Simulation with Video Foundation Models for Physical AI, 2026
[c] Bandyopadhyay et al., Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation, 2026

   Model      Benchmark      Original total      + UnStep total      Original FPS      + UnStep FPS   
SANAVBench T2V83.4383.451742
CosmosVBench I2V88.3888.661236

 

2. Reviewers otjn and scZc: Individual VBench dimensions

We fully agree that VBench aggregates can hide individual dimensions, which is why we included all 16 individual dimensions in Table 10 of the submitted paper. All UnStep scores in the paper (including Table 10) are averaged over five runs, as explained in Appendix C.1. However, the Self Forcing baseline in that table came from a single run, making it more sensitive to run-to-run variance. We therefore ran original Self Forcing five times and updated Table 10 in the revised paper with its averaged dimensions. Compared to the updated Self Forcing average, the largest decrease with UnStep is 1.52 points in dynamic degree, and the largest increase is 2.87 points in spatial relationship.

To assess whether these changes correspond to visual differences in video quality, we invited ML researchers to inspect generated videos from original checkpoints with and without UnStep. 75% reported no noticeable difference, 15% preferred the wrapped checkpoints, and 10% preferred the originals. We invite the reviewers to inspect the comparisons themselves at our anonymous project page: https://anonymous2192.github.io/unstep_website/

 

3. Reviewers otjn and scZc: Intermediate latency

We agree that first chunk and inter-chunk latencies are important to report for streaming. In the table below we report both (in milli seconds) for the original Self Forcing checkpoint and each cumulative UnStep configuration. First-chunk latency ends when the first RGB frames are available. Inter-chunk latency is the mean interval between consecutive RGB outputs across all later chunks of a 5s clip.

All configurations use four denoising steps for the first chunk. The first chunk also has no preceding attention context, so its main speedup comes from the runtime stack. Later chunks additionally benefit from step reduction and attention windowing. The cache pass slightly increases the first frame latency but doesn’t affect the overall FPS since it does not add a DiT pass.

     Configuration        First chunk RGB latency      Inter-chunk latency   
C0: Original SF536720
C2: C0 + Runtime stack241375
C3: C2 + Step reduction241262
C4: C3 + Limited attention241231
C5: C4 + Clean-cache refinement281225
C6: C5 + SVD281225

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.