- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UnStep is a training-free inference wrapper for distilled causal video diffusion models. It combines DiT and VAE runtime optimizations with fewer denoising steps and a restricted temporal attention window. Two mechanisms recover quality: reusing the existing clean-cache forward pass to refine the generated latent, and applying truncated SVD to selected attention value/output projections. On Self Forcing at 832 × 480, the paper reports 49.8 FPS versus 17.0 FPS for the original implementation, with better VBench score. The runtime-optimized baseline reaches 32.2 FPS before the schedule and context reductions. The same settings are also evaluated on other distilled checkpoints, GB200, image-to-video generation, and longer videos.
Critical strengths
- The clean-cache refinement makes productive use of an existing model evaluation. After step and context reduction, this operation raises VBench total from 83.87 to 84.33 without adding a DiT call or reducing reported throughput (Table 1). This is the most compelling contribution. It addresses the quality loss from shortening an already-distilled trajectory without further training.
- The system improvement is substantial and clearly broken down. Table 1 separates compilation, DiT/VAE runtime changes, compute reductions, and quality recovery. The fixed configuration also transfers to other distilled checkpoints and evaluation settings within the same backbone family. These results support the practical value of the approach, even if the generality of the quality claim requires more evidence.
Critical weaknesses
- Quality preservation is not sufficiently established. Aggregate gains hide tradeoffs, including a 4.04-point decrease in Self Forcing's dynamic degree. More importantly, the long-video evaluation disables cross-clip consistency, leaving long-range identity preservation under the restricted cache untested.
- Average FPS does not establish the streaming benefit. Cache refinement adds no DiT call, but makes the current output wait for the cache pass, unlike the original method. This may affect first-chunk latency, since both methods retain four initial denoising steps. Matched-runtime measurements of first-chunk latency and inter-chunk output intervals are needed.
Questions for authors
- Why is cross-clip consistency disabled in the long-video evaluation? Can you evaluate subject/background consistency across distant segments of the same video? This would directly affect my assessment of quality preservation under the restricted context.
- In the baseline and UnStep implementations, when do VAE decoding and RGB emission occur relative to the clean-cache pass? Please report first-chunk latency and inter-chunk output intervals under matched runtime optimizations, including a with/without-refinement comparison at the same denoising schedule and attention window.
- How was the V/O truncation range of layers 8–29 selected: fixed in advance, or selected through validation? What candidate ranges were considered, on which data, and how sensitive is the result to the starting layer? The LASER motivation supports avoiding early layers, but does not establish this particular boundary for the video DiT. What if we change the selected layers to 8-20 for example?
5 comments
Thanks authors for the answers. I have a further question here. Appendix A.7 shows that clean-cache refinement reduces VBench total from 84.31 to 83.88 under the original four-step Self Forcing setup, whereas it improves quality under the shortened schedule. Could you clarify why its effect reverses? This raises the concern that the benefit may be specific to particular configurations, rather than reflecting a generally effective refinement mechanism across models and inference settings.
Thanks for your question and your further engagement.
The clean-cache refinement and truncated SVD are designed to reduce errors introduced by the shortened schedule. Their parameters were tuned for the shortened schedule, so the same settings can be too aggressive when applied to the full schedule. Using less aggressive settings (SVD retention factor of 0.99 and a clean-cache noise level of 0.001), we obtained an increase of 0.1 points in VBench total over the baseline with the full schedule.
However, since UnStep is designed to recover quality after step reduction, we believe that its transfer using the same parameters and design choices across backbones (Wan, Cosmos, SANA) and generation tasks (text-to-video, image-to-video, long-video) provides more general evidence of effectiveness across models and inference settings than applying the same refinement parameters to the original full schedule.
Response to Reviewer otjn
Many thanks for your thorough review, for appreciating the contribution of our work, for noting that the improvements are substantial and clearly broken down, and for recognizing the practical value of our approach. We address your concerns below.
1. Quality preservation
Please see point 2 of our common response where we explained that the 4 point decrease in Self Forcing’s dynamic degree was due to run-to-run variance and that on average the max decrease is 1.52 points in dynamic degree and max increase is 2.87 in spatial relationship. We also created an anonymous project page with videos generated with and without UnStep for qualitative analysis.
Regarding long video evaluation, we did not selectively disable cross-clip consistency, we followed the official VBench-Long defaults which exclude this component. Following your request, we evaluated cross-clip consistency (subject and background), averaged across all segments of each video and also measured separately between distant segments: first vs last, first vs middle, and middle vs last. Scores are reported in the table below, and show similar or improved cross clip consistency also between distant frames.
2. Streaming latency
Please see point 3 of our common response, where we report first-chunk RGB latency and inter-chunk output intervals for every cumulative step of UnStep. In these measurements, VAE decoding and RGB emission occur before the cache pass without refinement (C0 - C4) and after it with refinement (C5, C6). Compared with original Self Forcing, UnStep reduces first chunk latency to about half and inter-chunk intervals to about a third, which enables earlier first frame emission and overall faster streaming.
3. V/O truncation range
Like all other UnStep design choices, the V/O truncation layer range of 8-29 was selected empirically on Self Forcing evaluated with VBench 5s text to video, then fixed for all other experiments. Following your request, we ablated the layer range, while keeping all other settings fixed. We tested the requested 8-20 range in addition to earlier or later starting layers, and reported the results in the table below. The results show limited sensitivity for small changes in the layer range.
We hope your quality preservation and streaming latency concerns have been correctly addressed, we are happy to clarify further!