acceptodds
Under review as a conference paper at ICLR 2027

FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity

Abstract

While the overall inference latency of Video Diffusion Transformers (DiTs) can be substantially reduced through model distillation, per-step inference latency remains a critical bottleneck. Existing acceleration paradigms primarily exploit redundancy across the denoising trajectory; however, we identify a limitation where these step-wise strategies encounter diminishing returns in few-step regimes. In such scenarios, fewer intermediate states can limit opportunities for feature reuse or predictive modeling, motivating complementary forms of acceleration. To overcome this, we propose Frame Interleaved Sparsity DiT (FIS-DiT), a training-free and operator-agnostic framework that shifts the optimization focus from the temporal trajectory to the latent frame dimension. Our approach is motivated by an intrinsic duality within this dimension: the existence of frame-wise sparsity that permits reduced computation, coupled with a structural consistency that calls for maintaining coverage of frame positions in the global spatiotemporal context. Leveraging this insight, we implement Frame Interleaved Sparsity (FIS) as an execution strategy that manipulates frame subsets across the model hierarchy, refreshing all latent positions without requiring full-scale block computation. Empirical evaluations on Wan 2.2 and HunyuanVideo 1.5 demonstrate that FIS-DiT reaches 2.4 speedup while maintaining comparable performance on VBench, providing a scalable and robust pathway toward real-time high-definition video generation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Video Diffusion Transformers, Few-Step Inference, Training-Free Acceleration, Frame-Interleaved Sparsity, Latent-Frame Redundancy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nau4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FIS-DiT reduces per-step computational cost via latent-frame sparse computation within denoising steps. It computes full spatial token on selected anchor frames, reconstructs omitted module outputs through local linear interpolation, rotates anchor phases across sparse blocks, and retains dense computation in sensitive blocks and steps identified through calibration.

On an evaluation subset of 100 prompts, FIS-Fast reports measured end-to-end speedups of 2.09–2.41× across 4-step Wan 2.2 text-to-video and 8-step HunyuanVideo 1.5 image-to-video setups, accompanied by minor variations in aggregate VBench scores. Ablation studies support the contributions of the interleaved scheduling and block/step protection mechanisms.

Critical strengths
  1. Clear Motivation: The paper clearly identifies the diminishing effectiveness of timestep-based acceleration in few-step video DiTs and motivates latent-frame sparsity as a complementary acceleration direction.

  2. Good Analysis: The authors provide systematic empirical analysis of intra-step temporal redundancy, including adjacent-frame dynamics, block-wise heterogeneity, and interpolation errors, which directly motivates the proposed design choices.

  3. Practical Method: FIS-DiT is training-free, lightweight, compatible with existing DiT inference pipelines, and preserves dense spatial layouts for efficient kernels such as FlashAttention.

  4. Comprehensive Experiments: The experiments cover two different video DiT backbones, multiple resolutions and few-step settings, diverse acceleration baselines, as well as targeted ablations on interleaving and risk-aware deployment.

Critical weaknesses

Limited Evidence of Quality Preservation Evidence of quality preservation is confined to aggregate automated metrics on a relatively small evaluation subset. The matched comparison across 100 prompts neither reports paired uncertainty nor results from repeated seeds, and it omits detailed breakdowns of motion or temporal sub-metrics.

Insufficient Execution Details for Independent Implementation The sparse execution details are insufficient to support an independent implementation. Generic module descriptions and block-level execution overviews leave ambiguities regarding the attention, FFN, and residual paths in the actual evaluation, how non-frame conditioning/text tokens are handled, and how temporal positional indices are managed after frame subsampling. Although the high-level execution flow is understandable, these nuances can simultaneously affect both outputs and computational costs.

Questions for authors

see weakness.

Reviewer_r9CGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates inference acceleration for few-step distilled Video DiTs and proposes the FIS-DiT framework, which exploits latent-frame redundancy rather than cross-timestep reuse to achieve sparse computation. Exploring intra-step frame sparsity is a promising direction, and the reported speedups demonstrate encouraging potential. However, in the current manuscript, there remains an empirical gap between the layer-selection metric and the actual feature errors, evaluations on recent representative backbones and popular video generation tasks beyond T2V are insufficient, and relevant few-step baselines and ablation studies remain incomplete. These limitations lead me to recommend a Weak Reject for the current version; however, I may reconsider my evaluation and potentially raise my score if the authors adequately address these concerns during the rebuttal.

Critical strengths
  1. The motivation to shift the acceleration focus from cross-timestep reuse to intra-step latent frame sparsity is well-founded, offering a complementary paradigm for few-step distilled video DiTs where step-wise redundancy degrades.
  2. The framework is entirely training-free and preserves contiguous spatial token layouts within anchor frames, allowing direct compatibility with FlashAttention without custom CUDA kernels or extra feature-cache memory overhead.
  3. The interleaved anchor scheduling is a neat and practical design that periodically refreshes frame positions across layers, effectively preventing approximation errors from accumulating repeatedly on fixed frames.
  4. The observed end-to-end inference speedups are solid across the tested settings, and the framework shows good composability when combined with existing timestep caching methods.
Critical weaknesses
  1. Why are middle blocks selected for sparsity when their relative feature error is 12.4%, versus roughly 4% at the entry and exit blocks? Please explain why low temporal CV is a useful selection criterion despite this higher error.

  2. How does FIS-DiT compare with a few-step method such as DisCa and a simple temporal-downsampling-plus-interpolation baseline? An ablation on boundary-frame protection would also help justify the design.

  3. Does linear interpolation oversmooth fast or non-rigid motion? Please show results on motion-heavy prompts with motion-specific metrics, since aggregate VBench scores over 100 prompts may miss these failures.

  4. How much inference time is spent on frame slicing, interpolation, and scattering? An operator-level latency breakdown and peak VRAM measurement would help explain the gap between FLOP savings and actual speedup.

Questions for authors

Please refer to the specific clarification requests and suggestions outlined in the Weaknesses section above for the rebuttal.

Reviewer_FNCpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FIS-DiT is a training-free framework that accelerates few-step video diffusion by computing selected latent frames and interpolating the outputs of skipped frames. It rotates the selected frames across transformer blocks while retaining full computation in sensitive blocks and final denoising steps. Experiments on Wan2.2 and HunyuanVideo-1.5 report up to approximately 2.4× speedup with comparable aggregate VBench scores.

Critical strengths
  • Practical acceleration without retraining. Frame-level sparsity reduces computation across attention and feed-forward operations while preserving compatibility with standard dense kernels.

  • Effective scheduling and protection mechanisms. Interleaved frame selection and dense computation in sensitive blocks and final steps improve the quality–efficiency trade-off, supported by component ablations.

Critical weaknesses
  • Insufficient robustness validation. Evaluation on 100 prompts and aggregate VBench scores does not adequately establish whether motion, dynamic textures, and semantic fidelity are preserved under aggressive frame sparsity.

  • Limited justification for the sparsity criterion. The calibration relies on temporal variation in feature magnitudes, which does not directly measure full-feature interpolation error or final video degradation. Its reliability across different content types remains insufficiently established.

Questions for authors
  • Does degradation increase with motion complexity, particularly for dynamic textures and rapid motion? A focused analysis of existing results—including motion continuity and flicker under interleaved reconstruction—would clarify whether aggregate VBench scores conceal temporal degradation.

  • How reliably does the CV-based calibration identify regions where interpolation is safe? Since feature-magnitude variation does not directly measure reconstruction error, evidence linking calibration scores to approximation errors or final quality would strengthen the justification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.