acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Causal Video Generation with Mixed Sparse Attention

Abstract

Causal video diffusion has emerged as a streaming alternative to bidirectional models that denoise entire clips before playback. These autoregressive models use a key/value cache, but attention costs grow with visible history. However, causal masking changes attention support, so directly adapting bidirectional sparse attention can degrade quality. Prior work also leaves attention patterns across chunks, denoising passes, and horizons underexplored. We find stable spatial-local heads, weakening recent-window support, and changing reads despite immutable cached keys. Therefore, Mixed Sparse Attention (MSA) combines static/content routing, horizon/cost gates, reusable statistics, and budgeting across passes. For 20-second Causal Forcing generation at 720p, MSA achieves up to 2.634 denoising and 1.550 end-to-end speedup over dense attention. Moreover, MSA achieves comparable VBench and VBench-Long scores to the matched sparse-attention baseline across six evaluated dimensions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
sparse attention, video generation, efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5jRUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MSA (Mixed Sparse Attention), a training-free mixed sparse attention inference scheduling framework for causal video diffusion. Without fine-tuning the model, the method dynamically selects sparse attention strategies during autoregressive video generation via offline head behavior calibration, runtime gated routing, global budget auditing, and incremental history statistics to accelerate long-video inference. The authors conduct experiments on the Causal-Forcing causal variant of Wan2.1‑1.3B and verify speed gains against multiple sparse attention baselines. They also introduce a real density auditing mechanism to address the issue of inflated speedup ratios in prior sparse methods. However, multiple limitations exist in the experimental evaluation that undermine the reliability of conclusions.

Critical strengths
  1. This work proposes global real density auditing, which measures the actual computational cost across all denoising steps, cache updates, and dense fallback operations. It resolves the common issue in prior sparse attention works that only report nominal sparsity and overestimate acceleration performance, bringing practical value to the evaluation paradigm of this field.
  2. For causal video diffusion scenarios, the authors systematically observe the behavior of different attention heads in the Transformer and design a gated dynamic mixed sparse scheduling scheme. The approach is plug-and-play at inference time and requires no parameter fine-tuning of the pre-trained model, achieving considerable inference speedup for long-video CF-long settings.
Critical weaknesses
  1. Insufficient experimental evaluation and weak generalization. All experiments are validated solely on a single AR model obtained via Causal-Forcing distillation of Wan2.1‑1.3B. Evaluation uses only a subset of VBench prompts, and main experiments adopt a single random seed, making it impossible to rule out metric bias introduced by sampling noise. Baselines lack representative recent bidirectional sparse attention methods for video, leading to incomplete comparisons.
  2. Counterintuitive experimental observations lack rigorous explanation. In the RF‑5s experiment, multiple quality metrics of MSA exceed those of dense Full Attention, while dense attention is generally the performance upper bound for sparse approximation methods.
Questions for authors
  1. Why does the evaluation not adopt the full set of VBench prompts and report the performance of all evaluation metrics?
  2. Some metrics of MSA outperform dense full attention, which is generally regarded as the theoretical upper bound of sparse attention methods. What are the potential explanations for this counterintuitive phenomenon?
  3. Will the authors consider incorporating more state-of-the-art bidirectional sparse attention methods (e.g., VSA, SLA) as additional baselines for comprehensive comparison?
  4. All experiments are only conducted on the AR model of Wan2.1-1.3B, which leads to limited generalization and insufficient persuasiveness. Could the authors supplement experimental results on other models to validate the effectiveness of the proposed MSA?
Reviewer_ndSrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method that accelerates model inference by reducing attention computations. The experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The motivation is clear, and the paper offers a novel approach to accelerating model inference by modifying the attention mechanism.
  2. The paper is easy to follow, and the proposed method avoids the need to retrain the entire model, substantially reducing computational costs.
Critical weaknesses
  1. The paper does not provide visual examples of videos generated using the proposed method. Does the method introduce any visual quality degradation or other negative effects? Qualitative comparisons are needed to assess these potential trade-offs. Quantitative metrics alone are insufficient, as benchmark scores can be susceptible to metric gaming and may not fully reflect perceptual quality.
  2. Although the method approaches the problem from a novel perspective, its design is relatively complex. More experiments are needed to establish its generalization and robustness across different models and settings.
Questions for authors

See weaknesses.

Reviewer_YsCrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores redundancies in the attention mechanism of causal video diffusion models (CVDMs). The authors observe that many approaches aiming at making CVDMs more efficient introduce artifacts such as subject inconsistency (Fig.1), and propose a training-free method to make attention more efficient in CVDMs at inference time.

Four key observations about the behaviour of attention in CVDMs motivate the proposed approach: the existence of a subset of heads consistently attending to the same spatial regions over time, with different temporal patterns (o1), the importance of long context (o2), variability of how KV caches are used throughout rollouts (o3), and the importance of evaluating "attention density" over every pass (o4). These observations are based on three prompts, presumably from Wan2.1-1.3B.

Each observation motivates an element of the proposed method, MSA:

  • o1: each head is assign a role
  • o2: routing gates
  • o3: append only pool history.

The method is then tested on generation quality and performance against other efficient attention variants.

Critical strengths

None

Critical weaknesses

The paper is extremely hard to read.

The paper does not do a good job at formulating the problem and explaining each proposed solution: it reads as an exploratory analysis, and turning this into a research paper will require a much stronger framing and more comprehensive evaluation.

Moreover, the paper claims that "full tables, temporal profiles, ablations and five RF frame-comparison grids" are in a "separate extended appendix", which is not provided to the reviewer.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.