acceptodds
Under review as a conference paper at ICLR 2027

Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation

Abstract

Diffusion Transformers (DiTs) have achieved strong performance in image and video generation, but the quadratic complexity of full attention makes high-resolution generation computationally expensive. Window attention offers an efficient alternative, yet existing methods face a practical trade-off: partitioned window attention typically achieves computational efficiency consistent with its theoretical complexity. However, isolated windows block cross-window interaction, often introducing visible grid-like artifacts in the generated results. Fine-grained sliding-window attention effectively restores interactions across neighboring windows and improves visual quality. However, its irregular computation patterns create a substantial gap between theoretical and practical speedups and require specialized kernels tailored to each hardware backend. To tackle these challenges, we propose BASA, a backend-agnostic sparse attention, which brings the best of both worlds: visual quality and practical acceleration. Specifically, BASA replaces visual self-attention with shifted local-window attention. By introducing a structured window-shifting scheme across DiT blocks, we allow tokens divided by window boundaries in one layer to communicate in the following layers, thereby achieving global information exchange and eliminating window-induced visual artifacts. Notably, our design introduces no additional irregular operators or customized kernels, making it readily deployable on existing attention backends and closing the gap between theoretical sparsity and practical acceleration. Experiments demonstrate that BASA achieves measured speedups exceeding 90% of the theoretical estimates on FLUX and delivers a 4.52 attention speedup on Wan while maintaining competitive generation quality.

Primary area:
generative models
Keywords:
Generative Diffusion Model, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vT9eOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BASA, a backend-agnostic sparse attention mechanism for accelerating high-resolution image and video generation with Diffusion Transformers (DiTs). The method combines shifted local-window attention across layers and denoising steps with pooled global K/V representations to enable cross-window communication while reducing attention computation. For video generation, it additionally introduces content-adaptive window routing to preserve fine-grained information. The approach uses standard attention backends without requiring customized sparse kernels, but requires LoRA-based dense-teacher distillation to maintain generation quality.

The authors evaluate BASA on FLUX.1-dev for image generation and Wan2.1-1.3B for video generation, comparing against Swin, CLEAR, and STA. Experiments show up to 2.68× and 4.52× attention-level speedups for image and video generation, respectively, while largely preserving generation quality. Ablation studies further examine the contributions of window shifting, global pooling, and routing.

Critical strengths

BASA achieves substantial measured attention speedups using standard SDPA backends, reaching up to 2.68× on FLUX and 4.52× on Wan, and the method is claimed to be backbone agnostic.

Critical weaknesses

As stated above, I have two main concerns. First, the method requires additional fine-tuning through dense-teacher distillation, but the paper does not compare against training-free sparse attention methods. This makes it difficult to assess the practical advantages of BASA, especially when considering the additional adaptation cost. Second, I am not fully convinced by the novelty of the proposed window-shifting mechanism. Similar ideas, including shifting attention windows across denoising steps, have already been explored in previous works such as HiDiffusion. I would like to see a clearer discussion of how BASA differs from these approaches and what improvements come specifically from its proposed shifting strategy.

Questions for authors
  1. How does BASA fundamentally differ from previous sliding window attention works, why it might yield better results?
  2. Could you compare BASA with existing training-free sparse attention methods under the same generation settings?
Reviewer_Vj1EOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes backend-agnostic sparse attention (BASA), a trainable sparse attention method for both image and video generation. BASA integrates both sliding window and Top-K attention mechanisms, balancing the trade-off between computational efficiency and model performance. The authors also design a distillation stategy to finetune a sparse version of a dense attention model. The finetune experiments on FLUX.1 and WAN 2.1 show that BASA outperforms previous sliding window attention algorithms.

Critical strengths
  • The paper introduces shifted window sparse attention, which improves standard sliding window attention by using different offsets in each layer. This design sounds
Critical weaknesses
  • The most critical weakness is the lack of comparison with other SOTA sparse attention methods. Current comparisons are limited to sliding window attention. Other trainable vision sparse attention should also be compared.For vision attention, SpargeAttention [1], SpargeAttention2 [2], and VSA [3] could be compared. For video attention, SVG [4] and SVG2 [5] could be compared. The lack of SOTA comparison makes it hard to evaluate the effectiveness of the proposed method.
  • Some method designs are closely similar to previous works. K/V-Pooled Global Memory is similar to the Hierarchical KV Enrichment in LLSA [6]. Content adaptive true-window routing is identity to the standard sparse Top-K attention used by most recent sparse attention methods. The similarity to previous works limit the technical novelty of the paper.

[1] SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

[2] SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning

[3] VSA: Faster Video Diffusion with Trainable Sparse Attention

[4] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

[5] Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutation

[6] Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers

Questions for authors

Please resolve the concerns in weakness section. I suggest to provide comparison with at least one SOTA sparse attention method, such as SpargeAttention, and clarity the difference between the proposed designs and previous designs.

Reviewer_Dn2LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BASA, which combines shifted local windows, pooled global K/V memory, and video-specific window routing to accelerate high-resolution diffusion generation using standard attention backends. Dense-teacher distillation with LoRA adapts the sparse models. Experiments report 2.443× full-MSA acceleration on FLUX and up to 4.52× attention acceleration on Wan, with broadly preserved quality metrics.

Critical strengths
  1. The method addresses the practical gap between theoretical sparsity and measured acceleration. BASA-32 improves image fidelity over the evaluated sparse baselines at competitive attention latency.
  2. The image efficiency analysis accounts for projection costs rather than reporting attention sparsity alone.
  3. Image and video ablations support the usefulness of the constituent components.
Critical weaknesses
  1. Some closely related literature is missing from the discussion. HilbertA (ICML 2026) combines local attention, cross-layer window shifts, and shared context for diffusion acceleration, and popular general sparse methods like SpargeAttention (ICML 2025).
  2. The ablations establish the contribution of each component but not the advantages of the proposed designs. Concretely, they do not isolate the interleaved shift schedule or content-adaptive routing against alternatives with comparable computation.
  3. End-to-end generation latency is missing.
  4. Backend independence is weakly supported by only the formulation, without extensive performance evaluated across backends.
  5. The Wan adaptation protocol lacks details.
  6. The linear-scaling statement applies only to the local-window branch. With fixed pooling factors, the full method retains an O(N2/(phpw))O(N²/(p_h p_w)) global-memory term, so its overall complexity remains quadratic in the number of spatial tokens. References:
Questions for authors
  1. What existing evidence isolates the benefits of BASA's particular shift schedule and routing rule over simpler alternatives at matched computation?
  2. What are the end-to-end dense and BASA runtimes under the reported generation protocols?
  3. Please specify the Wan adaptation data and training budget, including whether the same settings apply to the baselines.
  4. Tables 3 and 4 use different video baseline configurations. Which quality results correspond to the reported speedups?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.