- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MC-Sparse, a training-free sparse attention framework for accelerating diffusion transformers. The authors identify three sources of the dense–sparse attention gap: structural binding error, selection error, and discarded tail error. To address them, MC-Sparse combines token-level KV selection, Fast PDDP-based query grouping, cross-step reuse of attention indices, and temporal residual compensation. The experiments on video and 3D generation models show the inference efficiency and generation quality.
Critical strengths
- Practical and efficient implementation. The GPU-efficient query grouping and token-sparse attention kernel make fine-grained sparse attention computationally feasible, achieving meaningful speedup.
- Comprehensive experimental validation. The paper evaluates on video and 3D diffusion models, with enough comparisons demonstrating consistent improvements.
- Straightforward motivation. The motivation for accelerating DiT inference is clear and easy to understand. The proposed method directly addresses the limitations of existing sparse attention approaches through finer-grained token selection, query grouping, and residual compensation.
Critical weaknesses
- Limited methodological novelty. The identified structural binding errors in block-sparse attention are largely intuitive and have been explored in prior work. Similarly, temporal consistency across diffusion steps and residual compensation are established ideas. The proposed method seems to primarily combine existing techniques.
- Predominantly engineering contribution. While the implementation is efficient, the main contributions appear to lie in system optimization. The paper does not sufficiently establish what new insights instead of combining these components, which limits its novelty.
- Insufficient ablation studies. The paper lacks sufficient ablation studies on important hyperparameters, particularly the anchor step schedule and query group size. The rationale behind their selection is not clearly justified, and their impact on generation quality and inference efficiency remains unclear.
Questions for authors
- Anchor step selection: How are the anchor steps determined? Table 7 specifies fixed anchor steps (e.g., 10 and 26), but the rationale for these choices is unclear. How sensitive are generation quality and inference efficiency to the anchor schedule?
- Query block size: How is the query group size determined? Section 4.1 states that it is aligned with the GPU kernel's query tile size, but does not sufficiently justify this choice. Could the authors provide a sensitivity analysis showing how different group sizes affect attention fidelity and computational efficiency?
4 comments
Rebuttal for Reviewer m8j2 (2)
W2: New insights beyond combining components
Our analysis and experiments establish five findings that go beyond the individual components and shape the design:
- The oracle analysis bounds each direction and locates the bottleneck. Matched-density oracles give the best fidelity reachable by better selection, by removing KV binding, and by removing query binding (Fig. 2). They show that structural binding, not selection accuracy, is the current bottleneck, so better block selection alone cannot close the gap, and that a discarded tail remains even with exact per-query selection, which calls for compensation.
- Beyond sparse masks, query groups and residuals are also reusable. Prior work such as AdaSpa reuses sparse masks along the denoising trajectory. We find that query groups and dense–sparse residuals are stable enough to reuse as well (Fig. 4, Fig. 8). Reusing all three amortizes accurate grouping and token-level selection and enables compensation, while attention is still computed from the current Q, K, and V.
- Exact selection and reduced block binding make compensation itself more effective. The dense–sparse residual varies less across steps than the attention output, and exact selection stabilizes it further (Fig. 8). Correspondingly, the same residual correction gains 3.54 / 2.88 dB within MC-Sparse but only 1.25 / 1.45 dB on vanilla BSA at 20% / 25% density (Table 10).
- Tile-aligned grouping avoids the waste of variable-sized kernels. Variable-sized clusters only partially fill kernel tiles, so k-means grouping as in SVG2 executes 1.45× its nominal interactions after tile alignment, even with token-level KV (Tables 4 and 9). Avoiding this waste is the core motivation of Fast PDDP: it partitions queries directly into equal-sized, tile-aligned groups, removing the waste by construction (1.00×) at lower grouping cost (Table 5). Grouping methods should therefore be compared at matched realized density.
- Token-level selection and residual compensation reinforce each other. In a leave-one-component-out ablation in the same setting as Table 3 (25% density), removing token-level KV granularity from the full method costs 3.99 dB PSNR, far more than its 0.91 dB increment in Table 3, where it is added before compensation. This matches finding 3: compensation is far more effective with token-level selection. Removing any single component degrades all three fidelity metrics.
These findings are why hardware-aware choices, such as tile-aligned grouping and the token-sparse kernel, are part of the method rather than implementation details, as in STA and VSA.
Rebuttal for Reviewer m8j2 (3)
W3-1 & Q1: Anchor schedule
The first anchor, step 10, is the last dense warm-up step (Table 7), so its dense attention is already computed and exact selection and the residual add little cost. The second anchor, step 26, sits at the midpoint of the 50-step schedule, a natural untuned choice; we used {10, 26} for all video models.
On both Wan2.1-1.3B and 14B, adding a second anchor improves fidelity, and moving it from step 26 to step 18 gains about 0.40 dB PSNR at the same speed. A third anchor costs 5–8% in speed, and its benefit depends on placement: {10, 18, 26} gains 1.55 dB (1.3B) and 1.04 dB (14B), while {10, 34, 42} loses fidelity. We will add this rationale to Table 7 and describe {10, 26} as a practical default rather than an optimized choice.
Wan2.1-T2V-1.3B
| Anchor steps | ΔPSNR (dB) ↑ | ΔSSIM ↑ | Relative DiT speed ↑ |
|---|---|---|---|
| {10} | −2.507 | −0.0308 | 1.058× |
| {10, 18} | +0.398 | +0.0031 | 1.000× |
| {10, 26} (default) | 0.000 | 0.0000 | 1.000× |
| {10, 34} | −0.624 | −0.0056 | 1.000× |
| {10, 18, 26} | +1.552 | +0.0132 | 0.948× |
| {10, 26, 42} | +0.449 | +0.0047 | 0.948× |
| {10, 34, 42} | −0.475 | −0.0040 | 0.948× |
Wan2.1-T2V-14B
| Anchor steps | ΔPSNR (dB) ↑ | ΔSSIM ↑ | Relative DiT speed ↑ |
|---|---|---|---|
| {10} | −1.319 | −0.0151 | 1.084× |
| {10, 18} | +0.396 | +0.0039 | 1.000× |
| {10, 26} (default) | 0.000 | 0.0000 | 1.000× |
| {10, 34} | −0.386 | −0.0030 | 1.000× |
| {10, 18, 26} | +1.039 | +0.0100 | 0.921× |
| {10, 26, 42} | +0.248 | +0.0034 | 0.921× |
| {10, 34, 42} | −0.272 | −0.0012 | 0.921× |
W3-2 & Q2: Query group size
Fidelity is insensitive to the query group size C between 64 and 128, while C = 64 is slower, so we use C = 128, the native query tile size of our FlashAttention-3 kernel on Hopper GPUs (Sec. 4.1). In a small-scale check on Wan2.1-T2V-1.3B and 14B, using the same prompts for every configuration, PSNR differs by at most 0.04 dB and SSIM and LPIPS change in both directions, while the attention kernel with C = 64 runs at 0.97× and 0.89× the speed of C = 128.
A 64-query tile doubles the KV loads for the same queries and removes the overlap between two query subtiles within the kernel, which explains the slowdown. A larger C, such as 256, runs as two 128-query tiles that share one KV selection, so it brings no kernel speedup while forcing more queries to share a selection, a stronger query-binding constraint. We will add this rationale and the comparison to Sec. 4.1.
| Model | C | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Relative kernel speed ↑ |
|---|---|---|---|---|---|
| Wan2.1-T2V-1.3B | 64 | 22.91 | 0.8576 | 0.2077 | 0.971× |
| Wan2.1-T2V-1.3B | 128 | 22.92 | 0.8580 | 0.2063 | 1.000× |
| Wan2.1-T2V-14B | 64 | 26.46 | 0.8705 | 0.2161 | 0.890× |
| Wan2.1-T2V-14B | 128 | 26.50 | 0.8693 | 0.2172 | 1.000× |
We thank the reviewer again for the constructive feedback. In the revision, we will add both ablations, state the rationale for the anchor schedule (Table 7) and the query group size (Sec. 4.1), and improve the presentation of the method. We would be glad to clarify any remaining questions.
Kind Reminder for Discussion
Dear Reviewer m8j2,
Thank you again for engaging in the discussion. As the discussion period closes in less than a day, we would like to kindly ask whether our clarification of the contributions above addresses your remaining concern on novelty. If any point is unclear, please let us know and we will clarify it promptly.
Rebuttal for Reviewer m8j2 (1)
We thank the reviewer for the constructive feedback and for recognizing our practical and efficient implementation, comprehensive experimental validation, and clear motivation. We address each concern below; Q1 and Q2 are answered together with the related ablation weakness (W3).
W1: Novelty relative to prior work
Our contributions are (i) a systematic matched-density oracle analysis, which, to our knowledge, is the first to separate and quantify structural binding, selection, and discarded-tail errors (Fig. 2), and (ii) token-level KV selection, which departs from the block-level selection used by prior training-free sparse attention for video DiTs, together with a kernel that reaches 96–99% of the ideal sparse-attention speedup (Table 6). The table below states what MC-Sparse shares with the closest prior work and what it changes.
The borrowed residual correction is more effective in MC-Sparse than on its own. At 20% / 25% density, adding it to vanilla BSA improves PSNR by 1.25 / 1.45 dB, and adding it to PISA, which already compensates with block means, improves PSNR by only 0.49 / 0.78 dB, consistent with both mechanisms filling the same role. Within MC-Sparse, it improves PSNR by 3.54 / 2.88 dB (Table 10). Its large increment in Table 3 therefore reflects the combination with our grouping and selection designs rather than the correction alone.
To separate our designs from caching itself, we also compare with AdaSpa and DiTFastAttn, which reuse exact block selections and cached residuals respectively, on Wan2.1-14B-T2V at similar denoising speedups (1.48–1.55×). Neither method reports experiments on Wan, and AdaSpa's code is not released, so we reimplement AdaSpa following its paper; for both methods, we start from the configurations used in their papers and adjust them to reach a denoising latency close to that of MC-Sparse. MC-Sparse achieves 3.23 dB and 3.89 dB higher PSNR, so combining existing mechanisms with caching alone does not account for the gains. We will add both methods to the main comparison table.