acceptodds
Under review as a conference paper at ICLR 2027

MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers

Abstract

Sparse attention is a primary approach to reducing the latency of diffusion transformers in long-sequence generation tasks, such as video and high-resolution 3D asset generation. However, existing methods can degrade generation quality and fidelity at high sparsity levels. Through controlled oracle comparisons, we trace this degradation to three sources: constraints imposed by token grouping, inaccurate interaction selection, and the attention contributions lost when tokens are discarded. Guided by this analysis, we propose Meta-Cached Sparse Attention (MC-Sparse), a training-free framework that selects individual key-value (KV) tokens while organizing similar queries into tile-aligned groups for efficient GPU execution. MC-Sparse caches metadata comprising query groups, KV indices selected using exact attention probabilities, and residuals between dense and sparse attention outputs, and reuses them across subsequent denoising steps. Across video and 3D generation models, MC-Sparse achieves higher fidelity to dense-attention outputs and larger denoising speedups than existing sparse-attention baselines, without visible quality degradation. Relative to dense attention, it delivers a denoising speedup on Minimax-H3-Base and a speedup on 3D asset generation, both with negligible quality loss.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sparse attention, efficient attention, video generation, 3d generation, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m8j2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MC-Sparse, a training-free sparse attention framework for accelerating diffusion transformers. The authors identify three sources of the dense–sparse attention gap: structural binding error, selection error, and discarded tail error. To address them, MC-Sparse combines token-level KV selection, Fast PDDP-based query grouping, cross-step reuse of attention indices, and temporal residual compensation. The experiments on video and 3D generation models show the inference efficiency and generation quality.

Critical strengths
  1. Practical and efficient implementation. The GPU-efficient query grouping and token-sparse attention kernel make fine-grained sparse attention computationally feasible, achieving meaningful speedup.
  2. Comprehensive experimental validation. The paper evaluates on video and 3D diffusion models, with enough comparisons demonstrating consistent improvements.
  3. Straightforward motivation. The motivation for accelerating DiT inference is clear and easy to understand. The proposed method directly addresses the limitations of existing sparse attention approaches through finer-grained token selection, query grouping, and residual compensation.
Critical weaknesses
  1. Limited methodological novelty. The identified structural binding errors in block-sparse attention are largely intuitive and have been explored in prior work. Similarly, temporal consistency across diffusion steps and residual compensation are established ideas. The proposed method seems to primarily combine existing techniques.
  2. Predominantly engineering contribution. While the implementation is efficient, the main contributions appear to lie in system optimization. The paper does not sufficiently establish what new insights instead of combining these components, which limits its novelty.
  3. Insufficient ablation studies. The paper lacks sufficient ablation studies on important hyperparameters, particularly the anchor step schedule and query group size. The rationale behind their selection is not clearly justified, and their impact on generation quality and inference efficiency remains unclear.
Questions for authors
  1. Anchor step selection: How are the anchor steps determined? Table 7 specifies fixed anchor steps (e.g., 10 and 26), but the rationale for these choices is unclear. How sensitive are generation quality and inference efficiency to the anchor schedule?
  2. Query block size: How is the query group size determined? Section 4.1 states that it is aligned with the GPU kernel's query tile size, but does not sufficiently justify this choice. Could the authors provide a sensitivity analysis showing how different group sizes affect attention fidelity and computational efficiency?
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer m8j2 (1)

We thank the reviewer for the constructive feedback and for recognizing our practical and efficient implementation, comprehensive experimental validation, and clear motivation. We address each concern below; Q1 and Q2 are answered together with the related ablation weakness (W3).

W1: Novelty relative to prior work

Our contributions are (i) a systematic matched-density oracle analysis, which, to our knowledge, is the first to separate and quantify structural binding, selection, and discarded-tail errors (Fig. 2), and (ii) token-level KV selection, which departs from the block-level selection used by prior training-free sparse attention for video DiTs, together with a kernel that reaches 96–99% of the ideal sparse-attention speedup (Table 6). The table below states what MC-Sparse shares with the closest prior work and what it changes.

AspectClosest prior workSharedChanged in MC-SparseEvidence
Error analysisSVG2 (structural binding), AdaSpa (selection), PISA (discarded tail), each mainly addressing one source—Separates and quantifies all three sources at matched density, then addresses them jointlyFig. 2
KV selection granularityPrior training-free methods for video DiTs (e.g., SVG2, Sol-Attn, PISA, SVG-EAR, AdaSpa)—Selects individual KV tokens instead of KV blocksFig. 2, Table 6
Cross-step reuse of exact selectionAdaSpaExact selection at a few steps, reused at later stepsExact attention mass is aggregated per KV token within each query group, rather than per KV blockTable 3, Fig. 4; +3.23 dB PSNR over AdaSpa at similar speed (table below)
Query groupingSVG2Grouping semantically similar queriesFast PDDP partitions queries directly into equal-sized, tile-aligned groups, avoiding the computation that variable-sized clusters waste in partially filled kernel tiles; it also lowers grouping costTables 4, 9: k-means groups execute 1.45× their nominal interactions after tile alignment vs. 1.00× for Fast PDDP; Table 5 (grouping cost)
Discarded-tail compensationDiTFastAttnAdditive dense–sparse residual correctionFills the role of block-mean compensation (PISA, SVG-EAR), which token-level selection cannot use. Combined with our selection, it gains more than applied alone: the more accurate the selection, the more stable the residual across stepsTable 10: +3.54 / +2.88 dB in MC-Sparse vs. +1.25 / +1.45 dB on BSA and +0.49 / +0.78 dB on PISA (20% / 25% density); Fig. 8 (residual stability)

The borrowed residual correction is more effective in MC-Sparse than on its own. At 20% / 25% density, adding it to vanilla BSA improves PSNR by 1.25 / 1.45 dB, and adding it to PISA, which already compensates with block means, improves PSNR by only 0.49 / 0.78 dB, consistent with both mechanisms filling the same role. Within MC-Sparse, it improves PSNR by 3.54 / 2.88 dB (Table 10). Its large increment in Table 3 therefore reflects the combination with our grouping and selection designs rather than the correction alone.

To separate our designs from caching itself, we also compare with AdaSpa and DiTFastAttn, which reuse exact block selections and cached residuals respectively, on Wan2.1-14B-T2V at similar denoising speedups (1.48–1.55×). Neither method reports experiments on Wan, and AdaSpa's code is not released, so we reimplement AdaSpa following its paper; for both methods, we start from the configurations used in their papers and adjust them to reach a denoising latency close to that of MC-Sparse. MC-Sparse achieves 3.23 dB and 3.89 dB higher PSNR, so combining existing mechanisms with caching alone does not account for the gains. We will add both methods to the main comparison table.

MethodPSNR ↑SSIM ↑LPIPS ↓Imaging Quality ↑Background Consistency ↑Denoising speedup ↑
AdaSpa25.580.8940.17566.5495.451.55×
DiTFastAttn24.920.8710.21765.4095.751.48×
MC-Sparse (Ours)28.810.9120.12867.1296.701.53×
AuthorsOpenReview ↗

Rebuttal for Reviewer m8j2 (2)

W2: New insights beyond combining components

Our analysis and experiments establish five findings that go beyond the individual components and shape the design:

  1. The oracle analysis bounds each direction and locates the bottleneck. Matched-density oracles give the best fidelity reachable by better selection, by removing KV binding, and by removing query binding (Fig. 2). They show that structural binding, not selection accuracy, is the current bottleneck, so better block selection alone cannot close the gap, and that a discarded tail remains even with exact per-query selection, which calls for compensation.
  2. Beyond sparse masks, query groups and residuals are also reusable. Prior work such as AdaSpa reuses sparse masks along the denoising trajectory. We find that query groups and dense–sparse residuals are stable enough to reuse as well (Fig. 4, Fig. 8). Reusing all three amortizes accurate grouping and token-level selection and enables compensation, while attention is still computed from the current Q, K, and V.
  3. Exact selection and reduced block binding make compensation itself more effective. The dense–sparse residual varies less across steps than the attention output, and exact selection stabilizes it further (Fig. 8). Correspondingly, the same residual correction gains 3.54 / 2.88 dB within MC-Sparse but only 1.25 / 1.45 dB on vanilla BSA at 20% / 25% density (Table 10).
  4. Tile-aligned grouping avoids the waste of variable-sized kernels. Variable-sized clusters only partially fill kernel tiles, so k-means grouping as in SVG2 executes 1.45× its nominal interactions after tile alignment, even with token-level KV (Tables 4 and 9). Avoiding this waste is the core motivation of Fast PDDP: it partitions queries directly into equal-sized, tile-aligned groups, removing the waste by construction (1.00×) at lower grouping cost (Table 5). Grouping methods should therefore be compared at matched realized density.
  5. Token-level selection and residual compensation reinforce each other. In a leave-one-component-out ablation in the same setting as Table 3 (25% density), removing token-level KV granularity from the full method costs 3.99 dB PSNR, far more than its 0.91 dB increment in Table 3, where it is added before compensation. This matches finding 3: compensation is far more effective with token-level selection. Removing any single component degrades all three fidelity metrics.

These findings are why hardware-aware choices, such as tile-aligned grouping and the token-sparse kernel, are part of the method rather than implementation details, as in STA and VSA.

AuthorsOpenReview ↗

Rebuttal for Reviewer m8j2 (3)

W3-1 & Q1: Anchor schedule

The first anchor, step 10, is the last dense warm-up step (Table 7), so its dense attention is already computed and exact selection and the residual add little cost. The second anchor, step 26, sits at the midpoint of the 50-step schedule, a natural untuned choice; we used {10, 26} for all video models.

On both Wan2.1-1.3B and 14B, adding a second anchor improves fidelity, and moving it from step 26 to step 18 gains about 0.40 dB PSNR at the same speed. A third anchor costs 5–8% in speed, and its benefit depends on placement: {10, 18, 26} gains 1.55 dB (1.3B) and 1.04 dB (14B), while {10, 34, 42} loses fidelity. We will add this rationale to Table 7 and describe {10, 26} as a practical default rather than an optimized choice.

Wan2.1-T2V-1.3B

Anchor stepsΔPSNR (dB) ↑ΔSSIM ↑Relative DiT speed ↑
{10}−2.507−0.03081.058×
{10, 18}+0.398+0.00311.000×
{10, 26} (default)0.0000.00001.000×
{10, 34}−0.624−0.00561.000×
{10, 18, 26}+1.552+0.01320.948×
{10, 26, 42}+0.449+0.00470.948×
{10, 34, 42}−0.475−0.00400.948×

Wan2.1-T2V-14B

Anchor stepsΔPSNR (dB) ↑ΔSSIM ↑Relative DiT speed ↑
{10}−1.319−0.01511.084×
{10, 18}+0.396+0.00391.000×
{10, 26} (default)0.0000.00001.000×
{10, 34}−0.386−0.00301.000×
{10, 18, 26}+1.039+0.01000.921×
{10, 26, 42}+0.248+0.00340.921×
{10, 34, 42}−0.272−0.00120.921×

W3-2 & Q2: Query group size

Fidelity is insensitive to the query group size C between 64 and 128, while C = 64 is slower, so we use C = 128, the native query tile size of our FlashAttention-3 kernel on Hopper GPUs (Sec. 4.1). In a small-scale check on Wan2.1-T2V-1.3B and 14B, using the same prompts for every configuration, PSNR differs by at most 0.04 dB and SSIM and LPIPS change in both directions, while the attention kernel with C = 64 runs at 0.97× and 0.89× the speed of C = 128.

A 64-query tile doubles the KV loads for the same queries and removes the overlap between two query subtiles within the kernel, which explains the slowdown. A larger C, such as 256, runs as two 128-query tiles that share one KV selection, so it brings no kernel speedup while forcing more queries to share a selection, a stronger query-binding constraint. We will add this rationale and the comparison to Sec. 4.1.

ModelCPSNR ↑SSIM ↑LPIPS ↓Relative kernel speed ↑
Wan2.1-T2V-1.3B6422.910.85760.20770.971×
Wan2.1-T2V-1.3B12822.920.85800.20631.000×
Wan2.1-T2V-14B6426.460.87050.21610.890×
Wan2.1-T2V-14B12826.500.86930.21721.000×

We thank the reviewer again for the constructive feedback. In the revision, we will add both ablations, state the rationale for the anchor schedule (Table 7) and the query group size (Sec. 4.1), and improve the presentation of the method. We would be glad to clarify any remaining questions.

AuthorsOpenReview ↗

Kind Reminder for Discussion

Dear Reviewer m8j2,

Thank you again for engaging in the discussion. As the discussion period closes in less than a day, we would like to kindly ask whether our clarification of the contributions above addresses your remaining concern on novelty. If any point is unclear, please let us know and we will clarify it promptly.

Reviewer_1pjSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Review of MC-Sparse

Summary

This paper proposes MC-Sparse, a training-free sparse attention framework for diffusion transformers. Through controlled oracle comparisons, the authors identify structural binding, inaccurate selection, and discarded tail contributions as sources of approximation error. The proposed method combines token-level KV selection, tile-aligned query grouping, and cross-step reuse of selection metadata and dense–sparse output residuals. Experiments on video and 3D generation demonstrate improved fidelity to dense-attention outputs and reduced denoising latency.

Critical strengths

Strengths

  • The oracle comparisons provide a clear analysis of the limitations of block-sparse attention and motivate the proposed design.
  • The system implementation is substantial, covering efficient query grouping, exact selection, token-sparse attention kernels, and cache offloading.
  • The reported results show a favorable quality–latency trade-off across several diffusion models, with component ablations supporting the usefulness of the design.
Critical weaknesses

Weaknesses

  1. Limited methodological novelty. The approach primarily integrates established ideas, including query clustering, fine-grained sparsification, cross-step reuse, and additive residual compensation. While this integration appears effective and technically nontrivial, the methodological contribution seems incremental. The paper would benefit from more clearly distinguishing its new contributions from existing techniques and explaining what generalizable insight the proposed combination provides.

  2. Incomplete evaluation of video generation quality. The paper reports only two VBench dimensions: imaging quality and background consistency. These do not sufficiently characterize motion smoothness, temporal flickering, subject consistency, dynamic degree, or text alignment. PSNR, SSIM, and LPIPS measure similarity to dense-attention outputs rather than independently establishing generation quality. Thus, a broader scope of benchmarking like VBench and VBench2.0 is necessary for a better evaluation of the generation quality.

  3. Insufficient component and combination ablations. There are two distinct gaps in the ablation study:

    • Component combinations are not systematically evaluated. Table 3 adds components in a fixed order, while Tables 4 and 10 examine selected combinations. These experiments are useful, but do not establish the contribution of each component within the complete framework or whether simpler combinations achieve comparable results. For example, the authors could remove query regrouping from the full method while retaining exact token selection and compensation, or replace exact selection with approximate scoring while retaining the remaining components. Such controlled comparisons should report both final generation quality and denoising latency.

    • The configurations of individual components are insufficiently studied. In particular, the video experiments use two anchor steps at 10 and 26 without varying their number or placement. Since cross-step reuse is central to the method, the effects of refresh frequency on quality and latency need to be established. Figures 4 and 8 analyze local temporal stability, but do not replace experiments that vary the anchor schedule in the complete generation pipeline. Sensitivity to query-group size and dense warm-up length would also help justify the chosen settings. Without these analyses, the robustness of the reported results and the tuning required for other models or sampling schedules remain unclear.

  4. Incomplete characterization of cache costs. The method stores per-layer grouping information, KV indices, and output residuals. Although the appendix discusses offloading and prefetching, reporting peak GPU memory, CPU memory consumption, and transfer overhead for complete configurations would make the practical trade-offs clearer. It is also unclear how well the reported overlap of transfer and computation extends to longer sequences or hardware with lower host–device bandwidth.

Questions for authors
  1. Novelty: Could the authors clarify the main methodological contributions beyond combining existing techniques?

  2. Video quality: Could the authors report the complete scores for both VBench-1 and VBench-2 to provide a more comprehensive evaluation of video generation quality?

  3. Component combinations: Could the authors provide ablations of different component combinations, particularly to establish the necessity of the other components when residual compensation is enabled?

  4. Hyperparameter sensitivity: Could the authors provide sensitivity analyses for the key hyperparameters, particularly the number and placement of anchor steps, to justify the chosen settings and assess their robustness?

  5. Memory costs: Could the authors report the overall GPU and CPU memory overhead of the caching mechanism and its impact on practical efficiency?

8 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 1pjS (1)

We thank the reviewer for the thoughtful feedback and for recognizing the clarity of our oracle analysis, the substantial system implementation, and the favorable quality–latency trade-off across diffusion models. We have added the following results; each question is answered together with the related weakness.

  • Leave-one-component-out ablation with fidelity and latency (W3-1 & Q3)
  • Temporal flickering and motion smoothness on Minimax-H3-Base, with complete VBench and VBench 2.0 results to follow by November 17 (W2 & Q2)
  • Sensitivity to the anchor schedule and the query group size on Wan2.1-1.3B and 14B (W3-2 & Q4)
  • Peak GPU memory, CPU cache size, measured transfer costs, and scaling with sequence length (W4 & Q5)
AuthorsOpenReview ↗

Rebuttal for Reviewer 1pjS (2)

W1 & Q1: Novelty and generalizable insights

Our contributions are (i) a systematic matched-density oracle analysis, which, to our knowledge, is the first to separate and quantify structural binding, selection, and discarded-tail errors (Fig. 2), and (ii) token-level KV selection, which departs from the block-level selection used by prior training-free sparse attention for video DiTs, together with a kernel that reaches 96–99% of the ideal sparse-attention speedup (Table 6). The table below states what MC-Sparse shares with the closest prior work and what it changes.

AspectClosest prior workSharedChanged in MC-SparseEvidence
Error analysisSVG2 (structural binding), AdaSpa (selection), PISA (discarded tail), each mainly addressing one source—Separates and quantifies all three sources at matched density, then addresses them jointlyFig. 2
KV selection granularityPrior training-free methods for video DiTs (e.g., SVG2, Sol-Attn, PISA, SVG-EAR, AdaSpa)—Selects individual KV tokens instead of KV blocksFig. 2, Table 6
Cross-step reuse of exact selectionAdaSpaExact selection at a few steps, reused at later stepsExact attention mass is aggregated per KV token within each query group, rather than per KV blockTable 3, Fig. 4
Query groupingSVG2Grouping semantically similar queriesFast PDDP partitions queries directly into equal-sized, tile-aligned groups, avoiding the computation that variable-sized clusters waste in partially filled kernel tiles; it also lowers grouping costTables 4, 9: k-means groups execute 1.45× their nominal interactions after tile alignment vs. 1.00× for Fast PDDP; Table 5 (grouping cost)
Discarded-tail compensationDiTFastAttnAdditive dense–sparse residual correctionFills the role of block-mean compensation (PISA, SVG-EAR), which token-level selection cannot use. Combined with our selection, it gains more than applied alone: the more accurate the selection, the more stable the residual across stepsTable 10: +3.54 / +2.88 dB in MC-Sparse vs. +1.25 / +1.45 dB on BSA and +0.49 / +0.78 dB on PISA (20% / 25% density); Fig. 8 (residual stability)

The combination also yields findings that apply beyond this method:

  1. The oracle analysis bounds each direction and locates the bottleneck. Matched-density oracles give the best fidelity reachable by better selection, by removing KV binding, and by removing query binding (Fig. 2). They show that structural binding, not selection accuracy, is the current bottleneck, so better block selection alone cannot close the gap, and that a discarded tail remains even with exact per-query selection, which calls for compensation.
  2. Beyond sparse masks, query groups and residuals are also reusable. Prior work such as AdaSpa reuses sparse masks along the denoising trajectory. We find that query groups and dense–sparse residuals are stable enough to reuse as well (Fig. 4, Fig. 8). Reusing all three amortizes accurate grouping and token-level selection and enables compensation, while attention is still computed from the current Q, K, and V.
  3. Exact selection and reduced block binding make compensation itself more effective. The dense–sparse residual varies less across steps than the attention output, and exact selection stabilizes it further (Fig. 8). Correspondingly, the same residual correction gains 3.54 / 2.88 dB within MC-Sparse but only 1.25 / 1.45 dB on vanilla BSA at 20% / 25% density (Table 10).
  4. Tile-aligned grouping avoids the waste of variable-sized kernels. Variable-sized clusters only partially fill kernel tiles, so k-means grouping as in SVG2 executes 1.45× its nominal interactions after tile alignment, even with token-level KV (Tables 4 and 9). Avoiding this waste is the core motivation of Fast PDDP: it partitions queries directly into equal-sized, tile-aligned groups, removing the waste by construction (1.00×) at lower grouping cost (Table 5). Grouping methods should therefore be compared at matched realized density.
  5. Token-level selection and residual compensation reinforce each other. In the leave-one-component-out ablation (W3-1), removing token-level KV granularity from the full method costs 3.99 dB PSNR, far more than its 0.91 dB increment in Table 3, where it is added before compensation. This matches finding 3: compensation is far more effective with token-level selection. Removing any single component degrades all three fidelity metrics.

These findings are why hardware-aware choices, such as tile-aligned grouping and the token-sparse kernel, are part of the method rather than implementation details, as in STA and VSA.

AuthorsOpenReview ↗

Rebuttal for Reviewer 1pjS (3)

W2 & Q2: Video generation quality

We added the VBench dimensions the reviewer names that support custom prompts. On temporal flickering and motion smoothness, MC-Sparse (15%) scores on par with PISA and Sol-Attn on Minimax-H3-Base, with differences within 0.1.

VBench dimension ↑MC-Sparse (density=15%)PISASol-Attn
Temporal Flickering98.0898.0597.98
Motion Smoothness99.2599.2199.22

Complete VBench scores require each dimension's annotated prompts, which our sampled prompts do not cover, and VBench 2.0 requires its own new prompt set. These runs are still in progress under our compute budget. To discuss the other points with the reviewer early, we post this response now and will add the complete VBench and VBench 2.0 results to this thread by November 17.

We also report PSNR, SSIM, and LPIPS because a training-free approximation aims to preserve the dense model's outputs, and these metrics detect deviations that VBench scores often miss: VBench scores of sparse methods differ only slightly, and full attention itself does not score highest on every dimension. This follows the commonly used protocol of SVG-EAR, with coverage comparable to SVG2 and PISA.

W3-1 & Q3: Component combinations

We add a leave-one-component-out ablation in the same setting as Table 3 (Wan2.1-T2V-1.3B, 25% density), removing one component from the full method while keeping all others, including residual compensation; without exact selection, KV tokens are selected by mean-pooled scoring. The full method and the variant without residual compensation coincide with the corresponding rows of Table 3 (27.35 and 24.47 dB PSNR). Removing any component degrades PSNR, SSIM, and LPIPS, so no simpler combination matches the full method: each variant loses at least 0.65 dB PSNR while saving at most 2.3% of denoising latency. Within the full method, token-level KV granularity matters most (−3.99 dB), more than residual compensation (−2.88 dB); its effect is far larger than its 0.91 dB increment in Table 3, where it is added before compensation, because token-level selection and compensation reinforce each other.

ConfigurationΔPSNR (dB) ↑ΔSSIM ↑ΔLPIPS ↓Relative denoising latency ↓
Full MC-Sparse0.0000.00000.00001.000×
w/o exact KV selection (mean-pooled scoring)−0.707−0.0096+0.00850.987×
w/o token-level KV granularity−3.994−0.0792+0.06500.992×
w/o query grouping (Fast PDDP)−0.648−0.0073+0.00640.977×
w/o residual compensation−2.883−0.0457+0.04900.982×
Reviewer_GZVwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses matched-density oracle comparisons to decompose the quality gap between block-sparse and dense attention in diffusion transformers into structural binding, selection, and discarded-tail errors. It proposes MC-Sparse, a training-free method that, at a few anchor steps, forms tile-aligned query groups (fast PDDP), selects KV tokens by exact attention mass, and records the dense–sparse output residual, then reuses all three in later steps with a token-sparse kernel. On four video DiTs and an internal 3D model, it reports higher fidelity to dense outputs than prior sparse-attention baselines, with 1.52×–2.32× DiT-denoising speedups.

Critical strengths

1. Clear analysis and strong fidelity results.

The matched-density oracle comparison (Fig. 2) gives a clean picture of where block-sparse attention loses fidelity. MC-Sparse substantially improves fidelity to dense outputs across four video models and a 3D model (e.g., +4.8 dB PSNR over Sol-Attn on Minimax-H3-Base), with an efficient token-sparse kernel (96–99% of the ideal speedup, Table 6).

2. Well presented.

The paper is easy to follow, and the figures are clean and illustrate the analysis and pipeline well.

Critical weaknesses

1. Incremental contribution.

That sparse attention degrades quality at high sparsity is well known, and each of the three error sources identified by the analysis is already targeted by prior work the paper cites: semantic query grouping (SVG2), compensation for unselected blocks (PISA, SVG-EAR), and cross-step reuse (AdaSpa, DiTFastAttn), with the residual correction itself taken from DiTFastAttn. The method is mainly a combination of these ideas, and the largest single gain comes from the borrowed residual compensation (+2.3 to +3.5 dB in Table 3, vs. about +1 dB for each other component). Yet the paper does not compare against cross-step reuse methods such as DiTFastAttn or AdaSpa, so it is unclear how much of the improvement comes from the new design rather than from adding caching.

2. Evaluation and cost do not fully support the claims.

  • Quality evidence is thin. Each task uses only 50 prompts, with no variance or worst-case results. PSNR/SSIM/LPIPS only measure similarity to dense outputs, and the two VBench scores barely separate methods (ImgQual 68.54–69.00 across sparse methods vs. 69.20 for dense on Minimax-H3-Base).
  • Speedup claims are overstated. On Wan2.1-14B the speedup is essentially tied with Sol-Attn and PISA (1.52–1.53× vs. 1.51–1.52×), despite the abstract's claim of larger speedups. Reported densities are nominal and exclude anchor-step cost, which the paper itself estimates at Δ\Deltas ≈ 5–6%. The PISA speedup on HY3D-Internal is estimated (< 1.87×), not measured.
  • Memory cost is not discussed. The cache takes about 2.4 GB per layer on Wan2.1-14B (Table 11), roughly 100 GB over 40 layers, and requires CPU offloading. This is not discussed as a limitation, although the method targets long sequences.
  • Analysis scope is narrow. The oracle analysis and all quality ablations use only Wan2.1-1.3B at 480p.
Questions for authors
  1. How does MC-Sparse compare with cross-step reuse methods (e.g., DiTFastAttn, AdaSpa) at matched latency?
  2. How sensitive are quality and speed to the anchor schedule, which is fixed at {10, 26} for all video models?
  3. What is the total cache memory for Wan2.1-14B at 720p, and how does it scale with sequence length?
  4. Can you report effective density including anchor-step cost, and end-to-end latency including encoders and VAE?
  5. Can you report results over more prompts with variance, and for Minimax-H3-Base with the original (non-enhanced) prompts?
  6. Do the oracle-analysis conclusions hold for a 14B model at 720p?

Clarifications:

  1. What is 'MC-Sparse-Flash' in Sec. 5.2? It does not appear in Table 1. It seems undefined.
  2. For Minimax-H3-Base (T2AV), is sparsity applied only to the video branch, and is audio quality affected?
  3. Please state the query group size C and the KV budget K used in each setting.

Minor typo: 'cheep' → 'cheap' (Sec. 5.4). Line 470

6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer GZVw (1)

We thank the reviewer for the thoughtful feedback and for recognizing our clear analysis, strong fidelity results, and clear presentation. We have added the following results; each question is answered together with the related weakness.

  • Comparison with DiTFastAttn and AdaSpa at similar speed on Wan2.1-14B-T2V (W1-3 & Q1)
  • Original (non-enhanced) prompts and 200 enhanced prompts on Minimax-H3-Base, with per-prompt variance and audio metrics (W2-1 & Q5)
  • Effective density and end-to-end speedup (W2-2 & Q4)
  • Total cache memory, its scaling, and lossless bitmask compression (W2-3 & Q3)
  • Oracle analysis and cumulative ablation on Wan2.1-14B-T2V at 720p (W2-4 & Q6)
  • Anchor-schedule ablation on Wan2.1-1.3B and 14B (Q2)

In the revision, we will add DiTFastAttn and AdaSpa to the main comparison, revise the abstract's speedup wording, describe the anchor schedule as a default rather than a tuned choice, rename MC-Sparse-Flash, state the query group size C and the KV budget K, and fix the typo.

AuthorsOpenReview ↗

Rebuttal for Reviewer GZVw (2)

W1-1: Incremental contribution

Our contributions are (i) a systematic matched-density oracle analysis, which, to our knowledge, is the first to separate and quantify structural binding, selection, and discarded-tail errors (Fig. 2), and (ii) token-level KV selection, which departs from the block-level selection used by prior training-free sparse attention for video DiTs, together with a kernel that reaches 96–99% of the ideal sparse-attention speedup (Table 6). The table below states what MC-Sparse shares with the closest prior work and what it changes.

AspectClosest prior workSharedChanged in MC-SparseEvidence
Error analysisSVG2 (structural binding), AdaSpa (selection), PISA (discarded tail), each mainly addressing one source—Separates and quantifies all three sources at matched density, then addresses them jointlyFig. 2; reproduced on Wan2.1-14B (W2-4)
KV selection granularityPrior training-free methods for video DiTs (e.g., SVG2, Sol-Attn, PISA, SVG-EAR, AdaSpa)—Selects individual KV tokens instead of KV blocksFig. 2, Table 6
Cross-step reuse of exact selectionAdaSpaExact selection at a few steps, reused at later stepsExact attention mass is aggregated per KV token within each query group, rather than per KV blockTable 3, Fig. 4; +3.23 dB PSNR over AdaSpa at similar speed (W1-3)
Query groupingSVG2Grouping semantically similar queriesFast PDDP partitions queries directly into equal-sized, tile-aligned groups, avoiding the computation that variable-sized clusters waste in partially filled kernel tiles; it also lowers grouping costTables 4, 9: k-means groups execute 1.45× their nominal interactions after tile alignment vs. 1.00× for Fast PDDP; Table 5 (grouping cost)
Discarded-tail compensationDiTFastAttnAdditive dense–sparse residual correctionFills the role of block-mean compensation (PISA, SVG-EAR), which token-level selection cannot use. Combined with our selection, it gains more than applied alone: the more accurate the selection, the more stable the residual across stepsTable 10: +3.54 / +2.88 dB in MC-Sparse vs. +1.25 / +1.45 dB on BSA and +0.49 / +0.78 dB on PISA (20% / 25% density); Fig. 8 (residual stability)

The core motivation of Fast PDDP is hardware efficiency: variable-sized clusters only partially fill kernel tiles, so k-means grouping as in SVG2 executes 1.45× its nominal interactions after tile alignment, even with token-level KV (Tables 4 and 9). Fast PDDP partitions queries directly into equal-sized, tile-aligned groups, which removes this waste by construction.

The borrowed residual correction is more effective in MC-Sparse than on its own. At 20% / 25% density, adding it to vanilla BSA improves PSNR by 1.25 / 1.45 dB, and adding it to PISA, which already compensates with block means, improves PSNR by only 0.49 / 0.78 dB, consistent with both mechanisms filling the same role. Within MC-Sparse, it improves PSNR by 3.54 / 2.88 dB, consistent with Fig. 8, where exact selection keeps the residual more stable across steps. Likewise, in the comparison under W1-3, AdaSpa and DiTFastAttn, which reuse exact block selections and cached residuals respectively, trail MC-Sparse by 3.23 dB and 3.89 dB PSNR at similar speed (1.48–1.55×). Caching alone therefore does not account for the gains.

W1-2: Largest gain from the borrowed residual correction

Table 3 is cumulative, so the residual's increment there (+2.3 to +3.5 dB) is measured on top of exact token-level selection and query grouping. Applied to vanilla BSA or PISA instead, the same correction gains much less (Table 10; see W1-1), so the large increment reflects the combination rather than the correction alone. A leave-one-component-out ablation in the same setting as Table 3 (25% density) confirms this: within the full method, removing token-level KV granularity costs 3.99 dB PSNR, more than removing residual compensation (2.88 dB), so our token-level design matters more than the borrowed correction. We use this residual instead of block-mean compensation because token-level selection cuts across KV blocks, so block summaries cannot be reused.

W1-3 & Q1: Comparison with cross-step reuse methods

At similar denoising speedups on Wan2.1-14B-T2V, MC-Sparse achieves 3.23 dB and 3.89 dB higher PSNR than AdaSpa and DiTFastAttn, both of which also reuse information across denoising steps. For both methods, we start from the configurations used in their reported experiments and adjust them to reach a denoising latency close to that of MC-Sparse.

MethodPSNR ↑SSIM ↑LPIPS ↓Imaging Quality ↑Background Consistency ↑Denoising speedup ↑
AdaSpa25.580.8940.17566.5495.451.55×
DiTFastAttn24.920.8710.21765.4095.751.48×
MC-Sparse (Ours)28.810.9120.12867.1296.701.53×
AuthorsOpenReview ↗

Rebuttal for Reviewer GZWv (3)

W2-1 & Q5: Quality evidence

We added two evaluations on Minimax-H3-Base: the original 50 test prompts without prompt enhancement, and 200 prompts with the standard enhancement. Both report mean ± standard deviation across prompts and include audio metrics. MC-Sparse runs at 15% density, and PISA and Sol-Attn keep the same hyperparameters as in Table 1.

Video (mean ± std)

PromptsMethodPSNR ↑SSIM ↑LPIPS ↓Imaging Quality ↑Background Consistency ↑
50, originalPISA23.59 ± 2.680.824 ± 0.0690.232 ± 0.04570.04 ± 9.8892.80 ± 4.71
50, originalSol-Attn23.71 ± 2.840.829 ± 0.0710.223 ± 0.04670.10 ± 9.7593.06 ± 4.32
50, originalMC-Sparse (15%)27.30 ± 3.210.889 ± 0.0550.172 ± 0.04570.49 ± 9.6792.97 ± 4.68
200, enhancedPISA23.47 ± 2.350.822 ± 0.0670.241 ± 0.05069.54 ± 7.4490.46 ± 5.63
200, enhancedSol-Attn23.74 ± 2.560.828 ± 0.0700.231 ± 0.05369.78 ± 7.2590.73 ± 5.34
200, enhancedMC-Sparse (15%)27.08 ± 2.840.884 ± 0.0520.181 ± 0.04769.94 ± 7.3390.90 ± 5.02

For audio, MC-Sparse stays much closer to the full-attention audio.

Audio (mean ± std)

PromptsMethodSI-SDR (dB) ↑MR-STFT SC ↓CLAP vs. Full ↑CLAP audio–text ↑
50, originalFull Attention———0.407 ± 0.104
50, originalPISA−2.28 ± 4.670.778 ± 0.2390.892 ± 0.0720.396 ± 0.107
50, originalSol-Attn1.34 ± 3.240.638 ± 0.2540.947 ± 0.0350.406 ± 0.098
50, originalMC-Sparse (15%)10.72 ± 4.670.271 ± 0.1520.974 ± 0.0200.408 ± 0.107
200, enhancedFull Attention———0.429 ± 0.102
200, enhancedPISA−1.77 ± 3.750.751 ± 0.2350.892 ± 0.0820.425 ± 0.105
200, enhancedSol-Attn1.39 ± 3.140.585 ± 0.2080.947 ± 0.0270.429 ± 0.095
200, enhancedMC-Sparse (15%)9.67 ± 4.700.297 ± 0.1470.973 ± 0.0200.433 ± 0.099

On temporal flickering and motion smoothness, MC-Sparse (15%) scores on par with PISA and Sol-Attn, with differences within 0.1.

VBench dimension ↑MC-Sparse (density=15%)PISASol-Attn
Temporal Flickering98.0898.0597.98
Motion Smoothness99.2599.2199.22

Our evaluation (50 prompts per task, with Imaging Quality and Background Consistency from VBench) follows the commonly used protocol of SVG-EAR (ECCV 2026), with coverage comparable to SVG2 and PISA. VBench scores separate training-free attention approximations only weakly; even full attention does not score highest on every dimension. We therefore use them to check for noticeable degradation, and use PSNR, SSIM, and LPIPS to measure how closely each method preserves the dense output.

W2-2 & Q4: Speedup claims, effective density, and end-to-end latency

On Wan2.1-14B-T2V, MC-Sparse's speedup matches Sol-Attn and PISA (1.53× vs. 1.52×), and its advantage there is fidelity (+4.19 dB and +4.78 dB PSNR). We will therefore revise the abstract to claim higher fidelity than all evaluated baselines "while matching or exceeding their denoising speedups". On HY3D-Internal, PISA's released kernel does not run efficiently without modification (Appendix), so its <1.87× is an upper bound estimated from Sol-Attn's kernel latency, still below MC-Sparse's measured 2.32×.

All speedups are measured and include anchor-step costs. End-to-end speedups, which add condition encoding and VAE decoding, are at most 0.01× (Wan2.1-14B-T2V) and 0.09× (Minimax-H3-Base) below the denoising speedups. The effective density below also counts dense warm-up, block-mean compensation, and anchor overhead; like prior work, it excludes grouping and clustering costs, which the paper's 5–6% estimate includes.

Wan2.1-14B-T2V

MethodNominal densityEffective densityDenoising speedup ↑End-to-end speedup ↑
Full Attention100%100%1.00×1.00×
Sol-Attn31.6%45.28%1.52×1.51×
PISA30.0%45.25%1.52×1.51×
SVG231.5%45.20%1.34×1.34×
SVG-EAR25.3%41.49%1.38×1.38×
MC-Sparse (Ours)25.0%44.00%1.53×1.52×

Minimax-H3-Base

MethodNominal densityEffective densityDenoising speedup ↑End-to-end speedup ↑
Full Attention100%100%1.00×1.00×
Sol-Attn32.4%46.20%1.59×1.53×
PISA30.0%45.53%1.52×1.47×
MC-Sparse (density=25%)25.0%44.31%1.61×1.55×
MC-Sparse (density=15%)15.0%36.35%1.80×1.71×

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. Reviewers found the oracle analysis clear and well motivated, the implementation practical and substantial, and the fidelity and quality–latency results strong. Below we summarize our responses to the three shared concerns, with full tables in each individual response.

1. Novelty beyond combining existing techniques (GZVw, m8j2, 1pjS)

MC-Sparse's gains come from three designs that reinforce each other, not from adding existing mechanisms. They follow from our matched-density oracle analysis, to our knowledge the first to separate and quantify structural binding, selection, and discarded-tail errors (Fig. 2), which shows that structural binding is the current bottleneck.

  • Fast PDDP: tile-aligned grouping avoids the waste of variable-sized kernels. Variable-sized query clusters only partially fill kernel tiles, so k-means grouping as in SVG2 executes more interactions than its nominal budget. Avoiding this waste is the core motivation of Fast PDDP: it partitions queries directly into equal-sized, tile-aligned groups, removing the waste by construction at lower grouping cost, further reduced since submission.
  • Token-level KV selection. To our knowledge, MC-Sparse is the first training-free sparse attention for video DiTs to select individual KV tokens rather than KV blocks, with a token-sparse kernel close to the ideal speedup.
Wan2.1 (Tables 4–6)k-means, SVG2 kernelFast PDDP, our kernel
Executed / nominal interactions, token-level KV ↓1.45×1.00×
Attention recall at s = 0.2, token-level KV ↑0.9040.923
Grouping time per call on 14B, ms per layer ↓295.166.6 (173.4 in submission)
Kernel efficiency vs. ideal speedup, s = 0.2–0.5 ↑0.80–0.810.96–0.99
  • Residual compensation that is not simply additive. Token-level selection cannot use block-mean compensation (PISA, SVG-EAR), so we use DiTFastAttn's dense–sparse residual. Exact selection keeps this residual stable across steps (Fig. 8), so the same correction gains far more within MC-Sparse (Table 10):
Residual correction added toΔPSNR at 20% density
Vanilla BSA+1.25 dB
PISA (with block-mean compensation)+0.49 dB
MC-Sparse+3.54 dB

Conversely, removing token-level KV granularity from the full method costs 3.99 dB PSNR, far more than its 0.91 dB increment before compensation (Table 3). Caching alone does not explain the gains either: AdaSpa and DiTFastAttn, which reuse block selections and residuals, trail MC-Sparse by 3.23 and 3.89 dB PSNR at similar speed on Wan2.1-14B-T2V.

2. Memory overhead of cross-step caching (GZVw, 1pjS)

The cache is either kept in CPU memory or sharded across GPUs, and in both cases adds almost no latency.

  • Size. The per-layer size in Table 11 (about 2.4 GB on Wan2.1-14B-T2V) stores KV selections as INT32 indices. A lossless bitmask reduces the cache of all 40 layers from 95.87 GiB to 37.61 GiB.
  • Transfer. Prefetching one layer ahead from pinned CPU memory leaves only 0.05 ms of exposed upload per layer on Wan2.1-14B-T2V (0.14 ms on Minimax-H3-Base). Sparse attention takes about 4× as long as the upload (208.8 vs. 47.7 ms per layer), or about 9× with the bitmask, so the overlap holds with several times lower bandwidth.
  • Scaling. Residuals and query-group indices grow linearly with sequence length N, and bitmask selections as N²/C, the same order as attention computation. Longer sequences are typically served on more GPUs with sequence parallelism (e.g., xDiT), which shards the cache, so the cache per GPU grows no faster than the attention computation per GPU.

3. Evaluation and ablations (GZVw, m8j2, 1pjS)

Added experimentRaised byResult
VBench and VBench 2.0 on Minimax-H3-Base1pjSAt 1.80× speedup (PISA 1.52×, Sol-Attn 1.59×), close to full attention: VBench 72.08 vs. 72.20, both VBench 2.0 averages within 1.5 points
50 original and 200 enhanced prompts, mean ± SD, audioGZVw+3.3–3.7 dB PSNR over PISA and Sol-Attn in both settings, and 8.3–13.0 dB higher audio SI-SDR
Leave-one-component-out ablation (reverse of Table 3)1pjSEach removal loses at least 0.65 dB PSNR while saving at most 2.3% of latency
Oracle analysis and cumulative ablation on Wan2.1-14B-T2VGZVwTrends similar to those on Wan2.1-1.3B
Anchor schedule: 7 schedules on Wan2.1-1.3B and 14BGZVw, m8j2, 1pjS{10, 26} is an untuned default. {10, 18} gains about 0.40 dB at the same speed, and a third anchor costs 5–8% in speed
Query group size C = 64 vs. 128 on Wan2.1-1.3B and 14Bm8j2, 1pjSPSNR within 0.04 dB, while C = 64 runs at 0.97× and 0.89× the kernel speed

We thank the reviewers for their suggestions (e.g., the comparison with cross-step reuse methods, the hyperparameter ablations, and the memory overhead analysis). We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.