acceptodds
Withdrawn from ICLR 2027

Rethinking Sparse Attention in Few-Step Diffusion Transformer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-step diffusion compresses a multi-step teacher trajectory into a handful of student evaluations. Sparse attention further reduces cost by retaining only highly ranked interactions, but the student’s current ranking need not represent the corresponding teacher-window support. On aligned probes of a four-step Wan2.2 student, the top-b pool captures 51.91% of teacher attention mass and leaves a value-weighted residual. The useful core favors persistent interactions; its complement contains both missed salient blocks and a diffuse tail. This diagnosis motivates SPARE Attention: current-score candidates plus a small complement quota under a fixed sparse budget, with coarse routing and no teacher access at inference. A matched per-step-budget intervention reduces final Teacher40 RMSE from 0.959537 to 0.928340 (3.25%), improving all six reported records; larger quotas are not uniformly better. Separately, the original system configuration selects 14.84% of key blocks on the full 6,220-video VBench suite and denoises in 29.565 s, the shortest measured time among the timed sparse implementations, with Total comparable to the SLA family and PISA

Primary area:
generative models
Keywords:
DiT, Sparse Attention

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Q6P3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SPARE proposes a training-free sparse attention method for few-step diffusion transformers. The authors select the attention support by combining top-k sparse attention with randomly selected keys and demonstrate that it achieves improved performance in video-generation and other video-related tasks under a fixed sparse budget.

Critical strengths
  1. The paper addresses an important problem of efficiency in diffusion models.
  2. The method is plug-and-play and allows to use ready pretrained models.
  3. The method itself is very simple and the results are somewhat convincing over the baselines that the authors compared to.
Critical weaknesses
  1. Limited evaluation against other sparse attention methods. The paper mostly compares to SLA, but there are several methods that use quite similar approaches, like DFSAttn, SpargeAttention or XAttention.
  2. Hence, the research gap that the authors actually address is not clear, because a lot of the claims seem to be covered by the previous work.
  3. The paper's writing is extremely hard to follow. The structure of the paragraphs is chaotic.
Questions for authors
  1. Could you compare to the baselines I mentioned (or explain why these are not comparable if you think this is the case) and other relevant plug-and-play sparse attention baselines for video generation?
  2. Could you clearly state the scientific claim the paper is presupposing and explain how it was not covered before SPARE and how your results demonstrate it?
Reviewer_MFvgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies block-sparse self-attention on top of a 4-step distilled video diffusion student (Wan2.2-T2V-A14B). The usual rule keeps the top-b key blocks ranked by the student's current scores. The authors ask whether that top-b is a good proxy for what the 40-step teacher attends to over the corresponding 10-step window.

Three claimed contributions:

(1) Diagnosis. On 6 prompts (6 layers, 8 heads, 16 query rows), the student's top-b covers 51.9% of the teacher's window-averaged attention mass. The student favours "persistent" blocks (in the teacher's top set on most window steps). Of the uncovered 48%, 17% is missed teacher-high mass and 29.5% is a diffuse tail that never enters the teacher's own top set.

(2) SPARE Attention. Under a fixed per-step budget b_t, select n_t blocks from the top-b_t pool and o_t blocks uniformly from its complement; carry the previous step's support as selection state; use coarse 128-token query blocks for kernel speed. No teacher at inference, no fine-tuning.

(3) Experiments. With the keep schedule fixed, an outside quota of 16 lowers final-latent RMSE to the teacher from 0.9595 to 0.9283 (-3.25%, 6/6 records). A multi-mask-seed control in a different configuration gives SPARE minus pure top-b = -0.0173 [-0.0225, -0.0105]. On full VBench (6,220 videos) SPARE reaches Total 82.34 at 14.84% density in 29.57 s, vs. SLA80 (82.13, 19.92%, 32.58 s) and the dense student (81.29, 46.78 s). VBench-2.0 and T2V-CompBench are also reported.

Critical strengths
  1. Honest, fixed-budget experimental design. The locked comparison (Sec. 5.2) holds the per-step keep schedule constant, so the effect of HOW blocks are allocated is separated from HOW MANY are computed. The direction of the effect (non-zero outside quota lowers Teacher RMSE vs. pure top-b) is reproduced in two configurations, one of them multi-seed with a bootstrap CI that excludes zero. This is why I treat the paper as salvageable rather than unsound.

  2. Transparency. The authors separate the diagnostic residual, teacher-endpoint agreement and perceptual quality; state that retention of previous support gives no measurable benefit (A - B = 0.0006); and release the full budget sweep and all per-dimension scores on three benchmarks. The evidence needed to question the headline claims is in the paper itself.

Critical weaknesses
  1. The headline effect is confounded, and the un-confounded version of the method is prior work. By Sec. 4.2 / App. A.2, when n_t < b_t the blocks inside top-b_t are sampled at random, not taken as the deterministic top-n_t. Going from quota 0 to quota 16 therefore changes two things at once: 16 random outside blocks are added AND the choice of which 50/66 (or 64/80) core blocks survive is randomised. No control of the form "deterministic top-(b_t - o_t) + uniform outside blocks, no history" is reported, so the paper cannot say which change produces the RMSE drop. Since retention is shown to be inert, what remains is top-k plus random blocks (BigBird, Zaheer et al. 2020), which is neither cited nor compared. Supporting statistics are also thin: the locked result is 6 prompts, one mask seed, one generation seed, no per-record data; the best quota differs between the two configurations (16 in b*, 8 in b_sys) and all pairwise CIs among non-zero quotas include zero; b* was tuned on a dev cohort drawn from the same manifest as the 6 probe prompts (App. A.8).

  2. The metric that improves does not track quality, and the quality results do not favour the method. By Table 5, the smallest budget (keep 48) has the best VBench Total (82.60) and the worst Teacher RMSE (0.968); a 2.7x change in density moves RMSE by only 0.014, less than the quota does. Dense-student RMSE is not reported, so the 3.25% has no scale. On quality: SLA85 at identical density (76/512) has Total 82.35 >= 82.34; SLA90 has 82.61 at lower density. On VBench-2.0 SPARE scores 0.56 vs. 0.59 for both dense and SLA80 (Physics 0.50 vs. 0.58; Multi-View Consistency 0.349 vs. 0.471; Human Clothes 0.856 vs. 0.932); CompBench Generative Numeracy 0.460 vs. 0.529. A drop specifically in physical/multi-view consistency is the expected cost of swapping ranked blocks for random ones and is not mentioned in the abstract or conclusions. The speed claim does not isolate the method either: vs. SLA80, density (-25%), query-block size (128 vs. 64) and the selection rule all change at once; SLA85, the only equal-density baseline, is untimed; the 0.56% gap to Sol-Attn is called unreliable by the authors. "SLA80" is the LightX2V top-k backend without SLA's linear branch or fine-tuning, so "SLA family" is a misnomer.

Questions for authors

Q1. (clarification) In the locked experiment, is the core at quota 0 the deterministic top-b, and at quota 16 a uniform subsample of 50/66 and 64/80 from the top pool? Please give r_t and the realised retained/fresh split per quota.

Q2. (request) Teacher RMSE for the control "deterministic top-(b_t - o_t) + uniform outside blocks, no history", same geometry and budget.

Q3. (request) Teacher40 RMSE of the dense student and of SLA85 on the same 6 records, and wall-clock for SLA85 and for pure top-b on the same coarse query blocks. Without these, neither the RMSE scale nor the speed claim can be interpreted.

Q4. (request) The six per-record differences for quota 0 vs. 16, and any additional generation seeds or records.

Q5. (clarification) How do you account for the VBench-2.0 Physics and Multi-View Consistency drops? Has the zero-quota variant been run on VBench-2.0 to test whether random outside blocks are the cause?

Q6. (clarification) What fraction of window mass is covered by the teacher's OWN top-102 at a single step (teacher self-coverage), and how does the diagnosis change with window length? Needed to separate a distillation- specific mismatch from the ordinary softmax tail.

Suggestions not affecting the score: add a protocol table (configuration, cohort, seeds, metric, purpose); remove the traces of earlier review rounds and unify terminology; move VBench-2.0/CompBench and the RMSE-vs-Total disagreement into the main text; replace Figure 1 (MiniMax H3, Qwen-Image, no numbers) with an example that has quantitative backing; fix the duplicated VBench reference, "EXACTXACT" in A.10 and the repeated paragraphs in Sec. 3.3 / A.11.

Reviewer_oah5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article finds that the attention ranking of the current step in the few step diffusion model may miss some important interactions in the multi-step teacher trajectory. Based on this, SPARE is proposed: under a fixed sparse budget, high score candidates are retained, and some attention blocks are added from outside the candidate set. The paper verifies the methods of teacher endpoint error, video quality, and denoising speed, achieving approximately 1.58 times denoising acceleration under the main configuration.

Critical strengths
  1. The motivation for the method is clear, allocating quotas for interactions outside of the high score set within a fixed budget. The design is simple and does not require a teacher model for inference.
  2. Provided fixed budget ablation, complete VBench testing, and timing including selection expenses; The difference between teacher consistency and actual generation quality has also been clearly explained.
Critical weaknesses
  1. The scale of evidence for the core mechanism is relatively small. The key diagnosis and endpoint error experiments only involved 6 samples, and some experiments lacked uncertainty analysis of multiple seeds. The decrease in teacher endpoint error has not been consistently matched with an improvement in generation quality, and the additional value of the historical preservation mechanism has not been validated.
  2. The comprehensive advantages of quality and speed are not yet sufficient. SLA85/90/95 achieves comparable or higher VBench scores at the same or lower density, but without speed testing; The overall score of SPARE on VBench-2.0 is lower than that of dense models and SLA80. Therefore, the existing results are insufficient to confirm its comprehensive advantages.
Questions for authors
  1. Can fixed budget supplementary sampling improve both teacher endpoint error and generation quality on independent, larger samples, and multiple random seeds? What is the effect after removing the historical preservation constraint?
  2. Can we supplement the timing of SLA85/90/95 and compare the quality and speed under the same sparse density and block granularity to distinguish the contribution of selection strategy and implementation configuration?
Reviewer_imUDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether sparse support selected from a few-step student's current-step attention scores adequately represents the teacher steps that each student evaluation replaces. Using aligned offline probes of a four-step Wan2.2-T2V-A14B student, the authors report that the student's top-block candidate pool covers 51.91% of teacher-window attention mass, with the complement containing both missed teacher-high blocks and a diffuse never-top tail. They propose SPARE Attention, which forms a top-b candidate pool and divides a fixed per-step selection budget between that pool and its complement. Sampling is uniform within groups defined by pool membership and previous-support membership, with no teacher access at inference. In a locked experiment on six records, an outside quota of 16 at the last three steps reduces teacher-endpoint RMSE from 0.959537 to 0.928340. Separately, the original system configuration achieves a VBench Total of 82.34 on the full 6,220-video suite at 14.84% average block density, with a denoising time of 29.565 s measured on a development cohort.

Critical strengths
  1. The question is well posed. Treating the student's current-step ranking as an assumption to examine is a useful framing, and Section 1 clearly explains why that ranking may not capture the support relevant to a multi-step teacher window.

  2. The probe analyses provide concrete diagnostics: 51.91% paired teacher-mass coverage, 29.0% of teacher top-102 blocks outside the student top-102 in the separate rank audit, and selection rates of 64.4% versus 13.5% for persistent versus intermittent blocks. The authors also report negative and inconclusive findings in Appendices A.11, A.12, and F.1, which strengthens the transparency of their interpretation.

  3. The method is relatively simple to deploy, requiring no teacher access, dense reference prepass, or additional forward pass. The evaluation covers several benchmarks, including full-suite VBench, VBench-2.0 results across 18 dimensions, T2V-CompBench, and a 606-item image-editing transfer check using a local judge.

Critical weaknesses
  1. The locked endpoint comparison uses only six records and reports aggregate means and an all-six improvement count without per-record seed arrays for uncertainty estimation. Quota 16 has the lowest reported mean RMSE, but the robustness of its advantage over quotas 8 and 24 remains unclear. Appendix A.9's pairwise intervals among nonzero quotas contain zero, although that study uses a different keep schedule and state rule. The original three-mask-seed controls do support an improvement over pure top-b selection under their own configuration; my concern is specifically the uncertainty and generalizability of the locked experiment's 3.25% gain and its preference for quota 16.

  2. The paper supports allocating part of the budget outside the top-b candidate pool, but does not directly establish the final-endpoint advantage of its complement-sampling rule over selecting next-ranked complement blocks. Figure 2e compares local residual fits, and Section 3.5 explicitly states that these are not a direct random-versus-next-ranked endpoint comparison. Appendix F.1 also lacks this arm. A comparison that matches the per-step budget, complement quota, within-pool selection rule, and retention targets would strengthen the justification for the particular sampling rule.

  3. The headline 9.25% denoising-time reduction relative to SLA80 combines a change in selection and query geometry with lower average density: 14.84% versus 19.92%. Table 5 partly addresses this issue by reporting SPARE at the same 19.92% average density with a time of 30.370 s, compared with SLA80's 32.579 s; Appendix F.2 also reports an operator-level benefit from coarse query blocks. However, SLA85, which matches the main SPARE configuration's 14.84% average density and has a similar VBench Total (82.35 versus 82.34), remains untimed. The timing comparison at the main operating point is therefore incomplete, and the contribution of query geometry to the full denoising speedup remains unisolated.

Questions for authors
  1. Could the authors provide per-record endpoint errors for the policies in Figure 2f and, if available, results with additional mask and generation seeds? These would help estimate record-level uncertainty in the reported 3.25% improvement and assess whether quota 16 consistently outperforms the other tested nonzero quotas.

  2. Could the authors report a paired final-endpoint comparison between SPARE's state-constrained complement sampling and a next-ranked-complement alternative, matching the per-step budget, outside quota, within-pool selection rule, and retention targets? Please clarify how the retained/fresh constraints are handled so that the comparison isolates the complement-selection rule.

  3. In addition to the mean-102 result in Table 5, could the authors time SLA85 under the same denoising protocol to compare against SPARE at 14.84% average density? Separately, could they quantify how much of the full denoising-time difference is attributable to coarse query geometry under a fixed selection policy and per-step block budget, accounting for the kernel numerical differences reported in Appendix A.12?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.