- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SPIRA, a sparse attention method for block diffusion LLMs that reduces prefix-selection cost through a two-stage procedure. It retrieves a shared candidate set using routing queries derived from the block’s pre-RoPE query means, then rescores the candidates with the original post-RoPE queries to select the final top-K prefix keys. The selected set is reused across subsequent denoising steps within the block.
Critical strengths
-
Clear presentation. The paper is well written and easy to follow, with a clear progression from empirical observations to the proposed method and its evaluation.
-
Systematic analysis of query structure. Section 3 carefully examines the shared structure of pre-RoPE queries and the role of query-specific information after RoPE. The analysis provides a useful empirical motivation for shared candidate generation followed by query-specific rescoring.
-
Broad experimental coverage. The evaluation covers three block diffusion model families and multiple tasks on LongBench and RULER. The paper also includes ablations on the retained-key budget and candidate multiplier, evaluations across checkpoints with different block sizes, and latency and batch-scaling analyses.
Critical weaknesses
1. Missing comparisons with the closest baselines. The evaluation compares SPIRA with QUEST and SparseD, which were originally developed for autoregressive and fully bidirectional diffusion LLMs, respectively, but omits direct comparisons with MAGE and LoSA, which specifically target block diffusion LLMs. In particular, MAGE already selects a block-shared prefix subset at the first denoising step and reuses it throughout the remaining steps, a structure also adopted by SPIRA. Direct comparisons with these methods, especially MAGE, are necessary to establish SPIRA’s incremental performance gains over existing sparse attention methods for block diffusion LLMs.
2. Unclear incremental benefit of two-stage selection.
2-(a) Regarding estimation quality: the reference oracle used to measure top-K recall in Fig. 3 is obtained by aggregating exact prefix-attention weights across the block’s queries and selecting the top-K keys. This is the same selection criterion used by MAGE at the first step. The reported recall improvement therefore demonstrates a closer approximation to MAGE’s selection than routing alone. However, this internal ablation does not establish whether SPIRA’s two-stage selection ultimately better preserves the individual queries’ exact top-K subsets, which are known to be difficult to capture with a single shared subset. Comparing SPIRA with other sparse KV selection methods in terms of per-query top-K recall would help clarify whether its selector provides any additional improvement in estimation quality.
2-(b) Regarding efficiency: the motivation for using two-stage selection to reduce selection cost is clear, but its contribution to overall decode speedup remains uncertain. Since the selected sparse KV subset is reused throughout subsequent denoising steps, the additional savings from cheaper selection occur only at the first step of each block. Moreover, MAGE already overlaps selection with other computation and amortizes its initial dense-attention step across the block’s denoising steps. The paper should therefore quantify how much SPIRA reduces first-step latency relative to MAGE and how much this reduction contributes to total decode speedup. Without this analysis and a direct accuracy comparison, it is difficult to determine whether the two-stage design offers a meaningful improvement in the quality–latency trade-off.
3. Unclear benefit of the pre-RoPE routing formulation. Appendix B reports that replacing the proposed routing representation with the post-RoPE block mean changes top-K recall by less than 0.002 across all three models. This raises the question of what practical benefit the pre-RoPE formulation provides over simply averaging the post-RoPE queries. While the analysis of pre-RoPE alignment is informative, downstream accuracy and latency comparisons between these routing alternatives, with the same candidate budget and rescoring procedure, are needed to isolate the contribution of the proposed representation.
Questions for authors
-
Comparison with block diffusion baselines. Could you provide direct comparisons with MAGE on shared model checkpoints under matched decoding settings, reporting both downstream accuracy and decode latency?
-
Selection quality at the query level. How does SPIRA compare with MAGE in terms of estimation quality at the same shared sparsity budget, as measured by mean per-query recall against each query’s exact top-K subset? Evaluating the selectors on common reference queries would help isolate differences in selection quality. How does this comparison change as block size increases, and do any differences translate into downstream accuracy gains?
-
Contribution of cheaper selection to overall speedup. How much does SPIRA reduce first-step latency relative to MAGE, and what fraction of the total decode-latency reduction comes from this improvement? Please provide a breakdown of first-step and subsequent-step latency, and examine how the incremental speedup changes with the number of denoising steps per block.
-
Benefit of the pre-RoPE formulation. Appendix B reports that substituting the post-RoPE block mean changes top-K recall by less than 0.002. Does the proposed pre-RoPE routing provide a measurable downstream accuracy or latency advantage over this alternative when the candidate budget and rescoring procedure are held fixed? If the results are comparable, could you clarify the practical role of the pre-RoPE formulation in the method?
11 comments
Follow-up Questions
Thank you for the detailed responses and experiments. These partially address my concerns. To clarify the remaining points, I have the following follow-up questions.
My two remaining concerns are: (1) distinguishing SPIRA’s own system-level gains from those already provided by prior work through later-step sparse attention and reuse, and (2) whether reducing first-step selection FLOPs translates into meaningful latency savings over MAGE under memory-bound, long-context conditions. Both methods scan the full prefix keys, so FLOP reductions alone do not establish the practical latency benefit.
-
KV-cache memory traffic. In addition to the attention FLOPs reported in your response to Q3, could you provide the corresponding total KV-cache memory traffic for SPIRA and MAGE, including SPIRA’s routing and candidate rescoring?
-
Latency breakdown and the source of speedup. Could you report first-step, subsequent-step, and total block latency for both SPIRA and exact attention across context lengths under otherwise identical settings? This would clarify whether SPIRA materially reduces first-step latency and how much of the overall latency reduction comes from the first step versus subsequent steps.
Additional observation on the first two layers. In addition, I find it interesting that SPIRA applies sparse attention in the first two layers yet achieves accuracy comparable to MAGE, which retains exact attention in these layers. Could you analyze why block diffusion LLMs tolerate sparsity in these early layers, whereas AR sparse-attention methods such as QUEST typically keep them exact?
Response to the follow-up questions
Thank you for the detailed follow-up. We ran the measurements you asked for and answer each question below. Beyond the reuse of one block-shared set over the later steps, which SPIRA and MAGE share, SPIRA builds that set by scanning the prefix keys once and rescoring only the candidates, without values and without an exact forward. We measure this in DRAM traffic against MAGE and in latency against exact attention. All traffic and timing measurements use SDAR-4B on an A100-40GB at batch 1 with K=256 and m=8, with CUDA graphs enabled and prefill excluded.
KV-cache memory traffic
We measured DRAM reads per layer with Nsight Compute, covering every selection and attention kernel. Model weights, read equally by both methods, are left out. MAGE's kernels are not public, so its columns are composed from the same measured kernels following its Algorithm 1, with an exact first step and its select stream (its Section 4), and later steps with the two exact layers of its evaluation setup, averaged over the 36 layers.
| prefix | first step, SPIRA / MAGE composed (MB per layer) | later step, SPIRA / MAGE composed (MB per layer) | exact attention, every step (MB per layer) |
|---|---|---|---|
| 16K | 69 / 112 | 5.4 / 9.4 | 78 |
| 32K | 112 / 229 | 5.4 / 14.1 | 162 |
| 65K | 193 / 466 | 5.4 / 23.5 | 332 |
| 122K | 335 / 876 | 5.4 / 39.8 | 624 |
At 65K, SPIRA's first step reads 134 MB in the stage-1 key scan, 23 MB in candidate rescoring and 36 MB in routing, scoring intermediates, gather and attention. It reads no values outside the selected set, while MAGE's exact forward reads the keys and values of the whole prefix. SPIRA's first step therefore reads 1.6 to 2.6 times less, and per block SPIRA reads 1.7 to 3.1 times less than MAGE with 8 steps and 1.7 to 4.2 times less with 32. Against exact attention, SPIRA's traffic per block at 8 steps is 17% at 16K and 7% at 122K.
Latency breakdown
We timed each forward of a block for SPIRA and exact attention in the runtime of Section 5.2 at K=256 and m=8, both with FlashAttention so that they share one attention kernel.
| prefix | first step, exact / SPIRA (ms) | later step, exact / SPIRA (ms) | block of 8 steps, exact / SPIRA (ms) | block of 32 steps, exact / SPIRA (ms) |
|---|---|---|---|---|
| 8K | 109.2 / 20.3 | 11.8 / 10.0 | 207.9 / 105.5 | 509.1 / 361.3 |
| 32K | 108.9 / 20.6 | 17.4 / 10.0 | 252.9 / 105.8 | 703.2 / 361.4 |
| 65K | 109.1 / 23.6 | 25.6 / 10.0 | 319.0 / 108.9 | 971.4 / 365.5 |
| 98K | 109.9 / 26.4 | 34.1 / 10.1 | 390.0 / 112.4 | 1250.6 / 367.1 |
| 122K | 110.2 / 28.4 | 40.6 / 10.0 | 441.8 / 113.6 | 1457.4 / 372.4 |
Exact attention's first step stays near 110 ms at every prefix, close to Table 2's once-a-block cost (100–107 ms). Block times are measured end to end on blocks after the first of each generation, and include the commit forward and host time between forwards, so SPIRA's block is 1.97 to 3.89 times faster with 8 steps and 1.41 to 3.91 times with 32. SPIRA's first step, selection included, takes 20 to 28 ms. The first step accounts for 87% of the block saving at 8K and 25% at 122K with 8 steps, and 60% and 8% with 32, and the later steps carry most of the rest, with the commit forward and host time the remainder. Against MAGE, whose kernels are not public, we compare the traffic above.
The first two layers
SDAR-4B, Table 1's 1,350 LongBench documents:
| SPIRA, m=8 | K=256 | K=1024 |
|---|---|---|
| all layers sparse, as in Table 13 | 37.50 | 37.60 |
| first two layers exact | 37.01 | 37.97 |
Making the first two layers exact changes SPIRA's scores by −0.49 and +0.37. QUEST, which keeps them exact in Table 1, loses 4.7 and 3.2 points at K=256 and 1024 when they are made sparse.
On the same SDAR model, SPIRA keeps its scores without the exact layers while QUEST does not, suggesting that the tolerance depends on the selection procedure rather than being a property of the model. In these layers a block's masked queries are nearly identical before RoPE, since every masked position enters with the same [MASK] embedding (pairwise cosine 0.994 at layer 1), which suits one routing pass, and stage 2 then scores the candidate keys exactly with each query. QUEST selects pages by a bound and keeps these layers exact following its layer-wise analysis on an autoregressive model (Tang et al., 2024, Figure 3).
We thank the reviewer again for the follow-up, and we will add the traffic and latency tables to Appendix F and the first-two-layer ablation with the agreement of these layers to Appendix A.
Response to Reviewer n9hz
Thank you for the questions on MAGE, per-query recall and first-step cost, answered below. SPIRA itself uses no exact first step or exact layers. Only the run under MAGE's protocol gives both methods two exact layers.
Table 1 used m=2. We raised m up to 16 (Appendix G), and coverage reaches 0.94 at m=8, used below unless m=2 is stated. A larger m leaves the prefix read unchanged, since the read counts keys used in attention, not candidate scoring (Section 5), and adds at K=256 0.3 to 0.4 points of dense FLOPs and at most 1 ms per block.
Q1. Direct comparison with MAGE and LoSA
On a shared checkpoint under MAGE's own protocol, SDAR-8B-Chat with its six LongBench tasks, block 32 and threshold 0.95 over 1,200 documents, both methods keep two exact layers and SPIRA uses m=2. SPIRA and MAGE score 43.38 and 43.71 at K=256, 44.72 and 44.58 at K=512, and 45.91 and 45.74 at K=1024, with sign-test p 0.39, 0.91 and 0.60. With the same two exact layers, SPIRA still reads half as much of the prefix, 8.4% against 17.4% at K=256 (Appendix L).
Under our protocol, Table 1's seven LongBench tasks and 1,350 documents with SPIRA's decoder settings, MAGE runs as its paper specifies, with its two exact layers, and SPIRA with m=8. Per-task results are in Appendix I.
No cell differs significantly, every p 0.23 or more. At K=256, however, SPIRA reads 3.4 to 3.5% of the prefix and MAGE 17 to 27%, mostly in its exact first step and exact layers, which read the whole prefix (Appendix I). On RULER from 4K to 32K, SPIRA at K=1024 reads less than MAGE at K=256 and scores higher on all models (Figure 12). Decode cost is in Q3.
LoSA's K is per query and it runs a dense first step, so at K=256 it reads 13 to 20% of the prefix against SPIRA's 3.4 to 3.5%. At similar reads, SPIRA at K=1024 scores 37.60, 25.37 and 48.85 against LoSA's 35.27, 25.33 and 47.86 at K=256 (Appendix J).
Q2. Per-query recall against each query's own exact top-K
Both selectors, SPIRA at m=8, are scored on the same queries and later steps, each query against its own exact top-K by attention-mass recall (Appendix L).
On SDAR-4B they are within 0.004 at every trained block size from 8 to 64 and every K. SPIRA thus selects as well as MAGE per query, consistent with Q1, while reading far less of the prefix at K=256.
Q3. First-step cost and its share of the savings
We found no public implementation of MAGE. We implement its algorithm as published, but its latency depends on kernels we cannot reproduce faithfully. We therefore compare prefix-attention FLOPs at K=256, summed over documents from measured step counts, with dense at 100. Totals match the FLOPs rows of Appendix I.
SPIRA's selection costs a ninth or less of MAGE's exact first step. This is about 42% of the prefix-attention FLOP savings over MAGE on SDAR LongBench, 35% on LLaDA2 LongBench and 55% on LLaDA2 RULER 32K, and the rest comes mainly from MAGE's exact layers. As the number of forwards per block falls, MAGE's exact first step is amortized over fewer steps, while SPIRA's selection stays near 0.5. At a 122K prefix on SDAR-4B with K=256, SPIRA decodes a block 3.9 and 3.8 times faster than exact attention with 32 and 8 steps per block.
Q4. Pre-RoPE versus post-RoPE routing
Before RoPE, a block's queries split into a shared direction μ and residuals. Keys chosen with μ alone hold 0.847, 0.844 and 0.883 of the exact attention mass on the three models, nearly all the full queries hold (Appendix B), so one routing pass per block finds the candidates and stage 2 restores the per-query part. SPIRA forms the shared term R̄μ from the block mean alone, since R̄ has a closed form, while a post-RoPE mean rotates all B queries first. Measured with Nsight Compute, the post-RoPE route reads 4.2 times as much DRAM to form the routing vector (Appendix B).
We thank the reviewer again. We will add the RULER numbers against MAGE, the FLOPs breakdown above and the 122K timing to Appendices K, I and F.