acceptodds
Under review as a conference paper at ICLR 2027

SPIRA: Sparse Pre-RoPE Informed Routing Attention for Diffusion Language Models

Abstract

Block diffusion language models generate a block of tokens at a time by denoising masked positions in parallel. Sparse prefix attention reduces the number of prefix keys attended to during decoding, but selecting those keys is itself a full-prefix operation when each masked query scores the prefix independently. We find that the masked queries of a block share a common direction before RoPE. This structure supports shared candidate generation across the block, while final selection still depends on query-specific scores from the original position-dependent queries. SPIRA exploits this separation: one routing query per query head, constructed from the block’s pre-RoPE query mean, scans the prefix once per block to retrieve a candidate set, and the original queries rescore only those candidates to select the final keys. The selected keys are then reused throughout the block, so later denoising steps no longer read the full prefix. Across three block diffusion language models, SPIRA outperforms two sparse baselines on LongBench with 256 retained keys, exceeding the stronger baseline by up to 10.2 points. With 1024 keys, it comes within 0.06 points of exact attention on SDAR-4B-Chat-b64. On the same model, it achieves a 3.45× speedup in per-block decode latency at a 122k-token prefix.

Primary area:
generative models
Keywords:
diffusion large language models, sparse attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_n9hzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPIRA, a sparse attention method for block diffusion LLMs that reduces prefix-selection cost through a two-stage procedure. It retrieves a shared candidate set using routing queries derived from the block’s pre-RoPE query means, then rescores the candidates with the original post-RoPE queries to select the final top-K prefix keys. The selected set is reused across subsequent denoising steps within the block.

Critical strengths
  1. Clear presentation. The paper is well written and easy to follow, with a clear progression from empirical observations to the proposed method and its evaluation.

  2. Systematic analysis of query structure. Section 3 carefully examines the shared structure of pre-RoPE queries and the role of query-specific information after RoPE. The analysis provides a useful empirical motivation for shared candidate generation followed by query-specific rescoring.

  3. Broad experimental coverage. The evaluation covers three block diffusion model families and multiple tasks on LongBench and RULER. The paper also includes ablations on the retained-key budget and candidate multiplier, evaluations across checkpoints with different block sizes, and latency and batch-scaling analyses.

Critical weaknesses

1. Missing comparisons with the closest baselines. The evaluation compares SPIRA with QUEST and SparseD, which were originally developed for autoregressive and fully bidirectional diffusion LLMs, respectively, but omits direct comparisons with MAGE and LoSA, which specifically target block diffusion LLMs. In particular, MAGE already selects a block-shared prefix subset at the first denoising step and reuses it throughout the remaining steps, a structure also adopted by SPIRA. Direct comparisons with these methods, especially MAGE, are necessary to establish SPIRA’s incremental performance gains over existing sparse attention methods for block diffusion LLMs.

2. Unclear incremental benefit of two-stage selection.

2-(a) Regarding estimation quality: the reference oracle used to measure top-K recall in Fig. 3 is obtained by aggregating exact prefix-attention weights across the block’s queries and selecting the top-K keys. This is the same selection criterion used by MAGE at the first step. The reported recall improvement therefore demonstrates a closer approximation to MAGE’s selection than routing alone. However, this internal ablation does not establish whether SPIRA’s two-stage selection ultimately better preserves the individual queries’ exact top-K subsets, which are known to be difficult to capture with a single shared subset. Comparing SPIRA with other sparse KV selection methods in terms of per-query top-K recall would help clarify whether its selector provides any additional improvement in estimation quality.

2-(b) Regarding efficiency: the motivation for using two-stage selection to reduce selection cost is clear, but its contribution to overall decode speedup remains uncertain. Since the selected sparse KV subset is reused throughout subsequent denoising steps, the additional savings from cheaper selection occur only at the first step of each block. Moreover, MAGE already overlaps selection with other computation and amortizes its initial dense-attention step across the block’s denoising steps. The paper should therefore quantify how much SPIRA reduces first-step latency relative to MAGE and how much this reduction contributes to total decode speedup. Without this analysis and a direct accuracy comparison, it is difficult to determine whether the two-stage design offers a meaningful improvement in the quality–latency trade-off.

3. Unclear benefit of the pre-RoPE routing formulation. Appendix B reports that replacing the proposed routing representation with the post-RoPE block mean changes top-K recall by less than 0.002 across all three models. This raises the question of what practical benefit the pre-RoPE formulation provides over simply averaging the post-RoPE queries. While the analysis of pre-RoPE alignment is informative, downstream accuracy and latency comparisons between these routing alternatives, with the same candidate budget and rescoring procedure, are needed to isolate the contribution of the proposed representation.

Questions for authors
  1. Comparison with block diffusion baselines. Could you provide direct comparisons with MAGE on shared model checkpoints under matched decoding settings, reporting both downstream accuracy and decode latency?

  2. Selection quality at the query level. How does SPIRA compare with MAGE in terms of estimation quality at the same shared sparsity budget, as measured by mean per-query recall against each query’s exact top-K subset? Evaluating the selectors on common reference queries would help isolate differences in selection quality. How does this comparison change as block size increases, and do any differences translate into downstream accuracy gains?

  3. Contribution of cheaper selection to overall speedup. How much does SPIRA reduce first-step latency relative to MAGE, and what fraction of the total decode-latency reduction comes from this improvement? Please provide a breakdown of first-step and subsequent-step latency, and examine how the incremental speedup changes with the number of denoising steps per block.

  4. Benefit of the pre-RoPE formulation. Appendix B reports that substituting the post-RoPE block mean changes top-K recall by less than 0.002. Does the proposed pre-RoPE routing provide a measurable downstream accuracy or latency advantage over this alternative when the candidate budget and rescoring procedure are held fixed? If the results are comparable, could you clarify the practical role of the pre-RoPE formulation in the method?

11 comments
AuthorsOpenReview ↗

Response to Reviewer n9hz

Thank you for the questions on MAGE, per-query recall and first-step cost, answered below. SPIRA itself uses no exact first step or exact layers. Only the run under MAGE's protocol gives both methods two exact layers.

Table 1 used m=2. We raised m up to 16 (Appendix G), and coverage reaches 0.94 at m=8, used below unless m=2 is stated. A larger m leaves the prefix read unchanged, since the read counts keys used in attention, not candidate scoring (Section 5), and adds at K=256 0.3 to 0.4 points of dense FLOPs and at most 1 ms per block.

Q1. Direct comparison with MAGE and LoSA

On a shared checkpoint under MAGE's own protocol, SDAR-8B-Chat with its six LongBench tasks, block 32 and threshold 0.95 over 1,200 documents, both methods keep two exact layers and SPIRA uses m=2. SPIRA and MAGE score 43.38 and 43.71 at K=256, 44.72 and 44.58 at K=512, and 45.91 and 45.74 at K=1024, with sign-test p 0.39, 0.91 and 0.60. With the same two exact layers, SPIRA still reads half as much of the prefix, 8.4% against 17.4% at K=256 (Appendix L).

Under our protocol, Table 1's seven LongBench tasks and 1,350 documents with SPIRA's decoder settings, MAGE runs as its paper specifies, with its two exact layers, and SPIRA with m=8. Per-task results are in Appendix I.

model, exact scoreK=256, SPIRA / MAGEK=512K=1024
SDAR-4B, 38.0337.50 / 36.9036.72 / 36.8137.60 / 37.69
Fast-dLLM v2 1.5B, 26.2323.17 / 23.6024.85 / 24.9625.37 / 25.26
LLaDA2.0-mini, 49.2348.54 / 48.4448.44 / 48.0148.85 / 48.68

No cell differs significantly, every p 0.23 or more. At K=256, however, SPIRA reads 3.4 to 3.5% of the prefix and MAGE 17 to 27%, mostly in its exact first step and exact layers, which read the whole prefix (Appendix I). On RULER from 4K to 32K, SPIRA at K=1024 reads less than MAGE at K=256 and scores higher on all models (Figure 12). Decode cost is in Q3.

LoSA's K is per query and it runs a dense first step, so at K=256 it reads 13 to 20% of the prefix against SPIRA's 3.4 to 3.5%. At similar reads, SPIRA at K=1024 scores 37.60, 25.37 and 48.85 against LoSA's 35.27, 25.33 and 47.86 at K=256 (Appendix J).

Q2. Per-query recall against each query's own exact top-K

Both selectors, SPIRA at m=8, are scored on the same queries and later steps, each query against its own exact top-K by attention-mass recall (Appendix L).

model, block sizeK=256, SPIRA / MAGEK=1024, SPIRA / MAGE
SDAR-4B, 640.837 / 0.8360.899 / 0.898
LLaDA2.0-mini, 320.841 / 0.8350.915 / 0.917
Fast-dLLM v2, 320.866 / 0.8550.924 / 0.917

On SDAR-4B they are within 0.004 at every trained block size from 8 to 64 and every K. SPIRA thus selects as well as MAGE per query, consistent with Q1, while reading far less of the prefix at K=256.

Q3. First-step cost and its share of the savings

We found no public implementation of MAGE. We implement its algorithm as published, but its latency depends on kernels we cannot reproduce faithfully. We therefore compare prefix-attention FLOPs at K=256, summed over documents from measured step counts, with dense at 100. Totals match the FLOPs rows of Appendix I.

setting, forwards per blockmethodfirst stepexact layers, later stepssparse attentiontotal
SDAR-4B LongBench, 21SPIRA0.4, selection02.12.5
MAGE4.2, exact forward5.31.911.5
LLaDA2.0-mini LongBench, 13SPIRA0.602.32.9
MAGE5.59.42.016.9
LLaDA2.0-mini RULER 32K, 8SPIRA0.500.81.3
MAGE11.18.90.620.6

SPIRA's selection costs a ninth or less of MAGE's exact first step. This is about 42% of the prefix-attention FLOP savings over MAGE on SDAR LongBench, 35% on LLaDA2 LongBench and 55% on LLaDA2 RULER 32K, and the rest comes mainly from MAGE's exact layers. As the number of forwards per block falls, MAGE's exact first step is amortized over fewer steps, while SPIRA's selection stays near 0.5. At a 122K prefix on SDAR-4B with K=256, SPIRA decodes a block 3.9 and 3.8 times faster than exact attention with 32 and 8 steps per block.

Q4. Pre-RoPE versus post-RoPE routing

Before RoPE, a block's queries split into a shared direction μ and residuals. Keys chosen with μ alone hold 0.847, 0.844 and 0.883 of the exact attention mass on the three models, nearly all the full queries hold (Appendix B), so one routing pass per block finds the candidates and stage 2 restores the per-query part. SPIRA forms the shared term R̄μ from the block mean alone, since R̄ has a closed form, while a post-RoPE mean rotates all B queries first. Measured with Nsight Compute, the post-RoPE route reads 4.2 times as much DRAM to form the routing vector (Appendix B).

We thank the reviewer again. We will add the RULER numbers against MAGE, the FLOPs breakdown above and the 122K timing to Appendices K, I and F.

Reviewer_n9hzOpenReview ↗

Follow-up Questions

Thank you for the detailed responses and experiments. These partially address my concerns. To clarify the remaining points, I have the following follow-up questions.

My two remaining concerns are: (1) distinguishing SPIRA’s own system-level gains from those already provided by prior work through later-step sparse attention and reuse, and (2) whether reducing first-step selection FLOPs translates into meaningful latency savings over MAGE under memory-bound, long-context conditions. Both methods scan the full prefix keys, so FLOP reductions alone do not establish the practical latency benefit.

  1. KV-cache memory traffic. In addition to the attention FLOPs reported in your response to Q3, could you provide the corresponding total KV-cache memory traffic for SPIRA and MAGE, including SPIRA’s routing and candidate rescoring?

  2. Latency breakdown and the source of speedup. Could you report first-step, subsequent-step, and total block latency for both SPIRA and exact attention across context lengths under otherwise identical settings? This would clarify whether SPIRA materially reduces first-step latency and how much of the overall latency reduction comes from the first step versus subsequent steps.

Additional observation on the first two layers. In addition, I find it interesting that SPIRA applies sparse attention in the first two layers yet achieves accuracy comparable to MAGE, which retains exact attention in these layers. Could you analyze why block diffusion LLMs tolerate sparsity in these early layers, whereas AR sparse-attention methods such as QUEST typically keep them exact?

AuthorsOpenReview ↗

Response to the follow-up questions

Thank you for the detailed follow-up. We ran the measurements you asked for and answer each question below. Beyond the reuse of one block-shared set over the later steps, which SPIRA and MAGE share, SPIRA builds that set by scanning the prefix keys once and rescoring only the candidates, without values and without an exact forward. We measure this in DRAM traffic against MAGE and in latency against exact attention. All traffic and timing measurements use SDAR-4B on an A100-40GB at batch 1 with K=256 and m=8, with CUDA graphs enabled and prefill excluded.

KV-cache memory traffic

We measured DRAM reads per layer with Nsight Compute, covering every selection and attention kernel. Model weights, read equally by both methods, are left out. MAGE's kernels are not public, so its columns are composed from the same measured kernels following its Algorithm 1, with an exact first step and its select stream (its Section 4), and later steps with the two exact layers of its evaluation setup, averaged over the 36 layers.

prefixfirst step, SPIRA / MAGE composed (MB per layer)later step, SPIRA / MAGE composed (MB per layer)exact attention, every step (MB per layer)
16K69 / 1125.4 / 9.478
32K112 / 2295.4 / 14.1162
65K193 / 4665.4 / 23.5332
122K335 / 8765.4 / 39.8624

At 65K, SPIRA's first step reads 134 MB in the stage-1 key scan, 23 MB in candidate rescoring and 36 MB in routing, scoring intermediates, gather and attention. It reads no values outside the selected set, while MAGE's exact forward reads the keys and values of the whole prefix. SPIRA's first step therefore reads 1.6 to 2.6 times less, and per block SPIRA reads 1.7 to 3.1 times less than MAGE with 8 steps and 1.7 to 4.2 times less with 32. Against exact attention, SPIRA's traffic per block at 8 steps is 17% at 16K and 7% at 122K.

Latency breakdown

We timed each forward of a block for SPIRA and exact attention in the runtime of Section 5.2 at K=256 and m=8, both with FlashAttention so that they share one attention kernel.

prefixfirst step, exact / SPIRA (ms)later step, exact / SPIRA (ms)block of 8 steps, exact / SPIRA (ms)block of 32 steps, exact / SPIRA (ms)
8K109.2 / 20.311.8 / 10.0207.9 / 105.5509.1 / 361.3
32K108.9 / 20.617.4 / 10.0252.9 / 105.8703.2 / 361.4
65K109.1 / 23.625.6 / 10.0319.0 / 108.9971.4 / 365.5
98K109.9 / 26.434.1 / 10.1390.0 / 112.41250.6 / 367.1
122K110.2 / 28.440.6 / 10.0441.8 / 113.61457.4 / 372.4

Exact attention's first step stays near 110 ms at every prefix, close to Table 2's once-a-block cost (100–107 ms). Block times are measured end to end on blocks after the first of each generation, and include the commit forward and host time between forwards, so SPIRA's block is 1.97 to 3.89 times faster with 8 steps and 1.41 to 3.91 times with 32. SPIRA's first step, selection included, takes 20 to 28 ms. The first step accounts for 87% of the block saving at 8K and 25% at 122K with 8 steps, and 60% and 8% with 32, and the later steps carry most of the rest, with the commit forward and host time the remainder. Against MAGE, whose kernels are not public, we compare the traffic above.

The first two layers

SDAR-4B, Table 1's 1,350 LongBench documents:

SPIRA, m=8K=256K=1024
all layers sparse, as in Table 1337.5037.60
first two layers exact37.0137.97

Making the first two layers exact changes SPIRA's scores by −0.49 and +0.37. QUEST, which keeps them exact in Table 1, loses 4.7 and 3.2 points at K=256 and 1024 when they are made sparse.

On the same SDAR model, SPIRA keeps its scores without the exact layers while QUEST does not, suggesting that the tolerance depends on the selection procedure rather than being a property of the model. In these layers a block's masked queries are nearly identical before RoPE, since every masked position enters with the same [MASK] embedding (pairwise cosine 0.994 at layer 1), which suits one routing pass, and stage 2 then scores the candidate keys exactly with each query. QUEST selects pages by a bound and keeps these layers exact following its layer-wise analysis on an autoregressive model (Tang et al., 2024, Figure 3).

We thank the reviewer again for the follow-up, and we will add the traffic and latency tables to Appendix F and the first-two-layer ablation with the agreement of these layers to Appendix A.

Reviewer_h5qEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper exploits the similarity among masked queries in the pre-RoPE space within a block to share candidate KV retrieval, followed by rescoring with the original queries and reusing the selected KV entries across denoising steps to reduce long-context decoding overhead. Experiments show that the method largely preserves generation quality while achieving inference speedups.

Critical strengths

The observation that masked queries within the same block exhibit strong directional alignment in the pre-RoPE space is interesting.

Critical weaknesses

The shared routing query is constructed by averaging pre-RoPE queries. I understand the motivation behind this design, but directly averaging post-RoPE queries can also produce a shared routing query, and the paper reports that its Top-K recall is nearly identical to that of pre-RoPE routing. This raises the question of why the pre-RoPE alignment and mean-rotation design are necessary. This issue substantially affects my assessment of the paper's actual contribution.

Questions for authors

HERALD also uses a single query to replace multiple queries within a block for full-prefix selection. Despite being a closely related work, the paper does not provide a performance comparison with HERALD.

4 comments
AuthorsOpenReview ↗

Response to Reviewer h5qE

Thank you for asking what the pre-RoPE formulation adds when a post-RoPE mean gives nearly the same recall, and for pointing us to HERALD. In response, we measured how a block's pre-RoPE queries split into a shared direction and query-specific residuals on all three models, and compared SPIRA with HERALD on the same three model families.

W1. Why pre-RoPE alignment and the mean rotation are needed if a post-RoPE mean gives the same recall

Before RoPE, a block's queries separate into a part they share and a part that differs between them. This split says in advance what a post-RoPE mean contains, which part of the queries retrieves, and what is left for stage 2. Below we show the split, check its predictions, and give what forming the vector before RoPE saves.

First, the split. After RoPE, the masked queries of a block have a mean pairwise cosine of 0.769 on SDAR-4B, because RoPE turns each query by its own position. Before RoPE the cosine is 0.905, and it stays above the post-RoPE value on all fourteen checkpoints we measured, spanning training stages, model sizes, block sizes and architectures, including the autoregressive checkpoint SDAR starts from and full-sequence diffusion models (Appendix A). Each pre-RoPE query can thus be written as the block mean μ plus a residual ε_i.

Second, what each part does, measured on 12 LongBench documents at K=512 over all layers and KV heads of the three models. The K keys chosen with μ alone hold 0.847, 0.844 and 0.883 of the block's exact attention mass on SDAR-4B, Fast-dLLM v2 and LLaDA2.0-mini, against 0.851, 0.846 and 0.886 with the full queries, and lie within 0.05 of them in 5,078 of 5,088 cells. Keys chosen with the residuals alone hold 0.149, 0.233 and 0.178. Restricted to the slow half of the rotary pairs, selection holds 0.816, 0.801 and 0.754, and to the fast half 0.110, 0.301 and 0.154, with the slow half ahead in 5,062 of 5,088 cells (Appendix B). On all three models, retrieval is therefore carried by μ in the slow bands, while the residuals carry what differs between queries.

Third, what the split predicts. Eq. (2) in Section 3 writes the post-RoPE mean as R̄μ + δ, where δ is the mean of the rotated residuals. In the slow bands ‖δ‖ is about 0.2% of ‖μ‖ (Appendix B), so where retrieval happens it reduces to R̄μ, and the near-identical recall the reviewer notes is what the split predicts. In the fast bands δ is no longer small, but selection there holds less of the mass than in the slow bands on all three models. The split also says what a shared vector cannot do. On SDAR, the shared vector alone picks the final K at 0.913 recall, and rescoring its 2K candidates with each query at its own position raises this to 0.979 (Section 3). SPIRA is built on this division, with R̄μ for one routing pass per block and the residuals left to stage 2.

The same structure sets the cost. R̄ depends only on the block's positions and has a closed form shared by all layers, so SPIRA rotates one mean instead of B queries. In our implementation, measured with Nsight Compute on an A100 for an SDAR-4B block of 64 queries, forming the routing vector from post-RoPE queries reads 4.2 times more DRAM (Appendix B).

Q1. Comparison with HERALD

We implemented HERALD as its paper describes, with its block-centre [MASK] query scoring the full prefix once per block. We ran it on all three model families with the same decoder settings as SPIRA. The table gives LongBench scores over seven tasks and 1,350 documents, with the prefix read fraction defined in Section 5. Each cell lists SPIRA first with candidate multiplier m=8, so its scores differ from Table 1, which uses m=2. We raised m step by step in Appendix G and use m=8, where candidate coverage reaches 0.94. A larger m adds at most 1 ms per block and leaves the prefix read unchanged, and Appendix H lists m=2 and m=8 side by side.

model, dense scoreK=256 scoreK=256 readK=1024 scoreK=1024 read
SDAR-4B, 38.0337.50 / 36.643.5% / 8.8%37.60 / 37.7413.8% / 18.6%
Fast-dLLM v2 1.5B, 26.2323.17 / 23.743.4% / 10.3%25.37 / 25.4313.8% / 19.9%
LLaDA2.0-mini, 49.2348.54 / 48.423.5% / 13.1%48.85 / 48.7713.8% / 22.5%

At K=256, 512 and 1024 the scores are not significantly different, with every paired sign test at p of 0.32 or more. At K=256, SPIRA reads 2.5 to 3.7 times less of the prefix. On RULER, averaged over thirteen tasks from 4K to 32K, SPIRA at K=1024 reads 12.7 to 13.2% and scores 46.89, 44.40 and 86.22, while HERALD reads 11.6 to 13.3% and scores 45.55, 34.02 and 77.52, at K=512 on SDAR-4B and Fast-dLLM v2 and K=256 on LLaDA2.0-mini (Appendix K). Appendix H reports HERALD per task, and we will add it to Table 1 in the revised version.

We thank the reviewer again. The decomposition and the HERALD comparison are now in Appendices A, B, H and K.

Thank you for your careful evaluation of our work and for raising important questions about SPIRA's core contribution.

Your feedback led us to examine directly the role of pre-RoPE alignment and what distinguishes SPIRA from existing shared-query retrieval methods.

In response, we analyzed the shared and query-specific components of pre-RoPE queries across three model families. This analysis helped explain the near-identical recall of pre- and post-RoPE averaging while highlighting the complementary roles of shared retrieval and query-specific rescoring. We also compared SPIRA with HERALD under consistent decoding settings.

These findings clarify that SPIRA's contribution extends beyond constructing a shared routing query: it exploits the structure of masked queries to enable efficient retrieval followed by query-specific refinement.

We value your perspective, which has helped us sharpen both the conceptual framing and the experimental foundation of our work.

Thank you for the detailed response. However, I do not think my core concern has been fully addressed.

The additional cosine similarity, shared/residual decomposition, and frequency-band analyses help explain why pre-RoPE routing works and why the two averaging methods achieve similar recall. However, my main concern remains: if directly averaging post-RoPE queries achieves nearly identical Top-K recall, what substantial benefit do the pre-RoPE alignment and mean-rotation design provide? Moreover, the finding that the rotated residuals are very small further suggests that the two methods are nearly equivalent in the frequency bands that dominate retrieval.

Therefore, I will maintain my original rating.

Thank you for clarifying your remaining concern. We agree that, as the decomposition predicts, the post-RoPE mean and the pre-RoPE routing term retrieve nearly the same keys: substituting one for the other changes top-K recall by less than 0.002 across all three models (Appendix B).

A practical benefit of the pre-RoPE formulation is computational. The mean rotation R̄ has a closed form determined by the block's positions and rotary frequencies, allowing the same transformation to be reused across all layers. SPIRA therefore forms the routing vector by rotating a single block mean instead of all B queries. In our measured implementation, this reduces DRAM reads for routing-vector formation by 4.2× (Appendix B, Figure 6).

More importantly, the pre-RoPE analysis identifies a shared structure among masked queries: every masked position starts from the same [MASK] embedding, and pre-RoPE alignment exceeds post-RoPE alignment across all fourteen checkpoints examined (Appendix A, Table 4). The mean-rotation identity further makes the shared term analytically interpretable: at each rotary frequency, it applies a rotation corresponding to the block's center position, scaled by c_f = sin(Bθ_f/2)/(B sin(θ_f/2)), which depends only on the block size and rotary frequency. Thus, the shared term preserves the slow bands, which carry most of the retrieval mass, while attenuating the fast bands, where position-dependent rotations spread queries apart (Section 4, Eq. 3; Appendix B, Tables 6–7).

This analysis identifies what can be shared and what must remain query-specific. Stage 1 uses the shared direction to retrieve candidates, while Stage 2 retains query-specific residuals and positional phases through rescoring with the original post-RoPE queries. This query-specific rescoring raises recall from 0.913 to 0.979 on SDAR (Section 3, Figure 3a). Consistent with this distinction, restricting rescoring to a rank-eight residual basis constructed before rather than after RoPE reduces recall by 0.038–0.121 across the three models, with lower recall in 445/450, 449/450, and 450/450 cells for SDAR-4B, Fast-dLLM v2, and LLaDA2.0-mini, respectively (Section 3; Appendix A). A post-RoPE mean could also serve as the routing vector; nevertheless, identifying the shared and query-specific components of masked queries provided the basis for SPIRA's two-stage design.

We will make this distinction explicit in the final version. SPIRA's contribution lies in characterizing the shared pre-RoPE structure of masked queries and translating this insight into an efficient, training-free, two-stage block-level selection mechanism.

We appreciate your thoughtful question, which prompted us to examine the role of the pre-RoPE formulation more carefully and clarify its structural and computational contributions to SPIRA. Your feedback has helped us sharpen the conceptual framing of our work.

Reviewer_4sVXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPIRA, a sparse attention mechanism designed for block diffusion language models. SPIRA leverages the alignment of pre-RoPE masked queries to perform shared KV retrieval, then uses post-RoPE rescoring to pinpoint key positions. The approach achieves significant decoding speedups while preserving strong task performance.

Critical strengths
  1. The observation of strong pre-RoPE query alignment is insightful and well-supported by experiments across multiple models.
  2. The routing and rescoring mechanism is simple, intuitive, and training-free. Importantly, this sparse attention approach is uniquely designed to block diffusion language models; by capitalizing on the block-wise structure native to DLMs, it effectively converts this architectural property into substantial decoding speedups. I like the idea of Q-Compress to speedup sparse attention
  3. The proposed method achieves strong quality-efficiency trade-offs across evaluated benchmarks.
Critical weaknesses
  1. The proposed method heavily relies on the observation that masked queries exhibit strong alignment before RoPE. However, it is unclear whether this is a generalizable property of well-trained diffusion language models or merely an artifact of specific training dynamics, potentially associated with insufficient training or underdeveloped query representations. Although the authors demonstrate this phenomenon across three models, these observations alone do not fully establish its generality. I would appreciate a deeper explanation of why such alignment emerges and whether it is expected to persist across different training stages, model scales, and architectures.
  2. The large performance improvement over SparseD, particularly the approximately 10% gain at K=256, is somewhat surprising. SparseD performs full attention for the first two denoising steps and derives its sparse pattern from exact attention scores. Intuitively, this should provide relatively reliable information for identifying important KV positions. Have the authors investigated where SparseD's selected attention patterns fail and why SPIRA is able to identify better KV positions? A direct analysis of the selected KV patterns and their differences would help explain this substantial performance gap and provide stronger evidence for the effectiveness of the proposed routing mechanism.
Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4sVX

Thank you for the positive review and for raising the generality of the alignment and the gap to SparseD. In response, we measured the alignment on fourteen released checkpoints and compared the keys SPIRA and SparseD keep on RULER's needle tasks.

W1. Generality of the pre-RoPE alignment

The alignment comes from two properties of how masked positions enter the model. Every undecoded position in a block receives the same [MASK] embedding, and at the input, position enters only through RoPE, after the query projection. At layer 0 the masked queries are therefore identical before RoPE, and in later layers they differ only through the context each has gathered, which is largely shared. RoPE then rotates each query by its own position, and this is what spreads them apart.

If this explanation is right, pre-RoPE alignment should stay above post-RoPE alignment as training continues. We checked this on released checkpoints from different training stages, scales, block sizes and architectures. The table gives the mean pairwise cosine of a block's masked queries, measured on 350 LongBench documents per checkpoint, except 50 for the scale and block-size rows.

checkpointpre-RoPEpost-RoPE
Qwen3-4B-Base, SDAR's starting checkpoint0.9760.851
SDAR-4B-Chat-b640.9050.769
TraDo-4B (SDAR + RL)0.9470.819
SDAR-1.7B / 8B0.933 / 0.9160.690 / 0.784
SDAR-4B trained with block 8 / 160.896 / 0.9030.836 / 0.816
LLaDA-8B Base / Instruct / 1.50.855 / 0.789 / 0.7860.675 / 0.585 / 0.583
Dream-7B Base / Instruct0.918 / 0.9090.808 / 0.793
LLaDA-MoE-7B Base / Instruct0.837 / 0.7710.637 / 0.567

Pre-RoPE alignment is higher than post-RoPE alignment in every row. Training changes the level but not this ordering, and the gap does not shrink. It is 0.13, 0.14 and 0.13 along the SDAR line and 0.18, 0.20 and 0.20 along LLaDA. The table is in Appendix A.

W2. The gap to SparseD at K=256

On SDAR-4B at K=256, SPIRA scores 35.50 on LongBench and SparseD 25.32, and the gap shrinks as K grows, to 37.97 against 37.41 at K=1024. The RULER experiments in the SparseD paper keep 30% of the keys, while Table 1 gives every method the same fixed K. On a median LongBench prefix of 10.1K tokens, K=256 is 2.5% of the prefix, about a twelfth of that ratio.

To compare the two selections directly, we looked at the selected keys on RULER's needle tasks, where the position the model must find is known. On SDAR-4B at 4K and 8K with K=256, over 232 documents, the keys SparseD keeps contain 10.5% of the needle tokens, and the keys SPIRA keeps contain 41.6% (Appendix D).

We thank the reviewer again. The checkpoint table and the needle coverage are now in Appendices A and D.

We appreciate your insightful review and your positive assessment of SPIRA's key observations and efficiency gains.

Your comments motivated us to test the generality of pre-RoPE query alignment and to examine where SPIRA's advantage over SparseD comes from.

In response, we extended the alignment analysis to fourteen checkpoints across training stages, model scales, block sizes, and architectures, and compared the KV positions selected by SPIRA and SparseD on needle retrieval tasks.

These findings strengthen the empirical foundation of SPIRA and provide a clearer understanding of the structural properties and retrieval behavior underlying its effectiveness.

Your thoughtful feedback has been valuable in improving both the rigor and clarity of our research. Thank you again for your careful review.

Thanks for the detailed response. It solves my concerns about this submission. I would keep my score.

Comments on the paper

1 comment

We thank the AC and all reviewers for their careful evaluation. SPIRA makes two connected contributions. First, it characterizes a structural property of masked blocks: every masked position enters with the same [MASK] embedding, so a block's masked queries share a common direction before RoPE, and the mean-rotation identity analytically describes which rotary bands a block-shared vector preserves. Second, it translates this structure into a training-free, two-stage selector: one shared routing pass per block retrieves candidates, the original queries rescore only those candidates, and the selected keys are reused across the block's denoising steps. In its standard configuration, SPIRA requires no exact prefix attention at any step or layer.

Pre-RoPE structure and the two-stage design (h5qE, n9hz). Before RoPE, a block's masked queries are identical at the first layer and remain more aligned than after RoPE in later layers. Pre-RoPE alignment exceeds post-RoPE alignment on all fourteen released checkpoints we measured, spanning training stages, sizes, block sizes, and full-sequence diffusion models (Appendix A). The shared direction μ alone retains nearly as much exact attention mass as the full queries (0.847/0.844/0.883 vs 0.851/0.846/0.886 across three models). In each rotary band, the mean rotation corresponds to a rotation at the block's center position, scaled by c_f = sin(Bθ_f/2)/(B sin(θ_f/2)), so the shared term preserves the slow bands, which carry most of the retrieval mass (Eq. 3; Appendix B). Consistent with this analysis, a post-RoPE mean retrieves nearly the same keys (top-K recall difference below 0.002). The analysis identifies what can be shared and what must remain query-specific, providing the basis for SPIRA's two stages. Rescoring the candidates with the original post-RoPE queries, which carry each query's positional phase, raises recall from 0.913 to 0.979 on SDAR (Section 3). The closed form also lets SPIRA rotate one block mean instead of all B queries when forming the routing vector. Appendix B (Figure 6) reports the DRAM measurements.

Comparisons with HERALD, MAGE, and LoSA (h5qE, n9hz). Unlike SPIRA, HERALD selects with one [MASK] query, no rescoring and exact first two layers, MAGE uses exact attention at the first step and in two layers, and LoSA runs a dense first step, then recomputes only unsettled queries, each with its own selection. On LongBench across three models and K=256/512/1024, SPIRA (m=8) shows no significant score difference from HERALD (sign test, all p ≥ 0.32) or MAGE (all p ≥ 0.23), while reading 3.4–3.5% of the prefix at K=256, vs 8.8–13.1% for HERALD and 17–27% for MAGE. Under MAGE's own protocol on SDAR-8B, p = 0.39/0.91/0.60, with SPIRA reading 8.4% vs 17.4%. Prefix read (the fraction of prefix KV entries attended to) excludes candidate scoring; including selection, SPIRA's prefix-attention FLOPs are 2.5 vs MAGE's 11.5 on SDAR-4B and 2.9 vs 16.9 on LLaDA2.0-mini (dense = 100; Appendix I). On RULER (4K–32K), at similar prefix read (12.7–13.2% vs 11.6–13.3%), SPIRA at K=1024 scores 46.89/44.40/86.22 against HERALD's 45.55/34.02/77.52 at K=512 on SDAR-4B and Fast-dLLM v2 and K=256 on LLaDA2.0-mini. Per-query recall is within 0.011 of MAGE (Appendix L). LoSA reads 13–20% of the prefix at K=256 vs SPIRA's 3.4–3.5%; at similar read, SPIRA at K=1024 scores 37.60/25.37/48.85 vs LoSA's 35.27/25.33/47.86 at K=256 (Appendix J). Appendices H–L report every cell, including where a baseline scores higher.

Practical efficiency and baseline fairness (n9hz). Against SGLang's exact attention, which reuses one CUDA graph across blocks, SPIRA decodes a block 3.26–3.65× faster at a 122k prefix with K=256 (1.07–1.17× at 8k, 1.57–1.71× at 32k), and 3.17–3.60× with K=1024, prefill excluded (Appendices E, F). MAGE's kernels are not public, so we estimate its KV-cache traffic by composing measured kernel costs following its Algorithm 1, rather than comparing latency: SPIRA reads 1.7–4.2× less KV-cache DRAM per block (Appendix F).

Generality and retrieval quality (4sVX, n9hz). SPIRA outperforms QUEST and SparseD across three models at K=256 (by up to 10.2 points) and approaches exact-attention quality at K=1024. SparseD's own RULER experiments keep 30% of the keys; at K=256 on needle retrieval, SPIRA's keys contain 41.6% of needle tokens vs SparseD's 10.5% (Appendix D). Making SPIRA's first two layers exact changes its score by only −0.49/+0.37 at K=256/1024, while making QUEST's first two layers sparse reduces its scores by 4.7/3.2 points (Appendix A). These results suggest that tolerance to early-layer sparsity may depend on the selection procedure; whether this extends to MAGE, HERALD, or LoSA remains to be evaluated.

We thank all reviewers; their feedback helped us clarify SPIRA's contributions and test them against stronger baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.