acceptodds
Under review as a conference paper at ICLR 2027

Mask-Routed Sparse Attention for Long-Context Diffusion Language Models

Abstract

Diffusion language models apply bidirectional attention over the entire context at every denoising step, making each step substantially more expensive as context length grows. Sparsifying only at inference leaves the pretrained weights unchanged. We propose Mask-Routed Sparse Attention (MRSA), which further trains a diffusion language model to route using its native mask token. We append one mask to the end of each chunk and add a learned role offset to mark it as a routing slot. Every fourth layer uses a query with a bidirectional local window and the top-k chunks scored by that mask; all other layers use bidirectional sliding-window attention. Starting from Dream-v0-Base-7B extrapolated to 16K with YaRN, we compare MRSA with bidirectional NSA, DSA, and SWA under the same layout, key budget, and dense initialization, using the same Fast-dLLM decoder. Across three RULER probes from 16K to 128K, MRSA outperforms the other sparse methods when the YaRN factor is scaled to the evaluation length. At 128K, its mean accuracy is 18.4% higher than NSA, the next-best sparse model, while DSA and SWA remain near the floor. At the same length, decoding is 172× faster than full dense attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, sparse attention, long-context modeling, mask-token routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mKAcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes MRSA which is a continued-training sparse-attention method for masked diffusion LMs. Each 64-position chunk ends with the model's native mask token plus a learned role offset. On every fourth layer, this routing slot pools the chunk's keys into a summary, and each query attends to its local window plus the top-K=32K=32 chunks ranked by summary score. All other layers use bidirectional sliding-window attention (SWA). Starting from Dream-v0-Base-7B (YaRN-extended to 16K), MRSA, bidirectional NSA, DSA, SWA and two dense references are LoRA-trained for 1,000 steps on a 12\tfrac{1}{2} LM + 12\tfrac{1}{2} RULER objective; the sparse layers share a $\sim$2.56K-key budget. MRSA has the best RULER mean among sparse variants from 16K to 128K (47.7 vs 40.3 for NSA at 128K) and decodes 172×172\times faster than full dense at 128K. It ranks last among sparse variants on LongBench (25.4 vs 30.9 for SWA).

Critical strengths
  1. A controlled, budget-matched comparison with clear ablations. MRSA, NSA and DSA share the initialization, layout, $\sim$2.56K-key budget, training steps and decoder. Under these conditions MRSA leads every sparse baseline on every RULER probe and length (Table 1). With the YaRN factor fixed, its single-needle accuracy is 99.0 at 128K, against 25.0 for hybrid dense. Each component matters: without the role offset, the 16K mean falls from 53.8 to 11.5 (Table 6).
Critical weaknesses
  1. The long-context evidence is in-distribution, and the held-out benchmark goes the other way.

    • Half of the training loss is a RULER term over exactly the three evaluated probes (Eq. 13, Sec. 4.1), and the other ten RULER tasks are not reported.
    • On LongBench, which is not trained on, MRSA is the worst sparse variant (25.4 vs 30.9 for SWA, and 44.3 for full dense; Table 2). On passage retrieval it scores 5.6 vs 99.5 for full dense. The abstract reports only RULER.
    • So the paper shows that MRSA learns the trained probes better than other sparse adaptations, not that it improves long-context modeling.
  2. In my opinion, the supporting evidence is inconsistent or under-powered.

    • Fig. 5/Sec. 4.3 report clean 16K single-needle accuracy of 99.0 (EOS) and 94.0 (new token). Table 7 reports 10.0 and 99.0 for what appears to be the same setting.
    • Every number is a single run with no variance, and the 128K EOS ablation run "has not started" (App. B).
    • Latency is reported only against the dense models; NSA, DSA and SWA "are not timed" (Sec. 4.4). The timings are also hard to interpret:
      • Table 3's budgets imply ≈43×\approx 43\times fewer keys per query than full dense, yet the reported decode speedup is 172×172\times.
      • MRSA's per-token decode time falls from 10.55 ms at 16K to $\sim$3.7 ms at 64K (Fig. 6).
Questions for authors
  1. Please reconcile Fig. 5 with Table 7. Which result supports the claim that the native mask is needed?
  2. How do the variants compare on long-context tasks outside the training loss: the other RULER tasks, LongBench beyond 16K, or training without the RULER term?
  3. Can you time NSA, DSA and SWA under the same decoder and report denoising steps per token? Why does MRSA's decode get faster with longer context?
  4. Eq. 6 equals LSEj(q^i⊤kj/d)−KL(αc ∥ softmaxj(q^i⊤kj/d))\mathrm{LSE}_j(\hat q_i^\top k_j/\sqrt{d}) - \mathrm{KL}\big(\alpha_c \,\|\, \mathrm{softmax}_j(\hat q_i^\top k_j/\sqrt{d})\big), which is a lower bound on the chunk's log-sum-exp. Is this the intended rationale for the entropy term? In Eq. 8 it is combined with an exact local log-sum-exp computed with a different query, qiq_i. How are the two scales kept comparable?
Reviewer_FQjjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Mask-Routed Sparse Attention (MRSA), a continued-training sparse attention method for diffusion language models. It introduces routing mask tokens to summarize chunks and select relevant remote context, combining top-k chunk selection with bidirectional local attention. Experiments on Dream-7B report improvements over adapted sparse-attention baselines on three RULER tasks and substantial decoding speedups relative to dense attention.

Critical strengths
  1. The paper addresses the computational challenges of bidirectional attention in long-context diffusion language models. Reusing the model's native mask token for chunk routing is an interesting design choice.

  2. The paper includes ablations on the routing role offset, query residual, entropy term, and chunk summarization strategy. These experiments provide evidence that the proposed routing components affect synthetic retrieval accuracy.

Critical weaknesses
  1. The most significant concern is the poor performance on LongBench-v1. MRSA achieves an average score of only 25.4, compared with 44.3 for full dense attention and 30.9 for the simple sliding-window baseline. MRSA is the worst-performing sparse method in the aggregate LongBench comparison.

  2. The continued-training objective assigns half of its weight to RULER-style synthetic tasks. It substantially limits the generalization evidence provided by the RULER results.

  3. At 128K, MRSA achieves 47.7 average accuracy compared with 87.4 for full dense attention. The paper emphasizes superiority over the other sparse baselines, but the large absolute accuracy degradation relative to dense attention makes the overall quality-efficiency trade-off questionable.

Questions for authors

Please refer to the weakness section

Reviewer_tfdQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Mask-Routed Sparse Attention (MRSA), a method designed to scale long-context diffusion language models. MRSA uses native mask tokens to summarize individual sequence chunks, using a learned Top-K router to select relevant distant context. By combining local sliding-window attention with chunk-level sparse attention through a hierarchical weighting mechanism, MRSA balances local and global dependencies. Evaluated on Dream-v0-Base-7B across selected RULER benchmarks, MRSA outperforms adapted NSA, DSA, and SWA baselines while delivering substantial decoding speedups over dense attention.

Critical strengths
  1. Interesting mask-based routing design. Using the native mask token as a chunk-level routing representation is an interesting idea that leverages a distinctive characteristic of diffusion language models.

  2. The proposed method integrates chunk summarization, Top-K selection, and attention mass allocation into a unified framework. The routing mechanism can be optimized through the language modeling objective without requiring additional routing supervision.

Critical weaknesses
  1. Concerns about baseline results and missing comparisons. The results in Table 1 are surprising. NSA, DSA, and SWA have demonstrated strong performance in autoregressive LLMs, yet their adapted versions show substantial accuracy degradation, with DSA and SWA performing nearly at chance level. The paper provides limited explanation for these results. Furthermore, comparisons with existing sparse attention methods for DLMs, such as SparseD (ICLR 2026), are missing.

  2. Limited generalization beyond the training tasks. The RULER evaluation only covers three task families that are also used during continued training, raising concerns about generalization. More importantly, MRSA performs worse than all sparse baselines on LongBench (25.4 vs. 30.9 for SWA), suggesting limited effectiveness on general long-context tasks.

  3. Incomplete efficiency evaluation. The latency experiments only compare MRSA with dense and hybrid dense attention, without measuring NSA, DSA, or SWA. Since these are the main baselines, their latency results are necessary to evaluate the accuracy–efficiency trade-off.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.