- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work proposes MRSA which is a continued-training sparse-attention method for masked diffusion LMs. Each 64-position chunk ends with the model's native mask token plus a learned role offset. On every fourth layer, this routing slot pools the chunk's keys into a summary, and each query attends to its local window plus the top- chunks ranked by summary score. All other layers use bidirectional sliding-window attention (SWA). Starting from Dream-v0-Base-7B (YaRN-extended to 16K), MRSA, bidirectional NSA, DSA, SWA and two dense references are LoRA-trained for 1,000 steps on a LM + RULER objective; the sparse layers share a $\sim$2.56K-key budget. MRSA has the best RULER mean among sparse variants from 16K to 128K (47.7 vs 40.3 for NSA at 128K) and decodes faster than full dense at 128K. It ranks last among sparse variants on LongBench (25.4 vs 30.9 for SWA).
Critical strengths
- A controlled, budget-matched comparison with clear ablations. MRSA, NSA and DSA share the initialization, layout, $\sim$2.56K-key budget, training steps and decoder. Under these conditions MRSA leads every sparse baseline on every RULER probe and length (Table 1). With the YaRN factor fixed, its single-needle accuracy is 99.0 at 128K, against 25.0 for hybrid dense. Each component matters: without the role offset, the 16K mean falls from 53.8 to 11.5 (Table 6).
Critical weaknesses
-
The long-context evidence is in-distribution, and the held-out benchmark goes the other way.
- Half of the training loss is a RULER term over exactly the three evaluated probes (Eq. 13, Sec. 4.1), and the other ten RULER tasks are not reported.
- On LongBench, which is not trained on, MRSA is the worst sparse variant (25.4 vs 30.9 for SWA, and 44.3 for full dense; Table 2). On passage retrieval it scores 5.6 vs 99.5 for full dense. The abstract reports only RULER.
- So the paper shows that MRSA learns the trained probes better than other sparse adaptations, not that it improves long-context modeling.
-
In my opinion, the supporting evidence is inconsistent or under-powered.
- Fig. 5/Sec. 4.3 report clean 16K single-needle accuracy of 99.0 (EOS) and 94.0 (new token). Table 7 reports 10.0 and 99.0 for what appears to be the same setting.
- Every number is a single run with no variance, and the 128K EOS ablation run "has not started" (App. B).
- Latency is reported only against the dense models; NSA, DSA and SWA "are not timed" (Sec. 4.4). The timings are also hard to interpret:
- Table 3's budgets imply fewer keys per query than full dense, yet the reported decode speedup is .
- MRSA's per-token decode time falls from 10.55 ms at 16K to $\sim$3.7 ms at 64K (Fig. 6).
Questions for authors
- Please reconcile Fig. 5 with Table 7. Which result supports the claim that the native mask is needed?
- How do the variants compare on long-context tasks outside the training loss: the other RULER tasks, LongBench beyond 16K, or training without the RULER term?
- Can you time NSA, DSA and SWA under the same decoder and report denoising steps per token? Why does MRSA's decode get faster with longer context?
- Eq. 6 equals , which is a lower bound on the chunk's log-sum-exp. Is this the intended rationale for the entropy term? In Eq. 8 it is combined with an exact local log-sum-exp computed with a different query, . How are the two scales kept comparable?