acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Mask Attention: Trainable Sparse Attention with Fused Block Screening

Abstract

Dynamic sparse attention must keep selection overhead low to translate sparsity into GPU speedups. We introduce Dynamic Mask Attention (DMA), an algorithm–systems co-design that fuses learned block screening into IO-aware attention kernels. DMA uses a rank-1 query-key gate to derive cheap blockwise bounds on gate scores, reducing memory traffic and bypassing QK score computation and post-score operations on skipped tiles. This screening is integrated into forward, backward, and decoding kernels without materializing a dense routing matrix. The same gate modulates retained attention logits and is trained through the language-modeling objective. Across models ranging from 80M to 8B parameters, DMA improves the quality-efficiency trade-off in pre-training, long-context evaluation, and sparse adaptation, while maintaining quality comparable to full attention. On Qwen3-8B, DMA achieves an average score of 64.26 versus 62.50 for Full Attention across RULER-128K, LongBench, and benchmarks of reasoning, coding, and knowledge. It also delivers approximately 1.2 the end-to-end prefill and decode serving throughput of Full Attention under an 8K-input/24K-output workload.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Long-Context Transformers, Adaptive Sparse Attention, Block Screening, Efficient Attention, Systems for Large Language Models, Training and Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tmGJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DMA uses a rank-1 gate to screen attention blocks inside fused kernels and modulate retained logits. Forward and backward passes reconstruct their supports separately, introducing approximate gradients. Experiments cover training, long-context adaptation, and serving, reporting comparable or improved quality and approximately 1.2x prefill and decoding throughput under the tested workload.

Critical strengths

1.In-kernel screening explicitly addresses selection overhead, making the method more practically motivated than approaches evaluated only through sparsity ratios.

2.The evaluation covers training and inference, with measured serving throughput and useful ablations of gate rank, thresholds, and tile geometry.

Critical weaknesses

1.Appendix E.4 compares gradients with dense gated attention, not exact differentiation of the same forward-retained mask. Please add that control to isolate backward support mismatch and measure its effect on gradients and convergence.

2.The screening bound covers the gate, not the complete attention score. First-layer, shorter-context analyses do not establish reliability at 128K. Please measure omitted attention mass and output errors across layers and long inputs.

3.Efficiency accounting should include scanning all candidate blocks. The serving workload also uses only 8K input, unlike the 128K quality evaluation. Please provide a screening-cost breakdown and longer-input serving measurements.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Response (W1) to Reviewer tmGJ

We thank the reviewer for the careful review and the effort spent examining the method and experiments. We appreciate the recognition of the practical motivation for in-kernel screening, the training and serving evaluation, and the ablations on gate rank, thresholds, and tile geometry.

W1: Exact backward on the forward-retained support

We added the requested control by fixing the DMA forward computation, its retained support, and the upstream gradient. The reference computes exact gradients for that fixed forward-retained support. Comparing the implemented DMA backward with this reference isolates the additional gradient error introduced by reconstructing the backward support.

The table reports checkpoints after 200M, 1B, and 2B adaptation tokens. IoU measures the overlap between the original DMA forward and backward supports. The gradient-error columns use the new fixed-forward-support exact-backward reference.

LengthAdaptation tokensForward/backward IoU (%)Max. dQ errorMax. dK errorMax. dV error
1K200M99.062.00E-031.00E-034.00E-04
1K1B99.862.00E-038.00E-045.00E-04
1K2B99.852.00E-039.00E-044.00E-04
2K200M98.681.00E-033.00E-042.00E-04
2K1B99.768.00E-042.00E-041.00E-04
2K2B99.49.00E-042.00E-041.00E-04
4K200M98.987.00E-041.00E-046.00E-05
4K1B98.878.00E-042.00E-046.00E-05
4K2B99.475.00E-041.00E-046.00E-05
8K200M99.357.00E-049.00E-052.00E-05
8K1B99.682.00E-047.00E-052.00E-05
8K2B99.774.00E-047.00E-052.00E-05

Across these settings, the largest reported errors are 2×10−32\times10^{-3} for dQdQ, 10−310^{-3} for dKdK, and 5×10−45\times10^{-4} for dVdV. At 8K, the final checkpoint has errors of 4×10−44\times10^{-4}, 7×10−57\times10^{-5}, and 2×10−52\times10^{-5}, respectively. This directly quantifies the gradient difference associated with the independently reconstructed backward support.

The model-level training evidence comes from the four-scale pretraining study in Figure 5 and the 8B adaptation results in Table 3. The new control adds a local gradient comparison at three adaptation stages.

AuthorsOpenReview ↗

Response (W2) to Reviewer tmGJ

W2: Gate bounds and long-context accuracy

DMA screens candidate blocks using a bound on the learned gate, while the quality of sparse attention depends on the information retained after screening. The paper examines this connection in stages: Section 2.2 establishes the gate-side bound, Appendix C relates omitted attention mass to output error, Appendix E measures selector fidelity and computation allocation, and Table 3 with Appendix D.8 evaluates the full model at 128K. These results link the screening mechanism to its numerical behavior and long-context retrieval performance.

Section 2.2, Eqs. (3)-(4), gives an exact tilewise bound on the scalar gate and its log-sigmoid bias. Appendix A motivates the rank-1 gate when block-averaged query/key representations have a dominant mode. Appendix C connects sparse execution to omitted attention mass: Eq. (55) bounds the difference between the dense gated output and the renormalized retained-support output by

∥∥ou⋆−ou^∥∥2≤2Vmax⁡δu.\|\| o_u^{\star} - \hat{o_u} \|\|_{2} \leq 2V_{\max} \delta_u.

where δu\delta_u is the total probability mass assigned to skipped positions by the dense gated reference, and the value-vector norms in the row are bounded by Vmax⁡V_{\max}. This analysis identifies the quantity that governs the output approximation.

Appendix E.3 provides the corresponding selection and output measurements. In Table 11, the learned rank-1 selector achieves 99.13% block Recall@K at 62.3% sparsity, compared with 49.46% recall at 62.1% sparsity for L2 scoring. Their full-model RULER-8K scores are 92.52 and 74.61. This comparison supports the learned gate's ability to retain useful blocks at comparable computation. Table 12 reports recall above 99.1% at 1K, 2K, 4K, and 8K, with maximum output-error values of 10−310^{-3}, 7×10−47\times10^{-4}, 5×10−45\times10^{-4}, and 2×10−42\times10^{-4}, respectively, against the same checkpoint's screening-disabled gated operator. These first-layer measurements quantify selection fidelity and output accuracy across the measured lengths.

The cross-layer analysis in Appendix E.2, Table 10, covers layers 0, 11, 23, and 35 at 8K context. Its statistics average query heads at the same relative position across the eight KV groups. Retrieval retains more computation than QA at every matched layer and relative GQA position, with allocation also varying across depth. These results show that the learned screening responds to the task and to representations at different layers.

At 128K, Table 3 directly evaluates the complete model: DMA reaches 67.26 on RULER, compared with 58.18 for FullAttn, a gain of 9.08 points. Appendix D.8, Table 8, reports all 13 subtasks under the same 128K evaluation protocol. These results establish the model's long-context retrieval performance, complementing the operator measurements and cross-layer allocation analysis.

We hope that the cited results address the concern at complementary levels. Tables 11 and 12 show high selector recall and small measured output error on first-layer states from 1K to 8K; Table 10 shows how retained computation varies across sampled layers and tasks; and Table 3 with Table 8 measures long-range retrieval in the complete 128K model. The mass-based bound in Appendix C explains why retaining useful attention matters for output accuracy. Together, the numerical checks and the 128K task results support the effectiveness of DMA's gate-based screening in the evaluated settings.

AuthorsOpenReview ↗

Response (W3) to Reviewer tmGJ

W3: Candidate-block scanning and longer serving workloads

The runtime measurements include the candidate-block scan. The screening path computes the scalar query/key projections, obtains block extrema, and evaluates the four-corner gate test for each causal candidate tile. Full KV loading and token-level attention follow for retained tiles. This scan is part of the fused execution being timed.

The candidate scan can also be stated explicitly. With query and key tile sizes BqB_q and BkB_k, prefill visits O(N2/(BqBk))O(N^2/(B_qB_k)) candidate tile pairs. After block extrema are computed and made available, each candidate requires four scalar products, a maximum, and a threshold comparison. The scalar projections cost O(Ndh)O(Nd_h) per head, and retained token-level attention costs O(Nwdh)O(Nwd_h) for effective retained budget ww. With block extrema precomputed and reused, the arithmetic count for these operations is

O ⁣(Ndh+N2BqBk+Nwdh)O\!\left(Nd_h+\frac{N^2}{B_qB_k}+Nwd_h\right)

per head. The measured runtime includes metadata access and extrema handling in the implemented kernel. The Figure 3 attention-module timing also includes QKV projections; Appendix E.3 reports kernel timings on exported Q/K/V states.

We added long-input and long-output measurements using eight A100-80GB GPUs, data parallelism of eight, and eight concurrent requests. These are complete-request batch timings:

Input / output tokens per requestMethodTotal elapsed (s)Attention elapsed (s)Attention shareRequest speedup
128K / 128FullAttn61.4641.5267.60%1.00×
128K / 128DMA21.521.326.10%2.86×
128 / 128KFullAttn3,132.181,247.8239.80%1.00×
128 / 128KDMA1,944.1248.752.50%1.61×

For 128K input, attention accounts for 67.6% of FullAttn runtime, and DMA reduces total elapsed time from 61.46 s to 21.52 s. This provides a 2.86×2.86\times end-to-end gain in a workload with a long prompt. For long generation, the cache grows over the 128K-token output, and DMA reduces total elapsed time from 3,132.18 s to 1,944.12 s, giving 1.61×1.61\times over the complete run. The attention and total-runtime columns show how attention savings contribute to the observed serving improvement.

Reviewer_zNZJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Dynamic Mask Attention (DMA), a trainable block-sparse attention method. DMA learns an inexpensive rank-1 bilinear gate that provides lightweight statistics for screening attention tiles before loading their full key and value representations and computing attention scores. The gate is trained by incorporating its output as a log-sigmoid bias in the attention logits before softmax. During the backward pass, DMA constructs a sparse support from the same rank-1 projections, which may differ from the forward support. In both passes, tiles are skipped based on statistics computed on the fly. Experiments include A100 runtime measurements, associative recall, pre-training models ranging from 80M to 1.7B parameters, downstream evaluation, and sparse adaptation of Qwen3 8B. The results show favorable quality-efficiency trade-offs.

Critical strengths

There are essentially two things that I liked a lot in this work.

First, it provides a simple and practical effective method for accelerating softmax attention (the standard in most LLMs out there). The idea of using a rank-1 gate to inexpensively screen blocks before expensive KV movement is very interesting. The concept is not 100% novel, but the exact combination in this paper is. Also, I believe that using the same learned signal for screening and also for regularizing softmax provides a clear path for making the method trainable, which contrasts to previous works that focus solely on the inference regime.

Second, the paper provides a comprehensive empirical evaluation with many insightful analyses in the appendix (the presentation could be better here and there --- se my weakness). To outline things, it evaluates on synthetic retrieval, pre-training, downstream tasks, and adaptation of an 8B model, alongside module and serving performance. This breadth provides encouraging evidence for practical utility.

Critical weaknesses

I believe the baseline comparisons do not fully establish the benefit of DMA's learned sparsity. Let me clarify: although NSA is trainable, its use of compressed block summaries and selection at a different granularity (chunks) makes it a less direct comparison. AdaSplash-2 would provide a stronger test against an efficient, end-to-end trainable sparse attention method, but it is not evaluated. A SWA baseline with attention sinks is also missing. Since DMA scans from the diagonal toward earlier tokens, this simple baseline would help establish whether its learned selection offers meaningful gains beyond retaining local context and sink tokens. The other baselines are strong and are very good. In fact, I would also recommend including BLASST in downstream task results.

The specified screening rule appears to violate autoregressive causality? Equation 3 uses extrema over an entire query block, including queries later than an earlier token being evaluated. Changing a later query can therefore change whether a past key block is retained for that earlier token.

Presentation issues also make the method harder to follow: the introduction is quite repetitive, Figures 1 and 2 use notation before defining it and lack clear explanations of their arrows and execution flow. Another example is the retained-key budget (w) in Table 4 (easy to fix). Finally, the spacing around figures and tables also appears unusually tight? Consecutive tables lack sufficient separation, and several captions have no clear blank line before the main text resumes.

Questions for authors

Table 23 of the official Qwen3 technical report reports RULER-128K scores of 77.4 in non-thinking mode and 72.0 in thinking mode, both substantially above the 58.18 reported here. The official evaluation already uses YaRN with scaling factor 4, so length extrapolation alone does not explain the discrepancy. What happened in here?

How much retained computation lies outside the local window and initial sink tokens? Can you report this across layers and individual heads to clarify what DMA learns beyond those common patterns? For example, metrics like center of probability mass, variance, and normalized entropy can help.

What kernel framework was used for DMA and for the exact baseline?

Which threshold settings (tau) produced the main results?

3 comments
AuthorsOpenReview ↗

Response (W1/W2) to Reviewer zNZJ

We thank the reviewer for the time and care devoted to the review, and for recognizing the practical value of screening before KV movement, the shared signal for screening and training, and the breadth of the empirical analysis.

W1: SWA with sinks, learned selection, and baseline coverage

We added SWA with attention sinks under the MQAR protocol in Figure 4 and Appendix D.3. The task contains 512 key-value pairs with random tokens between relevant spans. SWA uses a window of 512 tokens; SWA + sink uses one initial sink token and a local window of 511. DMA uses the same threshold-calibration procedure for a comparable effective retained-key budget. All methods follow the same training and evaluation protocol.

The complete results are below. Entries are accuracies in percent.

Model dimensionSequence lengthSWASWA + sinkDMA
321K0.280.330.8
322K0.170.190.77
324K0.010.010.56
328K000.23
641K6.246.3818.29
642K1.241.3716.14
644K0.040.0412.34
648K000.41
1281K60.6761.8499.99
1282K16.4317.2399.99
1284K4.324.9372.32
1288K1.221.4553.21
2561K97.6497.9100
2562K32.1634.71100
2564K22.423.4299.99
2568K8.3211.1288.6

At model dimension 256, DMA reaches 99.99% at 4K and 88.60% at 8K, compared with 23.42% and 11.12% for SWA + sink. DMA achieves the highest accuracy at every tested dimension and sequence length. This experiment supports the value of adaptive selection for long-range associative retrieval with random interference.

The rank-1/L2 ablation in Appendix E.3 supplies a second comparison at closely matched realized sparsity: 62.3% for rank-1 and 62.1% for L2. Their block Recall@K values are 99.13% and 49.46%, and their full-model RULER-8K scores are 92.52 and 74.61. This links the learned selector's stronger block recall to improved retrieval quality at similar retained work.

AdaSplash-2 is a relevant trainable sparse-attention method. It produces differentiable sparsity through α\alpha-entmax and uses histogram initialization to accelerate the normalization threshold calculation. DMA screens blocks using a learned gate before full KV loading and QK computation, then applies gated softmax to the retained blocks. A matched from-scratch pretraining comparison with AdaSplash-2 requires training new models, which we cannot complete within the rebuttal period.

BLASST's downstream results appear in Table 3, with the complete LongBench and RULER subtasks in Appendix D.8, Tables 7 and 8. Its RULER-128K score is 65.92 and its six-metric average is 61.21; the corresponding DMA scores are 67.26 and 64.26.

W2: Autoregressive causality of block screening

DMA follows a causal execution order when screening attention blocks. For each query block, the kernel traverses causally accessible key blocks from the diagonal toward earlier positions. The running reference is initialized as φi=−∞\varphi_i=-\infty, ensuring that the diagonal block is always retained. Within each retained tile, standard token-level causal masking is applied before softmax, so each query aggregates information only from its current and preceding positions.

We further evaluate the numerical consistency of the screened attention operator in Appendix E.3, Table 12. Using the same checkpoint and the corresponding dense gated attention operator with screening disabled as the reference, DMA achieves maximum output errors ranging from 2×10−42\times10^{-4} to 1×10−31\times10^{-3} across sequence lengths of 1K–8K. These results show that block screening preserves close numerical agreement with dense gated attention under the evaluated settings.

AuthorsOpenReview ↗

Response (W3/Q2/Q3/Q4) to Reviewer zNZJ

Q2: The Qwen3-8B RULER-128K baseline

The backbone used in our experiment is Qwen3-8B-Base. The FullAttn row evaluates its checkpoint after dense continued training on 2B tokens at 32K context, followed by YaRN extension to approximately 131K tokens. Evaluation uses completion-format inputs with lm-eval-harness 0.4.13. The three decoding seeds are 0, 1, and 2, with temperature 0.6 and top-p 0.95.

Qwen3 technical report evaluates the post-trained Qwen3-8B model in non-thinking and thinking modes, with 128K scores of 77.4 and 72.0. These model and prompting configurations identify the distinction between the official evaluation and our Base-model adaptation experiment.

We evaluated the original Qwen3-8B-Base checkpoint with the same RULER configuration used for our adapted checkpoints:

CheckpointRULER-128K
Original Qwen3-8B-Base56.52
Qwen3-8B-Base after dense continued training58.18
Qwen3-8B-Base after DMA adaptation67.26

All three entries use the same evaluation configuration and average over three decoding seeds. Dense continued training improves the score by 1.66 points over the original Base checkpoint. DMA adds 9.08 points over the dense continued-training reference. This comparison establishes the effect of adaptation under our evaluation protocol.

Q2: What the computation-allocation analysis measures

Appendix E.2, Table 10 measures retained computation at layers 0, 11, 23, and 35 on retrieval tasks (MQ/MV) and QA tasks (QA1/QA2). Its G0G_0-G3G_3 columns average the query heads at the same relative position across the eight KV groups. At the panel labeled 1/N1/N, for example, layer 0, G0G_0 has 60.40% sparsity on retrieval and 73.81% on QA, corresponding to 39.60% and 26.19% retained computation. Retrieval retains more computation at every matched layer and GQA position in this sweep. This shows task-dependent allocation and variation with depth and head representations.

As shown in our response to W1, the SWA + sink MQAR control provides task-level evidence for the value of adaptive long-range selection under a comparable retained-key budget. The requested spatial statistic concerns the fraction of retained causal query-key pairs outside the union of the local window and initial sink positions, reported separately for each layer and individual head.

W3, Q3, Q4: Kernel framework, threshold settings, and retained budget

FullAttn uses the CUDA implementation of FlashAttention-2, and DMA is implemented in Triton 3.7.1. The module benchmarks run on an A100-SXM4-80GB GPU in bf16, as specified in Appendix D.2. The selected forward tile is 128×32128\times32 with four warps, and the backward tile is 32×12832\times128 with eight warps (Appendix E.5).

The main results use c=128c=128 in the Appendix E.2 parameterization:

τ=128N,θi=τiBN=128iBN2.\tau=\frac{128}{N},\qquad \theta_i=\tau\frac{iB}{N}=\frac{128iB}{N^2}.

The corresponding values of τ\tau are 0.015625 at 8K, 0.00390625 at 32K, and 0.0009765625 at 128K. These settings use a common coefficient and position-dependent schedule across lengths.

For DMA, ww denotes the effective retained-key budget induced by the threshold. It is observed from the retained computation. For methods with a fixed window or explicit selection budget, ww is a directly controlled setting.

W3: Figures, notation, and execution flow

The method proceeds from scalar query/key projections to block screening and then token-level attention on retained tiles. In Figures 1 and 2, AA and Δ\Delta denote the scalar query/key projections, φ~\tilde{\varphi} the candidate gate bound, φ\varphi the running reference, and θ\theta the screening margin.

The two panels of Figure 2 depict the screening and attention operations within the same fused tile loop. Scalar statistics determine whether a candidate tile is retained; retained KV tiles are then loaded and used for gated softmax attention. The arrows indicate data movement between HBM and SRAM and the query/key loop directions. Sections 2.1-2.4 and Appendix B give the corresponding equations and execution steps.

We appreciate for the reviewer's careful reading. If the paper is accepted, we will simplify the Introduction, refine Figures 1 and 2 with clearer labels for screening, KV loading, and forward/backward paths, and improve spacing around figures, tables, and captions. These presentation changes will make the method and its execution flow easier to follow.

Dear Reviewer zNZJ,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate any follow-up on our response, including the added SWA-with-sinks comparison, the original Qwen3-8B-Base evaluation, and the clarifications of causal screening, threshold settings, and kernel execution.

Please let us know if any concerns remain or would benefit from further clarification. Thank you for your time and consideration.

Reviewer_guv8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Mask Attention, a trainable block-sparse attention method. DMA projects each query and key to a scalar to screen KV tiles, and computes full token-level attention only within retained tiles. The same gate modulates retained attention logits so that it can receive language-modeling gradients. The screening is integrated into forward, backward, and decoding kernels. The paper evaluates both pretraining from scratch up to 1.7B parameters and sparse adaptation of Qwen3-8B.

Critical strengths

The paper is clearly written. Figures 1-4 are clearly presented, very helpful for understanding.

I appreciate that the paper implements sparse forward, backward, and decoding kernels rather than reporting only theoretical FLOP reductions.

The evaluation is broad. It includes associative recall, pretraining from scratch at several model sizes, standard and long-context downstream evaluation, sparse adaptation of an existing 8B model, and broad ablations.

Critical weaknesses
  1. The source of the quality gains over the related learned sparse-attention methods is unclear.

Though the paper compares with other learned sparse methods, like NSA/SeerAttention/HySparse, the experiments do not isolate which difference explains DMA's higher average accuracy. It is unclear whether the gain comes from better block recall, direct adaptation to the language-modeling loss, token-level reweighting within retained tiles, or others.

  1. The evidence for using DMA throughout large-scale pretraining is limited.

The largest model pretrained from scratch is 1.7B, while the 8B experiment is continued training. These experiments show that DMA can be trained and adapted at the tested scales, but do not show the sparse backward remains stable and beneficial in substantially larger pretraining runs.

  1. The baseline choices need clearer justification.

Different experiments in the paper use different baseline sets, which makes the results difficult to connect. In particular, HySparse is designed for from-scratch training, but it is omitted from the from-scratch experiments and evaluated only through attention adaptation, where its quality drops substantially. This setting may disadvantage HySparse.

  1. The full-model speed evidence is limited.

Figure 3 shows large attention-module speedups, but Table 3 shows the actual full-model throughput is limited. The full-model test also uses 8K input, while 128K is used only for quality evaluation.

Questions for authors
  1. Which specific difference from NSA and SeerAttention explains DMA's higher accuracy: selector recall, direct language-modeling supervision, logit modulation, or variable compute allocation?

  2. Can you summarize which baselines support from-scratch training, backward, prefill, decoding, and KV-cache reduction, and explain the baseline selection for each experiment?

  3. HySparse was primarily evaluated as a from-scratch architecture in its original paper. Is 2B-token attention-only adaptation sufficient for a fair comparison here?

  4. Why do the large attention-module speedups translate into only marginal full-model throughput?

5 comments
AuthorsOpenReview ↗

Response (W1/Q1) to Reviewer guv8

We thank the reviewer for the careful review and the time invested in evaluating our work. We appreciate the recognition of the clear presentation, the implemented forward, backward, and decoding kernels, and the breadth of the evaluation.

W1, Q1: What accounts for the quality gains?

DMA couples block selection to the language-modeling objective through the same gate used in retained attention. For a retained query-key pair, the score is

suv=qu⊤kv+log⁡σ((qu⊤wA)(kv⊤wΔ)).s_{uv}=q_u^\top k_v+\log\sigma\bigl((q_u^\top w_A)(k_v^\top w_\Delta)\bigr).

The log-sigmoid term provides a gradient path from the language-modeling loss to the gate parameters. Those parameters also determine the block-screening signal. Thus, logit modulation and direct language-modeling supervision are two parts of the same training mechanism. For comparison, NSA derives block importance from its compressed-attention branch, and SeerAttention supervises its gate by distilling pooled full-attention distributions.

The selector ablation in Appendix E.3 provides direct evidence for the value of the learned gate. At nearly identical realized sparsity, the rank-1 selector retains substantially more of the blocks selected by full-QK ranking and achieves higher retrieval accuracy:

SelectorRealized sparsityBlock Recall@KRULER-8K
L262.10%49.46%74.61 ± 0.72
Learned rank-162.30%99.13%92.52 ± 0.63

These results connect improved block selection with a 17.91-point increase in RULER-8K at comparable retained computation.

We also added SWA with attention sinks to the MQAR experiment. It retains one initial sink token and a local window of 511 tokens. SWA uses a window of 512, and DMA follows the threshold calibration for a comparable effective retained-key budget in Appendix D.3. At model dimension 256, the results are:

Sequence lengthSWASWA + sinkDMA
4K22.40%23.42%99.99%
8K8.32%11.12%88.60%

This comparison shows the benefit of adaptive selection for recovering distant associations amid random interference. Appendix E.2 adds evidence about computation allocation: retrieval retains more computation than QA at every sampled layer and relative GQA position. Together, these observations support the learned selector's retrieval accuracy and its ability to allocate computation according to the input.

AuthorsOpenReview ↗

Response (W2) to Reviewer guv8

W2: Training evidence and the sparse backward pass

Our from-scratch pretraining study spans approximately 80M, 200M, 680M, and 1.7B parameters. The 1.7B run uses 40,000 steps with 1M tokens per step, totaling 40B training tokens (Appendix D.4). After the initial 2,000-step dense warm-up (Appendix D.1), training proceeds with sparse execution. Figure 5 reports the resulting perplexity-compute trade-off across all four scales, and Tables 1 and 2 evaluate the trained 1.7B models on downstream tasks. These results provide model-level evidence of effective pretraining with DMA at scales up to 1.7B. The Qwen3-8B study further evaluates continued training of a larger backbone over 2B adaptation tokens.

To examine the approximate backward pass directly, we also compare DMA's implemented backward kernel with exact backward on the same forward-retained support. For each Qwen3-8B adaptation state, both computations use the same DMA forward output, retained tiles, and upstream gradient. This isolates the gradient difference introduced by reconstructing the backward support under a key-major traversal, which is the mechanism raised in the concern.

The table reports measurements at 1K-8K sequence lengths after 200M, 1B, and 2B adaptation tokens. Forward/backward IoU measures overlap between the supports selected by the two traversals. The error columns compare the implemented DMA backward against exact backward on the fixed forward support.

LengthAdaptation tokensForward/backward IoU (%)Max. dQ errorMax. dK errorMax. dV error
1K200M99.062.00E-031.00E-034.00E-04
1K1B99.862.00E-038.00E-045.00E-04
1K2B99.852.00E-039.00E-044.00E-04
2K200M98.681.00E-033.00E-042.00E-04
2K1B99.768.00E-042.00E-041.00E-04
2K2B99.49.00E-042.00E-041.00E-04
4K200M98.987.00E-041.00E-046.00E-05
4K1B98.878.00E-042.00E-046.00E-05
4K2B99.475.00E-041.00E-046.00E-05
8K200M99.357.00E-049.00E-052.00E-05
8K1B99.682.00E-047.00E-052.00E-05
8K2B99.774.00E-047.00E-052.00E-05

Across the 12 configurations, forward/backward IoU ranges from 98.68% to 99.86%. The largest reported errors relative to the fixed-support reference are 2×10−32\times10^{-3} for dQdQ, 10−310^{-3} for dKdK, and 5×10−45\times10^{-4} for dVdV. At 8K after 2B adaptation tokens, the respective errors are 4×10−44\times10^{-4}, 7×10−57\times10^{-5}, and 2×10−52\times10^{-5}. The support overlap and gradient measurements quantify the effect of DMA's backward reconstruction on the sampled large-model states. Together with the four-scale pretraining results, they provide complementary evidence on the model-level training behavior and numerical consistency of the sparse backward path.

AuthorsOpenReview ↗

Response (W3/Q2/Q3) to Reviewer guv8

W3, Q2, Q3: Baseline selection and the HySparse comparison

The baseline sets follow the question addressed by each experiment. The module, MQAR, and scaling studies compare FullAttn, SWA, MLA, NSA, and DMA under a shared training setup. The 1.7B downstream evaluation adds inference-time sparsification methods, including H2O, InfLLM, Quest, DAM, and Exact-Top, with SeerAttention and FlexPrefill also included in Table 2. The 8B suite examines conversion of an existing dense backbone and serving performance, using FullAttn, HySparse, BLASST, and DMA. FullAttn and DMA provide common references across these suites.

For the training-time and sparse-adaptation baselines, the following summary distinguishes training support, the accelerated execution stages, and the KV-memory mechanism. In the last column, capacity refers to stored KV states and traffic refers to the states read during attention.

MethodTraining and backwardPrefillDecodingKV-memory mechanism
FullAttnFrom-scratch training; exact backwardDenseDenseFull-sequence KV storage
SWAFrom-scratch training; windowed backwardWindowedWindowedWindow-bounded cache capacity
MLAFrom-scratch training; latent-attention backwardLatent attentionLatent attentionCompressed latent KV capacity
NSAFrom-scratch training; sparse backwardSparseSparseCompressed branch and selected raw-KV traffic
SeerAttention, original versionGate distillation and long-context fine-tuningSparseOriginal study targets prefillSelective block reads during prefill
HySparseArchitectural pretraining; attention-only adaptation hereHybrid full/sparseHybrid full/sparseCross-layer KV sharing and local cache capacity
BLASSTSparsity-aware fine-tuning with retained-support backwardSparseSparseReduced V traffic during decoding
DMAFrom-scratch training and adaptation; approximate sparse backwardSparseSparseReduced KV traffic through block screening

The training and inference descriptions follow the versions cited in the paper: NSA, SeerAttention, HySparse, BLASST, and MLA. Appendix D.2 specifies the additional retained-budget setting used for the MLA benchmark variant.

The additional 1.7B baselines evaluate inference-time sparsification on the trained checkpoints:

MethodLocationDocumented inference and memory mechanism
H2OTables 1 and 2Prompt-cache selection followed by dynamic eviction during decoding; retains heavy-hitter and recent tokens
InfLLMTables 1 and 2Memory retrieval during chunked input encoding and decoding; bounded GPU cache with CPU offloading
QuestTables 1 and 2Query-aware KV-page selection during decoding; full KV storage with selective reads
DAMTables 1 and 2Adaptive attention-position masking; decoding latency evaluated in the original study
Exact-TopTables 1 and 2Inference-time selection reference
FlexPrefillTable 2Input-dependent sparse prefill

HySparse is designed for architectural pretraining. Its comparison in Table 3 addresses a specific deployment question: the quality and throughput reached when converting the same Qwen3-8B backbone under a common 2B-token adaptation budget. The recipe uses 32K sequences, a learning rate of 2×10−42\times10^{-4}, attention-only updates, and a 1:3 full-to-sparse layer ratio. Under this recipe, HySparse reaches 0.56K decode tokens/s and DMA reaches 0.54K, while their average quality scores are 48.60 and 64.26. This comparison captures HySparse's throughput advantage and DMA's stronger quality after the specified adaptation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: DMA’s Contributions, Additional Evidence, and Review Discussion

Dear AC, SAC, and PC,

We thank the reviewers for their thoughtful evaluation. We summarize our contribution, the reviewers’ feedback, and the evidence provided during rebuttal.

TL;DR of Our Work

DMA investigates how learned attention selection can guide efficient execution before expensive KV loading and token-level QK computation. A shared rank-1 gate screens candidate blocks and modulates retained attention logits, providing a gradient path from the language-modeling objective to the screening parameters.

Fused forward, backward, and decoding implementations turn this mechanism into a trainable sparse-attention method. The evaluation connects selector quality and adaptive computation with pretraining at four scales from approximately 80M to 1.7B parameters, Qwen3-8B adaptation, and complete serving workloads.

Reviewers’ Recognition

guv8 recognizes the implemented forward, backward, and decoding kernels and the breadth of evaluation. zNZJ highlights screening before expensive KV movement and the shared learned signal connecting selection with trainability. tmGJ recognizes that in-kernel screening directly addresses selection overhead and appreciates the training, serving, and implementation ablations.

Reviewers’ Concerns and Our Responses

  • guv8 and zNZJ ask what supports the value of learned selection. We connected the selector ablation to retrieval quality: at closely matched sparsity, rank-1 selection achieves 99.13% block Recall@K and 92.52 RULER-8K, versus 49.46% and 74.61 for L2 selection. We also added SWA with sinks under the same associative-recall protocol. At dimension 256 and 8K context, DMA achieves 88.60%, versus 11.12% for SWA with sinks.

  • guv8 and tmGJ question the sparse backward approximation. We added the requested exact-backward reference on the same forward-retained support, fixing the forward computation and upstream gradient. Tests cover 1K–8K contexts at three adaptation stages. Support IoU is 98.68–99.86%, with reported maximum dQ/dK/dV errors of 2×10⁻³/10⁻³/5×10⁻⁴. This directly measures the additional error from backward support reconstruction.

  • zNZJ questions the Qwen3 baseline, and guv8 requests clearer baseline comparability. We evaluated the original Qwen3-8B-Base checkpoint under the same RULER-128K protocol: scores are 56.52 before adaptation, 58.18 after dense continued training, and 67.26 after DMA adaptation. We clarified the distinction from the official post-trained Qwen3 evaluation and documented baseline coverage. The HySparse comparison measures adaptation of the same backbone under a common 2B-token budget.

  • guv8 and tmGJ request stronger complete-system efficiency evidence. We clarified that timing includes candidate-block scanning and added complete-request measurements on eight A100-80GB GPUs with eight concurrent requests. DMA achieves 2.86× speedup for 128K-input/128-output requests and 1.61× for 128-input/128K-output requests. Runtime breakdowns show how attention’s share of execution determines the resulting system gains.

  • tmGJ asks about the gate bound and long-context reliability; zNZJ requests execution and presentation clarifications. We distinguished the gate-bound guarantee, measured operator fidelity at 1K–8K, and complete-model retrieval evaluation at 128K. We also explained traversal, within-tile masking, threshold settings, kernel frameworks, and retained-budget notation.

Discussion Status and Incorporation of Additional Evidence

Current ratings are 3/3/2. tmGJ maintained the initial score, citing the extent of revisions. Our rebuttal supplies the requested fixed-support backward control and long-context serving measurements, alongside clarifications of the bound and complete execution cost.

These additions strengthen the existing method’s empirical foundation. They can be incorporated through two additional columns in a main-text table, one appendix table, and focused textual clarifications, while retaining the submitted method and training procedure.

The combined evidence connects inexpensive learned screening, retrieval quality, measured backward fidelity, and practical serving gains. We thank the reviewers for helping strengthen this evidence and the chairs for their thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.