- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DMA uses a rank-1 gate to screen attention blocks inside fused kernels and modulate retained logits. Forward and backward passes reconstruct their supports separately, introducing approximate gradients. Experiments cover training, long-context adaptation, and serving, reporting comparable or improved quality and approximately 1.2x prefill and decoding throughput under the tested workload.
Critical strengths
1.In-kernel screening explicitly addresses selection overhead, making the method more practically motivated than approaches evaluated only through sparsity ratios.
2.The evaluation covers training and inference, with measured serving throughput and useful ablations of gate rank, thresholds, and tile geometry.
Critical weaknesses
1.Appendix E.4 compares gradients with dense gated attention, not exact differentiation of the same forward-retained mask. Please add that control to isolate backward support mismatch and measure its effect on gradients and convergence.
2.The screening bound covers the gate, not the complete attention score. First-layer, shorter-context analyses do not establish reliability at 128K. Please measure omitted attention mass and output errors across layers and long inputs.
3.Efficiency accounting should include scanning all candidate blocks. The serving workload also uses only 8K input, unlike the 128K quality evaluation. Please provide a screening-cost breakdown and longer-input serving measurements.
Questions for authors
See weakness.
5 comments
Response (W2) to Reviewer tmGJ
W2: Gate bounds and long-context accuracy
DMA screens candidate blocks using a bound on the learned gate, while the quality of sparse attention depends on the information retained after screening. The paper examines this connection in stages: Section 2.2 establishes the gate-side bound, Appendix C relates omitted attention mass to output error, Appendix E measures selector fidelity and computation allocation, and Table 3 with Appendix D.8 evaluates the full model at 128K. These results link the screening mechanism to its numerical behavior and long-context retrieval performance.
Section 2.2, Eqs. (3)-(4), gives an exact tilewise bound on the scalar gate and its log-sigmoid bias. Appendix A motivates the rank-1 gate when block-averaged query/key representations have a dominant mode. Appendix C connects sparse execution to omitted attention mass: Eq. (55) bounds the difference between the dense gated output and the renormalized retained-support output by
where is the total probability mass assigned to skipped positions by the dense gated reference, and the value-vector norms in the row are bounded by . This analysis identifies the quantity that governs the output approximation.
Appendix E.3 provides the corresponding selection and output measurements. In Table 11, the learned rank-1 selector achieves 99.13% block Recall@K at 62.3% sparsity, compared with 49.46% recall at 62.1% sparsity for L2 scoring. Their full-model RULER-8K scores are 92.52 and 74.61. This comparison supports the learned gate's ability to retain useful blocks at comparable computation. Table 12 reports recall above 99.1% at 1K, 2K, 4K, and 8K, with maximum output-error values of , , , and , respectively, against the same checkpoint's screening-disabled gated operator. These first-layer measurements quantify selection fidelity and output accuracy across the measured lengths.
The cross-layer analysis in Appendix E.2, Table 10, covers layers 0, 11, 23, and 35 at 8K context. Its statistics average query heads at the same relative position across the eight KV groups. Retrieval retains more computation than QA at every matched layer and relative GQA position, with allocation also varying across depth. These results show that the learned screening responds to the task and to representations at different layers.
At 128K, Table 3 directly evaluates the complete model: DMA reaches 67.26 on RULER, compared with 58.18 for FullAttn, a gain of 9.08 points. Appendix D.8, Table 8, reports all 13 subtasks under the same 128K evaluation protocol. These results establish the model's long-context retrieval performance, complementing the operator measurements and cross-layer allocation analysis.
We hope that the cited results address the concern at complementary levels. Tables 11 and 12 show high selector recall and small measured output error on first-layer states from 1K to 8K; Table 10 shows how retained computation varies across sampled layers and tasks; and Table 3 with Table 8 measures long-range retrieval in the complete 128K model. The mass-based bound in Appendix C explains why retaining useful attention matters for output accuracy. Together, the numerical checks and the 128K task results support the effectiveness of DMA's gate-based screening in the evaluated settings.
Response (W3) to Reviewer tmGJ
W3: Candidate-block scanning and longer serving workloads
The runtime measurements include the candidate-block scan. The screening path computes the scalar query/key projections, obtains block extrema, and evaluates the four-corner gate test for each causal candidate tile. Full KV loading and token-level attention follow for retained tiles. This scan is part of the fused execution being timed.
The candidate scan can also be stated explicitly. With query and key tile sizes and , prefill visits candidate tile pairs. After block extrema are computed and made available, each candidate requires four scalar products, a maximum, and a threshold comparison. The scalar projections cost per head, and retained token-level attention costs for effective retained budget . With block extrema precomputed and reused, the arithmetic count for these operations is
per head. The measured runtime includes metadata access and extrema handling in the implemented kernel. The Figure 3 attention-module timing also includes QKV projections; Appendix E.3 reports kernel timings on exported Q/K/V states.
We added long-input and long-output measurements using eight A100-80GB GPUs, data parallelism of eight, and eight concurrent requests. These are complete-request batch timings:
| Input / output tokens per request | Method | Total elapsed (s) | Attention elapsed (s) | Attention share | Request speedup |
|---|---|---|---|---|---|
| 128K / 128 | FullAttn | 61.46 | 41.52 | 67.60% | 1.00× |
| 128K / 128 | DMA | 21.52 | 1.32 | 6.10% | 2.86× |
| 128 / 128K | FullAttn | 3,132.18 | 1,247.82 | 39.80% | 1.00× |
| 128 / 128K | DMA | 1,944.12 | 48.75 | 2.50% | 1.61× |
For 128K input, attention accounts for 67.6% of FullAttn runtime, and DMA reduces total elapsed time from 61.46 s to 21.52 s. This provides a end-to-end gain in a workload with a long prompt. For long generation, the cache grows over the 128K-token output, and DMA reduces total elapsed time from 3,132.18 s to 1,944.12 s, giving over the complete run. The attention and total-runtime columns show how attention savings contribute to the observed serving improvement.
Response (W1) to Reviewer tmGJ
We thank the reviewer for the careful review and the effort spent examining the method and experiments. We appreciate the recognition of the practical motivation for in-kernel screening, the training and serving evaluation, and the ablations on gate rank, thresholds, and tile geometry.
W1: Exact backward on the forward-retained support
We added the requested control by fixing the DMA forward computation, its retained support, and the upstream gradient. The reference computes exact gradients for that fixed forward-retained support. Comparing the implemented DMA backward with this reference isolates the additional gradient error introduced by reconstructing the backward support.
The table reports checkpoints after 200M, 1B, and 2B adaptation tokens. IoU measures the overlap between the original DMA forward and backward supports. The gradient-error columns use the new fixed-forward-support exact-backward reference.
Across these settings, the largest reported errors are 2×10−3 for dQ, 10−3 for dK, and 5×10−4 for dV. At 8K, the final checkpoint has errors of 4×10−4, 7×10−5, and 2×10−5, respectively. This directly quantifies the gradient difference associated with the independently reconstructed backward support.
The model-level training evidence comes from the four-scale pretraining study in Figure 5 and the 8B adaptation results in Table 3. The new control adds a local gradient comparison at three adaptation stages.