- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MassAlloc Attention, which computes all legal causal QK scores but skips low-contribution post-score operations. Forward allocation uses the running online-softmax normalizer, while backward allocation uses the final normalizer saved during forward. The method requires no stored selection mask and supports training, prefill, and decoding with a shared tolerance. Experiments report low output and gradient errors relative to FullAttn, lower attention latency and training FLOPs, and comparable model-level performance on the evaluated tasks.
Critical strengths
-
A useful design point between dense and support-sparse attention. MALA preserves exact access to every causal QK score and only sparsifies the computation that follows. This avoids false negatives from a separate sparse selector while still reducing softmax, value-loading, PV, and backward work. The paired forward and backward formulation, including nested backward support without storing a selection mask, is the clearest technical contribution.
-
Strong empirical coverage. The evaluation goes beyond a kernel benchmark. It includes matched-work allocation, output and gradient errors, associative recall, operator latency, scaling-law training, and downstream model evaluation. In particular, the matched-work study helps separate the benefit of adaptive allocation from simply performing more computation. Across these evaluations, MALA remains close to FullAttn while reducing post-score work.
Critical weaknesses
-
The evidence does not isolate the value of full QK discovery from the value of adaptive allocation. The matched-work study provides convincing evidence that per-instance mass allocation is more effective than static allocation at the same post-score budget. However, it does not establish that computing every exact QK score is necessary to obtain this benefit. The sparse baselines in the associative-recall experiment differ from MALA not only in score coverage, but also in their allocation mechanisms, architectures, and training configurations. Moreover, the comparison is based partly on post-score work, whereas support-sparse methods can avoid QK computation and KV access at the cost of selection or routing overhead. The results therefore establish MALA as an effective operating point between FullAttn and support-sparse attention, but they do not demonstrate that full score discovery offers a generally superior quality-efficiency trade-off to adaptive support selection.
-
The backward rule uses attention mass as a proxy for gradient importance, but the implications of this approximation are not fully characterized. Backward can discard tiles that contributed to the forward output, so it is not the exact derivative of the approximate forward operator. Further, a tile’s gradient contribution depends on the upstream gradient, value vectors, and softmax coupling, not only on its attention probability. The paper acknowledges this distinction, and the reported gradient errors and end-to-end training results provide meaningful empirical support. However, these results do not determine whether the approximation behaves primarily as small perturbation noise or introduces systematic gradient bias. This does not invalidate the method, but it limits the conclusions that can be drawn from probability-based screening alone and deserves a more precise discussion.
Questions for authors
-
Efficiency regimes in which MALA should be preferred. MALA retains the full QK cost while reducing post-score work. In contrast, support-sparse approaches may avoid QK computation and KV access but incur selection, routing, or importance-estimation overhead. Could the authors clarify how these components affect the trade-off across training, prefill, and decoding? In particular, under what sequence lengths, head dimensions, or hardware regimes does post-score work dominate enough for MALA to be preferable, and when is full QK discovery expected to become the limiting cost?
-
Interpretation of the approximate backward pass. Backward may discard tiles that contributed to the forward output, so the resulting update is not the exact gradient of the approximate forward operator. How should this update be interpreted: as a small numerical approximation to that gradient, as a biased gradient-truncation rule, or as an implicit modification of the training procedure? Clarifying this point would help distinguish what follows from the method’s construction from what is supported only empirically.
-
Evidence concerning gradient bias. The reported relative errors for , , and are reassuring, but aggregate error does not indicate whether the approximation is directionally unbiased or whether errors are concentrated in particular layers, heads, positions, or training stages. Do the authors have evidence that backward screening does not systematically remove gradient contributions associated with low-probability but learning-relevant interactions? A qualitative discussion based on existing measurements would already help clarify this issue.
-
Relationship between the theoretical bound and empirical fidelity. With the canonical choice , the forward omitted-mass bound is substantially looser than the omitted mass observed in the experiments. How should this guarantee be interpreted in practice? Is the strong empirical fidelity mainly a property of the attention distributions produced by the evaluated models, and if so, what evidence suggests that this behavior remains stable across model scales, training stages, or distribution shifts?
4 comments
Response (W2/Q1/Q2) to Reviewer hqPE
Q1. Efficiency regimes and complete execution cost
The additional end-to-end measurements show larger training gains at long context, with serving gains at concurrency 16. We integrate MALA by replacing the attention kernel in Megatron-LM and vLLM.
Training
We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8, PP=1, and DP=16. The microbatch size is eight at 4K and one at 32K, with batch settings matched across methods at each scale and context length. FullAttn and MALA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.
| Model | Method | 4K Step (ms) | 4K Tokens/s | 32K Step (ms) | 32K Tokens/s |
|---|---|---|---|---|---|
| 1.7B | FullAttn | 212 | 2,478,176 | 288 | 1,818,080 |
| 1.7B | MoBA | 297 | 1,769,162 | 291.1 | 1,798,599 |
| 1.7B | DSA | 206.6 | 2,543,132 | 222.1 | 2,357,358 |
| 1.7B | MALA | 195.9 | 2,682,058 | 208.8 | 2,507,841 |
| 4B | FullAttn | 775 | 1,353,216 | 1,217 | 861,568 |
| 4B | MoBA | 1,080.6 | 970,555 | 1,230.1 | 852,412 |
| 4B | DSA | 755.5 | 1,388,096 | 940.9 | 1,114,375 |
| 4B | MALA | 717 | 1,462,597 | 885.1 | 1,184,697 |
| 8B | FullAttn | 1,711 | 919,136 | 2,363 | 665,728 |
| 8B | MoBA | 2,106.4 | 746,614 | 2,381.6 | 660,523 |
| 8B | DSA | 1,685.8 | 932,874 | 1,969.7 | 798,669 |
| 8B | MALA | 1,636 | 961,268 | 1,890.1 | 832,291 |
| 14B | FullAttn | 3,635 | 576,960 | 4,927 | 425,600 |
| 14B | MoBA | 4,294 | 488,456 | 4,960 | 422,760 |
| 14B | DSA | 3,593 | 583,666 | 4,230 | 495,725 |
| 14B | MALA | 3,510 | 597,459 | 4,089 | 512,827 |
MALA has the highest throughput among the four measured configurations at each scale and both lengths. Its throughput gain over FullAttn is 1.04–1.08× at 4K and 1.20–1.38× at 32K. At 14B, the 32K step decreases from 4,927 to 4,089 ms, giving a 20.5% throughput increase; the 4K throughput increase is 3.6%.
Serving
We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.
| Model | Method | Prefill Input Tokens/s | Decode Output Tokens/s |
|---|---|---|---|
| 14B | FullAttn | 46,746 | 1,542 |
| 14B | MoBA | 37,468 | 541 |
| 14B | DSA | 74,209 | 1,754 |
| 14B | MALA | 76,086 | 1,946 |
| 32B | FullAttn | 21,520 | 891 |
| 32B | MALA | 36,075 | 1,140 |
The prefill/decode gains over FullAttn are 1.63×/1.26× at 14B and 1.68×/1.28× at 32B. These results extend the original batch-one operator measurements to complete serving workloads with concurrency 16.
The trade-off follows the division of execution costs. In forward and prefill, MALA pays for complete QK discovery and avoids the softmax/value/PV path of skipped tiles. In backward, it recomputes QK, then screens before probability reconstruction and the corresponding gradient work. In decoding, it retains QK discovery and the complete KV cache while reducing low-contribution post-score execution. Support-sparse methods can reduce QK and key access, with their own selection, routing, and data-movement costs. The full-path measurements include these costs for the evaluated implementations.
The 14B training throughput increase is 3.6% at 4K and 20.5% at 32K. As retained post-score work decreases, full QK discovery becomes a larger fraction of the remaining attention cost. Head dimensions, attention concentration, and hardware balance affect this trade-off; the reported H100 measurements establish the gains for the tested configurations. Persistent KV-cache capacity remains unchanged.
W2 and Q2. Interpretation of the backward approximation
MALA backward is a deterministic gradient-screening approximation. Forward makes decisions using the evolving online normalizer; backward reuses the finalized forward normalizer. Under the same tiling and exact arithmetic, the latter gives a stricter test, so backward retained support is nested within forward support without saving a selection mask.
Backward evaluates the standard attention-gradient expressions on retained tiles and omits screened contributions, including some from tiles retained in forward. This is an additional approximation to the derivative with forward support held fixed. The screening is deterministic, and unbiasedness is not guaranteed. Local gradient comparisons measure its numerical effect; complete training experiments assess its optimization behavior.
Response (Q3/Q4) to Reviewer hqPE
Q3. Evidence from loss-derived gradients
We supplement the random-probe measurements with gradients generated by the cross-entropy training loss. The added experiment follows Appendix C.3, replacing only the upstream-gradient source. The table gives mean / P95 relative error (%) for each gradient tensor.
| Context | dQ Mean / P95 (%) | dK Mean / P95 (%) | dV Mean / P95 (%) |
|---|---|---|---|
| 1K | 0.3170 / 0.6056 | 0.1756 / 0.5059 | 0.0306 / 0.1603 |
| 2K | 0.2206 / 0.3518 | 0.1736 / 0.4838 | 0.0382 / 0.1757 |
| 4K | 0.2058 / 0.2894 | 0.1521 / 0.3020 | 0.0397 / 0.1694 |
| 8K | 0.1837 / 0.1991 | 0.1308 / 0.2201 | 0.0413 / 0.1614 |
| 16K | 0.1490 / 0.1733 | 0.1172 / 0.1258 | 0.0489 / 0.1522 |
| 32K | 0.1011 / 0.1613 | 0.1079 / 0.1186 | 0.0478 / 0.1539 |
Mean and P95 errors are lower than under the original Rademacher probe at every measured length. The largest mean errors for dQ/dK/dV are 0.3170%/0.1756%/0.0489%, and the largest P95 errors are 0.6056%/0.5059%/0.1757%. Training from 0.6B to 14B and continued training at 32B additionally support the approximation's usefulness during optimization.
Gradient contributions depend on , , and softmax coupling. These relative-error statistics characterize perturbation size and its measured tail; directional bias remains unmeasured. The evidence supports low gradient error under the sampled training loss and comparable model quality in the evaluated training settings.
Q4. Worst-case guarantee, empirical fidelity, and training adaptation
The mass bound supplies a worst-case guarantee; the realized attention distributions determine empirical fidelity. At , gives the stated 50% bound. The proof sums a threshold-based upper bound over the legal context. Actual skipped contributions can be much smaller, and tile-level screening retains an entire tile whenever any valid entry fails the skip test.
Across the original 1K–32K operator measurements at , mean omitted mass is at most . We additionally test on both MALA-trained and FullAttn-trained 14B checkpoints. The latter keeps its original dense-trained weights, with no MALA adaptation. At 32K:
| Checkpoint | τ | Forward / Backward Slots/Query | Mean Output Error (%) |
|---|---|---|---|
| MALA-trained | 0.5 | 1,178 / 1,164 | 0.0047 |
| MALA-trained | 1 | 1,066 / 1,058 | 0.0120 |
| MALA-trained | 2 | 1,015 / 998 | 0.0097 |
| MALA-trained | 4 | 994 / 982 | 0.0143 |
| FullAttn-trained | 0.5 | 1,192 / 1,141 | 0.0054 |
| FullAttn-trained | 1 | 1,072 / 1,043 | 0.0079 |
| FullAttn-trained | 2 | 1,024 / 1,001 | 0.0103 |
| FullAttn-trained | 4 | 1,002 / 978 | 0.0127 |
The two checkpoints show similar work and low local output errors across tolerances, including direct substitution on the FullAttn-trained model. These diagnostics test sensitivity to context length, tolerance, and training adaptation. The 0.6B–14B training and 32B continued-training results supply model-level evidence across scales. Fidelity under further distribution shifts remains dependent on the attention distributions encountered.
Dear Reviewer hqPE,
Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate any follow-up on our clarifications of full score discovery and backward approximation, together with the additional training-gradient measurements, FullAttn-trained checkpoint tests, and end-to-end efficiency results.
Please let us know if any concerns remain or would benefit from further clarification. Thank you for your time and consideration.
Response (W1) to Reviewer hqPE
We thank the reviewer for the careful evaluation and the precise questions about attribution and backward semantics. We appreciate your recognition of MALA as a useful design point, the paired forward and backward formulation, and the matched-work evidence for adaptive allocation.
W1. Full score accessibility and the role of the matched-work study
MALA's central design is to preserve exact causal score access and let the realized normalized contributions allocate post-score computation. Every legal causal interaction is scored before its post-score path can be skipped. Consequently, a distant interaction or one that is difficult to predict from a proxy signal can still be retained when its actual contribution meets the threshold. This accessibility comes with the full QK cost.
Table 1 holds post-score work fixed and varies allocation scope, then compares MALA's online decisions with a per-instance reference-mass oracle. Full QK discovery is shared by these controls. The experiment therefore isolates the value of input-dependent allocation and tests online decision fidelity; the independent benefit of exact versus approximate discovery is outside this comparison.
The complete-method comparisons evaluate the resulting quality–cost trade-off. MoBA and DSA retain their native parameterizations and selection mechanisms; FullAttn and MALA share the model architecture. Operator and end-to-end measurements include score computation, selection, and subsequent execution. Our claim is that full score access combined with adaptive post-score allocation provides an effective, trainable operating point, supported by model quality and measured execution gains.