acceptodds
Under review as a conference paper at ICLR 2027

MassAlloc Attention: Let Attention Allocate Its Own Compute

Abstract

Long-context full softmax attention (FullAttn) often assigns negligible normalized mass to much of the causal score space, yet dense kernels execute the complete post-score path after forming each QK tile. We introduce MassAlloc Attention (MALA), a fused attention primitive that preserves score access to every legal causal interaction and uses normalized contribution to allocate post-score computation. Forward uses its evolving online-softmax normalizer, while backward reuses the finalized normalizer to derive nested retained support using only standard attention state. A common tolerance governs training and inference, allowing the retained work to adapt across queries, heads, layers, and inputs. MALA reduces low-contribution post-score computation while retaining quadratic QK score discovery. A matched-work study at 8K isolates the benefit of distribution-adaptive allocation: under exactly matched total post-score work, MALA approaches a per-instance reference-mass oracle, with mean omitted mass of 0.0188% versus 0.0182%, while static allocations perform substantially worse. Across context lengths from 1K to 32K tokens, the same tolerance maintains low output and gradient errors relative to the reference operator. Across a broader controlled associative-recall comparison, MALA closely tracks FullAttn as context grows, reaching 89.67% accuracy at 8K compared with 89.97% for FullAttn. In an attention-operator benchmark at 128K tokens on 8 H100 GPUs with tensor parallelism, MALA reduces forward and backward latency during training by and and decoding latency during inference by relative to FullAttn, while retaining FullAttn-level per-rank peak operator memory. Across scaling-law training from 0.6B to 14B parameters on 128 H100 GPUs, MALA closely tracks FullAttn in perplexity while reducing total training FLOPs, with a 23.1% reduction at 14B during 32K-context training. The resulting 14B models and 32B models from separate continued training achieve comparable knowledge, reasoning, and long-context retrieval scores to FullAttn. These results indicate that allocating post-score computation according to normalized attention contributions can retain the evaluated capabilities of FullAttn while reducing attention computation.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Long-Context Transformers, Adaptive Sparse Attention, Online Softmax, Efficient Attention, Systems for Large Language Models, Training and Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hqPEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MassAlloc Attention, which computes all legal causal QK scores but skips low-contribution post-score operations. Forward allocation uses the running online-softmax normalizer, while backward allocation uses the final normalizer saved during forward. The method requires no stored selection mask and supports training, prefill, and decoding with a shared tolerance. Experiments report low output and gradient errors relative to FullAttn, lower attention latency and training FLOPs, and comparable model-level performance on the evaluated tasks.

Critical strengths
  • A useful design point between dense and support-sparse attention. MALA preserves exact access to every causal QK score and only sparsifies the computation that follows. This avoids false negatives from a separate sparse selector while still reducing softmax, value-loading, PV, and backward work. The paired forward and backward formulation, including nested backward support without storing a selection mask, is the clearest technical contribution.

  • Strong empirical coverage. The evaluation goes beyond a kernel benchmark. It includes matched-work allocation, output and gradient errors, associative recall, operator latency, scaling-law training, and downstream model evaluation. In particular, the matched-work study helps separate the benefit of adaptive allocation from simply performing more computation. Across these evaluations, MALA remains close to FullAttn while reducing post-score work.

Critical weaknesses
  • The evidence does not isolate the value of full QK discovery from the value of adaptive allocation. The matched-work study provides convincing evidence that per-instance mass allocation is more effective than static allocation at the same post-score budget. However, it does not establish that computing every exact QK score is necessary to obtain this benefit. The sparse baselines in the associative-recall experiment differ from MALA not only in score coverage, but also in their allocation mechanisms, architectures, and training configurations. Moreover, the comparison is based partly on post-score work, whereas support-sparse methods can avoid QK computation and KV access at the cost of selection or routing overhead. The results therefore establish MALA as an effective operating point between FullAttn and support-sparse attention, but they do not demonstrate that full score discovery offers a generally superior quality-efficiency trade-off to adaptive support selection.

  • The backward rule uses attention mass as a proxy for gradient importance, but the implications of this approximation are not fully characterized. Backward can discard tiles that contributed to the forward output, so it is not the exact derivative of the approximate forward operator. Further, a tile’s gradient contribution depends on the upstream gradient, value vectors, and softmax coupling, not only on its attention probability. The paper acknowledges this distinction, and the reported gradient errors and end-to-end training results provide meaningful empirical support. However, these results do not determine whether the approximation behaves primarily as small perturbation noise or introduces systematic gradient bias. This does not invalidate the method, but it limits the conclusions that can be drawn from probability-based screening alone and deserves a more precise discussion.

Questions for authors
  • Efficiency regimes in which MALA should be preferred. MALA retains the full QK cost while reducing post-score work. In contrast, support-sparse approaches may avoid QK computation and KV access but incur selection, routing, or importance-estimation overhead. Could the authors clarify how these components affect the trade-off across training, prefill, and decoding? In particular, under what sequence lengths, head dimensions, or hardware regimes does post-score work dominate enough for MALA to be preferable, and when is full QK discovery expected to become the limiting cost?

  • Interpretation of the approximate backward pass. Backward may discard tiles that contributed to the forward output, so the resulting update is not the exact gradient of the approximate forward operator. How should this update be interpreted: as a small numerical approximation to that gradient, as a biased gradient-truncation rule, or as an implicit modification of the training procedure? Clarifying this point would help distinguish what follows from the method’s construction from what is supported only empirically.

  • Evidence concerning gradient bias. The reported relative errors for dQdQ, dKdK, and dVdV are reassuring, but aggregate L2L_2​ error does not indicate whether the approximation is directionally unbiased or whether errors are concentrated in particular layers, heads, positions, or training stages. Do the authors have evidence that backward screening does not systematically remove gradient contributions associated with low-probability but learning-relevant interactions? A qualitative discussion based on existing measurements would already help clarify this issue.

  • Relationship between the theoretical bound and empirical fidelity. With the canonical choice τ=1\tau=1, the forward omitted-mass bound is substantially looser than the omitted mass observed in the experiments. How should this guarantee be interpreted in practice? Is the strong empirical fidelity mainly a property of the attention distributions produced by the evaluated models, and if so, what evidence suggests that this behavior remains stable across model scales, training stages, or distribution shifts?

4 comments
AuthorsOpenReview ↗

Response (W1) to Reviewer hqPE

We thank the reviewer for the careful evaluation and the precise questions about attribution and backward semantics. We appreciate your recognition of MALA as a useful design point, the paired forward and backward formulation, and the matched-work evidence for adaptive allocation.

W1. Full score accessibility and the role of the matched-work study

MALA's central design is to preserve exact causal score access and let the realized normalized contributions allocate post-score computation. Every legal causal interaction is scored before its post-score path can be skipped. Consequently, a distant interaction or one that is difficult to predict from a proxy signal can still be retained when its actual contribution meets the threshold. This accessibility comes with the full QK cost.

Table 1 holds post-score work fixed and varies allocation scope, then compares MALA's online decisions with a per-instance reference-mass oracle. Full QK discovery is shared by these controls. The experiment therefore isolates the value of input-dependent allocation and tests online decision fidelity; the independent benefit of exact versus approximate discovery is outside this comparison.

The complete-method comparisons evaluate the resulting quality–cost trade-off. MoBA and DSA retain their native parameterizations and selection mechanisms; FullAttn and MALA share the model architecture. Operator and end-to-end measurements include score computation, selection, and subsequent execution. Our claim is that full score access combined with adaptive post-score allocation provides an effective, trainable operating point, supported by model quality and measured execution gains.

AuthorsOpenReview ↗

Response (W2/Q1/Q2) to Reviewer hqPE

Q1. Efficiency regimes and complete execution cost

The additional end-to-end measurements show larger training gains at long context, with serving gains at concurrency 16. We integrate MALA by replacing the attention kernel in Megatron-LM and vLLM.

Training

We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8, PP=1, and DP=16. The microbatch size is eight at 4K and one at 32K, with batch settings matched across methods at each scale and context length. FullAttn and MALA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.

ModelMethod4K Step (ms)4K Tokens/s32K Step (ms)32K Tokens/s
1.7BFullAttn2122,478,1762881,818,080
1.7BMoBA2971,769,162291.11,798,599
1.7BDSA206.62,543,132222.12,357,358
1.7BMALA195.92,682,058208.82,507,841
4BFullAttn7751,353,2161,217861,568
4BMoBA1,080.6970,5551,230.1852,412
4BDSA755.51,388,096940.91,114,375
4BMALA7171,462,597885.11,184,697
8BFullAttn1,711919,1362,363665,728
8BMoBA2,106.4746,6142,381.6660,523
8BDSA1,685.8932,8741,969.7798,669
8BMALA1,636961,2681,890.1832,291
14BFullAttn3,635576,9604,927425,600
14BMoBA4,294488,4564,960422,760
14BDSA3,593583,6664,230495,725
14BMALA3,510597,4594,089512,827

MALA has the highest throughput among the four measured configurations at each scale and both lengths. Its throughput gain over FullAttn is 1.04–1.08× at 4K and 1.20–1.38× at 32K. At 14B, the 32K step decreases from 4,927 to 4,089 ms, giving a 20.5% throughput increase; the 4K throughput increase is 3.6%.

Serving

We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.

ModelMethodPrefill Input Tokens/sDecode Output Tokens/s
14BFullAttn46,7461,542
14BMoBA37,468541
14BDSA74,2091,754
14BMALA76,0861,946
32BFullAttn21,520891
32BMALA36,0751,140

The prefill/decode gains over FullAttn are 1.63×/1.26× at 14B and 1.68×/1.28× at 32B. These results extend the original batch-one operator measurements to complete serving workloads with concurrency 16.

The trade-off follows the division of execution costs. In forward and prefill, MALA pays for complete QK discovery and avoids the softmax/value/PV path of skipped tiles. In backward, it recomputes QK, then screens before probability reconstruction and the corresponding gradient work. In decoding, it retains QK discovery and the complete KV cache while reducing low-contribution post-score execution. Support-sparse methods can reduce QK and key access, with their own selection, routing, and data-movement costs. The full-path measurements include these costs for the evaluated implementations.

The 14B training throughput increase is 3.6% at 4K and 20.5% at 32K. As retained post-score work decreases, full QK discovery becomes a larger fraction of the remaining attention cost. Head dimensions, attention concentration, and hardware balance affect this trade-off; the reported H100 measurements establish the gains for the tested configurations. Persistent KV-cache capacity remains unchanged.

W2 and Q2. Interpretation of the backward approximation

MALA backward is a deterministic gradient-screening approximation. Forward makes decisions using the evolving online normalizer; backward reuses the finalized forward normalizer. Under the same tiling and exact arithmetic, the latter gives a stricter test, so backward retained support is nested within forward support without saving a selection mask.

Backward evaluates the standard attention-gradient expressions on retained tiles and omits screened contributions, including some from tiles retained in forward. This is an additional approximation to the derivative with forward support held fixed. The screening is deterministic, and unbiasedness is not guaranteed. Local gradient comparisons measure its numerical effect; complete training experiments assess its optimization behavior.

AuthorsOpenReview ↗

Response (Q3/Q4) to Reviewer hqPE

Q3. Evidence from loss-derived gradients

We supplement the random-probe measurements with gradients generated by the cross-entropy training loss. The added experiment follows Appendix C.3, replacing only the upstream-gradient source. The table gives mean / P95 relative error (%) for each gradient tensor.

ContextdQ Mean / P95 (%)dK Mean / P95 (%)dV Mean / P95 (%)
1K0.3170 / 0.60560.1756 / 0.50590.0306 / 0.1603
2K0.2206 / 0.35180.1736 / 0.48380.0382 / 0.1757
4K0.2058 / 0.28940.1521 / 0.30200.0397 / 0.1694
8K0.1837 / 0.19910.1308 / 0.22010.0413 / 0.1614
16K0.1490 / 0.17330.1172 / 0.12580.0489 / 0.1522
32K0.1011 / 0.16130.1079 / 0.11860.0478 / 0.1539

Mean and P95 errors are lower than under the original Rademacher probe at every measured length. The largest mean errors for dQ/dK/dV are 0.3170%/0.1756%/0.0489%, and the largest P95 errors are 0.6056%/0.5059%/0.1757%. Training from 0.6B to 14B and continued training at 32B additionally support the approximation's usefulness during optimization.

Gradient contributions depend on VV, dOdO, and softmax coupling. These relative-error statistics characterize perturbation size and its measured tail; directional bias remains unmeasured. The evidence supports low gradient error under the sampled training loss and comparable model quality in the evaluated training settings.

Q4. Worst-case guarantee, empirical fidelity, and training adaptation

The mass bound supplies a worst-case guarantee; the realized attention distributions determine empirical fidelity. At τ=1\tau=1, ϵ≤τ/(1+τ)\epsilon\leq\tau/(1+\tau) gives the stated 50% bound. The proof sums a threshold-based upper bound over the legal context. Actual skipped contributions can be much smaller, and tile-level screening retains an entire tile whenever any valid entry fails the skip test.

Across the original 1K–32K operator measurements at τ=1\tau=1, mean omitted mass is at most 6.2×10−56.2\times10^{-5}. We additionally test τ=0.5,1,2,4\tau=0.5,1,2,4 on both MALA-trained and FullAttn-trained 14B checkpoints. The latter keeps its original dense-trained weights, with no MALA adaptation. At 32K:

CheckpointτForward / Backward Slots/QueryMean Output Error (%)
MALA-trained0.51,178 / 1,1640.0047
MALA-trained11,066 / 1,0580.0120
MALA-trained21,015 / 9980.0097
MALA-trained4994 / 9820.0143
FullAttn-trained0.51,192 / 1,1410.0054
FullAttn-trained11,072 / 1,0430.0079
FullAttn-trained21,024 / 1,0010.0103
FullAttn-trained41,002 / 9780.0127

The two checkpoints show similar work and low local output errors across tolerances, including direct substitution on the FullAttn-trained model. These diagnostics test sensitivity to context length, tolerance, and training adaptation. The 0.6B–14B training and 32B continued-training results supply model-level evidence across scales. Fidelity under further distribution shifts remains dependent on the attention distributions encountered.

Dear Reviewer hqPE,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate any follow-up on our clarifications of full score discovery and backward approximation, together with the additional training-gradient measurements, FullAttn-trained checkpoint tests, and end-to-end efficiency results.

Please let us know if any concerns remain or would benefit from further clarification. Thank you for your time and consideration.

Reviewer_2GotOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MassAlloc Attention (MALA), a fused attention kernel for both training and inference. It computes QK scores for every causal tile, but skips the rest of the work for tiles whose largest score is small relative to the softmax normalizer, using a length-normalized threshold τ/L_q. In forward, this means skipping exp, V loading and PV; in backward, it skips probability reconstruction and the dP, dS, dQ, dK and dV computations. Forward uses the running normalizer, while backward reuses the final normalizer saved by forward, so the backward retained set is nested within the forward one without storing a mask. The same tolerance is used for training, prefill and decoding, and the KV cache is not reduced. At τ = 1, each query executes the post-score path for about 1K keys regardless of context length, so most of this work is skipped at long contexts. Experiments include a matched-work allocation study, operator fidelity from 1K to 32K, associative recall, operator latency up to 128K, scaling-law training from 0.6B to 14B, and model-level evaluation at 14B and 32B. MALA stays close to full attention in quality while reducing attention latency and long-context training FLOPs.

Critical strengths
  1. The method is simple and works for both training and inference. The skip rule only uses the softmax statistics that FlashAttention already keeps, so no mask, index or router is needed, and the backward retained set is automatically nested within the forward one. One tolerance is used across forward, backward, prefill and decoding, and the realized work adapts to each head and input.

  2. The evaluation is broad and reaches real model scale. Operator-level fidelity is measured directly against the same operator with skipping disabled, and models from 0.6B to 14B are trained with MALA, plus a 32B model by continued training. MALA stays close to full attention on knowledge, reasoning and RULER, while MoBA and DSA fall clearly behind at 14B. The matched-work study (Table 1) also shows that per-instance allocation is much better than static allocation, and that MALA's online decisions come close to an oracle using the final reference mass.

Critical weaknesses
  1. The closest related methods are not compared. As the paper's own related work notes, BLASST also skips work by comparing block maxima with running softmax statistics. The main difference in MALA is that it compares with the accumulated normalizer instead of the running maximum, together with the 1/L_q normalization, which removes length-dependent calibration. This is a reasonable refinement, but it needs an empirical comparison to show its benefit. Twilight, Top-θ and SpargeAttention are also only discussed in related work, while the experiments compare with methods that select keys before attention.

  2. The efficiency gains are bounded and mostly shown at the operator level. Since QK is always computed for every causal tile, the forward and decode savings are limited (at most about 2x in principle; 2.2x forward and 1.6x decode at 128K), and the KV cache is not reduced. Latency is measured only for the attention operator at batch size 1, and training gains are reported as FLOPs (2.5% at 4K and 23.1% at 32K for 14B) rather than wall-clock time. In addition, only τ = 1 is evaluated. At this setting the omitted mass is about 0.006%, so matching full attention in quality is expected, and the paper does not show how quality and speed change under more aggressive settings.

Questions for authors
  1. Could you compare with BLASST (and ideally SpargeAttention's in-kernel filter or Twilight) at similar post-score work? This would show what the normalizer-based rule adds over existing in-kernel skipping.

  2. How do quality and speed change with τ? A small sweep (e.g., 0.5, 1, 2, 4) would give a trade-off curve instead of a single point.

  3. Can you report end-to-end training time and decode throughput at batch size > 1, not only operator latency and FLOPs?

  4. All quality and fidelity results use models trained or continued-trained with MALA (the 14B checkpoint for Table 1 and Fig. 2, and the 32B model after 64B tokens of continued training from Qwen3-32B). How does MALA work as a drop-in kernel for a model trained with full attention, e.g., the original Qwen3-32B without continued training? This would show whether the low omitted mass and quality depend on training with MALA.

4 comments
AuthorsOpenReview ↗

Response (W1/W2/Q1/Q2) to Reviewer 2Got

Thank you for the detailed review and the time invested in evaluating our work. We appreciate your recognition of the unified training and inference primitive, the matched-work allocation study, and the model-scale evaluation. We have added a BLASST-rule comparison, tolerance sensitivity measurements, end-to-end training and serving results, and a direct operator test on a FullAttn-trained checkpoint.

W1 and Q1. Matched-work comparison with the BLASST rule

We compare the BLASST screening rule and MALA under the matched-work setting of Table 1. Both use approximately 1,024 mean post-score key slots per query. In this comparison, we calibrate the BLASST-rule thresholds by layer and head; MALA uses its common normalized-contribution tolerance. The table reports omitted probability mass and relative output error (%).

Selection RuleMean Slots/QueryMean Omitted Mass (%)P95 Omitted Mass (%)Mean Output Error (%)P95 Output Error (%)
Per-instance reference-mass oracle≈1,0240.01820.06640.01640.1013
BLASST rule≈1,0240.02030.08920.01870.1224
MALA≈1,0240.01880.06930.01740.1017

Both rules achieve low errors after matching work. MALA has lower mean and P95 errors and approaches the per-instance reference-mass oracle; its P95 output error is 0.1017%, close to the oracle's 0.1013%.

The accumulated softmax normalizer captures how much probability mass has already been retained. Tiles with the same score gap from the running maximum can therefore receive different decisions depending on that accumulated mass. The length-normalized threshold expresses the tolerance relative to the uniform per-key probability 1/Lq1/L_q, giving it a common interpretation across context lengths. MALA uses this control in forward and reuses the finalized normalizer in backward, yielding nested retained support without storing selection masks. One tolerance serves training, prefill, and decoding.

We focus the added empirical comparison on BLASST's closely related online-softmax screening rule. The gradient-fidelity and model-training experiments evaluate the paired backward mechanism. Twilight, Top-θ, and SpargeAttention are discussed in Related Work.

W2 and Q2. Sensitivity to the tolerance

We evaluate the requested quality–speed trade-off at τ=0.5,1,2,4\tau=0.5,1,2,4 using the existing 14B checkpoint, with fixed model weights. We measure operator quality through relative output and dQ/dK/dV errors, and characterize the computational side of speed through Work F/B: the mean post-score key slots executed per query in forward/backward. Lower Work F/B means less post-score computation. At 32K:

τWork F/B, Slots/QueryOutput Mean (%)dQ Mean (%)dK Mean (%)dV Mean (%)
0.51,178 / 1,1640.00470.09690.07780.0274
11,066 / 1,0580.01200.10110.10790.0478
21,015 / 9980.00970.10880.11630.0636
4994 / 9820.01430.13840.15590.0786

Increasing τ\tau from 0.5 to 4 reduces forward/backward work from 1,178/1,164 to 994/982 slots/query, approximately 15.6% in both passes. Across these settings, mean output error remains between 0.0047% and 0.0143%, and mean dQ/dK/dV errors remain at most 0.1384%/0.1559%/0.0786%. Thus, the table directly relates reduced executed work to the measured output and gradient fidelity.

Work F/B is a compute proxy for speed; full QK discovery and execution overhead also contribute to latency. The wall-clock measurements below use τ=1\tau=1. Diffuse rows can retain more work, including all forward tiles for a strictly uniform row at τ=1\tau=1.

AuthorsOpenReview ↗

Response (W2/Q3/Q4) to Reviewer 2Got

W2 and Q3. End-to-end training and serving throughput

MALA improves end-to-end throughput in both Megatron-LM and vLLM through an attention-kernel replacement. The measurements include full QK discovery, the allocation test, retained post-score computation, and the surrounding model execution.

Training

We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8, PP=1, and DP=16. The microbatch size is eight at 4K and one at 32K, with batch settings matched across methods at each scale and context length. FullAttn and MALA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.

ModelMethod4K Step (ms)4K Tokens/s32K Step (ms)32K Tokens/s
1.7BFullAttn2122,478,1762881,818,080
1.7BMoBA2971,769,162291.11,798,599
1.7BDSA206.62,543,132222.12,357,358
1.7BMALA195.92,682,058208.82,507,841
4BFullAttn7751,353,2161,217861,568
4BMoBA1,080.6970,5551,230.1852,412
4BDSA755.51,388,096940.91,114,375
4BMALA7171,462,597885.11,184,697
8BFullAttn1,711919,1362,363665,728
8BMoBA2,106.4746,6142,381.6660,523
8BDSA1,685.8932,8741,969.7798,669
8BMALA1,636961,2681,890.1832,291
14BFullAttn3,635576,9604,927425,600
14BMoBA4,294488,4564,960422,760
14BDSA3,593583,6664,230495,725
14BMALA3,510597,4594,089512,827

MALA has the highest throughput among the four measured configurations at each scale and both lengths. Its throughput gain over FullAttn is 1.04–1.08× at 4K and 1.20–1.38× at 32K. At 14B, the 32K step decreases from 4,927 to 4,089 ms, giving a 20.5% throughput increase; the 4K throughput increase is 3.6%.

Serving

We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.

ModelMethodPrefill Input Tokens/sDecode Output Tokens/s
14BFullAttn46,7461,542
14BMoBA37,468541
14BDSA74,2091,754
14BMALA76,0861,946
32BFullAttn21,520891
32BMALA36,0751,140

The prefill/decode gains over FullAttn are 1.63×/1.26× at 14B and 1.68×/1.28× at 32B. These results extend the original batch-one operator measurements to complete serving workloads with concurrency 16.

Cost and memory scope

MALA retains full QK computation and the full KV cache, with operator memory comparable to FullAttn. The approximately 2× forward limit describes an arithmetic model with equal QK and PV costs. Wall-clock savings also include softmax, value loading, and memory traffic, which explains how operator speedup can exceed that arithmetic ratio. The measurements above account for the complete execution path and show stronger training gains at 32K than at 4K.

Q4. Direct replacement on a FullAttn-trained checkpoint

We additionally test MALA directly on the FullAttn-trained 14B checkpoint, retaining its original weights and applying no MALA training adaptation. Each checkpoint is compared with its own screening-disabled reference. At τ=1\tau=1, the two checkpoints produce the following retained work, reported as forward / backward mean slots per query.

ContextMALA-Trained CheckpointFullAttn-Trained Checkpoint
1K469 / 461486 / 469
2K696 / 687709 / 700
4K907 / 897883 / 858
8K1,023 / 1,010989 / 971
16K1,048 / 1,0351,021 / 1,016
32K1,066 / 1,0581,072 / 1,043

On the FullAttn-trained checkpoint at 32K, the tolerance comparison gives:

τWork F/B, Slots/QueryOutput Mean (%)dQ Mean (%)dK Mean (%)dV Mean (%)
0.51,192 / 1,1410.00540.10320.08560.0232
11,072 / 1,0430.00790.10540.11230.0566
21,024 / 1,0010.01030.10740.11550.0745
41,002 / 9780.01270.14930.13340.0895

At τ=1\tau=1 and 32K, direct substitution gives a mean output error of 0.0079% and a largest mean gradient error of 0.1123%. The similar work and low errors across tolerances show that local operator fidelity also holds on the tested dense-trained 14B checkpoint without MALA adaptation. Model-level quality is evaluated separately through the paper's training and continued-training experiments.

Thank you for the author's response. Most of my concerns have been addressed, and I will maintain my positive score.

Thank you for the follow-up and for your careful evaluation. We appreciate your positive assessment and will incorporate the clarifications and additional results into the revised manuscript.

Reviewer_cYVZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MassAlloc Attention (MALA), a fused attention operator that dynamically allocates computation based on normalized attention contributions. Unlike conventional sparse attention methods that restrict the attention support before computing QK scores, MALA computes all causal QK scores and selectively skips subsequent computations for low-contribution tiles. In the forward pass, it uses the evolving online softmax normalizer to estimate tile contributions and applies a length-normalized threshold to determine which tiles to skip. In the backward pass, it reuses the final log-normalizer from the forward pass to identify tiles whose gradient computations can be omitted, without explicitly storing selection masks or tile indices. The paper provides a theoretical bound on the omitted attention probability mass and evaluates the approximation quality and efficiency through operator-level experiments, associative recall tasks, GPU kernel benchmarks, and large-scale model training. Results show that MALA achieves 2.2× and 3.0× speedups in forward and backward attention computation, respectively, and 1.6× faster autoregressive decoding at a 128K context length on 8 H100 GPUs. It also reduces total training FLOPs by 23.1% in 14B model training with a 32K context, while achieving performance comparable to Full Attention on the evaluated language modeling, reasoning, and retrieval benchmarks.

Critical strengths

Strength 1: A simple and adaptive compute allocation mechanism with a well-integrated forward and backward design. A key strength of this work is its formulation of attention sparsity as a distribution-conditioned compute allocation problem. Rather than relying on predefined sparse patterns or additional learned selection mechanisms, MALA directly leverages attention scores and native softmax statistics to dynamically determine which tiles require subsequent computation. The forward pass uses an evolving normalizer for tile selection, while the backward pass reuses the finalized normalizer from the forward pass, avoiding the need to explicitly store sparse masks or routing indices. This design is simple and technically elegant, and the same allocation principle can be applied across training, prefill, and autoregressive decoding.

Strength 2: Extensive experimental validation at both operator and model levels. The paper provides a comprehensive experimental evaluation covering matched-work comparisons, operator fidelity, associative recall, GPU kernel benchmarks, scaling-law experiments, and model-level evaluations. The results at 128K context length demonstrate meaningful reductions in attention computation latency, while the 14B and 32B model evaluations suggest that MALA maintains performance comparable to Full Attention on the evaluated language modeling, reasoning, and retrieval benchmarks. Furthermore, evaluating both forward and backward efficiency provides a more complete assessment of the method's computational benefits.

Critical weaknesses

Weakness 1: The theoretical analysis provides a relatively loose guarantee on the approximation error. The paper provides a theoretical upper bound on the omitted attention probability mass in the forward pass. However, under the default threshold setting of τ=1\tau=1, this bound allows up to 50% of the attention probability mass to be omitted, making it relatively loose compared with the much smaller approximation errors observed empirically. While the empirical results are encouraging, the current theoretical analysis does not fully explain why the approximation remains accurate in practice. A tighter characterization of the relationship between the threshold, omitted attention mass, and output approximation error would further strengthen the theoretical justification of MALA.

Weakness 2: The reliability of the backward approximation is not sufficiently characterized. The backward allocation mechanism uses attention contributions to determine which tiles can be skipped during gradient computation. However, the importance of a tile to the gradients is not necessarily determined solely by its attention probability, as gradient contributions also depend on the upstream gradients and value vectors. Although the paper provides empirical evaluations of gradient approximation errors, it does not establish a corresponding theoretical guarantee for the backward pass. Further analysis of the approximation under different upstream gradient distributions and training conditions would help clarify the reliability of this mechanism in practical training scenarios.

Weakness 3: The computational benefits are constrained by the full QK score computation. MALA retains the computation of all causal QK scores and only skips subsequent attention computations for low-contribution tiles. Consequently, the method does not reduce the quadratic complexity associated with QK score computation, which may limit its efficiency gains in settings where score computation accounts for a substantial portion of the attention cost. Although the reported kernel benchmarks demonstrate promising speedups, it would be useful to further characterize how the computational benefits scale with sequence length and different attention workloads. In addition, since MALA retains the full KV cache, its benefits do not extend directly to reducing the memory requirements of long-context inference.

Questions for authors
  1. Reliability of backward approximation: The backward pass uses attention contributions to determine which tiles can be skipped. However, small attention probabilities do not necessarily imply negligible gradient contributions, which also depend on the upstream gradients and value vectors. Can the authors clarify how the backward approximation behaves under different gradient distributions, and whether the reported gradient fidelity results are representative of actual training gradients? This is important for assessing whether the proposed mechanism can reliably reduce backward computation without affecting model optimization.

  2. Effectiveness of the default threshold: The paper adopts a unified threshold of 1 across different contexts, heads, and layers. How sensitive are the approximation accuracy and computational savings to this threshold, particularly for attention distributions that are relatively diffuse? Understanding this would help determine whether the reported efficiency-accuracy trade-off is robust or depends on the specific attention patterns in the evaluated models.

  3. Practical efficiency compared with existing sparse attention methods: MALA retains full causal QK score computation, whereas some existing sparse attention methods also reduce the cost of score computation. Under what sequence lengths and attention workloads does MALA provide a practical advantage over these methods in terms of end-to-end training or inference latency? Clarifying this comparison would help establish the practical scope of the proposed approach.

3 comments
AuthorsOpenReview ↗

Response (W1/W2/Q1) to Reviewer cYVZ

We thank the reviewer for the careful review and the time spent evaluating our work. We appreciate your recognition of distribution-conditioned compute allocation, the paired forward and backward design, and the operator- and model-level evaluation.

W1. The forward guarantee and empirical approximation error

The forward guarantee connects online screening to the total probability mass omitted from the final attention distribution. Under exact arithmetic, ϵ≤τ/(1+τ)\epsilon\leq\tau/(1+\tau) gives a worst-case bound of 50% at τ=1\tau=1. The proof bounds each skipped key relative to the current retained normalizer, uses the monotonic growth of that normalizer, and sums over the legal context.

This bound can be loose: skipped contributions can lie far below the threshold, and the tile-level rule retains an entire tile whenever any valid entry fails the skip condition. Given the actual omitted mass ϵ\epsilon and a bound Vmax⁡V_{\max} on value-vector norms, the appendix gives the output bound ∥o−o~∥≤2ϵVmax⁡\|o-\widetilde{o}\|\leq2\epsilon V_{\max}. This relates realized omitted mass to output error; the small empirical errors are measured directly.

Across 1K–32K contexts with the same τ=1\tau=1, mean omitted mass is at most 6.2×10−56.2\times10^{-5}. Direct substitution on the FullAttn-trained 14B checkpoint also gives a mean output error of 0.0079% at 32K and τ=1\tau=1. The training-gradient and tolerance experiments below further characterize the achieved fidelity.

W2 and Q1. Backward fidelity with actual training gradients

We additionally evaluate backward fidelity using upstream gradients produced by the cross-entropy training loss. We follow the operator-fidelity protocol of Appendix C.3 and replace the Rademacher probe with the loss-derived dOdO. Each comparison uses the same operator inputs and upstream gradient, with screening disabled for the FullAttn reference. The table reports mean relative errors (%); each cell gives Rademacher probe / training-loss dOdO.

ContextdQ Mean (%)dK Mean (%)dV Mean (%)
1K0.3832 / 0.31700.3467 / 0.17560.1680 / 0.0306
2K0.3099 / 0.22060.3018 / 0.17360.1497 / 0.0382
4K0.2526 / 0.20580.2559 / 0.15210.1436 / 0.0397
8K0.1988 / 0.18370.2177 / 0.13080.1276 / 0.0413
16K0.1598 / 0.14900.1545 / 0.11720.1150 / 0.0489
32K0.1217 / 0.10110.1086 / 0.10790.1077 / 0.0478

The P95 relative errors (%) are:

ContextdQ P95 (%)dK P95 (%)dV P95 (%)
1K0.9020 / 0.60560.6638 / 0.50590.2281 / 0.1603
2K0.7066 / 0.35180.5577 / 0.48380.2143 / 0.1757
4K0.5693 / 0.28940.4168 / 0.30200.2098 / 0.1694
8K0.4347 / 0.19910.3050 / 0.22010.1857 / 0.1614
16K0.3181 / 0.17330.2341 / 0.12580.1699 / 0.1522
32K0.2404 / 0.16130.1875 / 0.11860.1656 / 0.1539

Training-loss gradients give lower mean and P95 errors for dQ, dK, and dV at every measured length. Across lengths, their largest mean errors are 0.3170%, 0.1756%, and 0.0489%, respectively; the largest P95 errors are 0.6056%, 0.5059%, and 0.1757%.

Backward reuses the finalized forward normalizer, giving nested retained support and an additional gradient approximation whose effect depends on VV and dOdO. The two gradient sources test random directions and sampled training directions, respectively. Together with training from 0.6B to 14B and continued training at 32B, these measurements support the approximation's practical reliability. Support nesting establishes the structural relationship between the passes; gradient fidelity is supported empirically.

AuthorsOpenReview ↗

Response (W3/Q2/Q3) to Reviewer cYVZ

Q2. Tolerance sensitivity and diffuse attention

We additionally evaluate τ=0.5,1,2,4\tau=0.5,1,2,4 on the 14B checkpoint. Retained work is the mean number of post-score key slots per query. Each cell below reports forward / backward work.

Contextτ = 0.5τ = 1τ = 2τ = 4
1K489 / 474469 / 461439 / 426399 / 384
2K719 / 705696 / 687647 / 635625 / 615
4K1,071 / 1,046907 / 897842 / 828809 / 799
8K1,082 / 1,0521,023 / 1,010962 / 953899 / 871
16K1,113 / 1,0891,048 / 1,035980 / 961933 / 907
32K1,178 / 1,1641,066 / 1,0581,015 / 998994 / 982

At 32K, the corresponding mean relative output and training-gradient errors are:

τWork F/BOutput Mean (%)dQ Mean (%)dK Mean (%)dV Mean (%)
0.51,178 / 1,1640.00470.09690.07780.0274
11,066 / 1,0580.01200.10110.10790.0478
21,015 / 9980.00970.10880.11630.0636
4994 / 9820.01430.13840.15590.0786

Over this range, 32K forward work changes from 1,178 to 994 slots/query, mean output error remains between 0.0047% and 0.0143%, and the largest mean gradient error is 0.1559%. The tolerance controls screening; the score distribution and normalization state determine realized work.

Diffuse rows can retain more work. For a strictly uniform row, τ=1\tau=1 retains all forward tiles: each entry's contribution relative to the current normalizer is at least 1/Lq1/L_q. Larger tolerances relax this condition and can permit skipping. The approximately 1K work observed at long contexts emerges from the evaluated distributions.

W3 and Q3. Full QK cost and end-to-end efficiency

We additionally measure complete training and serving workloads in Megatron-LM and vLLM by replacing the attention kernel with MALA. The measurements include full QK discovery, allocation, retained post-score computation, and the surrounding model execution.

Training

We use Megatron-LM 0.18.2 on 128 H100 GPUs with TP=8, PP=1, and DP=16. The microbatch size is eight at 4K and one at 32K, with batch settings matched across methods at each scale and context length. FullAttn and MALA share the GQA architecture; MoBA and DSA retain their native attention parameterizations and kernels. The table reports complete iteration time and aggregate throughput.

ModelMethod4K Step (ms)4K Tokens/s32K Step (ms)32K Tokens/s
1.7BFullAttn2122,478,1762881,818,080
1.7BMoBA2971,769,162291.11,798,599
1.7BDSA206.62,543,132222.12,357,358
1.7BMALA195.92,682,058208.82,507,841
4BFullAttn7751,353,2161,217861,568
4BMoBA1,080.6970,5551,230.1852,412
4BDSA755.51,388,096940.91,114,375
4BMALA7171,462,597885.11,184,697
8BFullAttn1,711919,1362,363665,728
8BMoBA2,106.4746,6142,381.6660,523
8BDSA1,685.8932,8741,969.7798,669
8BMALA1,636961,2681,890.1832,291
14BFullAttn3,635576,9604,927425,600
14BMoBA4,294488,4564,960422,760
14BDSA3,593583,6664,230495,725
14BMALA3,510597,4594,089512,827

MALA has the highest throughput among the four measured configurations at each scale and both lengths. Its throughput gain over FullAttn is 1.04–1.08× at 4K and 1.20–1.38× at 32K. At 14B, the 32K step decreases from 4,927 to 4,089 ms, giving a 20.5% throughput increase; the 4K throughput increase is 3.6%.

Serving

We use vLLM 0.28 on eight H100 GPUs with TP=8 and total concurrency 16. The prefill-heavy workload has 128K input / 128 output tokens; the decode-heavy workload has 128 input / 128K output tokens. Throughput is measured over complete requests, using input tokens/s for the first workload and output tokens/s for the second.

ModelMethodPrefill Input Tokens/sDecode Output Tokens/s
14BFullAttn46,7461,542
14BMoBA37,468541
14BDSA74,2091,754
14BMALA76,0861,946
32BFullAttn21,520891
32BMALA36,0751,140

The prefill/decode gains over FullAttn are 1.63×/1.26× at 14B and 1.68×/1.28× at 32B. These results extend the original batch-one operator measurements to complete serving workloads with concurrency 16.

MALA retains quadratic QK computation and the full KV cache, with operator memory comparable to FullAttn. Its gains come from skipping low-contribution post-score execution using native attention state. Full QK becomes the limiting attention cost as retained post-score work decreases.

Dear Reviewer cYVZ,

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would appreciate any follow-up on our response, including the clarification of the forward guarantee, the additional gradient-fidelity measurements using training-loss gradients, the tolerance sensitivity results, and the end-to-end training and serving benchmarks.

Please let us know if any concerns remain or would benefit from further clarification. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: MALA’s Contributions, Additional Evidence, and Review Discussion

Dear AC, SAC, and PC,

We thank the reviewers for their thoughtful evaluation. We summarize our contribution, the reviewers’ feedback, and the evidence provided during rebuttal.

TL;DR of Our Work

MALA investigates whether attention can allocate its own computation using the probability distribution it realizes. Every legal causal interaction remains score-accessible; normalized contributions determine which post-score computations are executed. Forward uses the evolving softmax normalizer, and backward reuses its finalized state to obtain nested retained support using standard attention state. One tolerance serves training, prefill, and decoding.

The matched-work study isolates the value of input-dependent allocation and shows that MALA’s online decisions approach a per-instance reference-mass oracle. Operator measurements, associative recall, training from 0.6B to 14B, and evaluation at 14B and 32B connect this allocation principle to numerical fidelity, model quality, and practical execution.

Reviewers’ Recognition

cYVZ recognizes the distribution-conditioned allocation perspective and the integrated forward/backward design. 2Got highlights the simple unified mechanism, model-scale evaluation, and matched-work evidence. hqPE identifies the paired forward/backward formulation and nested support without stored selection masks as the clearest technical contribution, and recognizes MALA as a useful design point between dense and support-sparse attention.

Reviewers’ Concerns and Our Responses

  • cYVZ and hqPE question backward reliability under actual training gradients. We added measurements using upstream gradients generated by the cross-entropy loss. Across 1K–32K, mean and P95 errors are lower than under the original random probe at every measured length. The largest mean dQ/dK/dV errors are 0.3170%/0.1756%/0.0489%. We clarified the deterministic gradient approximation and its empirical validation.

  • cYVZ and hqPE ask how the theoretical bound relates to observed fidelity. We explained how the forward theorem bounds cumulative omitted mass using online normalization state. Actual fidelity is measured directly: mean omitted mass is at most 6.2×10⁻⁵ across the original 1K–32K tests. The backward measurements characterize approximation error; directional unbiasedness remains unmeasured.

  • 2Got requests a comparison with the closest screening approach. We added a BLASST-rule control at approximately 1,024 mean post-score slots/query. Following layer/head threshold calibration, both approaches achieve low errors; MALA has lower mean and P95 errors and approaches the reference-mass oracle. MALA uses a common normalized-contribution tolerance across execution stages.

  • cYVZ, 2Got, and hqPE ask about tolerance sensitivity and training adaptation. We evaluated τ=0.5, 1, 2, and 4 on MALA-trained and FullAttn-trained 14B checkpoints. At 32K, MALA’s forward/backward post-score work decreases by approximately 15.6%, while mean output error stays within 0.0047%–0.0143%. Direct substitution on the dense-trained checkpoint at τ=1 gives 0.0079% mean output error. These experiments measure operator fidelity and executed work.

  • All three reviewers ask about practical gains with full QK cost included. We added complete training and serving measurements. On 128 H100 GPUs, 32K training throughput improves by 1.20–1.38× across 1.7B–14B. On eight H100 GPUs with concurrency 16, 14B/32B serving gains are 1.63×/1.68× for prefill-heavy workloads and 1.26×/1.28× for decode-heavy workloads. These measurements include full QK discovery; KV-cache capacity remains unchanged.

Discussion Status

All three reviewers currently recommend weak acceptance. In the discussion, 2Got explicitly confirmed that most concerns had been addressed and maintained the positive score.

The additional evidence strengthens the connection between adaptive allocation, training fidelity, and complete-system efficiency. We will incorporate these results and clarifications into the revision. We thank the reviewers and chairs for their careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.