acceptodds
Under review as a conference paper at ICLR 2027

Where Does Attention Mass Live? A Matched-Budget Study of Query-Adaptive Attention Sparsification for Decoder LMs

Abstract

Attention sparsification is the dominant route to cheap long-context inference, yet practitioners choosing a masking rule face a crowded menu of heuristics with little guidance on which buys the most quality at a fixed kept-fraction budget. We present a controlled, matched-budget comparison of seven sparsification families—query-adaptive thresholding in logit and softmax-probability space, per-query top-k, sliding window, sink-plus-window, static block-sparse, and random masking—evaluated by exact masked-softmax teacher-forced loss on Qwen2.5 models at 0.5B and 1.5B parameters, across WikiText-2 and C4, at context lengths from 1,024 to 8,192. Three findings emerge. First, thresholding in probability space (retain entries whose softmax weight exceeds τ times the per-query maximum) matches full-attention loss within 0.007 nats/token at a realized kept-entry fraction of 13.5% at context 1,024 and within 0.030 nats at 4.1% at context 8,192 (fp32 paired residual: 0.008, CI [0.005, 0.010]); in the budget-matched comparison (8K context, n=8 paired documents, 4.1% versus 4.6% kept fraction) it beats magnitude-based top-k by 0.026 nats (paired 95% CI [0.018, 0.035]) at the lower budget. Second, the same rule defined in logit space—the natural first proposal—is consistently dominated, an honest negative result we trace to the multiplicative logit-ratio rule interpolating toward the row minimum rather than a calibrated gap from the maximum. Third, per-head budget profiles measured on one set of documents transfer almost perfectly to unseen documents (mean pairwise Pearson r = 0.98), but profile-guided reallocation—per-layer or per-head— does not improve on the uniform threshold (it costs +0.028 nats, significant on every paired test): the sparsity pattern is a stable property of the model, and a uniform threshold is already near-optimal in this family under proportional reallocation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
attention sparsification, long-context inference, KV cache, matched budget, softmax thresholding, query-adaptive masking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_j8ssOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work evaluates 7 different masking families with a common masked-softmax on qwen2.5-0.5b and 1.5b. Using teacher forced loss on wikitext-2 and c4 with context lengths of up to 8k. They find good loss budget tradeoffs when using relative to max prob thresholding. Consistent sparsity patterns per head. Failed attempts at distributing thresholds based on sparsity patterns and keeping the argmax of each query can be beneficial. This paper just provides comparisons between these methods.

Critical strengths
  1. Evaluating all masks through a shared attention implementation helps isolate selection rules from kernel-specific effects. The paired document-level analysis also provides useful evidence for the reported loss differences, including the 8K threshold-versus-top-k comparison.

  2. The head-profile measurements and four-way window ablation are potentially informative diagnostics. Reporting unsuccessful allocation attempts is valuable, provided the conclusions remain limited to the tested configurations.

Critical weaknesses
  • The central budget interpretation is inconsistent, and the evidence does not establish the claimed matched-budget frontier. Equation 2 averages retained fractions across query rows; it is not the fraction of all retained causal attention entries. At n=1,024, top-k-64 has an average row fraction of approximately 0.235 but a total entry fraction of approximately 0.121. More directly, Section 3.2 compares a bare window-128 budget of 0.234 with window+argmax at 0.386 and attributes the increase to argmax preservation. Under Equation 2, the bare window already has budget approximately 0.3845, and adding at most one entry per row cannot increase it by more than approximately 0.0073. These numbers mix two budget definitions. Moreover, Table 2 imports budgets from the frontier protocol rather than measuring them in the paired runs, and Table 3 omits allocation budgets entirely. The selected operating-point comparisons therefore do not establish the general frontier or uniform-threshold near-optimality claims.

  • The additional contribution over prior work is unclear, and the proposed mechanisms are overstated. Twilight [1] already explains the limitations of fixed-k budgeting and evaluates adaptive mass-based selection. Top-Theta [2] studies calibrated thresholds and head-dependent calibration, while BLASST [3] uses a closely related relative-to-maximum logit-gap criterion. The paper does not compare against oracle top-p or calibrated thresholds under its controlled protocol. Its probability-versus-logit interpretation is also misleading: p ≥ τp_max is exactly s ≥ s_max + ln τ, so the experiment compares parameterizations rather than inherently different spaces. Finally, the window ablation does not establish that sink benefits are content-adaptive argmax benefits: fixed sinks alone reduce loss from 4.82 to 2.499, recovering most of the improvement achieved by the combined variant at approximately 2.352. Likewise, poor results from two uncalibrated proportional allocation schemes do not demonstrate that uniform thresholds are near-optimal.

[1] Chaofan Lin et al. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning. 2025

[2] Konstantin Berestizshevsky, Renzo Andri, and Lukas Cavigelli. Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding. ICANN 2026

[3] Jiayi Yuan et al. BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding. MLSys 2026

Questions for authors
  • What budget was calculated for each run? Can you double check your window example in section 3.2 and calculate if your order still holds when calculating by total fraction of causal entries. With budgets being equal to each other (paired).

  • If we threshold relative to max do we still out perform a tighter top k search. Top p using mass and calibrated thresholds. At similar achieved budgets this would be helpful to know.

  • Were the different allocations in table 3 calibrated to match the uniform budget? If so what were those budgets. How do you know that section 7.3 shows content. It could be due to the benefits of keeping sinks and high scoring keys. Looks like there is a mistake in appendix A.5 for the -0.147.

Reviewer_ZtA5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors of the given paper provide the empirical examination of the seven sparsification rules. They examine the masking sparsification setup, where the attention weights are calibrated by the set of binary variables {Mij}ij\{M_{ij}\}_{ij}

As a result they propose the consideration of the following rules: logit-tresholding, softmax thresholding, per-query top-k, sliding window, sink plus window, block-sparse and random. Then the authors provide the experimental examination of these rules. All rules run in a single kernel that computes the exact softmax over the kept set, and quality is measured as teacher loss on Qwen2.5-0.5B/1.5B, WikiText-2 and C4, at contexts from 1K to 8K tokens. The paper reports several findings:

  1. As context grows, the advantage of adaptive rules over static patterns (sliding window, sink+window) increases, and softmax thresholding beats top-k at the most aggressive budgets.

  2. The same rule defined on logits is consistently worse, which the authors explain by the threshold depending on the row minimum.

  3. Per-head sparsity varies strongly across heads but is stable across documents, i.e. it is a property of the model.

Critical strengths
  1. Well-explained experiment sections with statistical significance: CIs and Wilcoxon statistics, not just mean losses.
Critical weaknesses
  1. The compared rules are idealized oracle masks (e.g., softmax thresholding requires computing the full attention row first), not practical methods. The paper itself is rather the toy exampled. The paper does not compare against established approaches: H20, Quest, NCA.

  2. As a consequence of the first part: what are the practical relevance of the winning rule?

Questions for authors
  1. What are the practical comparison to the current sota methods?

  2. See W1 - W2.

Reviewer_gEPEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is a computational study evaluating the impact of different attention sparsification mechanisms, that is, the attention matrix is formed and then sparsified. One of the main claimed contributions seems to be to provide a "matched-budget" study of where attention mass lives, although it is both hard to understand how this is insightful and early backed by the data since the matched-budget is not clear. The paper evaluates a set of methods on two different sizes of a Qwen model.

Critical strengths

The related work seems to be extensive, and although I often don't understand the motivation, the math is quite clear. Generally, the topic is interesting: it is very important to understand where much of the attention mass lives.

Critical weaknesses

I have several concerns w.r.t. this paper that make me lean towards rejection. I will outline my concerns below. I will put Questions separately, although many of my concerns are related to questions where I just don't know how the authors ended up with their respective decisions. For the moment, I recommend a score of 1, but I'm looking forward to seeing the authors responses.

Soundness

  • I think one of my main concerns is that I fail to see or understand how this is "matched-budget"? I have trouble understanding how this budget is properly derived (see section 3.3 for instance), I don't get how this makes any sense and I am happy to hear the authors thoughts. it seems like you're deriving a budget which should be held equal, but when I take a look at Table 1 I see an apples-to-pears comparison between a lot of different budgets? Why is B not constant across methods? How is that a matched-budget, matched wrt what? Please explain.
  • The models used are two models of exactly the same family, from 2024 (pre-reasoning if I'm not mistaken), and are not bigger than 1.5B parameters. This feels very insufficient for a computational study.
  • As the authors themselves state, the evaluation is on perplexity only, and thus not very informative.
  • The context windows considers seem relatively small.

Clarity

  • In the first contribution, you write that the statement basically holds by construction. I agree. Whats surprising about this then? What should I learn from the fact that you demonstrate something that is supposed to hold by construction?
  • Generally, I find the motivation and derivation of the paper relatively hard to follow.

Minor

  • This is of course your choice, but it feels like many parts are written by LLMs, I might be mistaken. "degrades gracefully" is a dead giveaway for me, but I don't want to accuse the authors of anything that they might not have done in practice.
Questions for authors
  • Out of curiosity and maybe lack of knowledge on my side: how relevant is this attention sparsification, if the attention matrix has to be completely realized anyways? I understand that it makes complete sense to only compute parts of the matrix, but computing it upfront and then sparsifying does not seem to yield any inference benefits? What am I missing here?
  • Why are logic-ratio and softmax threshold not spanning the same range (Line 260)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.