acceptodds
Under review as a conference paper at ICLR 2027

Retrieval Capacity of Self-Attention Under Competition

Abstract

How many tokens from its context does a language model actually use, and what determines that number? We study this question through self-attention. Without retraining, we retain only the tokens with the highest attention weights at each head, layer, and query, keeping their original weights unchanged. By varying the selected set size and measuring the increase in negative log-likelihood (NLL), we estimate the effective attention set size needed to stay within a chosen loss tolerance. Relatively small selected sets can keep NLL close to the full-attention baseline, although the required size varies across models. Attention-based selection substantially outperforms random selection. Selected sets exhibit geometric structure, although geometric separation alone does not establish that model loss is preserved. Extending context while evaluating the same prediction targets increases the required set size, while its fraction of context decreases over the tested range. Experiments with a fixed supporting fact show that additional background pushes its tokens down the attention ranking and reduces their attention mass. Renormalizing the retained weights can substantially reduce the required set size, showing that it also depends on how selected representations are combined. Conditional theoretical models explain how competition and attention-mass retention can produce growing set sizes without more distinct information to retrieve. These results provide a way to measure effective attention set size in language models and investigate its dependence on context, competition, and aggregation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, geometry, explainability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ei4bOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates how many context tokens need to be retained in self-attention to preserve language-model prediction performance and how this requirement changes with context length. The authors introduce an operational measure of effective attention set size, N∗N^*, based on Top-N selection of tokens ranked by attention weights or weighted-value contribution norms, with functional sufficiency evaluated through NLL degradation. They examine the geometric separation of selected and unselected contributions, compare attention-based selection with random baselines across pretrained language models, and show that the estimated required set size generally increases with context length. To investigate attention competition, they conduct controlled BABILong experiments that keep annotated supporting information fixed while increasing background length, finding that additional context displaces supporting tokens in attention rankings and reduces their attention mass. The paper further examines the effects of renormalizing retained attention weights and presents conditional theoretical analyses of how ranking competition and attention-mass preservation can produce increasing effective set sizes.

Critical strengths

1. Controlled investigation of attention competition with fixed supporting information

The controlled retrieval experiments in Section 3.3 provide a useful complement to the natural-language context-length experiments. By keeping the annotated supporting fact fixed while increasing background length, the authors reduce the confounding effect of additional task-relevant information and examine how competition affects the ranking, attention mass, and retention of supporting tokens. This experimental design provides more direct evidence for the role of attention competition than context-length scaling measurements alone.

2. Functional evaluation of attention selection and the role of normalization.

The paper goes beyond descriptive attention statistics by systematically intervening on selected attention contributions and measuring the resulting degradation in model predictions across multiple pretrained language models. The additional comparison between deletion and renormalized attention is particularly valuable, as it demonstrates that effective attention set size depends not only on token selection but also on how retained contributions are aggregated. The accompanying local error analysis provides a clear mathematical explanation for this dependence.

Critical weaknesses

1. Strong dependence on the intervention design and insufficient attribution of context-length effects

The central empirical finding that effective attention set size increases with context length depends strongly on the specific intervention used to define it. Sections 3.1–3.3 primarily employ post-softmax deletion without renormalizing the retained attention weights, which differs from standard sparse attention masking, where the retained weights are normalized over the selected tokens. The authors themselves show in Section 3.4 and Appendix E.1 that renormalization can dramatically reduce the required set size and even eliminate the observed growth across background conditions in some settings. For example, in BABILong, Gemma-7B's estimated set size increases from 64 to 760 under deletion but remains at 32 under renormalization. Although the paper acknowledges this intervention dependence, the main context-length results are still based on deletion, while renormalized interventions are not evaluated as systematically across context lengths and models. Furthermore, the language-modeling deletion and renormalization comparisons in Appendix E.1 do not have verified document-level pairing or identical numerical configurations. Consequently, it remains unclear to what extent the reported growth in effective attention set size reflects intrinsic retrieval requirements rather than sensitivity to the particular deletion-based aggregation procedure.

More broadly, the paper does not sufficiently disentangle the mechanisms underlying this growth. The authors identify several possible factors, including the acquisition of additional useful information, increased ranking competition, and sensitivity to discarded attention contributions. Section 3.2 shows that longer natural contexts improve full-model predictions while increasing the required set size, whereas Section 3.3 uses fixed annotated supporting facts to provide evidence of increased competition. However, the experiments do not directly investigate how these mechanisms contribute to the observed growth in the same setting. In particular, the natural-language modeling experiments do not analyze whether newly introduced context tokens provide useful predictive information, displace previously relevant tokens in attention rankings, or primarily affect the retained attention mass. The resulting interpretation of context-length dependence therefore remains largely descriptive, without sufficiently isolating the mechanisms responsible for the observed effects.

2. Limited connection between geometric, theoretical, and functional analyses.

The relationship between the geometric properties of selected attention contributions and their functional importance remains insufficiently established. Figure 2 reports stronger geometric separation for attention-selected sets than for random sets, but the geometric metrics exhibit nonmonotonic behavior as the selected-set size increases. Figure 4 then reports correlations between geometric F-scores and NLL degradation across varying Top-N budgets. Since both quantities depend on N, these correlations may largely reflect their shared dependence on the selection budget rather than a meaningful geometry–function relationship. Appendix C.2 further shows that excluding small N values substantially weakens or even reverses several correlations. More importantly, it remains unclear how the geometric separation analysis in Section 3.1 contributes to understanding the context-length dependence and attention competition investigated in Sections 3.2–3.3. The paper does not establish whether these geometric properties help explain or predict the observed changes in effective attention set size as context length increases, leaving the geometric analysis relatively disconnected from the central empirical findings.

The theoretical analysis is also primarily conditional on idealized assumptions whose empirical relevance remains insufficiently examined. Appendix F characterizes how ranking competition and attention-mass retention can produce growing selected-set requirements under assumptions about attention-score distributions, score dependence, and value-vector properties. However, the paper does not systematically evaluate whether these assumptions hold in pretrained Transformers or whether the derived mechanisms quantitatively explain the observed context-length trends. Although Appendix E.4 provides an exploratory analysis of discarded-vector statistics and finds that loss-sensitive diagnostics predict effective set size better than geometric separation, this analysis is limited to a fixed context length and does not directly validate the theoretical assumptions. Consequently, the geometric and theoretical analyses remain insufficiently connected to the observed functional retrieval behavior.

Questions for authors

Clarification Requests

1. Consistency of deletion and renormalization evaluations.

Appendix E.1 acknowledges that the deletion and renormalization evaluations do not have verified document-level pairing or identical numerical configurations. Why were these settings not matched, given the importance of this comparison to the paper's central conclusions? Could the authors repeat the evaluations using identical document windows, token positions, and numerical settings, and extend the matched comparison across the context lengths and background conditions in Sections 3.2–3.3? This would help establish whether the reported growth of N∗N^* persists under renormalized attention.

2. Choice of loss-tolerance definition.

Sections 3.1–3.2 define effective attention set size using relative NLL degradation, whereas Section 3.3 uses a fixed 0.10-nat candidate-answer-loss increase. Could the authors clarify why the tolerance definition changes and why 0.10 nats was selected? Do the main context-length trends remain consistent under alternative absolute and relative tolerances, particularly given the changes in full-model answer loss across background conditions?

Suggestions for Further Experiments and Presentation

1. Disentangling useful information acquisition and attention competition.

Sections 3.2 and 3.3 suggest that increasing context length can simultaneously introduce additional useful information and increase competition among attention sources, but their respective contributions to the growth of N∗N^\ast remain unclear. Could the authors use the existing nested-context evaluation in Section 3.2 to investigate how newly introduced context tokens affect attention allocation and predictive performance? Specifically, it would be valuable to compare attention mass, contribution norms, Top-N inclusion frequencies, and ranking changes for newly introduced versus previously available context tokens while keeping the prediction targets fixed. Controlled interventions on the newly introduced context sources could further determine whether they provide useful predictive information or primarily displace previously relevant tokens. Relating these changes to target NLL and the growth of N∗N^* would provide a more direct mechanistic explanation of the observed context-length dependence, beyond the separate observations of improved language-modeling performance and increased competition in the controlled QA experiments.

2. Contribution ranking after the attention output projection.

Equation (2) defines contribution-based ranking using the norm of the weighted value vector, ∣αivi∣2|\alpha_i v_i|_2, before the attention output projection. Why is the contribution measured in the pre-projection value space rather than using the projected contribution αiviWO(h)\alpha_i v_i W_O^{(h)} written to the residual stream? The latter is more directly connected to the functional effect of an attention head and is commonly considered in attention-head circuit analyses, as in Elhage et al. (2021), A Mathematical Framework for Transformer Circuits. Since the output projection does not generally preserve vector norms, the two representations may induce different token rankings and need not reflect the same functional importance. Could the authors clarify the motivation for this choice and compare the two ranking criteria to assess whether output-projected contributions provide a more informative measure of token importance?

3. Sensitivity to intervention locations.

Section 3.2 evaluates NLL only on the final 128 target tokens while applying Top-N deletion at all query positions. Could the authors compare this setting with target-only interventions that leave earlier context-position attention unrestricted? This would help distinguish the attention requirements of target predictions from the effects of perturbing earlier contextual representations, which the existing final-target versus all-token scoring comparison does not isolate.

4. Interpretation of the geometry–loss relationship.

Figure 4 reports correlations between geometric F-scores and NLL degradation across varying Top-N budgets, while Figure 2 shows nonmonotonic geometric trends and Appendix C.2 demonstrates substantial sensitivity to excluding small N values. Could the authors clarify whether these correlations reflect anything beyond the shared dependence on N, and explain the nonmonotonic geometric behavior? Comparing geometric separation and functional sensitivity at fixed N across different heads, layers, or selection methods, potentially using the localized interventions in Appendix E.3, would provide stronger evidence for a meaningful geometry–function relationship.

5. Empirical validation of the theoretical assumptions.

The theoretical results largely characterize the consequences of assumed attention-score distributions and deletion-based aggregation, but their connection to actual Transformer attention statistics remains insufficiently established. Could the authors examine whether attention-score distributions and dependencies remain stable across context lengths, whether the value-vector norm and alignment assumptions are supported empirically, and how the selected and discarded value means μS\mu_S and μT\mu_T differ across heads and layers? In particular, testing whether retained attention mass predicts local output perturbations, and whether these perturbations relate to downstream NLL, would help assess the relevance of Propositions 4–6. Although Appendix E.4 finds that loss-sensitive diagnostics are more predictive of N∗N^* than geometric separation, it does not systematically validate the theoretical assumptions or explain the observed context-length scaling.

6. Presentation of contribution-ranking results.

The Attention Top-N and Contribution Top-N curves in Figure 3 appear nearly identical, making the reported improvement for Llama-2-7B difficult to identify visually. Could the authors present the quantitative comparisons from Appendix C.3 more explicitly in the main text, for example by reporting effective set sizes at fixed NLL tolerances? The reported reductions from 85 to 60 on OpenWebText and from 101 to 65 on WikiText-103 would provide clearer support for this conclusion.

Reviewer_wkG1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors discuss effective attention set size using self-attention. In particular, the research question authors consider is how many tokens from its context does language model actually use and what determines the cardinallity of the selected set? The authors retain highest attention weights at each head, layer and query to determine the effective attention set size. The selection of contextual tokens at each layer and head give important tokens locally but ignores the global usage of the contextual tokens for a llm perspective. The geometric comparison of selected tokens based on Mudarisov et. al provides an interesting properties of the tokens important from attention scores perspective.

Critical strengths

Strengths:

  1. The paper addresses an interesting perspective of importance contextual tokens in LLMs, and thus leads to effective token size based on the self-attention at layer, head and query level.

  2. The random token selection provides a good control to compare the selected tokens based on the change in language modeling loss.

  3. The deletion versus renormalization comparison is a useful ablation. It shows that the measured set size depends on how the retained values are aggregated, and not only on which tokens are ranked highest.

Critical weaknesses
  1. The authors study effective attention set size by independently selecting the top-N context positions at each query, attention head, and layer. However, it does not explain how contextual vectors are aggregated and propagated across different heads and layers in LLMs. In deeper layers, representations incorporate information from multiple context positions through preceding layers' attention. Figure 2 discusses geometric precision, recall, and the extremal separability score, which are averaged across heads and layers. Such averaging hides differences in attention behavior across layers and heads, particularly because deeper-layer attention operates on highly contextualized representations.

  2. The experiments in the paper support the local selected set sufficiency, but provide very few experiments to interpret the estimated set size as the overall retrieval capacity of the Transformer.

  3. The Figure 2 shows clear separation for geometric scores between randomly selected tokens and attention selected tokens. It is not clear why different models have geometric scores very close to each other.

  4. The paper will strengthened by discussing the results more clearly as provided in Figures 2, 3, and 4.

Questions for authors
  1. Prior approaches such as attention rollout and attention flow explicitly consider information propagation across layers and residual connections. A comparison with these methods, together with head- and layer-specific analyses, would help establish whether the observed local selection patterns correspond to meaningful retrieval of original context tokens.

  2. Why global selection of contextual tokens is not considered. How does the results change if we consider rollout/flow for attention aggregation[1] at layer level.

  3. Why different LLMs have geometric scores very close to each other in Figure 2. The discussion on this will clear how the local selection of contextual tokens affects overall model performance.

[1] Abnar & Zuidema, "Quantifying Attention Flow in Transformers", ACL 2020

Reviewer_r9y2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how many context tokens an LLM needs to maintain its prediction performance and how this number changes with context length. The authors propose a method that keeps only the top-N tokens based on attention weights or contribution magnitudes and measures the resulting increase in prediction loss without retraining. Experiments across multiple language models show that a relatively small number of tokens can preserve performance, but the required number generally increases as the context becomes longer. The authors further show that adding irrelevant background tokens makes it harder for attention to retrieve relevant information. They also find that renormalizing the remaining attention weights can reduce the number of required tokens.

Critical strengths
  1. A clear and useful framework for measuring effective attention set size. Instead of relying only on attention weights to identify important tokens, the authors directly measure how removing tokens affects model performance. Experiments across nine language models provide useful evidence that a relatively small number of tokens can preserve prediction performance.
  2. Well-designed experiments. The authors conduct controlled experiments by keeping prediction targets or supporting facts fixed while increasing context length. These experiments help explain how additional background tokens affect attention and information retrieval. The authors also investigate attention renormalization and show that it can greatly reduce the required number of tokens. These findings provide useful insights into how token competition and attention aggregation affect model performance.
Critical weaknesses

The proposed metric is highly sensitive to attention normalization. The main experiments remove low-ranked tokens without renormalizing the remaining attention weights. This changes the magnitude of the attention output, so the measured effective attention set size may reflect the loss of attention mass rather than the number of truly useful tokens. For example, renormalization reduces the required set size for Gemma-7B from 226 to only 7. However, the authors do not repeat their main context-length experiments with renormalization. This makes it difficult to determine whether the observed increase in required tokens is mainly caused by token competition or the chosen measurement method.

Questions for authors

Does the effective attention set size still increase with context length when the remaining attention weights are renormalized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.