- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates how many context tokens need to be retained in self-attention to preserve language-model prediction performance and how this requirement changes with context length. The authors introduce an operational measure of effective attention set size, , based on Top-N selection of tokens ranked by attention weights or weighted-value contribution norms, with functional sufficiency evaluated through NLL degradation. They examine the geometric separation of selected and unselected contributions, compare attention-based selection with random baselines across pretrained language models, and show that the estimated required set size generally increases with context length. To investigate attention competition, they conduct controlled BABILong experiments that keep annotated supporting information fixed while increasing background length, finding that additional context displaces supporting tokens in attention rankings and reduces their attention mass. The paper further examines the effects of renormalizing retained attention weights and presents conditional theoretical analyses of how ranking competition and attention-mass preservation can produce increasing effective set sizes.
Critical strengths
1. Controlled investigation of attention competition with fixed supporting information
The controlled retrieval experiments in Section 3.3 provide a useful complement to the natural-language context-length experiments. By keeping the annotated supporting fact fixed while increasing background length, the authors reduce the confounding effect of additional task-relevant information and examine how competition affects the ranking, attention mass, and retention of supporting tokens. This experimental design provides more direct evidence for the role of attention competition than context-length scaling measurements alone.
2. Functional evaluation of attention selection and the role of normalization.
The paper goes beyond descriptive attention statistics by systematically intervening on selected attention contributions and measuring the resulting degradation in model predictions across multiple pretrained language models. The additional comparison between deletion and renormalized attention is particularly valuable, as it demonstrates that effective attention set size depends not only on token selection but also on how retained contributions are aggregated. The accompanying local error analysis provides a clear mathematical explanation for this dependence.
Critical weaknesses
1. Strong dependence on the intervention design and insufficient attribution of context-length effects
The central empirical finding that effective attention set size increases with context length depends strongly on the specific intervention used to define it. Sections 3.1–3.3 primarily employ post-softmax deletion without renormalizing the retained attention weights, which differs from standard sparse attention masking, where the retained weights are normalized over the selected tokens. The authors themselves show in Section 3.4 and Appendix E.1 that renormalization can dramatically reduce the required set size and even eliminate the observed growth across background conditions in some settings. For example, in BABILong, Gemma-7B's estimated set size increases from 64 to 760 under deletion but remains at 32 under renormalization. Although the paper acknowledges this intervention dependence, the main context-length results are still based on deletion, while renormalized interventions are not evaluated as systematically across context lengths and models. Furthermore, the language-modeling deletion and renormalization comparisons in Appendix E.1 do not have verified document-level pairing or identical numerical configurations. Consequently, it remains unclear to what extent the reported growth in effective attention set size reflects intrinsic retrieval requirements rather than sensitivity to the particular deletion-based aggregation procedure.
More broadly, the paper does not sufficiently disentangle the mechanisms underlying this growth. The authors identify several possible factors, including the acquisition of additional useful information, increased ranking competition, and sensitivity to discarded attention contributions. Section 3.2 shows that longer natural contexts improve full-model predictions while increasing the required set size, whereas Section 3.3 uses fixed annotated supporting facts to provide evidence of increased competition. However, the experiments do not directly investigate how these mechanisms contribute to the observed growth in the same setting. In particular, the natural-language modeling experiments do not analyze whether newly introduced context tokens provide useful predictive information, displace previously relevant tokens in attention rankings, or primarily affect the retained attention mass. The resulting interpretation of context-length dependence therefore remains largely descriptive, without sufficiently isolating the mechanisms responsible for the observed effects.
2. Limited connection between geometric, theoretical, and functional analyses.
The relationship between the geometric properties of selected attention contributions and their functional importance remains insufficiently established. Figure 2 reports stronger geometric separation for attention-selected sets than for random sets, but the geometric metrics exhibit nonmonotonic behavior as the selected-set size increases. Figure 4 then reports correlations between geometric F-scores and NLL degradation across varying Top-N budgets. Since both quantities depend on N, these correlations may largely reflect their shared dependence on the selection budget rather than a meaningful geometry–function relationship. Appendix C.2 further shows that excluding small N values substantially weakens or even reverses several correlations. More importantly, it remains unclear how the geometric separation analysis in Section 3.1 contributes to understanding the context-length dependence and attention competition investigated in Sections 3.2–3.3. The paper does not establish whether these geometric properties help explain or predict the observed changes in effective attention set size as context length increases, leaving the geometric analysis relatively disconnected from the central empirical findings.
The theoretical analysis is also primarily conditional on idealized assumptions whose empirical relevance remains insufficiently examined. Appendix F characterizes how ranking competition and attention-mass retention can produce growing selected-set requirements under assumptions about attention-score distributions, score dependence, and value-vector properties. However, the paper does not systematically evaluate whether these assumptions hold in pretrained Transformers or whether the derived mechanisms quantitatively explain the observed context-length trends. Although Appendix E.4 provides an exploratory analysis of discarded-vector statistics and finds that loss-sensitive diagnostics predict effective set size better than geometric separation, this analysis is limited to a fixed context length and does not directly validate the theoretical assumptions. Consequently, the geometric and theoretical analyses remain insufficiently connected to the observed functional retrieval behavior.
Questions for authors
Clarification Requests
1. Consistency of deletion and renormalization evaluations.
Appendix E.1 acknowledges that the deletion and renormalization evaluations do not have verified document-level pairing or identical numerical configurations. Why were these settings not matched, given the importance of this comparison to the paper's central conclusions? Could the authors repeat the evaluations using identical document windows, token positions, and numerical settings, and extend the matched comparison across the context lengths and background conditions in Sections 3.2–3.3? This would help establish whether the reported growth of persists under renormalized attention.
2. Choice of loss-tolerance definition.
Sections 3.1–3.2 define effective attention set size using relative NLL degradation, whereas Section 3.3 uses a fixed 0.10-nat candidate-answer-loss increase. Could the authors clarify why the tolerance definition changes and why 0.10 nats was selected? Do the main context-length trends remain consistent under alternative absolute and relative tolerances, particularly given the changes in full-model answer loss across background conditions?
Suggestions for Further Experiments and Presentation
1. Disentangling useful information acquisition and attention competition.
Sections 3.2 and 3.3 suggest that increasing context length can simultaneously introduce additional useful information and increase competition among attention sources, but their respective contributions to the growth of remain unclear. Could the authors use the existing nested-context evaluation in Section 3.2 to investigate how newly introduced context tokens affect attention allocation and predictive performance? Specifically, it would be valuable to compare attention mass, contribution norms, Top-N inclusion frequencies, and ranking changes for newly introduced versus previously available context tokens while keeping the prediction targets fixed. Controlled interventions on the newly introduced context sources could further determine whether they provide useful predictive information or primarily displace previously relevant tokens. Relating these changes to target NLL and the growth of would provide a more direct mechanistic explanation of the observed context-length dependence, beyond the separate observations of improved language-modeling performance and increased competition in the controlled QA experiments.
2. Contribution ranking after the attention output projection.
Equation (2) defines contribution-based ranking using the norm of the weighted value vector, , before the attention output projection. Why is the contribution measured in the pre-projection value space rather than using the projected contribution written to the residual stream? The latter is more directly connected to the functional effect of an attention head and is commonly considered in attention-head circuit analyses, as in Elhage et al. (2021), A Mathematical Framework for Transformer Circuits. Since the output projection does not generally preserve vector norms, the two representations may induce different token rankings and need not reflect the same functional importance. Could the authors clarify the motivation for this choice and compare the two ranking criteria to assess whether output-projected contributions provide a more informative measure of token importance?
3. Sensitivity to intervention locations.
Section 3.2 evaluates NLL only on the final 128 target tokens while applying Top-N deletion at all query positions. Could the authors compare this setting with target-only interventions that leave earlier context-position attention unrestricted? This would help distinguish the attention requirements of target predictions from the effects of perturbing earlier contextual representations, which the existing final-target versus all-token scoring comparison does not isolate.
4. Interpretation of the geometry–loss relationship.
Figure 4 reports correlations between geometric F-scores and NLL degradation across varying Top-N budgets, while Figure 2 shows nonmonotonic geometric trends and Appendix C.2 demonstrates substantial sensitivity to excluding small N values. Could the authors clarify whether these correlations reflect anything beyond the shared dependence on N, and explain the nonmonotonic geometric behavior? Comparing geometric separation and functional sensitivity at fixed N across different heads, layers, or selection methods, potentially using the localized interventions in Appendix E.3, would provide stronger evidence for a meaningful geometry–function relationship.
5. Empirical validation of the theoretical assumptions.
The theoretical results largely characterize the consequences of assumed attention-score distributions and deletion-based aggregation, but their connection to actual Transformer attention statistics remains insufficiently established. Could the authors examine whether attention-score distributions and dependencies remain stable across context lengths, whether the value-vector norm and alignment assumptions are supported empirically, and how the selected and discarded value means and differ across heads and layers? In particular, testing whether retained attention mass predicts local output perturbations, and whether these perturbations relate to downstream NLL, would help assess the relevance of Propositions 4–6. Although Appendix E.4 finds that loss-sensitive diagnostics are more predictive of than geometric separation, it does not systematically validate the theoretical assumptions or explain the observed context-length scaling.
6. Presentation of contribution-ranking results.
The Attention Top-N and Contribution Top-N curves in Figure 3 appear nearly identical, making the reported improvement for Llama-2-7B difficult to identify visually. Could the authors present the quantitative comparisons from Appendix C.3 more explicitly in the main text, for example by reporting effective set sizes at fixed NLL tolerances? The reported reductions from 85 to 60 on OpenWebText and from 101 to 65 on WikiText-103 would provide clearer support for this conclusion.