acceptodds
Under review as a conference paper at ICLR 2027

Beyond Multiplicity Correction: Sharp Retrieval Thresholds for Attention with Correlated Distractors

Abstract

Redundant context affects attention in two distinct ways: multiplicity adds normalization mass, while score variation within correlated groups can produce extreme individual scores. We separate these effects in a planted retrieval model with exponentially many Gaussian distractors organized into supplied clusters whose number and size both grow exponentially with dimension. The model is a two-level generalized random energy model, whose classical free energy yields exact retrieval thresholds. Gaussian interpolation further shows that increasing positive correlation at fixed token count lowers the distractor free energy, unlike adding duplicates. Our main result establishes a sharp limitation of count correction, which subtracts the log cluster size from pre-softmax scores: this correction removes multiplicity but retains within-cluster extremes. Even when the inverse temperature is chosen after observing all scores, count correction requires a signal at least , where is the exponential growth rate of the number of clusters. By contrast, averaging scores before exponentiation suppresses token-level fluctuations and lowers the optimal signal threshold to , where is the within-cluster correlation. This gap persists after adding exponentially many singleton distractors, so group size alone does not identify the target, and extends, under explicit cumulant regularity assumptions, to fixed finite hierarchies of independent-increment scores. Finite-sample, stability, and participation results complement the asymptotic analysis. A controlled retrieval experiment with frozen sentence embeddings illustrates both sides of the distinction: averaging helps when the relevant item is a singleton, but can discard useful evidence when a relevant group contains multiple matching items.

Primary area:
learning theory
Keywords:
attention, long-context retrieval, correlated distractors, redundancy, softmax, phase transitions, generalized random energy model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8m17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

The paper studies single-step softmax retrieval with a planted relevant item and correlated distractors organized into supplied groups. In a hierarchical Gaussian score model, it compares ordinary attention, count correction, and averaging scores before exponentiation. Its main result establishes a sharp count-correction threshold that cannot be improved by choosing the temperature after observing the scores, whereas averaging attains a lower threshold under the stated target assumptions. The paper also develops finite-array, hierarchical, and non-Gaussian results and discusses relevant-group extensions. Gaussian simulations, frozen sentence-embedding retrieval, and an internal-attention intervention in frozen DistilBERT illustrate the distinction between the operators and their limitations.

Critical strengths
  1. A precise separation beyond a fixed-temperature comparison. The uniform failure result over data-dependent temperatures in Theorem 3 makes the separation in Theorem 5 more informative than the deterministic Jensen ordering alone. It establishes that, within the model, temperature tuning cannot remove the limitation caused by the within-group fluctuations retained by count correction. I consider this the central technical contribution.

  2. Clear attribution and informative experiments. The manuscript distinguishes its new claims from classical REM/GREM ingredients and clearly separates numerical checks of the theory from illustrations on learned scores. The sentence experiment includes validation-based temperature selection, paired uncertainty estimates, and multiple-relevant-item conditions. Together with the adverse downstream results in Appendix K, these cases help explain when averaging removes useful evidence and support a careful interpretation of the findings.

Critical weaknesses

My main concern is the significance and scope of the result. It has two related parts:

  1. Limited scope of the main comparison. The main separation leaves the relevant score unchanged while averaging removes within-group variation among distractors. Appendix H.2 permits identical-score relevant copies; under full correction, taking the relevant group to be as large as the distractor groups preserves the group-mass comparison. Thus, group-size clues are not essential to this mechanism. However, the relevant contribution remains unaffected by averaging in this construction. The exact thresholds quantify the benefit of suppressing distractor fluctuations, but their implications for balancing this benefit against the need to preserve useful relevant matches remain limited.

  2. Limited empirical support for broader implications. The sentence experiment demonstrates a meaningful calibrated benefit, increasing mean relevant mass from 0.342 with count correction to 0.440 with averaging. However, the comparison reverses with four relevant keys: 0.626 for count correction versus 0.465 for averaging (Table 1). Appendix K further shows that increasing relevant attention mass need not improve correct-candidate probability. These results help describe the operators' behavior, but they leave the broader importance of the sharp separation unclear to me.

These observations are consistent with the paper's stated limitations and do not undermine its soundness. I also recognize that a theoretical contribution need not demonstrate downstream performance gains. Nevertheless, despite the technical value of the result covering all temperature choices, I am not yet persuaded that this specialized separation provides a sufficiently substantial advance in understanding the operator tradeoff to warrant acceptance.

Questions for authors
  1. Clarification of the conceptual contribution. Theorem 3 establishes a stronger conclusion than the same-temperature Jensen ordering, while Appendices J and K illustrate both benefits and failures of averaging. Could the authors clarify which additional insight from the sharp thresholds and the result covering all temperature choices is most important for understanding the tradeoff between suppressing distractor fluctuations and preserving useful relevant matches? In particular, how should this insight be interpreted when relevant groups contain different useful matches, beyond the identical-copy setting of Appendix H.2?

An explanation grounded in the existing analysis or reported results would help me reassess the significance of the contribution. This is a clarification request, rather than a request for a new general theorem or a large-model training study.

Reviewer_gYkmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies retrieval of an informative key from correlated Gaussian distractors using a single attention layer. The distractors are organized into clusters, with both the number of clusters and the number of keys per cluster growing exponentially with the embedding dimension dd. Assuming the informative key's query–key score scales as sdsd, the authors derive a retrieval threshold h(β)h(\beta) for ss as d→∞d\to\infty, where β\beta is the inverse temperature. The informative key receives asymptotically all attention when s>h(β)s>h(\beta) and vanishing attention when s<h(β)s<h(\beta). They compare these thresholds for standard attention and two modifications: count correction, which divides each distractor token’s unnormalized weight by its cluster size, and score averaging, which assigns each cluster a single weight by exponentiating its mean score.

Critical strengths

None

Critical weaknesses

Although the question may interest some researchers in theory, the presentation is difficult for human to follow and requires substantial revision to meet publication standards. Beyond the analysis based on standard attention, the settings considered are insufficiently motivated, and their relevance to practical problems is unclear. Given its emphasis and scope, the paper may be better suited to a more specialized theoretical venue.

Questions for authors

I don't have any question.

Reviewer_YQqoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies softmax attention retrieval with correlated distractors using a planted retrieval model. It compares ordinary attention, count correction, and score averaging, deriving sharp retrieval thresholds for the latter two methods. The main result shows that score averaging achieves a lower retrieval threshold than count correction, even when the latter uses an adaptively selected temperature. A controlled retrieval experiment validates the theoretical findings.

Critical strengths

The paper provides both theoretical analysis and empirical validation.

Critical weaknesses

My main concern is that the setting is highly idealized, and its connection to attention mechanisms in real Transformers is not sufficiently justified.

First, the paper assumes a planted relevant item whose attention logit satisfies X∗/d→sX_*/d \to s, while all distractor scores follow a mixed Gaussian distribution. Such a direct assumption on the attention logit is highly simplified and should be justified at least with some experimental results. It is unclear how such a score distribution would arise from actual query and key representations.

Similarly, the paper assumes that both the number of distractor clusters and the number of tokens within each cluster grow exponentially with the embedding dimension. Why is exponential growth chosen, and how does this scaling relate to realistic Transformer architectures and context lengths?

Since the subsequent theoretical results rely on these assumptions, I believe it is important for the authors to justify the relevance of these assumptions to practical attention mechanisms. Otherwise, the results just offer a mathematical characterization of "a softmax function under an artificially constructed random clustering model".

Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.