- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper formalizes performance degradation in long-context LLMs as an extreme-value interference problem in attention. By deriving a theoretical upper bound for context poisoning, the paper claims to prove that maintaining retrieval accuracy requires the decisive-evidence margin to scale as . Based on this, the authors propose several architectural suggestions and evaluate the effects of distractor count and confusability on model recall and reasoning through controlled experiments.
Critical strengths
- This paper introduces extreme-value theory to mathematically model attention competition. This perspective shifts the interference source from average noise to"maximum extreme-value noise.
- Comparing unrelated filler, same-domain, same-format, and near-alias distractors confirms that score aliasing induced by structural overlap is more damaging than raw context length.
Critical weaknesses
- Despite the novel angle, the paper suffers from serious internal contradictions and logical gaps. The weaknesses focus primarily on Eq.(7), whose bound rests on multiple strong assumptions (e.g., Gaussian-distributed distractor scores, and a quantifiable, constant relationship between evidence attention and answer correctness.):
1.1 The Gaussian i.i.d. assumption, the paper's own strong premise, is falsified by the paper's own experiments, causing the proof to collapse. It shows Shapiro-Wilk rejects normality, Ljung-Box rejects independence.
1.2 ρ may be a linear or nonlinear function rather than a fixed value. The cited Du et al. (2025) shows that even with perfect retrieval, long context alone harms performance, implying ρ decreases with N. If so, treating ρ as a constant ignores the core difficulty the paper itself cites.
1.3 An internal contradiction exists between ρ and . The bound assumes attention high score = correct, which is unproven. But the introduction of ρ (ranging from 0 to 1) shows high score correct, so the causal chain may or may not hold. Yet assumes it holds deterministically. The two assumptions directly contradict.
1.4 The theorem assumes one decisive evidence and one attention head. Real multi-hop tasks require aggregating scattered evidence, and multi-head attention allows cross-head compensation. The constraint lacks corollaries in these settings.
- The paper only offers three high-level suggestions, none of which are implemented or empirically validated. The paper's substantive engineering contribution is minimal.
- Gemma 3 (12B) has a 96.1% API error rate (72 valid out of 1,840 calls). Claude has 35.4%. Failures concentrate on the hardest, longest queries, so the surviving data systematically under-represent the hardest cases. Core conclusions rest on highly truncated data.
- It uses conventional BM25 retrieval gating. At 512K, the gain is only +0.070 EM on 100 questions, with p = 0.092, not statistically significant.
Questions for authors
None
We thank the reviewer for identifying assumptions whose formal role and empirical interpretation should be separated more clearly. We respectfully disagree that the appendix diagnostics cause the proof to collapse.
Gaussianity and independence. Theorem 1 is a conditional mathematical result and remains valid under its stated assumptions. The appendix does not observe the raw logits Zi in Eq. (5): it analyzes the logarithm of sentence-aggregated post-softmax attention after standardization. Because this statistic includes softmax normalization, sentence aggregation, position effects, and pooling across heads and layers, its non-Gaussianity is not a direct falsification of Eq. (5). We will correct the appendix sentence saying that this analysis tests the assumption "directly."
We also note that the Ljung-Box test does not reject independence in every setting; for example, the reported Gemma values are p=0.958 and p=0.320 at N=20 and N=50. The measured statistic nevertheless departs substantially from the Gaussian i.i.d. abstraction and should be treated only as a diagnostic.
The first step of the existing proof can be stated without Gaussianity or independence. Let
\qquad b=\gamma+\tau\log((1-\rho)/\rho).$$ Under $S_\star\le\gamma$ and the same one-sided faithfulness condition, $$A_N\le a_0+(1-a_0)\Pr(M_N\le b).$$ The i.i.d. Gaussian expression $\Phi(b/\sigma)^N$ is a closed-form specialization that yields the stated extreme-value scale. We will present the result in this order and describe the Gaussian case as an interpretable specialization, not a calibrated model of all transformer heads. **Meaning of $\rho$.** There is no assumption that high attention guarantees correctness. The condition is one-sided: $$\Pr(\hat y=y^\star\mid \alpha_{\star,N}<\rho)\le a_0.$$ For $\alpha_{\star,N}\ge\rho$, the proof uses only the trivial upper bound of one on correctness. Thus downstream reasoning may fail even when evidence attention is high, and there is no contradiction. Fixed $\rho$ is a uniform modeling assumption. If $\rho$ is replaced by $\rho_N$, the bound applies pointwise with the corresponding threshold, but the fixed-$\rho$ asymptotic need not follow; we will state this explicitly. Perfect retrieval alone does not imply a particular functional form for $\rho_N$, because evidence presence, attention allocation, and downstream use are distinct. **Scope and empirical role.** The theorem intentionally analyzes one evidence-selection bottleneck with one decisive item. It does not claim to model cross-head compensation or general multi-hop aggregation; these are limitations of scope, not contradictions in the stated result. The multi-hop gate experiment tests only the qualitative candidate-reduction/recall tradeoff. The endpoint failures limit the Claude and Gemma ablations, but they are not premises of Theorem 1. GPT-4.1 completed the fixed-length study, and the separate 700-pair gate experiment had zero endpoint errors. We will nevertheless report those results conservatively: the strongest gate effect is not statistically conclusive. The proposed losses and architectures will be presented as theory-derived implications and future directions, not as validated engineering contributions.