acceptodds
Under review as a conference paper at ICLR 2027

What Does Attention Retain? Information Loss and Sufficient Moments for In-Context Regression

Abstract

Success on Gaussian in-context regression can conceal information loss in a context representation. We show that, for any fixed moment budget, smooth noise laws can remain arbitrarily close to Gaussian and match its moments through twice the prescribed degree exactly, yet make the best decoder of that budget arbitrarily asymptotically inefficient. Each noise law is fixed before the sample-size limit. The mechanism is an exact characterization of the statistical experiment retained by complete mixed moments of covariates and labels: for centered isotropic absolutely continuous designs under the stated conditions, its local information is the squared norm of the noise score projected onto polynomials of one lower degree. This strong experiment limit covers every measurable decoder. Full local information is retained exactly for polynomial log noise densities of admissible degree; there, the same moments are sufficient at every sample size. Common-source affine attention realizes the degree-two boundary, while a scalar source-side score operation reaches the full-data endpoint. We connect the theory to standard causal softmax Transformers using independent risk evaluations and a task-preserving equal-Gram diagnostic. Reversible model drift witnesses a finite-context Bayes penalty for every Gram decoder, independently of a fitted control's quality. Fresh evaluations give positive numerical witnesses under Laplace and quartic noise, with a Gaussian negative control. The results separate computational performance from the statistical adequacy of a context representation.

Primary area:
learning theory
Keywords:
in-context learning, Transformers, learning theory, statistical sufficiency, Fisher information, moment compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MkpDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how much information is lost when an in-context regression prompt is compressed into low-order moments, and how this loss limits prediction accuracy. It then shows that given raw data, affine attention models can only utilize gram information, and that a source operation can overcome this limitation.

Critical strengths

The paper studies a meaningful question: what information an attention layer must retain for accurate in-context regression prediction. It characterizes the information retained by finite-degree moments in Theorem 3.1 and establishes when these moments are loseless for estimating the regression coefficient in Theorem 3.2.

Critical weaknesses

My major concern is that the writing of this paper needs substantial improvement. The paper lacks a clear logic flow, and there are some notations and non-standard terminologies that authors used without prior definition or proper reference, which makes it difficult to follow. For example, in eq.(2), it is unclear to me what is the meaning of RT∗R_T^{*} and Rfull∗R_{full}^{*}. In eq.(3), I could not find the definitions of σ2\sigma^2 and IfI_f until line 254 and line 273. In line 190 it is confusing what the author meant by "hierarchy" and "classification". Similar issues exist throughout the paper. For basic readability, the authors should at least clearly define all notations and non-standard technical terms upon first use, and refer readers to the equations where appropriate.

Besides clarity issues, the contribution of this paper appears limited. On transformer side, the major claim of this paper is that linear attention layers can only utilize Gram information of the prompt, which is not surprising; the proposed source operation seems tailored to the specific in-context regression setting rather than a general approach, and I did not see the benefits of this approach over simply providing higher-order moments to the model. Also, the connection between the theoretical results in section 4 and the experiments in section 5 is not clear to me.

Questions for authors
  1. Please see the weakness part. The whole paper needs a substantial revision to be readable.
  2. In line 285, what exactly does "source operation" refer to? Can you provide a concrete formula and explain how can it be applied in practice, and how does it relate to the experiments in Section 5?
  3. What is the advantage of this source operation over processing the raw data directly, such as providing higher-order moments or other features to the model? Also, can this approach be generalized to other settings beyond in-context regression?
Reviewer_r6vKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the information lost in in-context regression when an attention memory discards higher-order moments of the prompt. The authors analyze complete degree-PP moment summaries, construct near-Gaussian noise distributions for which moment-based estimators lose arbitrarily much Fisher information, and show that a class of affine attention models depends on the prompt only through its degree-two summary, the augmented Gram matrix. Limited experiments are provided to illustrate these effects, but they do not constitute significant evidence for the claims.

Critical strengths

The distinction between the information discarded by a representation and the computational limitations of its decoder is interesting. Extending the results to arbitrary measurable decoders and the exact moment matching of the near-Gaussian construction could be core contributions.

Nevertheless, the relevance of the work is hard to assess in its current presentation. Because of the poor positioning, it is also hard to see how the results apply to standard architectures, or whether the main information-theoretic results are new compared with classical ones.

Critical weaknesses

1. Clarity. The manuscript lacks clarity to a degree that requires a substantial rewrite. This is my main concern, and it affects my assessment of every other aspect of the paper. This is a list of examples:

  • Introduction. Even after the first two pages, I found it difficult to understand what the authors claim about attention and how the contributions fit together. The introduction starts with a clear question but quickly becomes a dense sequence of results that are nearly impossible to understand without first reading the rest of the paper. Terms such as "strong limit of the retained statistical experiment," "common-source affine attention," "full-data efficiency" and "reversible model drift" appear before they are given any meaning.
  • Figure 1. The figure gives little explanation of the setting and already uses IfI_f and J4J_4 before they are defined.
  • Problem setup (Section 3.1). The same issue persists beyond the opening pages. When the task is introduced on page 3, it is not stated what π\pi is in the data model, whether the outputs satisfy Yi∈RY_i \in \mathbb{R}, or what the shape of WW is. The quantities RT⋆R^\star_T and Rfull⋆R^\star_{\mathrm{full}} are used without definition. Many similar omissions make the text hard to follow for a human reader.
  • Section 3.2. The section moves from the empirical moments of (X,Y)(X, Y) to vP(e)v_P(e), BPB_P and aPa_P, and then states a theorem involving JPJ_P and the Le Cam distance without preparation. The role of ee is left implicit. More importantly, there is no explanation of why these quantities capture the information retained by the prompt moments. The reader has to hold this connection in mind while moving back and forth to the appendix and deciphering the notation. The explanations are too compressed.

In my view, these presentation issues alone warrant rejection, and they cannot be resolved within the discussion period without a rewrite that would amount to a different manuscript.

2. Reliance on the appendices. Important assumptions and theorems are deferred to the appendix. For example, Section 4 invokes the conditions of Theorems F.4 and F.7 without stating them. The experimental discussion repeatedly directs the reader to supplementary sections, and the explanations given in the main text carry little weight on their own. The main text should provide enough explanation and assumptions to assess the central claims independently.

3. Connection between the theory and attention architectures. The link between the classical statistical results and attention is poorly developed. The architectural results are restricted to an affine class that excludes, for example, softmax and causal masks. The experiments on standard Transformers therefore lie outside the scope of the theorems, yet they are presented as evidence for them.

4. Strength of the experimental evidence. The experiments do not support the empirical claims of the abstract.

  • The 26.6% Laplace improvement in the abstract holds at N=128N = 128, the largest training length (Table 1). At N=256N = 256 the ordering reverses under every noise law (Tables 5 and 7).
  • Under Gaussian noise, where ridge is Bayes-optimal, the Transformer is worse than ridge at every evaluated length (Tables 1, 4 and 5), and its risk was still decreasing at the final training budget (Appendix D.3). This suggests undertraining, and makes it difficult to attribute the Laplace gain to information retention rather than to optimization effects.
Questions for authors
  1. The AI use statement says that proofs, computations, references and claims underwent "targeted checks." Which results, experiments and manuscript sections did these checks cover, and what kind of human verification did they involve?
  2. What finite-NN risks are obtained for the full-data and moment-restricted estimators in the near-Gaussian construction?
  3. All results fix dd as N→∞N \to \infty. What survives in the proportional regime, d/N=O(1)d/N = O(1)?
Reviewer_WNv6OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the in-context linear regression via the information perspective. It focuses on 1) what is the best achievable prediction accuracy when a decoder has access only to the summary of the task; 2) which source-side features can be extracted before aggregation to overcome the limitation of the summary. This study provides a new criterion for evaluating attention memories.

Critical strengths
  • The paper clearly distinguishes information lost through compression from computational limitations in using the retained information, which offers a new problem formulation.
  • The experiments use independent training seeds, separate evaluation and replication datasets, retain rejected transitions and negative contributions, and report both failures in longer-context extrapolation and statistically inconclusive results. Particularly valuable is the acknowledgment that, under quartic noise, using additional predictive information does not necessarily translate into a clear improvement in overall prediction risk.
Critical weaknesses
  • Although the paper offers theoretical insights into information retention, its empirical evaluation is restricted to low-dimensional synthetic regression tasks, with no validation on real-world datasets. It should be discussed that how the insights in the paper can be implemented in the studies on real-world models and data.
Questions for authors
  • Is the approach in this study irrelevant to the specific in-context task (i.e., linear regression in this study)? Can it be extended to analyze other tasks such as IV regression?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.