acceptodds
Under review as a conference paper at ICLR 2027

In-context classification: Calibration and noise in linear attention with softmax readout

21 views

Abstract

We study in-context binary classification in a simple attention model trained to minimise a logistic loss. Each prompt has a latent logistic teacher, and the learner must infer its direction from Bernoulli-labeled context examples. We derive an asymptotic prediction for the test logit in a high-dimensional regime with context length and total training prompts . Our formula cleanly separates calibration, test-time context noise, and non-isotropic parameter-estimation noise as interpretable effects on in-context performance. Our analysis also identifies the ridgeless separability regime and addresses the effects of context length, pretraining-set size, regularisation, and sampling temperature on in-context classification performance. The theory's predictions closely agree with finite-dimensional simulations and fully trained linear-attention models, while experiments with standard softmax attention show qualitatively related behaviour beyond the regime assumed by our analysis.

Primary area:
learning theory
Keywords:
In-context learning, transformers, classification, logistic regression, cavity method, exact formula, random matrix theory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sXy6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper develops a high-dimensional theory of in-context binary classification using linear attention with a softmax readout. It characterizes the test logit distribution as a scaled teacher signal plus noise arising from finite context length and anisotropic parameter estimation. The theory predicts classification error, cross-entropy loss, and a separability threshold for ridgeless training. As far as I understand, this analysis relies on heuristic cavity approximations without rigorous justification.

Critical strengths

I found this work interesting as an extension of studies on proportional high-dimensional statistical theory (e.g., Sur & Candès, 2019) to the ICL setting.

Critical weaknesses
  1. I feel that the theoretical analysis is somewhat incomplete. As this is a learning theory work, I believe the theoretical results are the most crucial part. However, many (rough) approximations are used in the proofs, and I do not think the reliability of these results is very guaranteed. While there are approaches that are not fully mathematically rigorous, such as the cavity approach or the replica method, I consider it is still necessary to clarify what is mathematically established and what is not.

  2. In my view, the motivation and contributions of this paper are somewhat obscure.

  • First, Liu et al. (2025) analyze the excess risk and optimal rates for conditional probability estimation in multi-class ICL. While the theoretical framework differs, the authors do not articulate their exact differentiation from Liu et al. when contextualizing prior work on classification ICL. Consequently, the claims regarding motivation and contributions in the Introduction appear somewhat overstated.
  • Furthermore, the core idea of decomposing the prediction of trained Linear Attention into a signal component and multiple noise terms is already known in prior work (e.g., Takanami et al., 2026, although it is a regression setting).

So, I consider the motivation is ambiguous and this paper does not offer significant new insights.


Liu et al. (2025). In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality. arXiv:2508.08673.

Takanami et al. (2026). Learning Linear Regression with Low-Rank Tasks In-Context. AISTATS 2026.

Questions for authors
  1. To improve the presentation and soundness scores as well as the rate, could you clarify the motivation and contributions?

  2. Regarding the wording: since 'calibration' is a technical term in statistics and ML community that specifically means 'the predicted probabilities match the true probabilities,' I think it would be better to use alternative phrasing. Could you consider this?

3 comments
AuthorsOpenReview ↗

Response to reviewer sXy6

Weakness 1: what is rigorously vs numerically established?

This is a fair point, thank you for raising this. We detail the status of our claims as follows.

  • Numerically established: the concentration of a^0,a^1,a^2\hat{a}_0, \hat{a}_1, \hat{a}_2. See Appendix A, Figure 9.
  • Numerically established: the application of past results to derive the separability boundary τc\tau_c. See Appendix C, Figure 11.

We have adjusted the language in our work to reflect that these arguments are heuristic, and our results are numerically-justified conjectures.

Weakness 2: obscure motivation and contributions

Thanks for identifying these papers and bringing them to our attention. We acknowledge similarities and differences as follows:

Liu et al. establish risk bounds and rates for multiclass conditional-probability estimation using more realistic and more complex architectures. Our work addresses a complementary question in, granted, a simpler model: how finite pretraining data and context length shape the joint distribution of the predicted and teacher logits. This characterisation allows us to calculate the conditional class probability P(Y=1∣R=r)P(Y=1\mid R=r) and compare it with the model’s prediction σ(r)\sigma(r). The additional calibration analysis makes this distinction explicit by examining how signal scaling and noise jointly produce underconfidence or overconfidence.

We also certainly agree that decomposing linear-attention predictions into signal and noise is not a novel principle. In particular, Takanami et al. show an algorithmic signal and both memorisation-based and structure-dependent noise in their low-rank regression setting. Our contribution is to derive a corresponding characterisation for classification with logistic training, and examine its consequences for probability calibration. We have revised the related-work discussion and contribution statements to acknowledge these.

Question 1: clarifying motivation and contributions

Our main goal is to understand how a simple attention-based model can learn a distribution, and how finite-data affects the distribution (and its calibration with the true logit distribution) that has been learned. We view this as a highly relevant question: many other works have shown that the ability of models to learn ICL depends heavily on the properties of the pretraining data, as both context length and pretraining example sequences affect the model's ability to both distinguish tasks and correctly learn the task distribution. In the case of a binary classification setting, we expect the probabilistic nature of the task to be especially interesting in this question. Indeed, we find precisely that good calibration can occur even when the model retains high classification error, due to different effects of the data finiteness: we discuss this further below. We view this as a valuable contribution that would allow other works to build on our framework, to tackle questions about train-test alignment, task diversity, or scaling laws.

Question 2: technical use of the term calibration

This is a great point, and we thank the reviewer for giving us the opportunity to add further results here. Since we have derived the distribution of the output logit RR jointly with the true logit GtrueG_\text{true}, we can write the conditional distribution for Gtrue∣RG_\text{true} \mid R and use this to find the probability that the true label is "1" given a particular value of RR. Namely, we can easily write the curve p(R)=P(Y=1∣R)p(R) = \mathbb{P}(Y=1 \mid R ). Now if the model is calibrated, in the true statistical sense of the word, then we would have p(R)=σ(R)p(R) = \sigma(R). We can thus plot the discrepancy between p(R)p(R) and σ(R)\sigma(R) against RR for a variety of data configurations α,τ\alpha,\tau. We have additional results about that have been added to the updated draft, see Figure 4.

Thank you for your response. But did you update your manuscript? I cannot see your revision so far.

AuthorsOpenReview ↗

Resubmitted revised version

Hi thanks for responding! We annotated our main changes in blue; please let us know if it's still not visible?

Reviewer_z4iCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies in-context binary classification using linear attention with a softmax readout. It derives a high-dimensional characterization of the predicted logit distribution, decomposing it into a scaled teacher signal, test-time context noise, and parameter-estimation anisotropy. The analysis provides expressions for classification error and cross-entropy loss and examines the effects of context length, pretraining data size, regularization, and sampling temperature. It also identifies a separability threshold where longer training contexts may worsen generalization. The theoretical predictions are validated through numerical simulations and comparisons with fully trained linear-attention models, while additional experiments with standard softmax attention explore their applicability beyond the reduced model.

Critical strengths
  1. The paper presents a systematic and technically nontrivial theoretical analysis of in-context binary classification. The theoretical framework provides quantitative insights into the effects of context length, pretraining data size, and regularization, going beyond purely qualitative observations.
  2. The paper adapts cavity-based analysis to the structured features arising from in-context classification with linear attention. It derives self-consistency equations characterizing the learned parameters and uses them to obtain quantitative predictions of the test logit distribution and classification errors. This represents a meaningful technical contribution beyond a straightforward application of standard logistic regression theory.
  3. The paper provides numerical simulations across different parameter settings, demonstrating close agreement between the predicted logit distributions and classification errors and the observed results.
Critical weaknesses
  1. Many of the paper's qualitative conclusions are consistent with established intuitions from statistical learning theory. For example, longer test-time contexts reduce sampling noise, more pretraining data mitigates parameter-estimation noise, and separability can cause instability in unregularized logistic regression. While the quantitative characterization is valuable, the paper could better highlight novel insights that go beyond these expected behaviors.
  2. While the numerical simulations provide support for the theoretical predictions, given the broader scope of ICLR, additional experiments on real-world classification datasets would strengthen the paper by demonstrating the practical relevance and applicability of the theoretical findings.
  3. While the paper acknowledges qualitative discrepancies between standard softmax attention and the linear-attention theory in Figure 6, it would be beneficial if the authors could provide further insights into the underlying mechanisms responsible for these differences, particularly the observed discrepancies in logit variance and calibration.
Questions for authors

Result 2 relies on a heuristic extension of Koriyama & Bellec (2025) to structured, non-Gaussian features. Can the authors provide further justification for this transfer argument, or characterize the conditions under which the predicted separability threshold remains valid?

3 comments
AuthorsOpenReview ↗

Response to reviewer z4iC: Weaknesses

Weakness 1: unclear novelty and positioning

This is a fair point. To address this, we have added recent results, relying on our asymptotical exact characterisation of the joint distribution between the true logit and the model logit, that allow us to discuss calibration: what is the probability that the true label is 1, given a value of model logit RR? Is this σ(R)\sigma(R)? Are we under- or overconfident? We find that the different data-effects, namely context-length rescaling and batch-size isotropy, can interact nontrivially, allowing for data configurations where the logit is seemingly well-calibrated despite having high classification error. This unintuitive behaviour (calibration first worsening as loss improves) has been observed in non-theoretical settings, such as https://arxiv.org/pdf/2312.04021. Additionally, we show numerically that almost-perfect calibration is achievable with finite α\alpha (context length ratio) and τ\tau (sequence ratio) in Figure 5, and provide a theoretical suggestion as to what the optimal scaling should be in Result 4.

Weakness 2: real-world settings for classification

We agree that this would be an interesting extension of this work, and for that reason we have spent more time in the introduction discussing how our results relate to empirical findings by others. For the purposes of a clear narrative, we feel it is better to save additional experiments to a separate work. The goal of this work is primarily theoretical: we have generalised the theoretical study of ICL by linear attention to a classification task, and found good qualitative results that explain observed phenomena.

Weakness 3: softmax vs linear attention

We absolutely wish to further explore this. We would extend the calibration study of linear attention by computing similar calibration curves for a trained softmax model. This would have the form P(Y=1∣R)\mathbb{P}(Y = 1\mid R ), where RR is the model's outputed logit, and PP is computed numerically from the distribution of the trained model's outputted logits. We would compare this to the perfectly calibrated curve σ(R)\sigma(R). We have not yet included this result yet, as more numerics are needed to smoothly and cleanly compute these P(R)P(R) curves, and more clearly separate effects due to different data parameters.

AuthorsOpenReview ↗

Response to reviewer z4iC: Questions

Question 1: when is the separability result and universality claim valid?

This is a great question, and one that we wish we could provide a better answer to at this time. There are quite a few universality results that seem they might be applicable to our setting, but unfortunately aren't the right approach. The main problems in applying known results come down to the fact that

  • This is an ICL setting, so the task varies across prompts. There is no single teacher w∗w^* that determines the label probabilities for all prompts; that is, no fixed w∗w^* satisfies P(yi=1∣hi)=σ(hi⊤w∗)P(y_i=1 \mid h_i)=\sigma(h_i^\top w^*) for all ii.
  • Recall our covariates that we are trying to apply universality to are these hh vectors, which can be thought of for simplicity as having the structure vec(xqvT)\text{vec}(x_q v^T). Because of this lower-rank structure, these hh's are not subgaussian, which rules our the application of many universality results that require subgaussianity.
  • There are results that work with non-subgaussian covariates but we have not seen these yet in a logistic setting.

For the time being, we cannot provide a full proof or direct argument of the universality here, although there is probably some path towards arguing it, perhaps using concentration inequalities for tensors. We instead provide a new plot (Figure 11) in Appendix C showing that our predictions numerically match for a range of α\alpha values.

Reviewer_z4iCOpenReview ↗

Thank you for your rebuttal

The rebuttal partially addresses my concern regarding novelty by providing additional calibration results. However, the concerns regarding real-world validation and the discrepancy between softmax and linear attention remain largely unresolved. Therefore, I will maintain my original assessment and score.

Reviewer_hMr1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies in-context binary classification where each prompt comes from a latent logistic teacher, and analyses a linear-attention model with softmax readout in a proportional regime (context length ℓ∝d\ell \propto d, pretraining prompts n∝d2n \propto d^2). It derives an asymptotic law for the test logit that splits into a teacher-aligned scaling term, test-context noise, and anisotropy from finite pretraining data, plus a predicted separability threshold for the ridgeless estimator. The predictions are checked against finite-dimensional simulations and trained linear-attention models, with a preliminary softmax-attention comparison.

Critical strengths
  1. Well-written theory paper.

  2. The right object is analysed: the paper characterises the joint law of the model's pre-softmax score and the teacher's score, which is what makes calibration a first-class quantity: cross-entropy and error rate both follow from that single object.

  3. The reduction is genuinely clean. A trained attention model with d(d+1) effective parameters collapses to two order parameters, an isotropic part and an anisotropy measure, and the score is a slope times the teacher score plus independent Gaussian noise.

Critical weaknesses
  1. The setting is narrow: a single linear-attention layer, a reduced d×(d+1) estimator, and a simple data model, Gaussian covariates with a perfectly aligned teacher, so pretraining and test prompts share the same task prior.

  2. Nothing is proved: Appendix A repeatedly marks its own steps as assumptions made without proof, and Result 2 is a heuristic transfer of an existing theorem.

  3. The estimator is idealised: the theory describes the exact global minimiser of the empirical loss, ridgeless in the headline results, whereas trained models come from finite-time gradient descent.

  4. The novelty claim would be sharper if contrasted with existing theory for in-context classification (e.g., arXiv:2410.11778 and arXiv:2604.11613).

Questions for authors
  1. Is the fixed point of the self-consistency system unique, and how is it solved numerically?

  2. Does the blow-up at τc\tau_c, and the claim that longer training context can be harmful, survive at nonzero ridge or under finite-time gradient descent?

  3. Do trained models satisfy the parameter-level prediction a^1=tr⁡A^≈a1\hat a_1=\operatorname{tr}\hat A\approx a_1, a^2=d∣∣A^∣∣F2≈a2\hat a_2=d||\hat A||_F^2\approx a_2 and a^0=∣∣a^∣∣2≈0\hat a_0=||\hat a||^2\approx 0 (eqs. (9) and (17)), with A^,a^\hat A,\hat a read off via the reduction in eq. (6), or is only the output logit distribution compared?

  4. Is it possible to extend this theory for multi-layer linear attention?

3 comments
AuthorsOpenReview ↗

Response to reviewer hMr1: Weaknesses

Weakness 1: narrow task setting

We acknowledge that the task is simple, yet still we see many interesting phenomenological results even in the task aligned case. We agree with you that the more general case of pretraining and test tasks drawn from different priors would be highly interesting. Indeed, our current calculation provides some intuition about what might happen. Suppose tasks are now drawn from N(0,C)\mathcal{N}(0,C) for some structure matrix CC with tr(C)=1\text{tr}(C) = 1 and tr(C2)=o(1)\text{tr}(C^2) = o(1) (i.e. sufficient rank of CC). We see that (1) the rotational invariance argument of the population minimiser will no longer hold in general, so Γ=[A,a]\Gamma = [A, a] will have a different population solution that informs our finite-sample solution and order parameters; and (2) the self-consistency equations for all scalars, including the form of the final model output logit will depend on the training covariance. The test covariance will appear only as the averaging quantity for errors such as test CE or classification.

Weakness 2: conjectures without proofs

Thank you for this point, we indeed agree that various locations of the proof could be better formalised. To aid with this, we now provide simulations of empirical a^0,a^1,a^2\hat{a}_0, \hat{a}_1, \hat{a}_2 values converging in dd to our derived solutions; we also study the variance. Regarding the use of the Koriyama–Bellec boundary in our setting requires some additional universality argument for our features that we have not yet worked out. Note that the effective feature dimension p=d(d+1)p = d(d+1) indeed places us in their proportional regime of samples nn proportional to pp. However, our features hh are non-Gaussian and have tensor product structure, and there is no clear existing universality result that establishes directly the transfer here. We thus have changed our language: we will keep this predicted boundary location as a conjecture. We also provide numerical validation of this conjecture in our new Figure 11 in Appendix C.

Weakness 3: idealised estimator; ridge vs ridgeless setting

We acknowledge that the reduced y=⟨H,Γ⟩y = \langle H, \Gamma \rangle model is indeed a simplification of the full linear attention, but this is one made under a principled understanding of how the parameters of linear attention transform under gradient decent: the specific components of the key, query, and value matrix, when initialised at small values, converge to 0 under gradient descent. We also show in Figure 7 the alignment of our logit result with the finite-step trained fully-parameterised linear attention architecture.

Weakness 4: novelty and citations

Thanks! We acknowledge these works and discuss them further in our revised introduction.

AuthorsOpenReview ↗

Response to reviewer hMr1: Questions

Question 1: fixed point uniqueness and numerics

Thanks for this question. At this time, we are unable to prove uniqueness; this is very difficult to analyse analytically due to the complexity of this fixed point equation and all the different coupled and implicitly-defined variables. Studying this numerically is easier: The system being solved here is multi-dimensional, namely we are solving for (a1,a2)=(f(a1,a2),g(a1,a2))(a_1,a_2) = (f(a_1,a_2), g(a_1,a_2)) and so one could study the top eigenvalues of the Jacobian of this map and see if they are bounded by 1. We have done this numerically (Figure 10) and find that there are indeed regions of the (slightly transformed) parameter space for a1,a2−a12a_1,a_2-a_1^2 where the singular values of the fix-point Jacobian are bounded by 1. In this region, this means that the map R2→R2R^2 \to R^2 defined by the fixed point equation is contractive, which is indicative of a unique fixed point solution. It is also worthwhile noting that this region of contractiveness in the Jacobian only occurs in our simulations for τ>τc\tau > \tau_c, as expected.

Regarding how the solutions are computed numerically, we give an explanation of our fixed-point solver in Appendix A.

Question 2: τc\tau_c blowup for nonzero-ridge and gradient descent.

This is a nice question, thank you. We specifically demonstrate that there is a blowup in error at 0 ridge for all τ<τc\tau < \tau_c. That said, loss is non-monotonic in τ\tau even at nonzero ridge, which can be seen in Figure 2b. We conjecture that the location of this nonmonotonicity is related to the location of τc\tau_c; however doing proper numerics for this is not something we've had time to do yet. This nonmonotonicity, which would depend on training context length, could thus lead to a similar effect even for nonzero ridge or finite gradient steps: blindly increasing training context length at low τ\tau might worsen error.

Question 3: behaviour of a0,a1,a2a_0,a_1,a_2

We have provided additional figures giving numerical evidence for a^1≈a1,a^2≈a2,a^0≈0\hat{a}_1 \approx a_1, \hat{a}_2 \approx a_2, \hat{a}_0 \approx 0; this can be found in Appendix A.

Question 4 Is it possible to extend this theory for multi-layer linear attention?

Here we provide a speculative answer: https://arxiv.org/abs/2510.01098 shows that depth i.e. looped or deep linear attention is very useful when there is no single statistic for the tokens xx. In the work here, this is obviously not the case; all tokens are taken independently from the same distribution N(0,Id)\mathcal{N}(0,I_d). That said, we speculate that similar techniques from the above-mentioned paper might be useful towards this goal. Specifically, they also consider the reduced-Γ\Gamma model of linear attention and furthermore use an isotropic Γ∝Id\Gamma \propto I_d ansatz to solve the gradient flow equations. Perhaps a similar approach might work in our logistic setting, although we have not yet tried this.

Thanks for the rebuttal. The revised introduction resolves my concern about novelty and citations. I also appreciate the additional numerical validations and the discussions about potential extensions to more general settings. However, my main concerns (narrow setting, conjectures without proofs, and idealized estimator) seem still valid so I keep my evaluation as weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.