- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper develops a high-dimensional theory of in-context binary classification using linear attention with a softmax readout. It characterizes the test logit distribution as a scaled teacher signal plus noise arising from finite context length and anisotropic parameter estimation. The theory predicts classification error, cross-entropy loss, and a separability threshold for ridgeless training. As far as I understand, this analysis relies on heuristic cavity approximations without rigorous justification.
Critical strengths
I found this work interesting as an extension of studies on proportional high-dimensional statistical theory (e.g., Sur & Candès, 2019) to the ICL setting.
Critical weaknesses
-
I feel that the theoretical analysis is somewhat incomplete. As this is a learning theory work, I believe the theoretical results are the most crucial part. However, many (rough) approximations are used in the proofs, and I do not think the reliability of these results is very guaranteed. While there are approaches that are not fully mathematically rigorous, such as the cavity approach or the replica method, I consider it is still necessary to clarify what is mathematically established and what is not.
-
In my view, the motivation and contributions of this paper are somewhat obscure.
- First, Liu et al. (2025) analyze the excess risk and optimal rates for conditional probability estimation in multi-class ICL. While the theoretical framework differs, the authors do not articulate their exact differentiation from Liu et al. when contextualizing prior work on classification ICL. Consequently, the claims regarding motivation and contributions in the Introduction appear somewhat overstated.
- Furthermore, the core idea of decomposing the prediction of trained Linear Attention into a signal component and multiple noise terms is already known in prior work (e.g., Takanami et al., 2026, although it is a regression setting).
So, I consider the motivation is ambiguous and this paper does not offer significant new insights.
Liu et al. (2025). In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality. arXiv:2508.08673.
Takanami et al. (2026). Learning Linear Regression with Low-Rank Tasks In-Context. AISTATS 2026.
Questions for authors
-
To improve the presentation and soundness scores as well as the rate, could you clarify the motivation and contributions?
-
Regarding the wording: since 'calibration' is a technical term in statistics and ML community that specifically means 'the predicted probabilities match the true probabilities,' I think it would be better to use alternative phrasing. Could you consider this?
3 comments
Thank you for your response. But did you update your manuscript? I cannot see your revision so far.
Resubmitted revised version
Hi thanks for responding! We annotated our main changes in blue; please let us know if it's still not visible?
Response to reviewer sXy6
Weakness 1: what is rigorously vs numerically established?
Weakness 2: obscure motivation and contributions
Question 1: clarifying motivation and contributions
Question 2: technical use of the term calibration