- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work investigates in-context linear regression with task-dependent observation noise using a one-layer linear-attention Transformer. Under isotropic assumptions, it establishes that the optimal estimator reduces to scalar shrinkage of the empirical input-output cross-moment and derives exact finite-sample bias–variance formulas. It characterizes how task-noise heterogeneity affects the dispersion of conditional excess risk, revealing a non-monotonic dependence on the feature-to-sample ratio. It further compares the estimator with optimal ridge regression, decomposing the benefits of covariance adaptation and identifying regimes where the normalized risk gap vanishes. Experimental results with multilayer softmax Transformers provide additional empirical support for the theoretical findings.
Critical strengths
- Theoretical Analysis of Heterogeneous Task Noise.
The paper quantifies fluctuations in conditional risk under task-dependent noise variances. In particular, it shows that these fluctuations vary non-monotonically with respect to the ratio of feature dimension to in-context sample size (), reaching a peak at a location distinct from the standard interpolation threshold associated with double descent.
- Empirical Validation.
The key finite-sample results are not restricted to Gaussian distributions and hold under relatively mild moment conditions. Furthermore, experiments using multi-layer Softmax Attention Transformers demonstrate that the performance gap between a simple scalar estimator and Ridge regression is partially closed, examining connections to phenomena beyond the stylized theoretical model. I find this valuable as it effectively bridges the gap between theory and practice.
Critical weaknesses
- Limited Novelty.
The equivalence between the population-optimal estimator of a one-layer Linear Attention model and a single step of gradient descent is already known from Ahn et al. (2023) and Mahankali et al. (2024). In my understanding, many of the paper's core results rely primarily on a bias–variance analysis of this known estimator combined with classical high-dimensional Ridge regression theory (Dobriban & Wager, 2018)). Additionally, Bai et al. (2023) have studied adaptive ICL in mixture models with task-dependent noise, and Barnfield et al. (2026) have explored the limitations of one-layer Attention and adaptation via multi-layer Attention under different covariance settings. While I recognize that the specific theoretical formulations differ from these prior works, I feel that the new conceptual insights provided are insufficient for acceptance.
1. Ahn et al. (2023). Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS 2023.
2. Mahankali et al. (2024). One Step of Gradient Descent Is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention. ICLR 2024.
3. Dobriban & Wager (2018). High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification. Annals of Statistics, 46(1), 247–279.
4. Bai et al. (2023). Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS 2023.
5. Barnfield et al. (2026). Multi-layer Cross-attention Is Provably Optimal for Multi-modal In-context Learning. arXiv:2602.04872.
Questions for authors
Could you clarify your points about the above weakness? There's no need to contest every comment. Please highlight the strengths of your manuscript, as I may consider revising the score upwards if persuaded.
We thank the reviewer for careful and constructive feedback.
(#1 comparison) This paper mainly studies the noise-varying setting under a general task-noise distribution. Most prior work considered a fixed noise level (Ahn et al.; Mahankali et al.; Dobriban & Wager), and at most extends to a categorical distribution (see below; we also summarize other differences together).
For reference, we summarize the prior results:
(#2 contribution) We agree that the population-optimal estimator of one-layer linear attention reduces to a single step of gradient descent (Ahn et al.; Mahankali et al.), and that the ridge limits are classical (Dobriban & Wager); we do not claim either. What is new is what this estimator implies when the noise level varies across tasks:
Results by construction show what can be represented; ours are exact with no free parameter, so they predict risk levels and can be tested directly. We test them on real-world data (general comment; Sec. 7): 98 occupations in the American Community Survey, each a regression task with its own noise level. On these data: \fbox{The scalar rule's excess risk is within 3.4 \\% of Prop. 4.1} The dispersion of the variance term peaks at the grid point nearest γ=ν/(1+ν) as the theory predicts The oracle regret is 1.03 to 1.14 times Prop. C.1 The scalar-class results also extend to non-isotropic E[ww⊤], where the reduced-class optimum is a closed-form fixed map (App. F.10; our reply to Reviewer HuD9, #2).