acceptodds
Under review as a conference paper at ICLR 2027

When Are Linear Transformers Nearly Ridge-Optimal? Task-Varying Noise and Covariance Adaptation for In-Context Regression

Abstract

Test-time demonstrations for in-context learning may have noise levels that are shared within a task yet vary across tasks. We study this heterogeneity in in-context linear regression through a tractable one-layer linear-attention transformer. Under isotropic tasks, the population-optimal estimator reduces to scalar shrinkage of the empirical input-output cross-moment. We derive exact finite-sample bias–variance formulas and the optimal population scaling, then quantify two limitations of this rule. First, population tuning does not adapt to a new task’s noise level. Although average risk depends only on the mean noise variance, the dispersion of excess conditional risk is exactly proportional to the variance of task noise, with a coefficient determined by normalized estimation variance. In the proportional limit, this coefficient peaks below the interpolation threshold and decreases throughout the overparameterized regime. Second, scalar shrinkage cannot adapt to the realized empirical covariance. We decompose its risk gap to optimally tuned ridge into a squared-bias reduction and a variance increase, and derive rates showing that the normalized additive gap vanishes in data-rich, highly overparameterized, and weak-signal regimes. Experiments with multi-layer softmax-attention transformers show partial recovery of the scalar-to-ridge gap where covariance adaptation offers substantial gains.

Primary area:
learning theory
Keywords:
transformer, task-varying noise, covariance adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kGa2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work investigates in-context linear regression with task-dependent observation noise using a one-layer linear-attention Transformer. Under isotropic assumptions, it establishes that the optimal estimator reduces to scalar shrinkage of the empirical input-output cross-moment and derives exact finite-sample bias–variance formulas. It characterizes how task-noise heterogeneity affects the dispersion of conditional excess risk, revealing a non-monotonic dependence on the feature-to-sample ratio. It further compares the estimator with optimal ridge regression, decomposing the benefits of covariance adaptation and identifying regimes where the normalized risk gap vanishes. Experimental results with multilayer softmax Transformers provide additional empirical support for the theoretical findings.

Critical strengths
  1. Theoretical Analysis of Heterogeneous Task Noise.

The paper quantifies fluctuations in conditional risk under task-dependent noise variances. In particular, it shows that these fluctuations vary non-monotonically with respect to the ratio of feature dimension to in-context sample size (d/nd/n), reaching a peak at a location distinct from the standard interpolation threshold associated with double descent.

  1. Empirical Validation.

The key finite-sample results are not restricted to Gaussian distributions and hold under relatively mild moment conditions. Furthermore, experiments using multi-layer Softmax Attention Transformers demonstrate that the performance gap between a simple scalar estimator and Ridge regression is partially closed, examining connections to phenomena beyond the stylized theoretical model. I find this valuable as it effectively bridges the gap between theory and practice.

Critical weaknesses
  1. Limited Novelty.

The equivalence between the population-optimal estimator of a one-layer Linear Attention model and a single step of gradient descent is already known from Ahn et al. (2023) and Mahankali et al. (2024). In my understanding, many of the paper's core results rely primarily on a bias–variance analysis of this known estimator combined with classical high-dimensional Ridge regression theory (Dobriban & Wager, 2018)). Additionally, Bai et al. (2023) have studied adaptive ICL in mixture models with task-dependent noise, and Barnfield et al. (2026) have explored the limitations of one-layer Attention and adaptation via multi-layer Attention under different covariance settings. While I recognize that the specific theoretical formulations differ from these prior works, I feel that the new conceptual insights provided are insufficient for acceptance.


1. Ahn et al. (2023). Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning. NeurIPS 2023.

2. Mahankali et al. (2024). One Step of Gradient Descent Is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention. ICLR 2024.

3. Dobriban & Wager (2018). High-Dimensional Asymptotics of Prediction: Ridge Regression and Classification. Annals of Statistics, 46(1), 247–279.

4. Bai et al. (2023). Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS 2023.

5. Barnfield et al. (2026). Multi-layer Cross-attention Is Provably Optimal for Multi-modal In-context Learning. arXiv:2602.04872.

Questions for authors

Could you clarify your points about the above weakness? There's no need to contest every comment. Please highlight the strengths of your manuscript, as I may consider revising the score upwards if persuaded.

1 comment

We thank the reviewer for careful and constructive feedback.

(#1 comparison) This paper mainly studies the noise-varying setting under a general task-noise distribution. Most prior work considered a fixed noise level (Ahn et al.; Mahankali et al.; Dobriban & Wager), and at most extends to a categorical distribution (see below; we also summarize other differences together).

Bai et al. (NeurIPS 2023, Thm 17/I.1)Vladymyrov et al. (NeurIPS 2024, Thm 5.2)Barnfield et al. (2026, Thm 4.1, 6.2–6.3)Ours
Noise distributioncategorical σkk=1K\\{\sigma_k\\}_{k=1}^K (mixing law Λ\Lambda)two values (σ1\sigma_1 or σ2\sigma_2)The noise level varies across prompts only as a by-product of the latent factor model. Within each prompt, the noise is Gaussian with variance ζ2/(1+∥m∥2)\zeta^2/(1+\lVert m\rVert^2), which is coupled with the task vector ww and the covariate covariance Σx=I+mm⊤\Sigma_x = I + mm^\top through the latent parameters (m,ζ)(m,\zeta).general law of σ\sigma (E[σ4]<∞\mathbb{E}[\sigma^4]<\infty)
(n,d)(n,d)non-asymptotic upper bound for sufficiently large (d,n)(d,n) with d/n≤10d/n\le 10n→∞n\to\infty (fixed dd, i.e., d/n→0d/n\to 0)n→∞n\to\infty (fixed dd, i.e., d/n→0d/n\to 0) and depth T→∞T\to\inftyExact at any finite (d,n)(d,n), plus the proportional limit d/n→γ∈(0,∞)d/n\to\gamma\in(0,\infty)
theory typeexistence by constructionexistence by constructionimpossibility for every single-layer parameter + gradient-flow training dynamics: restricted 1-/2-parameter multi-layer cross-attention converges to the Bayes-optimal predictorquantification at the optimum (not constructed but derived)
Othersconstruction uses O(K)O(K) heads (with MLPs) and the guarantee bound is O~(log⁡K)1/3\tilde O(\log K)^{1/3}. So, the analysis is tied to finite KK (categorical distribution).beyond two levels, only empiricalcross-attention model; population covariance Σx=I+mm⊤\Sigma_x=I+mm^\top varies across prompts; in the infinite-context limit, noise is averaged out and plays no role in the optimization; Gaussian dataone-layer linear attention; non-Gaussian x,εx,\varepsilon allowed

For reference, we summarize the prior results:

(Bai et al., 2023, Thm 17/I.1) For sufficiently large n,dn,d with d/n≤10d/n\le10, there exists a transformer with O(log⁡n)O(\log n) layers and O(K)O(K) heads such that it is nearly Bayes-optimal with the gap ≤O~((log⁡Kn)1/3)\leq \tilde O\Big(\big(\frac{\log K}{n}\big)^{1/3}\Big)

(Vladymyrov et al., 2024, Thm 5.2) As n→∞n\to\infty, there exists a two-layer transformer that converges to the optimal ridge.

(Barnfield et al., 2026, Thm 4.1, 6.2–6.3) No single-layer attention can be Bayes-optimal, but a TT-layer linear cross-attention model trained by gradient flow becomes Bayes-optimal as the context length n→∞n\to\infty and then the depth T→∞T\to\infty.

(#2 contribution) We agree that the population-optimal estimator of one-layer linear attention reduces to a single step of gradient descent (Ahn et al.; Mahankali et al.), and that the ridge limits are classical (Dobriban & Wager); we do not claim either. What is new is what this estimator implies when the noise level varies across tasks:

  • Exact risk at finite (d,n)(d,n) under a general law of the noise level (table in #1).
  • The dispersion of risk across tasks. It is non-monotone in d/nd/n, with its peak at ν/(1+ν)\nu/(1+\nu), distinct from the interpolation threshold, which the reviewer noted as a strength.
  • The two adaptation limits, quantified separately: the oracle regret for task-noise adaptation (Prop. C.1), and the scalar-to-ridge gap for covariance adaptation, with where it peaks and how large it is (Thms. 6.1–6.3, Table 3).

Results by construction show what can be represented; ours are exact with no free parameter, so they predict risk levels and can be tested directly. We test them on real-world data (general comment; Sec. 7): 98 occupations in the American Community Survey, each a regression task with its own noise level. On these data: \fbox{The scalar rule's excess risk is within 3.4 \\% of Prop. 4.1} The dispersion of the variance term peaks at the grid point nearest γ=ν/(1+ν) as the theory predicts\fbox{The dispersion of the variance term peaks at the grid point nearest $\gamma=\nu/(1+\nu)$ as the theory predicts} The oracle regret is 1.03 to 1.14 times Prop. C.1\fbox{The oracle regret is 1.03 to 1.14 times Prop. C.1} The scalar-class results also extend to non-isotropic E[ww⊤]\mathbb E[ww^\top], where the reduced-class optimum is a closed-form fixed map (App. F.10; our reply to Reviewer HuD9, #2).

Reviewer_HuD9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies in-context linear regression with a one-layer linear self-attention model whose tasks share a latent weight vector but differ in noise level. It shows the population-optimal rule reduces to scalar shrinkage of the empirical cross-moment, derives exact finite-sample risk formulas, and quantifies two limitations, no adaptation to a task's noise level and no adaptation to the realized feature covariance, supported by experiments with multi-layer softmax transformers.

Critical strengths
  1. Well written theory paper that addresses a specific problem in depth.

  2. The theory is supported by reasonable experiments.

Critical weaknesses
  1. The setting is narrow in ways that matter: one layer, isotropic priors, a population-optimal rather than trained rule, and a fixed context length that is not listed among the limitations although the optimal coefficient depends explicitly on it.

  2. The empirical bridge to trained transformers is limited, with the seeded studies confined to a single small setting.

  3. The contribution is confined to essentially one axis of task heterogeneity, and in two of the regimes where the rule is called nearly ridge-optimal both rules also collapse to the null predictor.

Questions for authors
  1. Can the findings be connected to real transformers?

  2. What is the value of this work considering the narrow gap it targets?

6 comments

(Part 1/2) We thank the reviewer for a precise review. The setting is narrow by design, so that every quantity has a closed form, but we relax it where possible, by extension, reduction, or experiment (#1–#4):

#1 (population vs trained) The paper already compared trained models with the population-optimal rule: Fitting (with N tasks) converges to the population optimum with a N-CLT (Prop. B.2), and gradient training recovers it\fbox{Fitting (with $N$ tasks) converges to the population optimum with a $\sqrt{N}$-CLT (Prop. B.2), and gradient training recovers it} In addition to the theoretical result (Prop. B.2), we provide empirical evidence:

  • (App. E.5) In 280 seeded runs, trained one-layer models essentially coincide with the population-optimal θ∗Gxy\theta^\ast G_{xy} (their risk is within 0.4% and the fitted scalar is within 2% of θ∗\theta^*).
  • (Fig. 2, 20) Trained models' squared bias, variance, and risk follow Prop. 4.1 across γ\gamma and ν\nu.
  • (real-world data) On 98 ACS occupations, the trained Tφ,κT_{\varphi,\kappa} is within 0.05% of the scalar optimum (its θ=φκ\theta=\varphi\kappa within 1% of the tuned θ^\hat\theta on average for n≥8n\ge8; App. H), though it trains on only about 78 recurring tasks.

#2 (anisotropic feature and prior) The paper discussed anisotropic features and prior in App. F:

  • (Props. F.1, F.2) The optimum becomes a generally non-scalar fixed map BGxyBG_{xy} for any Gaussian covariance or centrally symmetric laws.
  • (Props. F.4, F.5) The scalar class retains exact risk and dispersion formulas under anisotropic non-Gaussian features.
  • (App. F.3) The scalar-to-ridge gap extends to ReLU random features.
  • cf. Task-prior anisotropy reduces to feature anisotropy. With E[ww⊤]=Σw\mathbb E[ww^\top]=\Sigma_w, the change of variables w~=Σw−1/2w\tilde w=\Sigma_w^{-1/2}w, x~=Σw1/2x\tilde x=\Sigma_w^{1/2}x leaves every label and the fixed-map class BGxy\\{BG_{xy}\\} invariant, so Props. F.1–F.2 (any Gaussian covariance, or centrally symmetric laws) already cover it. Σw∝Id\Sigma_w\propto I_d is needed only for the optimum to be scalar (Prop. 3.2; Remark F.3). We add this as a remark (Remark F.10).
  • With E[ww⊤]=Σw\mathbb E[ww^\top]=\Sigma_w, the scalar class depends on it only through Tr,Σw\mathrm{Tr}\\,\Sigma_w, so Props. 4.1, 5.1, B.2 and C.1 hold with τ2=Tr,Σw/d\tau^2=\mathrm{Tr}\\,\Sigma_w/d. If also E[w]=0\mathbb E[w]=0 and third moments vanish, the reduced-class optimum is the closed-form fixed map B∗GxyB^\ast G_{xy}, B∗=1dΣwM−1B^*=\tfrac1d\Sigma_wM^{-1}, M=E[GxyGxy⊤]M=\mathbb E[G_{xy}G_{xy}^\top] (Sec. 8, App. F.10).
  • (real-world data) On the ACS data, E[ww⊤]\mathbb E[ww^\top] is strongly anisotropic (top eigenvalue 44% of the trace, against 12.5%), yet the scalar rule's excess risk is within 3.4% of Prop. 4.1. B∗B^*, built from training occupations, is within 0.9% of the optimal fixed map fitted to them, 16–28% below the scalar rule for n≥64n\ge64, and within 6% of trained one-layer linear attention.

#3 (varying context length) “Training on a varying context length n∼π” ≡ fixed-length training at the effective length neff:=(Eπ[1/n])−1\fbox{“Training on a varying context length $n\sim\pi$” $\equiv$ fixed-length training at the effective length $n_{\text{eff}}:=(\mathbb{E}_\pi[1/n])^{-1}$}

The equivalence is exact, both at every finite dd and in the limit d/neff→γd/n_{\text{eff}}\to\gamma. Precisely, for n∼πn\sim\pi drawn independently of the task and the demonstrations, with one scaling for all nn: since ψd,n\psi_{d,n} is affine in nn (App. B.1), the optimal scaling and the π\pi-averaged squared bias, variance and risk exactly coincide with those of Prop. 4.1 at n=neffn=n_{\text{eff}} for every finite dd, and have the same limits with d/neff=d,Eπ[1/n]→γd/n_{\text{eff}}=d\\,\mathbb{E}_\pi[1/n]\to\gamma in place of d/n→γd/n\to\gamma. We state this in Sec. 8.

(Part 2/2)

#4 (real transformer) To relax the restriction to a single stylized setting, we add to Fig. 1 (right) \fbox{GPT-2 small (85M parameters: 12 layers, 12 heads, MLP, d_model=768), trained on up to 0.6B tokens, using three seeds}

beside our softmax-attention transformer (12 layers, 4 heads, D=192D=192, no MLP; used in both panels of Fig. 1 and in Fig. 17), which uses linear input (d+1→Dd{+}1\to D) and output (D→1D\to 1) projections, softmax attention with residual connections and pre-LayerNorm (plus a final LayerNorm), and two D×DD\times D matrices (P,Q)(P, Q) per head and layer. We observe that the models' task-averaged risks are ordered as follows (Fig. 1 left, Fig. 17), from highest to lowest: (i) scalar theory = 1-layer > (ii) fixed optimal ridge > practical transformer (≈ Bayes optimum)→n→∞ (iii) adaptive optimal ridge\fbox{(i) scalar theory = 1-layer > (ii) fixed optimal ridge > practical transformer ($\approx$ Bayes optimum)$\underset{n\to\infty}{\to}$ (iii) adaptive optimal ridge} (i) ξˉ[1+ν(1+ν)γ(1+ν)γ+ν]> (ii) ξˉ[1+γmγ(−γξˉ/τ2)]> (iii) Eξ[ξ[1+γmγ(−γξ/τ2)]]\fbox{(i) $\bar\xi\Big[1+\frac{\nu(1+\nu)\gamma}{(1+\nu)\gamma+\nu}\Big] >$ (ii) $\bar\xi\big[1+\gamma m_\gamma(-\gamma\bar\xi/\tau^2)\big]>$ (iii) $\mathbb{E}_{\xi}\Big[\xi\big[1+\gamma m_\gamma(-\gamma \xi/\tau^2)\big]\Big]$} The practical transformer behaves as if it estimates the noise level in context and regularizes each task accordingly.

  • All models in Fig. 1 and Fig. 17 are now retrained with three seeds at every (n,d)(n,d) (App. G).
  • (real-world data) On the ACS data, trained one- and two-layer linear attention are about 20–30% below the scalar rule for n≥64n\ge64 (four layers do no better than two). The gain is about that of B∗GxyB^*G_{xy} (#2), which uses no noise information; the value of task-noise adaptation for the scalar rule (Prop. C.1's mean oracle regret) is at most 2.8% of the excess risk there with signal and noise made independent (#5 for the observed pairing).

#5 (beyond one axis of heterogeneity) The ACS data have a second axis: across occupations, ∣w∣2/d\\|w\\|^2/d moves with the noise level ξ\xi (correlation 0.45). We now let ww depend on σ\sigma (Sec. 8, App. C.7):

  • Prop. 4.1 is unchanged, and Prop. 3.2 holds if E[σ2w]=0\mathbb E[\sigma^2w]=0.
  • Prop. 5.1's dispersion gains two signal terms from E[∣w∣2/d∣ξ]\mathbb E[\\|w\\|^2/d\mid\xi]; with ww independent of σ\sigma, it reduces to Prop. 5.1.
  • The task-noise oracle must model the dependence: on the ACS data, the oracle that assumes independence raises the excess risk by 6–8% at n≥128n\ge128.

Other axes, such as task-varying feature covariances, we leave to future work.

#6 ("nearly ridge-optimal" collapses to the null predictor) Agreed. App. D.5.1 already calls the claim "unambiguous, if also uninformative" as γ→∞\gamma\to\infty or ν→0\nu\to0. We want to emphasize that "nearly ridge-optimal" is metric-dependent as γ→0\gamma\to 0: at ν=1\nu=1, ridge still removes about half of the scalar's excess risk (Cor. D.6). The informative content is therefore where the gap peaks and how large it is (Table 3, App. D.5):

ν\nu110100
peak γ\gamma0.2890.6830.890
peak height Δmax⁡/E[σ2]\Delta_{\max}/\mathbb{E}[\sigma^2]0.0982.8742.0
share of scalar excess risk removed27%67%89%

(real-world data) On the ACS data (ν=0.036\nu=0.036), optimal ridge is within 0.5% of the scalar rule at every γ\gamma, yet at γ=1/64\gamma=1/64 the scalar rule's excess risk is only 31% of the null predictor's (0.0111 against ν\nu; Table 16): near ridge-optimality need not mean collapse to the null predictor.

#7 (value of this work, given the narrow gap) The gap is narrow only in regimes the theory identifies, and locating them is part of the contribution.

  • It is not narrow in general. At its peak, ridge removes 27%, 67% and 89% of the scalar rule's excess risk at ν=1\nu=1, 10 and 100 (#6), and the theory locates the peak from ν\nu alone.
  • Where it is narrow, the framework is a diagnostic. On the ACS data (ν=0.036\nu=0.036), the theory predicts, and the data confirm, that neither adaptation pays much for the scalar rule (ridge within 0.5%; mean oracle regret at most 2.8% of the excess risk with signal and noise made independent, while an oracle that ignores their dependence hurts, #5). Trained models' gain of about 20–30% is then traced to the non-isotropic E[ww⊤]\mathbb E[ww^\top], whose optimum #2 gives in closed form.

Thanks for the rebuttal and detailed revisions with new results. I appreciate the new experiments (specifically, Transformer and real-world data ones) and revisions. However, I still think that the contribution is thin since it is confined to essentially one axis of task heterogeneity (varying task noise), which significantly limits the potential of this work. Therefore, I keep my rating.

Reviewer_wsLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper uses a single-layer linear attention layer to separately quantify the costs of lacking task-specific noise adaptation and empirical covariance adaptation. Specifically, it trains the model with different task-related noise, and compares the results to the oracle estimator for quantifying the value of noise and covariance adaptation.

Critical strengths
  • The proof of the theorem in section 6 is detailed and rigorous. The discussion about the boundary of the study in the appendix is sufficient.
  • The motivation of this paper is good. It does not only report the performance degradation on high-noise tasks, but also reveal whether the model is adapting to noise or exploiting the empirical covariance structure.
Critical weaknesses
  • The organize of the main body in this paper is a little bit messy, and lacks of the experiments settings, which leads to hard for understanding the experiments shows in the main body.
  • Although the paper offers theoretical insights, its empirical evaluation is restricted to low-dimensional synthetic regression tasks, with no validation on real-world datasets. It should be discussed that how the insights in the paper can be implemented in the studies on real-world models and data.
Questions for authors
  • Could authors give the implementation details of the experiments in main body, and explain how these experiments related to the theory part?
  • Could authors discuss the potential real-world applications of the insights in this study?
2 comments

We thank the reviewer for careful and constructive feedback.

(#1 empirical settings)

  • All detailed experimental settings are collected in App. G (G.1 data, G.2 architectures, G.3–G.4 training protocols), and each main-text figure points there, keeping the main text for the closed-form results.
  • To make the main text self-contained, as the reviewer suggests, we have also added a short setup paragraph to the main text (end of Sec. 3).
  • We have expanded the empirical study: we added a larger transformer (see #2 larger models) and retrained the models in Fig. 1 and Fig. 17 with three random seeds.
  • What each main-text figure tests:
    • Fig. 1 (left): Prop. 5.1, the dispersion of excess risk across noise levels and its peak at d/n=ν/(1+ν)d/n=\nu/(1+\nu).
    • Fig. 1 (right): Thm. 6.1's hierarchy (scalar rule vs optimal ridge) and Thm. 6.3's small-gap regimes.
    • Fig. 2: Prop. 4.1, the squared bias, variance and risk of trained simplified models, beside ridge's (Thm. 6.2).

See below for the experimental settings (for the synthetic experiments; App. H gives the ACS settings):

settingcommon
optimizerAdamW
lr schedule100 warmup steps + cosine decay to 0
gradient clip1.0
batch size4000 fresh tasks per step
datax∼N(0,Id/d)x \sim N(0, I_d/d), w∼N(0,τ2Id)w \sim N(0, \tau^2 I_d), y=w⊤x+εy = w^\top x + \varepsilon
reportedmean, min–max over three seeds per (n,d)(n, d)
modeltheorysimplified modelL-layer linear transformerpractical transformerGPT-2 small
architecture-Tφ,κT_{\varphi,\kappa}LL linear-attention layers12 layers, 4 heads, D=192D=192, no MLP12 layers, 12 heads, D=768D=768, MLP
lr-0.010.011.6e-4 (=10−5(768/D)2)(=10^{-5}(768/D)^2)1e-4
weight init-std 0.1std 0.1P, Q: std 0.02$/\sqrt{D}$; read-in/out: std 0.02std 0.02 (residual outputs 0.02/\sqrt{2L}\)
steps-100010kFig. 1 left: 10k; right: 500500 (Fig. 1 right)
n,dn,dn,d→∞n, d \to \infty, d/nd/n fixedfinite n,dn,dfinite n,dn,dfinite n,dn,dfinite n,dn,d
training noise-σ2∼U(0,σmax⁡2)\sigma^2 \sim U(0,\sigma_{\max}^2) per tasksamesamesame
evaluationclosed form100 noise levels σ2∼U(0,σmax⁡2)\sigma^2 \sim U(0,\sigma_{\max}^2), 20k tasks eachsamesamesame

(#2 larger models) To relax the restriction to a single stylized setting, we add to Fig. 1 (right)

\fbox{GPT-2 small (85M parameters: 12 layers, 12 heads, MLP, d_model=768), trained on up to 0.6B tokens, using three seeds}

beside our softmax-attention transformer (12 layers, 4 heads, D=192D=192, no MLP; used in both panels of Fig. 1 and in Fig. 17), which uses linear input (d+1→Dd{+}1\to D) and output (D→1D\to 1) projections, softmax attention with residual connections and pre-LayerNorm (plus a final LayerNorm), and two D×DD\times D matrices (P,Q)(P, Q) per head and layer.

(#3 real-world data) We added a test on real-world data (general comment; Sec. 7, App. H): 98 occupations in the American Community Survey, each a task with its own noise level, with workers' actual log wages as outputs (no synthetic noise) and features whitened so that E[xx⊤]=Id/d\mathbb E[xx^\top]=I_d/d holds exactly. The considered models are limited to 1-/2-layer linear transformers. With no free parameter: \fbox{The scalar rule's excess risk is within 3.4 \\% of Prop. 4.1} The dispersion of the variance term peaks at the grid point nearest γ=ν/(1+ν) as the theory predicts\fbox{The dispersion of the variance term peaks at the grid point nearest $\gamma=\nu/(1+\nu)$ as the theory predicts} The oracle regret is 1.03 to 1.14 times Prop. C.1\fbox{The oracle regret is 1.03 to 1.14 times Prop. C.1} At this weak signal (ν=0.036\nu=0.036), optimal ridge is within 0.5% of the scalar rule, as Sec. 6 predicts. The same steps carry the theory to other data: define the tasks, measure ν\nu and the spread of noise levels, and compare the closed forms with the data.

(#4 real-world applications of the insights) The closed forms work as diagnostics; the ACS test shows how.

  • Average risk hides task-to-task heterogeneity: the dispersion across tasks of the variance term of the risk has a closed form (Prop. 5.1), with a peak at d/n=ν/(1+ν)d/n=\nu/(1+\nu) that the ACS data reproduce (#3).
  • What task-noise adaptation is worth, before building it: Prop. C.1 gives the mean oracle regret from ν\nu and the spread of noise levels; on the ACS data, at most 2.8% of the excess risk with signal and noise made independent. On the observed occupations, noisier ones carry more signal, and App. C.7 shows how the oracle must change.
  • Where a trained model's gains come from: comparing it with the scalar rule, ridge and the oracle separates task-noise from covariance adaptation; on the ACS data, trained linear attention's gain of about 20–30% over the scalar rule comes from the non-isotropic E[ww⊤]\mathbb E[ww^\top], not the noise level.

Thanks for your response and explanation. My concerns have beed solved, and I maintain the score at weak acceptance.

Comments on the paper

1 comment

To all reviewers: changes in the revision, and a test on real-world data

We thank the reviewers. Changes in the revised PDF are in blue:

  • Real-world data (new; Sec. 7, App. H): a test of the theory on 98 occupations in the American Community Survey (below).
  • A larger transformer: GPT-2 small (12 layers, 12 heads, MLP, 85M parameters, three seeds) in Fig. 1 (right), beside our 12-layer, 4-head softmax-attention transformer; the models in Fig. 1 and Fig. 17 added with three seeds (App. G).
  • The experimental setup in the main text (wsLY).
  • Theory (Sec. 8, now "Extensions, Limitations, and Future Work"): non-isotropic features E[xx⊤]\mathbb E[xx^\top] and non-isotropic tasks E[ww⊤]\mathbb E[ww^\top] (App. F), ww that depends on σ\sigma (App. C.7), and varying context length (HuD9 #2, #5, #3).
  • Space: Table 1 moved to App. D (now Table 2).

Real-world data. Each occupation is a task, with its residual variance as its noise level ξ\xi (ACS 2022–2024, IPUMS USA; 894,500 workers). A prompt is n=512,…,1n=512,\dots,1 workers from one occupation (γ=d/n∈[1/64,8]\gamma=d/n\in[1/64,8]), with their actual log hourly wages as outputs: no synthetic noise. The d=8d=8 features (age, sex, degree, and interactions) are whitened within each occupation, so E[xx⊤]=Id/d\mathbb E[xx^\top]=I_d/d holds exactly. Here ν=0.036\nu=0.036 and Var(ξ)/ξˉ2=0.145\mathrm{Var}(\xi)/\bar\xi^2=0.145. Rules are fitted on training occupations and scored on held-out ones (five folds), in closed form where possible.

The paper's predictionOn the ACS data
Prop. 4.1: excess risk of the scalar rulewithin 3.4% at every γ\gamma
Prop. 5.1: the dispersion of the variance term peaks at γ=ν/(1+ν)=0.034\gamma=\nu/(1+\nu)=0.034peak at the nearest grid point, γ=0.035\gamma=0.035; height 0.75 of the finite-(d,n)(d,n) theory's (the residual variance depends on xx)
Prop. C.1: mean oracle regret, each occupation scored at every ACS noise level (signal and noise independent, as Sec. 5 assumes)1.03–1.14 times the theory; at most 2.8% of the excess risk
Sec. 6, weak signal: optimal ridge ≈\approx scalar rulewithin 0.5% at every γ\gamma
Trained simplified model Tφ,κT_{\varphi,\kappa}within 0.05% of the scalar optimum

Where the data depart. E[ww⊤]\mathbb E[ww^\top] is far from isotropic (its top eigenvalue carries 44% of the trace, against 12.5% under isotropy), and ξ\xi correlates with ∣w∣2/d\\|w\\|^2/d (0.45; App. C.7): on the observed occupations, the oracle that assumes independence raises the excess risk by 6–8% at n≥128n\ge128. Trained one- and two-layer linear attention are about 20–30% below the scalar rule at n≥64n\ge64; with residuals rescaled across noise levels, they do not show the oracle's pattern (we train only linear attention on these data). The closed-form optimum B∗GxyB^*G_{xy} for non-isotropic E[ww⊤]\mathbb E[ww^\top] (App. F.10) is within 6% of the one-layer model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.