acceptodds
Under review as a conference paper at ICLR 2027

Interpreting Multi-layer Transformers for In-context Linear Regression with Varying Covariance

Abstract

We study how multi-layer softmax-attention transformers perform in-context linear regression in the challenging regime where the covariate distribution varies across sequences, in contrast to the fixed-distribution setting considered in prior work. We first show that conclusions from prior work break down in this regime, and that transformer depth becomes essential for achieving low in-context error. Through a population-limit interpretation of the trained transformer, we reveal that it solves linear regression by implementing a variant of gradient descent whose parameters depend strongly on model depth, corresponding to a consistent structure in the learned weight matrices. We also discover the non-convergent property of the learned algorithm. Building on this insight, we show that with a chain-of-thought-style intermediate step, the transformer can solve in-context instrumental variable (IV) regression, applying the same algorithm whenever the same regime studied here appears in the IV regression. Our findings provide new evidence that multi-layer transformers learn distinct in-context algorithms under more complex regression scenarios, bridging the gap between empirical performance and interpretable algorithmic behavior.

Primary area:
interpretability and explainable AI
Keywords:
In-context Learning, Looped Transformer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2SBeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies attention-only transformers, primarily with tied weights, trained on linear-regression sequences with covariances that vary across tasks. It identifies structured KQ and OV circuits and uses Gaussian exponential tilting to derive an affine recursion in the population limit. Under the observed circuit structure, this analysis motivates a two-parameter iterative algorithm that transforms both covariates and labels. The experiments compare this algorithm with the trained networks, examine depth extrapolation, and extend the architecture to a supervised two-stage instrumental-variable (IV) regression setting.

Critical strengths
  • Varying the covariance across tasks provides a useful test of algorithmic interpretations, since a fixed-covariance setting can allow the model to encode a single covariance matrix in its weights.

  • The paper examines agreement between the proposed algorithm and the trained networks through both prediction error and input sensitivity. The appendix also considers higher dimensions, a broader eigenvalue range, linear-attention controls, and untied models. These are relevant checks of the interpretation's scope.

  • The Gaussian population-limit calculation gives a compact description of repeated attention updates. The deterioration under additional unrolling is also an interesting observation about the learned finite-depth procedure.

Critical weaknesses
  • The connection between the population theorem and Algorithm 1 is not fully established. The finite-sample softmax update is treated as an empirical-moment update, but the resulting approximation error and its accumulation across layers are not controlled.

  • The novelty relative to GD++ remains unclear. Prior work already combines gradient-like updates with covariance-dependent transformations of the covariates. A direct comparison would help identify which aspects of the proposed algorithm are new.

  • The IV experiment evaluates prediction of the conditional mean given the instrument, which does not establish recovery of the structural effect. Direct regression of y on z would provide a useful control for assessing the benefit of the two-stage model.

  • The performance gap for the one-layer model does not by itself establish that depth is necessary. Algorithm 1 can already represent one-step GD, leaving optimization, finite-sample softmax effects, and differences in training budgets as possible explanations for the gap.

Questions for authors
  • Could Corollary 1 be stated as a population identity, with the finite-sample approximation presented separately? What controls the approximation error across layers? Also, are the interpreted step sizes extracted from the trained circuits or independently optimized for each comparison?

  • What is the precise relationship between the proposed algorithm and GD++, including the covariate transformation and the signs in the label updates and readout? Which new conclusion specifically depends on allowing the covariance to vary across tasks?

  • How does the IV model compare with direct in-context regression of y on z under the same query-generation protocol? Does the evaluation include structural-parameter error?

7 comments

Weaknesses

W1, Q1: About population identity for Corollary 1 and the error accumulation across layers.

Thank you for identifying this gap. We will separate the exact population result from its empirical-moment approximation, clarify the scope of the error analysis, and correct the description of the interpreted step sizes.

1. Population identity and Algorithm 1.

Under the assumptions of Theorem 1, with zero population mean and the exact structure in Eq. (5), we will restate Corollary 1 as

y_\ell^{(t+1)}=y_\ell^{(t)}+\langle u_2,\omega\rangle(\Sigma_{12}^{(t)})^\top x_\ell^{(t)},$$ where $\Sigma_{11}^{(t)}=[\Sigma^{(t)}]_{1:d,1:d}$ and $\Sigma_{12}^{(t)}=[\Sigma^{(t)}]_{1:d,d+1}$ are the covariate-covariance and covariate-label cross-covariance blocks of the joint context-token covariance $\Sigma^{(t)}$ propagated by Theorem 1. Algorithm 1 replaces these quantities with empirical second moments computed from its own current context tokens. We will describe it as an empirical-moment approximation. **2. Error propagation across layers.** We agree that the population identity and its finite-sample approximation should be distinguished. Below, we analyze **the single-layer approximation error** and **its propagation across layers**. Let $\Sigma^{(t)}$ denote the population covariance of the complete token. Write $z_\ell^{(t)}$ for the population token and $z_{\ell,L}^{(t)}$ for the actual finite-sample token, initialized identically. With the learned pattern structure, the population update is $$z_\ell^{(t+1)}=A^{(t)}z_\ell^{(t)},\qquad A^{(t)}= \begin{pmatrix} I+\eta_x\Sigma_{11}^{(t)} & 0\\\\ \eta_y(\Sigma_{12}^{(t)})^\top & 1 \end{pmatrix}.$$ **Single-layer error.** For a layer supplied with independent Gaussian context tokens and a fixed query, replacing the population attention integrals by finite-sample softmax produces an $O_p(L^{-1/2})$ error. This follows from the joint central limit theorem for the softmax numerator and denominator: their second moments are finite under Gaussian inputs, and the population denominator is positive. The error constant depends on the covariance, circuits, and query. Separately, replacing population second moments by empirical moments also produces an $O_p(L^{-1/2})$ update error. Specifically, $$\mathbb E\left\\| \frac1L\sum_{\ell=1}^L z_\ell^{(t)}z_\ell^{(t)\top} -\Sigma^{(t)} \right\\|_F^2= \frac{ (\operatorname{tr}\Sigma^{(t)})^2+ \operatorname{tr}\\!\left((\Sigma^{(t)})^2\right) }{L}.$$ Thus, on the same independent Gaussian inputs, **the single-layer softmax update and empirical-moment update differ by $O_p(L^{-1/2})$**.

Propagation across layers. Fix the model parameters, depth TT, and token index ℓ\ell. Let zℓ,L(t)z_{\ell,L}^{(t)} and zℓ(t)z_\ell^{(t)} denote the finite-sample and population states, initialized identically. The accumulated error is the difference zℓ,L(t)−zℓ(t)z_{\ell,L}^{(t)}-z_\ell^{(t)} after tt updates. Writing the actual update as zℓ,L(t+1)=A(t)zℓ,L(t)+rℓ(t)(L)z_{\ell,L}^{(t+1)}=A^{(t)}z_{\ell,L}^{(t)}+r_\ell^{(t)}(L), we obtain the exact decomposition

r_\ell^{(t)}(L).$$ The **first term preserves convergence to zero**: since $A^{(t)}$ is finite and independent of $L$, an $o_p(1)$ error at layer $t$ remains $o_p(1)$ after multiplication by $A^{(t)}$. It therefore remains to establish $r_\ell^{(t)}(L)=o_p(1)$. To control the residual, we first propagate exponential-square moment bounds $\frac{1}{L}\sum_{\ell'=1}^L e^{\alpha_t\\|z_{\ell',L}^{(t)}\\|^2}\le C_t$. Here $\alpha_t>0$ and $C_t<\infty$ are auxiliary tail-control constants that can be chosen independently of $L$ at any prescribed confidence level and fixed depth. At $t=0$, the bound follows from Gaussian exponential moments and Markov's inequality, choosing $0<\alpha_0<1/(2\\|\Sigma^{(0)}\\|)$. For any query vector $q$, the exponential-tilting inequality bounds the norm of each head's softmax-weighted context mean by $\sqrt{\frac{\log C_t}{\alpha_t}}+\frac{\\|\Phi^{(h)}q\\|}{\alpha_t}$. Thus, the complete layer update grows at most linearly in $\\|q\\|$, allowing the moment bound to propagate to the next layer by choosing a sufficiently small positive $\alpha_{t+1}$. These moment bounds control the tails; convergence is supplied by a separate induction. The initial context empirical distribution converges to $\mathcal N(0,\Sigma^{(0)})$ by the law of large numbers. Suppose it converges to $\mathcal N(0,\Sigma^{(t)})$ at layer $t$. On bounded token sets, weak convergence gives convergence of the attention numerators and denominators, uniformly over bounded query sets. The exponential-square moment bound makes their discarded tails uniformly small. Since the Gaussian population denominator is at least $1$, taking the ratio is valid. If $F_L^{(t)}(q)$ denotes the complete actual layer update using the current context, then for every fixed radius $R>0$, $\sup_{\\|q\\|\le R}\\|F_L^{(t)}(q)-A^{(t)}q\\|\xrightarrow{p}0$. Applying these updates to the context tokens, with the same tail control, shows that the next-layer empirical distribution converges to $\mathcal N(0,A^{(t)}\Sigma^{(t)}A^{(t)\top})=\mathcal N(0,\Sigma^{(t+1)})$. This closes the induction without assuming independence at intermediate layers. For each fixed $\ell$, the Gaussian initial state and the linear-growth bound imply $z_{\ell,L}^{(t)}=O_p(1)$ at fixed depth. The locally uniform approximation can therefore be evaluated at this data-dependent token: $$r_\ell^{(t)}(L)=F_L^{(t)}(z_{\ell,L}^{(t)})-A^{(t)}z_{\ell,L}^{(t)}=o_p(1).$$ Since the initial accumulated error is zero, both terms in the error recursion consequently vanish by induction. Equivalently, expanding the recursion gives $$\\|z_{\ell,L}^{(T)}-z_\ell^{(T)}\\|\le\sum_{t=0}^{T-1}\left(\prod_{s=t+1}^{T-1}\\|A^{(s)}\\|\right)\\|r_\ell^{(t)}(L)\\|=o_p(1),$$ where the empty product equals $1$. For fixed $T$, this is a finite sum with amplification factors independent of $L$. Hence $$\max_{t\le T} \\|z_{\ell,L}^{(t)}-z_\ell^{(t)}\\| =o_p(1)$$ Consequently, we **establishes an $o_p(1)$ discrepancy between the actual transformer and Algorithm 1 across layers at fixed depth**. Thus, the $z_{\ell,L}^{T}$ approximation error remains **bounded in probability and vanishes as $L$ increases for every fixed $T$**. We also conduct the experiments to empirically see the error of the finite sample across layers. We test on a three-layer two-head transformer. The mse between the transformer output and the algorithm are: | Prompt Length $L$ | 50 | 100 | 150 | 200 | 250 | 300 | 350 | 400 | |---|---|---|---|---|---|---|---|---| | Layer 1 error | 0.0194 | 0.0084 | 0.0073 | 0.0055 | 0.0043 | 0.0037 | 0.0036 | 0.0033 | | Layer 2 error | 0.0208 | 0.0092 | 0.0077 | 0.0063 | 0.0055 | 0.0045 | 0.0041 | 0.0039 | | Layer 3 error | 0.0264 | 0.0124 | 0.0083 | 0.0071 | 0.0066 | 0.0051 | 0.0047 | 0.0044 | **3. Source of the interpreted step sizes.** In all transformer-interpreted-algorithm comparisons, the step sizes are extracted directly from the corresponding trained circuits:$\eta_x=\langle u_1,\omega\rangle,\qquad \eta_y=\langle u_2,\omega\rangle.$ No independent optimization of these step sizes is performed for Algorithm 1.

W2, Q2: Unclear novelty related to GD++ and new aspects of the proposed algorithm.

Thank you for comments.

1. Relationship to GD++.

In [1], GD++ is expressed as

&\begin{pmatrix}x_j\\\\ y_j\end{pmatrix} \leftarrow \begin{pmatrix}x_j\\\\ y_j\end{pmatrix} + \begin{pmatrix}-\gamma XX^\top x_j\\\\-\Delta W x_j\end{pmatrix}, \\\\ &\Delta W=-\frac{\eta}{N}\sum_{i=1}^{N}(Wx_i-y_i)x_i^\top. \end{aligned}$$ Here, $X=[x_1,\ldots,x_N]$ contains the context covariates, $N$ is the context length, $W$ is the reference linear model, $\eta$ is the GD step size, and $\gamma$ controls the covariate transformation. The update applies to both context and query tokens. For scalar labels and the zero-initialized residual-token construction ($W=0$), this becomes, in our notation, $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}-\gamma\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}-\frac{\eta}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)}, \\\\ \hat y_q&=-y_q^{(T)}. \end{aligned}$$ By comparison, Algorithm 1 uses $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}+\frac{\eta_x}{L}\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}+\frac{\eta_y}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)}, \\\\ \hat y_q&=y_q^{(T)}. \end{aligned}$$ Although with the similar formulation, the algorithm learned by GD++ is **based on the setting that covariance is the same across sequences**. Under this setting, the covariance matrix will be encoded into the transformer weights as the pre-condition. In GD++, they assume the covariance matrix is identity so the learned transformer patterns is diagonal. However, in some other studies [2] and our experiments in Appendix C.3, training with a fixed anisotropic covariance produces covariance-dependent preconditioning in the KQ circuits. Consequencely, the condition for the gradient-like update algorithm in prior work is the training and testing sequences share the same covariance matrix $\Sigma$ for the covariates since **$\Sigma^{-1}$ appears as a pre-condition** in their interpreted algorithm, which can be formulated with our notation as: $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}+\frac{\eta_x}{L}\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}\Sigma^{-1}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}+\frac{\eta_y}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}\Sigma^{-1}x_\ell^{(t)}, \end{aligned}$$ If the covariance matrix varies across sequences, this algorithm does not exist anymore because the $\Sigma^{-1}$ can not be learned during the training. **2. What is new under varying covariances.** With multi-layer softmax transformers trained on the in-context linear regression with varying covariance, we ontain the observed structure in Eq. (5), which suggests that **the learned parameters do not encode a particular task-specific $\Sigma$** and the scalar coefficients are shared across tasks with different covariance matrix for covariates. This implies that we discover a new gradient-based algorithm under the new training setting. Compared to the algorithm in [1, 2], this algorithm does not related to any task-specific $\Sigma$ and does not have the $\Sigma^{-1}$ as a component in the formulation. Consequenctly, our new observation under varying covariance setting is that transformer learns a new GD-based algorithm that does not take $\Sigma^{-1}$ as a pre-condition as that in the prior work.
Reviewer_DdDvOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies transformers' behavior on in-context linear regression tasks with zero-mean covariates whose covariance varies across sequences. The major empirical finding is a parameter pattern that emerges in multi-layer looped transformers. This pattern can be interpreted as a concrete update rule in the population limit.

Critical strengths

This paper is generally clear and well-written. The pattern that emerges during training is an interesting finding. Theorem 1 and Corollary 1 interpret this pattern by showing a concrete update rule in the population limit, which is a non-trivial finding.

Critical weaknesses
  1. The paper only analyzes the parameter pattern of the trained transformer. I feel it would be more interesting if the authors could show how training leads to such a pattern, as this would explain why the pattern emerges.
  2. The transformer is trained on zero-mean covariates, while Theorem 1 allows non-zero means and corollary 1 then returns to the zero-mean setting, which is kind of distracting. If the aim is to show that a transformer trained on zero-mean covariates generalizes to non-zero-mean settings, it would be more convincing if the authors can provide empirical evidence that the model still performs well in this setting.
  3. The training setting for the IV transformer seems highly tailored and is not a general approach. The connection of this experiment to the main part is also not fully clear.
Questions for authors
  1. Please see weakness part.
  2. The IV regression training objective (10) supervises both the intermediate prediction x^\hat{x} and the final prediction y^\hat{y}. How much does the model's performance depend on explicit first-stage supervision? What happens if the same two-stage model is trained using only the final prediction error on yy with gradients flowing through both stages (as this is the training setting in the main part)?
8 comments

Weaknesses

W1: Training dynamics

Thank you for this suggestion. We tracked the training dynamics of a two-layer, two-head looped softmax transformer with the same settings in Section 4. We visualize KQ and OV heatmaps to examine the evolution of the matrix structure, training and validation prediction errors to track performance, coefficient trajectories (ω,u1,u2\omega, u_1, u_2) to characterize parameter changes, and effective step sizes (ηx,ηy\eta_x, \eta_y) to summarize the corresponding update strengths. Approximately diagonal patterns become visible within the early training steps (around 10,000 in our experiment, 1/15 of the entire training stage), alongside a substantial reduction in prediction error. During subsequent training, these patterns become more pronounced while their coefficients continue to evolve. Prediction error decreases more gradually, and the effective step sizes become comparatively stable despite continued changes in individual coefficients. With these empirical observations, we will have a empirical conclusion that in the training trajectory, an approximately diagonal structure gradually emerges early in training. Subsequently, the model continues to refine this structure and adjust its coefficients, while the prediction error and effective step sizes gradually stabilize. We add these experiments (Figure 7 and Figure 8 in Appendix B.1) in the appendix of our revised version of paper as the complementary studies on the training dynamics.

A complete training-dynamics theory would additionally need to track gradients through this coupled recursion and prove that optimization approaches the diagonal-circuit structure. We do not establish this result and will state the limitation clearly. Extending the analysis to these training dynamics, potentially through a reduced or manifold-based characterization of the evolving parameters and distributions, is an important direction for future work.

W2: Non-zero-mean settings

Thank you for pointing out this ambiguity. The aim of the Theorem 1 is to characterizes the population dynamics of a transformer with fixed weights under a general Gaussian context distribution. With the zero-mean settings in our experiments (also widely applied in related literatures), we have the empirical results for the learned patterns of transformer (Figure 2). With this pattern and the data settings, we then have the Corollary 1 and the interpreted algorithm. Therefore, our aim is not to show that model trained on zero-mean covariates generalizes to non-zero-mean settings, and this generalization is not guaranteed by the Theorem 1. We actually use a general theorem to interpret a specific algorithm case.

Following the your suggestion, we train a three-layer, two-head transformer on covariates with a non-zero-mean setting (mean is set to 0.51d0.5\mathbf{1}_d), keeping all other settings unchanged. The learned KQ and OV circuits exhibit approximately the same diagonal structure as in Figure 2. Combining this observation with Theorem 1 motivates the following corollary.

Corollary. Under the assumptions of Theorem 1 and the idealized circuit structure in Equation (5), let μx(t)\mu_x^{(t)} and μy(t)\mu_y^{(t)} denote the feature and label components of μ(t)\mu^{(t)}. Define

\bar{y}_\ell^{(t)}=y_\ell^{(t)}-\mu_y^{(t)}.$$ With the population approximation, for every $\ell\in[L]\cup\\{q\\}$, we have $$\begin{aligned} \bar{x}_\ell^{(t+1)} &=\bar{x}_\ell^{(t)}+\frac{\langle u_1,\omega\rangle}{L}\sum_{\ell'=1}^L {\bar{x}^{(t)}_{\ell'}} {\bar{x}^{(t)}_{\ell'}}^\top\cdot \bar{x}_\ell^{(t)},\\\\ \bar{y}_\ell^{(t+1)} &=\bar{y}_\ell^{(t)}+\frac{\langle u_2,\omega\rangle}{L}\sum_{\ell'=1}^L\bar{y}^{(t)}_{\ell'}{\bar{x}^{(t)}_{\ell'}}^\top\cdot \bar{x}_\ell^{(t)}, \end{aligned}$$ with the mean $\mu_x$ and $\mu_y$ updated by $$\begin{aligned} \mu_x^{(t+1)} &=\left(1+\sum_{h=1}^{H}u_1^{(h)}\right)\mu_x^{(t)}+\frac{\langle u_1,\omega\rangle}{L}\sum_{\ell'=1}^{L}\bar x_{\ell'}^{(t)}\bigl(\bar x_{\ell'}^{(t)}\bigr)^\top\mu_x^{(t)},\\\\ \mu_y^{(t+1)} &=\left(1+\sum_{h=1}^{H}u_2^{(h)}\right)\mu_y^{(t)}+\frac{\langle u_2,\omega\rangle}{L}\sum_{\ell'=1}^{L}\bar y_{\ell'}^{(t)}\bigl(\bar x_{\ell'}^{(t)}\bigr)^\top\mu_x^{(t)}. \end{aligned}$$ **Interpreted algorithm.** Replace $\eta_x=\langle u_1,\omega\rangle$, $\eta_y=\langle u_2,\omega\rangle$, $a_x=\sum_{h=1}^{H}u_1^{(h)}$,and $a_y=\sum_{h=1}^{H}u_2^{(h)}$, the interpreted algorithm is: $$\begin{aligned} \bar{x}_\ell^{(t+1)} &\gets \bar{x}_\ell^{(t)} + \eta_x/L\cdot \sum_{\ell'=1}^L\bar{x}_{\ell'}^{(t)} \bar{x}_{\ell'}^{(t)^\top}\cdot \bar{x}_\ell^{(t)},\\\\ \bar{y}_\ell^{(t+1)} &\gets \bar{y}_\ell^{(t)} + \eta_y/L\cdot\sum_{\ell'=1}^L\bar{y}_{\ell'}^{(t)} \bar{x}_{\ell'}^{(t)^\top} \cdot \bar{x}_\ell^{(t)},\\\\ \mu_x^{(t+1)} &\gets(1+a_x)\mu_x^{(t)}+\frac{\eta_x}{L}\sum_{\ell'=1}^{L}\bar x_{\ell'}^{(t)}\bigl(\bar x_{\ell'}^{(t)}\bigr)^\top\mu_x^{(t)},\\\\ \mu_y^{(t+1)} &\gets(1+a_y)\mu_y^{(t)}+\frac{\eta_y}{L}\sum_{\ell'=1}^{L}\bar y_{\ell'}^{(t)}\bigl(\bar x_{\ell'}^{(t)}\bigr)^\top\mu_x^{(t)},\\\\ \hat{y}_q^{(T)} &=\bar{y}_q^{(T)}+\mu_y^{(T)}, \end{aligned}$$ where $\hat{y}_q^{(T)}$ is the final output. The algorithm therefore consists of two parts: applying Algorithm 1 to centered covariates $\bar{x}_\ell$ and labels $\bar{y}_\ell$, and propagating the context mean according to Theorem 1. The final prediction restores the propagated label mean. This gives an approximate interpretation of the finite-context transformer under its approximately diagonal circuit structure. We test the performance of the transformer and the interpreted algorithm. The prediction errors (measured by mse) of these two models with different sequence lengths are: | Prompt Length $L$ | 50 | 100 | 150 | 200 | 250 | 300 | 350 | 400 | |---|---|---|---|---|---|---|---|---| | Error of 3-layer TF | 0.27256 | 0.06454 | 0.03722 | 0.02737 | 0.02151 | 0.01748 | 0.01545 | 0.01426 | | Error of Interpreted algorithm | 0.27998 | 0.07227 | 0.03757 | 0.02748 | 0.02182 | 0.01742 | 0.01574 | 0.01442 | This shows that the behavior of the interpreted algorithm is nearly indistinguishable from the actual transformer, which evaluate the correctness of our interpretation.

W3: Highly tailored training settings for IV tasks and unclear connections to main part.

Thank you for raising these concerns. We will clarify the rationale for the two-stage design and its connection to our main mechanistic findings.

1. Rationale for the two-stage design. Two-stage least squares (2SLS) [1] and the minimax estimator [2] are two major, widely used approaches to solving linear IV problems. The 2SLS procedure consists of two linear regression stages: in the first stage, it regresses the endogenous covariate xix_i on an instrument ziz_i to obtain fitted values x^i\hat{x}_i; in the second stage, it regresses the outcome yiy_i on x^i\hat{x}_i. This procedure can be naturally adapted to the in-context learning setting [3] by implementing both stages as in-context linear regressions. Building on this idea, we propose a natural extension of in-context regression to 2SLS and design a multilayer transformer that mirrors this decomposition: an initial block predicts x^i\hat{x}_i, and a deeper block regresses yiy_i on x^i\hat{x}_i.

2. Connection to the main experiments. We use IV regression as a case study to investigate whether the conclusions and the interpreted algorithm from the main part of the paper extend to other downstream tasks. For in-context IV regression, following the 2SLS procedure, a natural approach to IV problems, allows us to formulate two in-context linear regression stages. The second stage naturally gives rise to the same challenging setting considered in the main part: the covariance matrix varies across sequences. Specifically, under Eq. (8), the ideal population first-stage regression satisfies E[xi∣zi,θ]=θ⊤zi\mathbb{E}[x_i\mid z_i,\theta]=\theta^\top z_i, and Cov⁡(E[xi∣zi,θ]∣θ)=θ⊤θ\operatorname{Cov}\left(\mathbb{E}[x_i\mid z_i,\theta]\mid\theta\right)=\theta^\top\theta. Thus, the covariance of the fitted covariates depends on the sequence-specific task. Based on the conclusions of the main analysis, we expect the transformer to exhibit the same pattern and learn the same algorithm. This expectation is supported by conducting the experiments on IV tasks and observing the matching KQ/OV structure in Figure 12, the close agreement in prediction errors between the second-stage transformer and the interpreted algorithm in Figure 6(b), and the aligned input sensitivities in Figure 6(c). Consequently, our main part analysis provides an interpretation of the transformer behavior and learned algorithm when covariance varies across sequences, a setting that differs from those considered in prior work [4]. The IV regression experiments demonstrate that our analysis can be naturally applied to downstream tasks that share this setting.

We will add some descriptions for clarify the connection between the IV part and the main part in the introduction clearly in our revised paper.

Questions

Q1:

Thank you for this suggestion. We conducted an additional ablation using a one-layer transformer for the first stage and a three-layer transformer for the second stage. We removed the stop-gradient operation between the stages and trained both transformers jointly using only the final prediction loss, i.e., the second term of Eq. (10).

Under this setting, the model does not learn an accurate solution to the IV task within the tested training budget. After training, the final prediction mse error is approximately 5.4 for predicting yqy_q and 4.7 for the intermediate prediction x^q\hat{x}_q relative to xqx_q with sequence length LL at 450450, compared with 0.246 for 2SLS and 0.279 for transformer trained on Eq. (10) on the same tasks. Moreover, we can not find a clear pattern structure for the learned KQ and OV matrices for both two stages.

These results suggest that explicit first-stage supervision plays an important role in learning the intended two-stage decomposition under the tested architecture and training budget. Without this intermediate target, the first stage does not recover accurate fitted regressors in the original coordinates, therefore, we do not observe the second stage reliably recovering the regression mechanism identified in the main part of the paper. This ablation therefore supports the importance of intermediate supervision in guiding the model toward an effective IV solution.

References

[1] Basmann, Robert L. "A generalized classical method of linear estimation of coefficients in a structural equation."

[2] Dikkala, Nishanth, et al. "Minimax estimation of conditional moment models."

[3] Liang, Haodong, Krishna Balasubramanian, and Lifeng Lai. "Transformers handle endogeneity in in-context linear regression."

[4] He, Jianliang, et al. "In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention."

Reviewer_p8jGOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Considers in-context linear regression using multi-layer, multi-head attention networks with a residual structure and with parameters shared between layers.

Four contributions:

First, an empirical study demonstrating structural features of the KQ and OV matrices that emerge.

Second, a result that replaces sample averages by expectations in the transformer model and computes an explicit idealized recursion in this case for the sequence of layer outputs and their distribution for Gaussian inputs.

Third, plugging in the observed transformer parameters gives an algorithmic interpretation of the regression algorithm. (This is a beautiful discovery. I think there is a more intuitive way to write the algorithm that gives some indication of why it works - I'll share it below.)

Fourth, it considers an intrumental variables regression problem and observes that the attention network learns the same algorithmic structure observed earlier.

Critical strengths

The paper identifies a fascinating algorithmic structure to the linear regression approach discovered by transformer models.

Critical weaknesses

None

Questions for authors

These are more suggestions:

Presentation of Algorithm 1:

Why not write it in the usual linear regression notation as covariate data in an nxd matrix X_0 and response n-vector y_0 with last entry zero, then the algorithm is the recurrence X_{t+1}=X_t(I-alpha X_t^\top X_t); y_{t+1}=(I+beta X_t X_t^\top)y_t. (From your Figure 2, the coefficients alpha and beta are both positive, that is, the sign in the X recurrence seems to be negative.)

It's an interesting family of prediction methods.

The Gram matrix K_t= X_t X_t^\top evolves according to K_{t+1} = K_t(I-alpha K_t)^2. Clearly, its eigenvectors stay constant, so this is a polynomial recurrence that gives a trajectory for the eigenvalues, and for small alpha they'll decrease like lambda_t ~ 1/(2 alpha t + lambda_0).

For an eigenvector v_i with eigenvalue lambda_{i,0}=0, v_i^\top y_T = v_i^\top y_0 prod_t (1 + beta lambda_{i,t}) = v_i^\top y_0

For an eigenvector v_i with eigenvalue lambda_{i,0}>0, v_i^\top y_T = v_i^\top y_0 prod_t (1 + beta lambda_{i,t}) ~ v_i^\top y_0 T^{beta/(2 alpha)}.

So y_T is close to (I+(c-1)P)y_0 where c is a constant and P is the projection onto the column space of X_0. And then e_n^T (I+(c-1)P)y_0 is a constant times the OLS estimate of the last response.

(But it's not clear why this should be approximately true for T=3.)

Minor points:

Throughout, the equation references have a latex typo ("Equation equation 6" instead of "Equation (6)").

Sections 1 and 2:

Missing reference. Trained Transformers Learn Linear Models In-Context, Ruiqi Zhang, Spencer Frei, Peter L. Bartlett, arXiv:2306.09927. Beyond showing that 'linear transformers can implement optimization algorithms', that paper shows how gradient methods lead to parameters that solve the fixed covariance linear regression problem. And it gives a proof of the failure mode in the linear case with a varying input covariance.

l173: Why are the parameter matrices described as 'projection' matrices?

l256-262: What does 'unrolled more than once' mean? In particular, a 3-layer network is unrolled 3 times, since there are 3 layers with identical parameters. So in the 3-layer looped transformer of Fig 3(a), does 'unrolled twice' mean a 6-layer transformer?

3 comments

Questions

Q1: Presentation of Algorithm 1:

We thank the reviewer for suggesting the standard regression notation and the spectral interpretation. We agree that both improve the presentation, however, we would add two slight modifications based on your comments as belows.

  • The algorithm can be written as Xt+1=Xt(Id−αXt⊤MXt)X_{t+1}=X_t(I_d-\alpha X_t^\top M X_t) and yt+1=(In+βXtXt⊤M)yty_{t+1}=(I_n+\beta X_tX_t^\top M)y_t, where M=diag⁡(In−1,0)M=\operatorname{diag}(I_{n-1},0) is the fixed prefix mask, and the prediction is the last entry of yTy_T. The mask is necessary and aligns with our interpreted algorithm because the query receives updates but does not contribute to the keys/values or empirical moments.
  • Accounting for the prefix mask M=diag⁡(In−1,0)M=\operatorname{diag}(I_{n-1},0), the interpreted algorithm is Xt+1=Xt(Id−αXt⊤MXt),  yt+1=(In+βXtXt⊤M)ytX_{t+1}=X_t(I_d-\alpha X_t^\top M X_t),~~ y_{t+1}=(I_n+\beta X_tX_t^\top M)y_t, with en⊤y0=0e_n^\top y_0=0. Its exact finite-depth prediction admits the spectral representation yq(T)=en⊤[In+∑i=1r(gi,T−1)Pi]y0,y_q^{(T)}=e_n^\top\left[I_n+\sum_{i=1}^{r}(g_{i,T}-1)P_i\right]y_0, where Pi=X0vivi⊤X0⊤Mλi,0P_i=\frac{X_0v_iv_i^\top X_0^\top M}{\lambda_{i,0}} and gi,T=∏t=0T−1(1+βλi,t)g_{i,T}=\prod_{t=0}^{T-1}(1+\beta\lambda_{i,t}). We examine scaled-OLS interpretation at T=3T=3, we extracted the scalar coefficients from a trained three-layer, two-head transformer. For a preselected context with L=350L=350 and d=5d=5, the gains gi,Tg_{i,T} at T=3T=3 are (g1,3,…,g5,3)≈(2.001,2.031,2.037,2.039,2.041)(g_{1,3},\ldots,g_{5,3})\approx(2.001,2.031,2.037,2.039,2.041). These gains are close, supporting scaled OLS as a useful finite-depth approximation. For this approximation, we can choose the context-dependent common gain cT=1r∑i=1rgi,Tc_T=\frac{1}{r}\sum_{i=1}^{r}g_{i,T}, i.e., c3≈2.030c_3\approx2.030. Therefore, with T=3T=3, the transformer output on yqTy_q^{T} can be interpreted as y^qT=(c3−1)y^OLS\hat y_q^{T}=(c_3-1) \hat y_{OLS}. We have tested the mse error between the prediction of the transformer and prediction of the interpreted algorithm from this perspective, and the results support our interpretation:
Prompt Length LL50100150200250300350400
error0.14780.02650.01360.00980.00770.00650.00570.0051

Since the α\alpha and β\beta depend on the number of transformer layers TT, we have not tried a very large TT, but we think the same conclusion can be obtained.

Minor points:

Thanks for your correction. We will fix the typo and add the reference in our revised paper in introduction on failure with a varying input covariance setting.

For describing the parameter matrices as 'projection' matrices, we will replace "projection matrices" with "learnable matrices" to avoid ambiguity. For the meaning of unroll, in Figure 3(a), "unrolled twice" means repeating the entire trained three-layer computation twice, resulting in six applications of the shared attention layer.

Thanks for the response. That all makes sense. It's interesting to see the low error already with T=3.

Thanks for your time and effort throughout the review and rebuttal process, and we greatly appreciate your comments and constructive suggestions for improving our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.