- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies attention-only transformers, primarily with tied weights, trained on linear-regression sequences with covariances that vary across tasks. It identifies structured KQ and OV circuits and uses Gaussian exponential tilting to derive an affine recursion in the population limit. Under the observed circuit structure, this analysis motivates a two-parameter iterative algorithm that transforms both covariates and labels. The experiments compare this algorithm with the trained networks, examine depth extrapolation, and extend the architecture to a supervised two-stage instrumental-variable (IV) regression setting.
Critical strengths
-
Varying the covariance across tasks provides a useful test of algorithmic interpretations, since a fixed-covariance setting can allow the model to encode a single covariance matrix in its weights.
-
The paper examines agreement between the proposed algorithm and the trained networks through both prediction error and input sensitivity. The appendix also considers higher dimensions, a broader eigenvalue range, linear-attention controls, and untied models. These are relevant checks of the interpretation's scope.
-
The Gaussian population-limit calculation gives a compact description of repeated attention updates. The deterioration under additional unrolling is also an interesting observation about the learned finite-depth procedure.
Critical weaknesses
-
The connection between the population theorem and Algorithm 1 is not fully established. The finite-sample softmax update is treated as an empirical-moment update, but the resulting approximation error and its accumulation across layers are not controlled.
-
The novelty relative to GD++ remains unclear. Prior work already combines gradient-like updates with covariance-dependent transformations of the covariates. A direct comparison would help identify which aspects of the proposed algorithm are new.
-
The IV experiment evaluates prediction of the conditional mean given the instrument, which does not establish recovery of the structural effect. Direct regression of y on z would provide a useful control for assessing the benefit of the two-stage model.
-
The performance gap for the one-layer model does not by itself establish that depth is necessary. Algorithm 1 can already represent one-step GD, leaving optimization, finite-sample softmax effects, and differences in training budgets as possible explanations for the gap.
Questions for authors
-
Could Corollary 1 be stated as a population identity, with the finite-sample approximation presented separately? What controls the approximation error across layers? Also, are the interpreted step sizes extracted from the trained circuits or independently optimized for each comparison?
-
What is the precise relationship between the proposed algorithm and GD++, including the covariate transformation and the signs in the label updates and readout? Which new conclusion specifically depends on allowing the covariance to vary across tasks?
-
How does the IV model compare with direct in-context regression of y on z under the same query-generation protocol? Does the evaluation include structural-parameter error?
7 comments
Propagation across layers. Fix the model parameters, depth , and token index . Let and denote the finite-sample and population states, initialized identically. The accumulated error is the difference after updates. Writing the actual update as , we obtain the exact decomposition
r_\ell^{(t)}(L).$$ The **first term preserves convergence to zero**: since $A^{(t)}$ is finite and independent of $L$, an $o_p(1)$ error at layer $t$ remains $o_p(1)$ after multiplication by $A^{(t)}$. It therefore remains to establish $r_\ell^{(t)}(L)=o_p(1)$. To control the residual, we first propagate exponential-square moment bounds $\frac{1}{L}\sum_{\ell'=1}^L e^{\alpha_t\\|z_{\ell',L}^{(t)}\\|^2}\le C_t$. Here $\alpha_t>0$ and $C_t<\infty$ are auxiliary tail-control constants that can be chosen independently of $L$ at any prescribed confidence level and fixed depth. At $t=0$, the bound follows from Gaussian exponential moments and Markov's inequality, choosing $0<\alpha_0<1/(2\\|\Sigma^{(0)}\\|)$. For any query vector $q$, the exponential-tilting inequality bounds the norm of each head's softmax-weighted context mean by $\sqrt{\frac{\log C_t}{\alpha_t}}+\frac{\\|\Phi^{(h)}q\\|}{\alpha_t}$. Thus, the complete layer update grows at most linearly in $\\|q\\|$, allowing the moment bound to propagate to the next layer by choosing a sufficiently small positive $\alpha_{t+1}$. These moment bounds control the tails; convergence is supplied by a separate induction. The initial context empirical distribution converges to $\mathcal N(0,\Sigma^{(0)})$ by the law of large numbers. Suppose it converges to $\mathcal N(0,\Sigma^{(t)})$ at layer $t$. On bounded token sets, weak convergence gives convergence of the attention numerators and denominators, uniformly over bounded query sets. The exponential-square moment bound makes their discarded tails uniformly small. Since the Gaussian population denominator is at least $1$, taking the ratio is valid. If $F_L^{(t)}(q)$ denotes the complete actual layer update using the current context, then for every fixed radius $R>0$, $\sup_{\\|q\\|\le R}\\|F_L^{(t)}(q)-A^{(t)}q\\|\xrightarrow{p}0$. Applying these updates to the context tokens, with the same tail control, shows that the next-layer empirical distribution converges to $\mathcal N(0,A^{(t)}\Sigma^{(t)}A^{(t)\top})=\mathcal N(0,\Sigma^{(t+1)})$. This closes the induction without assuming independence at intermediate layers. For each fixed $\ell$, the Gaussian initial state and the linear-growth bound imply $z_{\ell,L}^{(t)}=O_p(1)$ at fixed depth. The locally uniform approximation can therefore be evaluated at this data-dependent token: $$r_\ell^{(t)}(L)=F_L^{(t)}(z_{\ell,L}^{(t)})-A^{(t)}z_{\ell,L}^{(t)}=o_p(1).$$ Since the initial accumulated error is zero, both terms in the error recursion consequently vanish by induction. Equivalently, expanding the recursion gives $$\\|z_{\ell,L}^{(T)}-z_\ell^{(T)}\\|\le\sum_{t=0}^{T-1}\left(\prod_{s=t+1}^{T-1}\\|A^{(s)}\\|\right)\\|r_\ell^{(t)}(L)\\|=o_p(1),$$ where the empty product equals $1$. For fixed $T$, this is a finite sum with amplification factors independent of $L$. Hence $$\max_{t\le T} \\|z_{\ell,L}^{(t)}-z_\ell^{(t)}\\| =o_p(1)$$ Consequently, we **establishes an $o_p(1)$ discrepancy between the actual transformer and Algorithm 1 across layers at fixed depth**. Thus, the $z_{\ell,L}^{T}$ approximation error remains **bounded in probability and vanishes as $L$ increases for every fixed $T$**. We also conduct the experiments to empirically see the error of the finite sample across layers. We test on a three-layer two-head transformer. The mse between the transformer output and the algorithm are: | Prompt Length $L$ | 50 | 100 | 150 | 200 | 250 | 300 | 350 | 400 | |---|---|---|---|---|---|---|---|---| | Layer 1 error | 0.0194 | 0.0084 | 0.0073 | 0.0055 | 0.0043 | 0.0037 | 0.0036 | 0.0033 | | Layer 2 error | 0.0208 | 0.0092 | 0.0077 | 0.0063 | 0.0055 | 0.0045 | 0.0041 | 0.0039 | | Layer 3 error | 0.0264 | 0.0124 | 0.0083 | 0.0071 | 0.0066 | 0.0051 | 0.0047 | 0.0044 | **3. Source of the interpreted step sizes.** In all transformer-interpreted-algorithm comparisons, the step sizes are extracted directly from the corresponding trained circuits:$\eta_x=\langle u_1,\omega\rangle,\qquad \eta_y=\langle u_2,\omega\rangle.$ No independent optimization of these step sizes is performed for Algorithm 1.W2, Q2: Unclear novelty related to GD++ and new aspects of the proposed algorithm.
Thank you for comments.
1. Relationship to GD++.
In [1], GD++ is expressed as
&\begin{pmatrix}x_j\\\\ y_j\end{pmatrix} \leftarrow \begin{pmatrix}x_j\\\\ y_j\end{pmatrix} + \begin{pmatrix}-\gamma XX^\top x_j\\\\-\Delta W x_j\end{pmatrix}, \\\\ &\Delta W=-\frac{\eta}{N}\sum_{i=1}^{N}(Wx_i-y_i)x_i^\top. \end{aligned}$$ Here, $X=[x_1,\ldots,x_N]$ contains the context covariates, $N$ is the context length, $W$ is the reference linear model, $\eta$ is the GD step size, and $\gamma$ controls the covariate transformation. The update applies to both context and query tokens. For scalar labels and the zero-initialized residual-token construction ($W=0$), this becomes, in our notation, $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}-\gamma\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}-\frac{\eta}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)}, \\\\ \hat y_q&=-y_q^{(T)}. \end{aligned}$$ By comparison, Algorithm 1 uses $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}+\frac{\eta_x}{L}\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}+\frac{\eta_y}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}x_\ell^{(t)}, \\\\ \hat y_q&=y_q^{(T)}. \end{aligned}$$ Although with the similar formulation, the algorithm learned by GD++ is **based on the setting that covariance is the same across sequences**. Under this setting, the covariance matrix will be encoded into the transformer weights as the pre-condition. In GD++, they assume the covariance matrix is identity so the learned transformer patterns is diagonal. However, in some other studies [2] and our experiments in Appendix C.3, training with a fixed anisotropic covariance produces covariance-dependent preconditioning in the KQ circuits. Consequencely, the condition for the gradient-like update algorithm in prior work is the training and testing sequences share the same covariance matrix $\Sigma$ for the covariates since **$\Sigma^{-1}$ appears as a pre-condition** in their interpreted algorithm, which can be formulated with our notation as: $$\begin{aligned} x_\ell^{(t+1)}&=x_\ell^{(t)}+\frac{\eta_x}{L}\sum_{\ell'=1}^{L}x_{\ell'}^{(t)}x_{\ell'}^{(t)\top}\Sigma^{-1}x_\ell^{(t)},\\\\ y_\ell^{(t+1)}&=y_\ell^{(t)}+\frac{\eta_y}{L}\sum_{\ell'=1}^{L}y_{\ell'}^{(t)}x_{\ell'}^{(t)\top}\Sigma^{-1}x_\ell^{(t)}, \end{aligned}$$ If the covariance matrix varies across sequences, this algorithm does not exist anymore because the $\Sigma^{-1}$ can not be learned during the training. **2. What is new under varying covariances.** With multi-layer softmax transformers trained on the in-context linear regression with varying covariance, we ontain the observed structure in Eq. (5), which suggests that **the learned parameters do not encode a particular task-specific $\Sigma$** and the scalar coefficients are shared across tasks with different covariance matrix for covariates. This implies that we discover a new gradient-based algorithm under the new training setting. Compared to the algorithm in [1, 2], this algorithm does not related to any task-specific $\Sigma$ and does not have the $\Sigma^{-1}$ as a component in the formulation. Consequenctly, our new observation under varying covariance setting is that transformer learns a new GD-based algorithm that does not take $\Sigma^{-1}$ as a pre-condition as that in the prior work.
Weaknesses
W1, Q1: About population identity for Corollary 1 and the error accumulation across layers.
Thank you for identifying this gap. We will separate the exact population result from its empirical-moment approximation, clarify the scope of the error analysis, and correct the description of the interpreted step sizes.
1. Population identity and Algorithm 1.
Under the assumptions of Theorem 1, with zero population mean and the exact structure in Eq. (5), we will restate Corollary 1 as
y_\ell^{(t+1)}=y_\ell^{(t)}+\langle u_2,\omega\rangle(\Sigma_{12}^{(t)})^\top x_\ell^{(t)},$$ where $\Sigma_{11}^{(t)}=[\Sigma^{(t)}]_{1:d,1:d}$ and $\Sigma_{12}^{(t)}=[\Sigma^{(t)}]_{1:d,d+1}$ are the covariate-covariance and covariate-label cross-covariance blocks of the joint context-token covariance $\Sigma^{(t)}$ propagated by Theorem 1. Algorithm 1 replaces these quantities with empirical second moments computed from its own current context tokens. We will describe it as an empirical-moment approximation. **2. Error propagation across layers.** We agree that the population identity and its finite-sample approximation should be distinguished. Below, we analyze **the single-layer approximation error** and **its propagation across layers**. Let $\Sigma^{(t)}$ denote the population covariance of the complete token. Write $z_\ell^{(t)}$ for the population token and $z_{\ell,L}^{(t)}$ for the actual finite-sample token, initialized identically. With the learned pattern structure, the population update is $$z_\ell^{(t+1)}=A^{(t)}z_\ell^{(t)},\qquad A^{(t)}= \begin{pmatrix} I+\eta_x\Sigma_{11}^{(t)} & 0\\\\ \eta_y(\Sigma_{12}^{(t)})^\top & 1 \end{pmatrix}.$$ **Single-layer error.** For a layer supplied with independent Gaussian context tokens and a fixed query, replacing the population attention integrals by finite-sample softmax produces an $O_p(L^{-1/2})$ error. This follows from the joint central limit theorem for the softmax numerator and denominator: their second moments are finite under Gaussian inputs, and the population denominator is positive. The error constant depends on the covariance, circuits, and query. Separately, replacing population second moments by empirical moments also produces an $O_p(L^{-1/2})$ update error. Specifically, $$\mathbb E\left\\| \frac1L\sum_{\ell=1}^L z_\ell^{(t)}z_\ell^{(t)\top} -\Sigma^{(t)} \right\\|_F^2= \frac{ (\operatorname{tr}\Sigma^{(t)})^2+ \operatorname{tr}\\!\left((\Sigma^{(t)})^2\right) }{L}.$$ Thus, on the same independent Gaussian inputs, **the single-layer softmax update and empirical-moment update differ by $O_p(L^{-1/2})$**.