- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper, based on the in-context weight learning task, studies a variant (stage 1 is the normal gd training, and stage 2 changes the groud-truth each step to w^*. the d, n are also in another regime compared to previous work). Follow this paper's setting, the author proves learning convergence by PL inequalities, and shows that this kind of learning can let the model learn a better/more efficient optimization algorithm, even not trained on any labels of it.
The experiment corresponds to the theory.
Critical strengths
I think the paper investigate an interesting case of chain of thought and is well-written.
Critical weaknesses
see the questions below
Questions for authors
- assumption 3.1 is strong because you initialize by identity matrix. is it to reduce the model to a cleaner form?
- what's the use of the positional encoding? I didn't understand the reason for using it in the proof sketch. I also didn't understand why this makes the difference in experiment. (I couldn't get the intuition). and to me, this task shouldn't use any information from the position, as the current weight is already in the query token.
- can you elaborate in more details about line 419 "Moreover, in Stage 1, we remove teacher forcing and define the loss using iterates generated autoregressively by the transformer itself".
- my mostly concerned question is what the work implied for reality/empirical things. I didn't find good analogy to stage 2 (even you said that it can be analogy of RL, but I think it's still different because you still have supervision over every steps)
5 comments
Response to Reviewer XY9M, Part 2
Q3. Elaborate more on "Moreover, in Stage 1, we remove teacher forcing and define the loss using iterates generated autoregressively by the transformer itself".
A3. We clarify the settings as follows.
-
Theoretical setting (Stage 1 with teacher forcing): In Section 2.2, Equation (3), the input matrix at step () is constructed using the teacher iterates : and the Stage 1 loss supervises the one-step prediction from the teacher-forced input: \mathcal{L}_{\mathrm{GD}}(\mathbf{V},\mathbf{W}) = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|F_{\mathrm{LSA}}(\mathbf{Z}_t^{\mathrm{GD}};\mathbf{V},\mathbf{W})_{[d+2:2d+1,-1]} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right].
-
Experimental setting (Stage 1 without teacher forcing): In our experiments, we remove teacher forcing so that at step , the transformer takes its own autoregressively generated iterates as input. Here, for , we define and The experimental Stage 1 loss then supervises this autoregressively generated trajectory against the GD teacher iterates : \mathcal{L}_{\mathrm{GD}}^{\mathrm{exp}}(\mathbf{V},\mathbf{W}) = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|F_{\mathrm{LSA}}(\mathbf{Z}_t;\mathbf{V},\mathbf{W})_{[d+2:2d+1,-1]} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right] = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|\mathbf{w}_{t+1} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right].
Strictly following the theoretical setting with teacher forcing would make Stage 1 training much easier, because the input does not depend on the model parameters. Here we present the setting without teacher forcing to demonstrate that our theoretical conclusion on learning Heavy-Ball with the two-stage procedure can be observed in more general settings. [1] also gave similar comments in their experiments, where they removed teacher forcing during training and defined the loss using the transformer's autoregressively generated iterates. We have revised Lines 418-424 to explain this setup in greater detail.
Response to Reviewer XY9M, Part 3
Q4. Practical implication? I didn't find good analogy to stage 2.
A4. The Stage 2 objective can be viewed as an RL-like objective with a per-step reward and a discount factor of . In the paper (Lines 182-187), we acknowledge the differences from standard RL. Our analogy to RL serves as a high-level motivation rather than a direct equivalence.
A key takeaway from our study is a concrete theoretical demonstration of the capability of transformers to go beyond merely imitating a teacher algorithm and to discover more efficient algorithms through training:
-
This finding may provide motivation for several practical applications, such as scientific prediction tasks and operator-learning methods for solving partial differential equations. In these applications, there often exist standard numerical algorithms that do not rely on AI or deep learning. By showing that transformers have the potential to outperform such methods, our work provides a theoretical basis for applying deep learning in these and other related fields.
-
Our result may also have implications for AI for Math. Mathematical data often contain intermediate solution steps, such as derivations or proof steps, which can serve as supervision for training models. Our result suggests that transformers may have the potential not only to imitate these provided solution paths, but also to discover different and potentially new proof strategies.
References
[1] Huang et al., Transformers learn to implement multi-step gradient descent with chain of thought. ICLR 2025.
[2] Ahn et al., Transformers learn to implement preconditioned gradient descent for in-context learning. NeurIPS 2023.
[3] Gatmiry et al., Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.
[4] Takanami and Pehlevan, An asymptotic theory of chain-of-thought in in-context learning. NeurIPS 2026.
[5] Zheng et al., On mesa-optimization in autoregressively trained transformers: Emergence and capability. NeurIPS 2024.
[6] Wu et al., Looped Transformers with Layer Normalization Provably Learn the Power Method, arXiv:2606.00605
Response to Reviewer XY9M, Part 1
Thank you for your insightful and helpful comments. We have carefully considered each of your points and have provided our responses below. Since mathematical equations take up a large number of characters, we have split our response into three parts.
A1. Assumption 3.1 makes assumptions such as an almost-zero initialization with certain nonzero blocks and a frozen matrix block, which have also been considered in comparable theoretical studies [1, 2, 3, 4, 5, 6].
The assumption that one block of V is initialized as a scalar multiple of the identity matrix is a technical assumption, which does enable a cleaner theoretical analysis. Similar assumptions have also been considered in [5, 6], where certain blocks of the attention weight matrices are initialized as scalar multiples of the identity matrix.
We emphasize that this assumption is used for the theoretical analysis, and our experiments further show that the phenomenon persists under Gaussian random initialization and without freezing any blocks. This suggests that the insights from our theoretical analysis remain relevant under milder and more practical settings.
A2. Positional encodings play a key role in enabling the model to learn Heavy-Ball. The intuition is that the Heavy-Ball update rule for wt+1HB (Equation (7) in the paper) relies on two past iterates wtHB,wt−1HB. Therefore, it is important for the CoT steps to carry explicit information about the iteration index, which allows the model to distinguish the linear model weights associated with different optimization steps.
Our experiments in Figure 2 further support this intuition: without positional encodings, the transformer performs similarly to gradient descent and does not learn Heavy-Ball. This empirical observation is consistent with the theoretical explanation above. We have revised the paper around Lines 434-441 to make this point clearer.
Finally, positional encodings are a standard component of transformer-based sequence modeling and are commonly incorporated when modeling CoT steps. Thus, we believe that studying this setting with positional encodings is both natural and reasonable.