acceptodds
Under review as a conference paper at ICLR 2027

Transformers with CoT Discover Heavy-Ball Acceleration Through Training

Abstract

Modern transformers often solve complex tasks by generating and refining intermediate reasoning steps. To develop a theoretical understanding of chain-of-thought (CoT) reasoning, a notable recent study by Huang et al. (2025) viewed transformers as algorithm executors, and showed that when transformers’ CoT steps are supervised by a gradient-descent teacher, they can learn to implement gradient descent for in-context weight prediction in linear regression. In this paper, we study the same in-context weight prediction task but ask whether CoT reasoning can go beyond merely imitating the teacher algorithm. We show that, after initial supervision from a gradient-descent teacher, further trajectory-level supervision using the true regression weights leads the model to implement the heavy-ball method, thereby achieving acceleration over the original teacher. Notably, this accelerated algorithm emerges without supervision from any heavy-ball teacher. Our results provide a concrete theoretical example in which transformer reasoning can discover new algorithms through training, rather than merely imitate the algorithm used for supervision.

Primary area:
learning theory
Keywords:
transformer, chain of thought, heavy-ball

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XY9MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper, based on the in-context weight learning task, studies a variant (stage 1 is the normal gd training, and stage 2 changes the groud-truth each step to w^*. the d, n are also in another regime compared to previous work). Follow this paper's setting, the author proves learning convergence by PL inequalities, and shows that this kind of learning can let the model learn a better/more efficient optimization algorithm, even not trained on any labels of it.

The experiment corresponds to the theory.

Critical strengths

I think the paper investigate an interesting case of chain of thought and is well-written.

Critical weaknesses

see the questions below

Questions for authors
  1. assumption 3.1 is strong because you initialize by identity matrix. is it to reduce the model to a cleaner form?
  2. what's the use of the positional encoding? I didn't understand the reason for using it in the proof sketch. I also didn't understand why this makes the difference in experiment. (I couldn't get the intuition). and to me, this task shouldn't use any information from the position, as the current weight is already in the query token.
  3. can you elaborate in more details about line 419 "Moreover, in Stage 1, we remove teacher forcing and define the loss using iterates generated autoregressively by the transformer itself".
  4. my mostly concerned question is what the work implied for reality/empirical things. I didn't find good analogy to stage 2 (even you said that it can be analogy of RL, but I think it's still different because you still have supervision over every steps)
5 comments
AuthorsOpenReview ↗

Response to Reviewer XY9M, Part 1

Thank you for your insightful and helpful comments. We have carefully considered each of your points and have provided our responses below. Since mathematical equations take up a large number of characters, we have split our response into three parts.

Q1. Assumption 3.1 is strong. One block is initialized to be proportional to the identity matrix. Is this assumption made to reduce the model to a cleaner form?

A1. Assumption 3.1 makes assumptions such as an almost-zero initialization with certain nonzero blocks and a frozen matrix block, which have also been considered in comparable theoretical studies [1, 2, 3, 4, 5, 6].

The assumption that one block of V\mathbf{V} is initialized as a scalar multiple of the identity matrix is a technical assumption, which does enable a cleaner theoretical analysis. Similar assumptions have also been considered in [5, 6], where certain blocks of the attention weight matrices are initialized as scalar multiples of the identity matrix.

We emphasize that this assumption is used for the theoretical analysis, and our experiments further show that the phenomenon persists under Gaussian random initialization and without freezing any blocks. This suggests that the insights from our theoretical analysis remain relevant under milder and more practical settings.

Q2. Why consider positional encoding? I didn't understand the reason for using it in the proof sketch. I also didn't understand why this makes the difference in experiment. I couldn't get the intuition.To me, this task shouldn't use any information from the position, as the current weight is already in the query token.

A2. Positional encodings play a key role in enabling the model to learn Heavy-Ball. The intuition is that the Heavy-Ball update rule for wt+1HB\mathbf{w}_{t+1}^{\mathrm{HB}} (Equation (7) in the paper) relies on two past iterates wtHB,wt−1HB\mathbf{w}_t^{\mathrm{HB}}, \mathbf{w}_{t-1}^{\mathrm{HB}}. Therefore, it is important for the CoT steps to carry explicit information about the iteration index, which allows the model to distinguish the linear model weights associated with different optimization steps.

Our experiments in Figure 2 further support this intuition: without positional encodings, the transformer performs similarly to gradient descent and does not learn Heavy-Ball. This empirical observation is consistent with the theoretical explanation above. We have revised the paper around Lines 434-441 to make this point clearer.

Finally, positional encodings are a standard component of transformer-based sequence modeling and are commonly incorporated when modeling CoT steps. Thus, we believe that studying this setting with positional encodings is both natural and reasonable.

AuthorsOpenReview ↗

Response to Reviewer XY9M, Part 2

Q3. Elaborate more on "Moreover, in Stage 1, we remove teacher forcing and define the loss using iterates generated autoregressively by the transformer itself".

A3. We clarify the settings as follows.

  • Theoretical setting (Stage 1 with teacher forcing): In Section 2.2, Equation (3), the input matrix ZtGD\mathbf{Z}_t^{\mathrm{GD}} at step tt (0≤t<T0 \le t < T) is constructed using the teacher iterates w0,w1GD,…,wtGD\mathbf{w}_0, \mathbf{w}_1^{\mathrm{GD}}, \dots, \mathbf{w}_t^{\mathrm{GD}}: ZtGD=[X0d×10d×1⋯0d×1y00⋯00d×nw0w1GD⋯wtGD01×n11⋯10T×ne0e1⋯et],\mathbf{Z}_t^{\mathrm{GD}} = \begin{bmatrix} \mathbf{X} & \mathbf{0}_{d\times 1} & \mathbf{0}_{d\times 1} & \cdots & \mathbf{0}_{d\times 1} \\\\ \mathbf{y} & 0 & 0 & \cdots & 0 \\\\ \mathbf{0}_{d\times n} & \mathbf{w}_0 & \mathbf{w}_1^{\mathrm{GD}} & \cdots & \mathbf{w}_t^{\mathrm{GD}} \\\\ \mathbf{0}_{1\times n} & 1 & 1 & \cdots & 1 \\\\ \mathbf{0}_{T\times n} & \mathbf{e}_0 & \mathbf{e}_1 & \cdots & \mathbf{e}_t \end{bmatrix}, and the Stage 1 loss supervises the one-step prediction from the teacher-forced input: \mathcal{L}_{\mathrm{GD}}(\mathbf{V},\mathbf{W}) = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|F_{\mathrm{LSA}}(\mathbf{Z}_t^{\mathrm{GD}};\mathbf{V},\mathbf{W})_{[d+2:2d+1,-1]} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right].

  • Experimental setting (Stage 1 without teacher forcing): In our experiments, we remove teacher forcing so that at step tt, the transformer takes its own autoregressively generated iterates w0,w1,…,wt\mathbf{w}_0, \mathbf{w}_1, \dots, \mathbf{w}_t as input. Here, for t=0,1,…t=0,1,\ldots, we define wt+1:=FLSA(Zt;V,W)[d+2:2d+1,−1]\mathbf{w}_{t+1} := F_{\mathrm{LSA}}(\mathbf{Z}_t;\mathbf{V},\mathbf{W})_{[d+2:2d+1,-1]} and Zt=[X0d×10d×1⋯0d×1y00⋯00d×nw0w1⋯wt01×n11⋯10T×ne0e1⋯et].\mathbf{Z}_t = \begin{bmatrix} \mathbf{X} & \mathbf{0}_{d\times 1} & \mathbf{0}_{d\times 1} & \cdots & \mathbf{0}_{d\times 1} \\\\ \mathbf{y} & 0 & 0 & \cdots & 0 \\\\ \mathbf{0}_{d\times n} & \mathbf{w}_0 & \mathbf{w}_1 & \cdots & \mathbf{w}_t \\\\ \mathbf{0}_{1\times n} & 1 & 1 & \cdots & 1 \\\\ \mathbf{0}_{T\times n} & \mathbf{e}_0 & \mathbf{e}_1 & \cdots & \mathbf{e}_t \end{bmatrix}. The experimental Stage 1 loss then supervises this autoregressively generated trajectory against the GD teacher iterates wt+1GD\mathbf{w}_{t+1}^{\mathrm{GD}}: \mathcal{L}_{\mathrm{GD}}^{\mathrm{exp}}(\mathbf{V},\mathbf{W}) = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|F_{\mathrm{LSA}}(\mathbf{Z}_t;\mathbf{V},\mathbf{W})_{[d+2:2d+1,-1]} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right] = \frac{1}{2T}\mathbb{E}\left[\sum_{t=0}^{T-1}\left\\|\mathbf{w}_{t+1} - \mathbf{w}_{t+1}^{\mathrm{GD}}\right\\|^2\right].

Strictly following the theoretical setting with teacher forcing would make Stage 1 training much easier, because the input ZtGD\mathbf{Z}_t^{\mathrm{GD}} does not depend on the model parameters. Here we present the setting without teacher forcing to demonstrate that our theoretical conclusion on learning Heavy-Ball with the two-stage procedure can be observed in more general settings. [1] also gave similar comments in their experiments, where they removed teacher forcing during training and defined the loss using the transformer's autoregressively generated iterates. We have revised Lines 418-424 to explain this setup in greater detail.

AuthorsOpenReview ↗

Response to Reviewer XY9M, Part 3

Q4. Practical implication? I didn't find good analogy to stage 2.

A4. The Stage 2 objective can be viewed as an RL-like objective with a per-step reward rt=−∣wt+1−w⋆∣2r_t = -\\|\mathbf{w}_{t+1} - \mathbf{w}^{\star}\\|^2 and a discount factor of 11. In the paper (Lines 182-187), we acknowledge the differences from standard RL. Our analogy to RL serves as a high-level motivation rather than a direct equivalence.

A key takeaway from our study is a concrete theoretical demonstration of the capability of transformers to go beyond merely imitating a teacher algorithm and to discover more efficient algorithms through training:

  • This finding may provide motivation for several practical applications, such as scientific prediction tasks and operator-learning methods for solving partial differential equations. In these applications, there often exist standard numerical algorithms that do not rely on AI or deep learning. By showing that transformers have the potential to outperform such methods, our work provides a theoretical basis for applying deep learning in these and other related fields.

  • Our result may also have implications for AI for Math. Mathematical data often contain intermediate solution steps, such as derivations or proof steps, which can serve as supervision for training models. Our result suggests that transformers may have the potential not only to imitate these provided solution paths, but also to discover different and potentially new proof strategies.


References

[1] Huang et al., Transformers learn to implement multi-step gradient descent with chain of thought. ICLR 2025.

[2] Ahn et al., Transformers learn to implement preconditioned gradient descent for in-context learning. NeurIPS 2023.

[3] Gatmiry et al., Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.

[4] Takanami and Pehlevan, An asymptotic theory of chain-of-thought in in-context learning. NeurIPS 2026.

[5] Zheng et al., On mesa-optimization in autoregressively trained transformers: Emergence and capability. NeurIPS 2024.

[6] Wu et al., Looped Transformers with Layer Normalization Provably Learn the Power Method, arXiv:2606.00605

Reviewer_qdnaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies one-layer linear self-attention with CoT for in-context weight prediction in linear regression following Huang et al. (2025). It proposes two-stage training: Stage 1 supervises CoT steps with a gradient-descent teacher following existing work, and Stage 2 drops the teacher and trains against the true weights. Theorem 3.1 shows both stages converge under gradient flow and that the learned iterates approximate the optimally tuned time-varying Heavy-Ball method. Small experiments are done and consistent with the theory.

Critical strengths
  • The research question is natural and the mechanism is clearly explained
  • Handling Stage 2 without teacher forcing appears technically non-trivial and differs from prior work.
  • The paper is clearly written with the gaps between theoretical assumptions and the experimental setup.
Critical weaknesses
  • The result is that gradient flow reaches the minimiser of the Stage-2 objective. Since the iterates are polynomials in S (Lemma 4.2) and the optimal polynomial under the MP law is Heavy-Ball, Heavy-Ball seems to be the only possible outcome of successful training. This is a valuable convergence result, but calling it the discovery of a new algorithm overstates it.
  • Strong assumptions such as very few CoT steps, a specific sparse initialization with one block frozen, linear attention, population gradient flow. The step-index positional encodings give the model the time-varying coefficients that Heavy-Ball requires.
  • Experiments are small
Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer qdna, Part 1

We appreciate your supportive feedback and thoughtful questions. Please see our responses below. Due to the large number of characters required by the mathematical equations, we have separated our response into two parts.

Q1. Because of Lemma 4.2, Heavy-Ball seems to be the only possible outcome of successful training. This is a valuable convergence result, but calling it the discovery of a new algorithm overstates it.

A1. Thank you for your insightful comment. You are right that the correspondence between the optimal polynomial and the Heavy-Ball method is a key theoretical insight that motivates us to study whether transformers can also learn Heavy-Ball without a teacher. However, we believe that building such a connection among (i) transformer training analysis, (ii) the matrix polynomial form of the objective function, and (iii) the characterization of Heavy-Ball as the minimizer of the unconstrained matrix polynomial objective should be considered a contribution of our work. We sincerely hope that the clarity of our analysis and presentation does not lead to the mistaken impression that our results are easy to derive or predictable.

We use the term "discovery" only in the sense that Heavy-Ball iterates are not used during training, which distinguishes our work from recent results on learning gradient descent [1]. We have carefully gone through the abstract and main text to make the context around the word "discover" explicit, so that its meaning is clear and the wording is not interpreted as making a stronger claim than intended.

Technically, we would also like to clarify that the Stage-2 objective for the transformer is still different from the unconstrained matrix polynomial objective for which Heavy-Ball is the minimizer:

  • The polynomials implemented by the transformers are defined in Lemma 4.6. By definition, for all 0≤t<T0 \le t < T, the degree-(t+1)(t+1) coefficient of the polynomial pt+1,θˉ(τ)(S)p_{t+1,\bar{\boldsymbol{\theta}}(\tau)}(\mathbf{S}) is given by θ~1(τ)θ~2(τ)t\widetilde{\theta}_1(\tau)\widetilde{\theta}_2(\tau)^t, where θ~1(τ)\widetilde{\theta}_1(\tau) and θ~2(τ)\widetilde{\theta}_2(\tau) are scalar auxiliary parameters. This imposes a constraint on the form of, and the relationships among, the highest-order coefficients of the polynomials pt+1,θˉ(τ)(S)p_{t+1,\bar{\boldsymbol{\theta}}(\tau)}(\mathbf{S}). In comparison, the time-varying Heavy-Ball method defined in Equation (7) (and characterized in Lemma 4.9 as the unconstrained minimizer pt⋆=arg⁡min⁡deg⁡(p)≤t,p(0)=0E∣p(S)w⋆−w⋆∣2p_t^\star = \arg\min_{\deg(p)\le t, p(0)=0} \mathbb{E}\\|p(\mathbf{S})\mathbf{w}^\star - \mathbf{w}^\star\\|^2) is not subject to this constraint, as its parameters (αt,βt)t=0T−1\\{(\alpha_t, \beta_t)\\}_{t=0}^{T-1} can vary freely across steps.

  • Because of this constraint on the leading coefficients, minimizing the Stage-2 objective does not trivially yield the unconstrained optimal polynomial pt⋆p_t^\star. Instead, a key part of our analysis is to prove that the minimizer satisfies ∣θ~2⋆∣<1\vert\widetilde{\theta}_2^\star\vert < 1 (specifically, −θ~2⋆≤1−(d/n)2128-\widetilde{\theta}_2^\star \leq 1 - \frac{(d/n)^2}{128}), so that the leading coefficient θ~1⋆(θ~2⋆)t\widetilde{\theta}_1^\star(\widetilde{\theta}_2^\star)^t decays exponentially with tt. Only through this mechanism does the constrained degree-(t+1)(t+1) polynomial pt+1,θ~′⋆p_{t+1,\widetilde{\boldsymbol{\theta}}'^\star} approach the unconstrained optimal degree-tt polynomial pt⋆p_t^\star for T/2≤t<TT/2 \le t < T (Lemma 4.10), yielding wt+1(τ)≈wtHB\mathbf{w}_{t+1}(\tau) \approx \mathbf{w}_t^{\mathrm{HB}} in Theorem 3.1.

AuthorsOpenReview ↗

Response to Reviewer qdna, Part 2

Q2. Strong assumptions such as very few CoT steps, a specific sparse initialization with one block frozen, linear attention, population gradient flow. The step-index positional encodings give the model the time-varying coefficients that Heavy-Ball requires.

A2. We address your comments as follows.

  • Very few CoT steps, a specific sparse initialization with one block frozen, linear attention, population gradient flow: Prior theoretical work [1] made very similar assumptions. Specifically, [1] studied a one-layer linear self-attention model trained under population gradient flow, assumed T=Θ(log⁡d)T = \Theta(\log d) CoT steps, and used a similar nearly-zero sparse initialization with a frozen block in the weight matrices. Since our work directly extends the framework of [1] from imitating a gradient-descent teacher to discovering an accelerated algorithm, adopting a comparable setting is a natural and standard choice. Beyond [1], sparse initialization, frozen blocks, and linear attention have also been studied in several recent works [2, 3, 4, 5, 6]. Moreover, our experiments in Section 5 demonstrate that models trained with online stochastic Adam, starting from Gaussian random initialization and without freezing any blocks, still exhibit similar training behavior. The new experiments added in the revision (Appendix B) further show that softmax transformers can also, to a certain extent, discover accelerated algorithms, suggesting that the phenomenon is not an artifact of linear attention.

  • The step-index positional encodings give the model the time-varying coefficients that Heavy-Ball requires: Your understanding is correct that positional encodings play a key role in enabling the model to learn Heavy-Ball. The intuition is that the Heavy-Ball update rule for wt+1HB\mathbf{w}_{t+1}^{\mathrm{HB}} (Equation (7) in the paper) relies on two past iterates wtHB,wt−1HB\mathbf{w}_t^{\mathrm{HB}}, \mathbf{w}_{t-1}^{\mathrm{HB}}. Therefore, it is important for the CoT steps to carry explicit information about the iteration index, which allows the model to distinguish the linear model weights associated with different optimization steps. Consistent with this intuition, our experiments in Figure 2 show that the model cannot learn Heavy-Ball without positional encodings. We also note that incorporating positional encodings into CoT steps is common and standard, and therefore our setting is natural and reasonable. We have revised the paper around Lines 434--441 to include a more detailed explanation.

Q3. Experiments are small.

A3. Our work is primarily theoretical and follows the theoretical setting studied in a line of recent works [1, 2]. Our experiments are comparable in scale to those in these works: [1] considers d=10,n=20d=10, n=20, and [2] considers d=5,n=10d=5, n=10 and d=5,n=20d=5, n=20, while our experiments use (d,n)=(12,24)(d,n)=(12,24) and (16,32)(16,32). The goal of our experiments is to demonstrate that the trained transformer closely matches Heavy-Ball and achieves acceleration over the gradient-descent teacher, and to further show that this phenomenon persists in settings beyond our theoretical assumptions, including Gaussian random initialization without freezing any blocks and online stochastic Adam training. We believe that these experiments are well aligned with the theoretical focus of the paper and provide evidence for the intended claims.


References

[1] Huang et al., Transformers learn to implement multi-step gradient descent with chain of thought. ICLR 2025.

[2] Ahn et al., Transformers learn to implement preconditioned gradient descent for in-context learning. NeurIPS 2023.

[3] Gatmiry et al., Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.

[4] Takanami and Pehlevan, An asymptotic theory of chain-of-thought in in-context learning. NeurIPS 2026.

[5] Zheng et al., On mesa-optimization in autoregressively trained transformers: Emergence and capability. NeurIPS 2024.

[6] Wu et al., Looped Transformers with Layer Normalization Provably Learn the Power Method, arXiv:2606.00605

Thanks for the detailed response which helps me get better understanding now. I will keep my initial score and vote for an acceptance.

Thank you very much for your positive feedback and for maintaining your recommendation. We are glad that our response helped clarify the work, and we appreciate your time and consideration.

Reviewer_h63jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies two-stage training for in-context linear regression. The model first learns from a GD teacher, then trains on its own generated iterates using the true regression weights. The paper shows that the learned trajectory approaches Heavy-Ball and improves on the GD teacher.

Critical strengths
  1. The second training stage is an interesting extension of earlier work on learning multi-step gradient descent. The model uses its own outputs and learns to improve on the teacher’s trajectory. The connection to Heavy-Ball is very interesting, since no Heavy-Ball trajectories are provided during training.

  2. The paper provides a technically substantial analysis of both training stages. It studies training convergence and connects the learned iterates to Heavy-Ball through matrix polynomials.

Critical weaknesses

The main limitation is the use of linear attention. I understand it is widely adopted in the training dynamics literature of transformers for tractable analysis. However, its attention coefficients can be signed and are not normalized, which makes it natural to express gradient updates and weighted combinations of previous iterates. This is especially helpful for the regression task studied here and simplifies the connection to an optimization algorithm. Thus, it remains unclear whether the same learning behavior appears with a softmax attention equiped with rigorous retrival mechanism, where the coefficients are nonnegative and normalized.

Questions for authors

Have the authors run experiments beyond linear attention, such as softmax attention within the same synthetic regression setup? As i mentioned in the weakness, I am unsure whether the Heavy-Ball-like behavior depends on this structure. Checking whether similar acceleration and trajectories appear with softmax attention would help show how closely the proposed explanation relates to standard transformer dynamics.

3 comments
AuthorsOpenReview ↗

Response to Reviewer h63j

Thank you very much for your thoughtful and supportive comments. We have addressed your comments below.

Q1. The main limitation is the use of linear attention. Experiments with softmax attention?

A1. You are right that the setting of linear self-attention is standard in the related theoretical literature, as it can naturally express iterative algorithms such as gradient descent. Related works [1, 2, 3, 4, 5, 6] often focus on such a setting with linear self-attention, and this motivates us to study the same setting.

Following your suggestion, we have added new experiments in Appendix B of the revised paper:

Retaining the exact synthetic data setup and autoregressive CoT rollout from Subsection 2.1, we replace the single-head linear attention layer with an HH-head softmax attention layer: FSM(Z;Vh,Whh=1H):=Z+1H∑h=1HVhZsoftmax⁡(ZTWhZ),F_{\mathrm{SM}}\bigl(\mathbf{Z}; \\{\mathbf{V}_h, \mathbf{W}_h\\}_{h=1}^H\bigr) := \mathbf{Z} + \frac{1}{H}\sum_{h=1}^H \mathbf{V}_h \mathbf{Z} \operatorname{softmax}\left(\mathbf{Z}^{\mathsf{T}}\mathbf{W}_h\mathbf{Z}\right), where softmax⁡(⋅)\operatorname{softmax}(\cdot) is applied column-wise so that the attention weights in every head and column are nonnegative and sum to one (full experimental details are provided in Appendix B).

In our experiments, we consider two cases: H=4H = 4 and H=64H = 64.

As shown in Figure 4 (Appendix B), across both settings (d,n)=(16,32)(d,n) = (16,32) and (12,24)(12,24), the 64-head softmax transformer trained with our two-stage procedure closely matches the Heavy-Ball method, whereas the model trained only in Stage 1 performs similarly to gradient descent. Moreover, in Figure 5 (Appendix B), even after two-stage training, the 4-head softmax transformer cannot achieve a convergence rate comparable to that of the Heavy-Ball method.

These results suggest that, because softmax attention cannot naturally express Heavy-Ball dynamics, it requires a large number of heads to effectively learn acceleration in the second stage of training. Nevertheless, the performance of the 64-head softmax transformer confirms that the ability of two-stage CoT training to discover Heavy-Ball acceleration is not merely an artifact of using unnormalized or signed linear attention.


References

[1] Huang et al., Transformers learn to implement multi-step gradient descent with chain of thought. ICLR 2025.

[2] Ahn et al., Transformers learn to implement preconditioned gradient descent for in-context learning. NeurIPS 2023.

[3] Gatmiry et al., Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.

[4] Takanami and Pehlevan, An asymptotic theory of chain-of-thought in in-context learning. NeurIPS 2026.

[5] Zheng et al., On mesa-optimization in autoregressively trained transformers: Emergence and capability. NeurIPS 2024.

[6] Wu et al., Looped Transformers with Layer Normalization Provably Learn the Power Method, arXiv:2606.00605.

Thanks to the authors for the response and additional experiments! It's interesting to see that softmax attention can also learn similar Heavy-Ball behavior. I understand how challenging it is to analyze more realistic transformers. Still, I feel that the focus on Heavy-Ball acceleration in the linear-attention setting makes the work somewhat of a theoretical exercise, and I'm not sure how naturally this connects to practical transformers. Overall, I think the work makes a solid contribution, so I'll maintain my recommendation for acceptance.

Thank you for the thoughtful feedback and for maintaining your recommendation. We appreciate your supportive and insightful comments. We agree that extending the theory to more realistic transformer architectures is an important next step, and we view this as a valuable direction for future work. We also note that linear attention has been used in operator learning [1], where the goal of learning solution operators for numerical tasks is closely related to our study of learning algorithms. Thus, we believe such areas may offer concrete settings in which our theoretical insights can be further investigated.


References

[1] Cao, Choose a transformer: Fourier or galerkin. NeurIPS 2021

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.