acceptodds
Withdrawn from ICLR 2027

A Comparative Study of In-Context Linear Regression by Transformers with Chain-of-Thought

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers have demonstrated strong capabilities in multi-step reasoning, with chain-of-thought (CoT) emerging as a powerful prompting and modeling technique. To theoretically understand the capabilities of transformers with CoT, a line of recent work has studied settings in which transformers generate the iterates of an algorithm either through layer-wise propagation, which can be viewed as a form of ”implicit CoT” (Saunshi et al., 2025), or through autoregressive ”explicit CoT” steps, which directly produce the intermediate iterates (Huang et al., 2025a). However, it remains unclear how the transformer and CoT setup affect the model's capabilities. To better understand these mechanisms in a controlled and mathematically concrete setting, we study a single-layer linear self-attention model on two in-context linear regression tasks: weight prediction and query prediction. Using multi-step gradient descent (GD) as the teacher algorithm, we compare when implicit and explicit CoT can or cannot simulate the target iterative computation. Our empirical and theoretical results reveal a sharp contrast in this stylized setting. On weight prediction, implicit CoT fails whereas explicit CoT succeeds; on query prediction, explicit CoT fails, whereas implicit CoT can simulate the GD trajectory up to a fixed output sign transformation and succeeds directly with an added scratchpad. We identify two corresponding bottlenecks: context overwriting during recurrent updates and insufficient writable state for autoregressive token generation. We further show that adding a scratchpad resolves the failure of implicit CoT on weight prediction and enables explicit CoT on query prediction. In addition, we provide a construction showing that the computation realized by explicit CoT can be exactly implemented by implicit CoT with a scratchpad.

Primary area:
learning theory
Keywords:
Implicit Chain-of-Thought, Explicit Chain-of-Thought, In-Context Learning, Transformer, Scratchpad

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_r9G8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores two existing paradigms of chain-of-thought reasoning, implicit and explicit CoT, specifically in linear self-attention transformers. Both approaches have been studied in previous work to understand the settings in which they can be useful. This paper brings these approaches into a unified framework and compares their expressivity on two tasks, i.e. weight prediction and query prediction.

The paper first recovers the previously known results that explicit CoT can implement multi-step gradient descent for weight prediction, while implicit CoT can do so for query prediction with a sign correction. More importantly, the authors establish new negative results showing implicit CoT cannot implement the same multi-step GD trajectory for weight prediction, and explicit CoT cannot implement even a single GD step for query prediction under their respective formulations.

They attribute these limitations to context overwriting in implicit CoT and limited writable state in explicit CoT. This motivates introducing an additional scratchpad that allows both methods to overcome these limitations. The authors provide theoretical results on expressivity, training convergence and generalization, including results supporting the scratchpad constructions, along with experiments on linear regression and nonlinear LEGO tasks to support their findings

Critical strengths

(1) The paper brings together two existing paradigms. These are often studied separately, so evaluating them in a unified framework is interesting. More importantly, this comparison highlights the role of scratchpads in overcoming their different limitations, which could potentially help with designing future architectures.

(2) The math is quite comprehensive, and the experiments are selected appropriately to highlight the differences between the methods.

Critical weaknesses

(1) The paper in general seems to overstate the significance and novelty. The positive results for implicit and explicit CoT in their respective settings are largely from existing works, and the primary new theoretical contributions are the negative results. However, I am not sure how surprising these results are given that the two methods operate quite differently, and the failures depend on specific choices of input encoding and architecture. In particular, it is not clear why we should expect either method to work in a setting designed around the other in the first place. While the negative results are interesting for highlighting their differences, I am not sure how much broader insight they provide. I think the most interesting contribution is showing how scratchpads can overcome these different limitations within the same framework. More emphasis should be put on these. Scratchpads improving transformers is not an entirely new concept, so experiments need to meaningfully highlight this.

(2) The paper is quite heavily limited to the linear self-attention setting. The authors mention that some results can be extended to regular softmax attention, and provide an additional LEGO experiment with softmax and the scratchpads. However, the evidence (experimental or theoretical) stating that the GD phenomenon carries over to softmax in some shape or form even if not strictly is either not there or does not receive enough emphasis. More direct experiments addressing the softmax setting are either not present or not appropriately emphasized.

Questions for authors

(1) What are the main new insights from the negative results? What broader implications do these results have, in particular what is new about the scratchpad construction compared to previous work?

(2) Do the authors expect the GD phenomena to carry over to regular softmax transformers? Can you provide direct experiments on weight and query prediction with softmax attention to support this?

1 comment

1. (W1+Q1) The paper in general seems to overstate the significance and novelty. The positive results for implicit and explicit CoT in their respective settings are largely from existing works, and the primary new theoretical contributions are the negative results. However, I am not sure how surprising these results are given that the two methods operate quite differently, and the failures depend on specific choices of input encoding and architecture. In particular, it is not clear why we should expect either method to work in a setting designed around the other in the first place. While the negative results are interesting for highlighting their differences, I am not sure how much broader insight they provide. I think the most interesting contribution is showing how scratchpads can overcome these different limitations within the same framework. More emphasis should be put on these. Scratchpads improving transformers is not an entirely new concept, so experiments need to meaningfully highlight this. What are the main new insights from the negative results? What broader implications do these results have, in particular what is new about the scratchpad construction compared to previous work?

A1: Thank you for this insightful comment. We agree that we should state the scope and novelty of contributions more carefully. We would like to clarify that only two of the eight settings in our comparison come from prior work: explicit CoT for weight prediction with a scratchpad, following [1], and the looped transformer for query prediction without a scratchpad, following [2]. We will clearly identify these two cases as existing results rather than contributions of this work. The remaining six settings complete the comparison across model formulation, task, and embedding dimension. For the four successful cases, we provide constructive expressivity results and corresponding training guarantees. For the two failure cases, we provide expressivity lower bounds showing that the target GD updates cannot be exactly implemented under the specified encodings. We believe the main contribution is not any individual case, but the systematic upper- and lower-bound characterization across these eight controlled settings.

We also agree that the general usefulness of scratchpads is already known and is not itself our contribution. Our contribution is to characterize the representation capacity provided by a scratchpad within a controlled theoretical framework. We believe that our results are more appropriately framed as upper- and lower-bound results for looped transformers and transformers with CoT under different embedding dimensions. In fact, we believe our analysis can lead to matching upper and lower bounds in each case. We will revise the paper accordingly.

Regarding the broader implications, since our work is theoretical and focuses on a controlled setting, we believe its implications are most direct for areas such as AI for Science, where linear attention mechanisms are often considered [3] and where models are expected to learn tasks that may also be addressed by classical numerical algorithms. For these tasks, our work suggests that scratchpad-like hidden dimensions may serve as an important computational resource, enabling transformer models to emulate certain iterative algorithms more efficiently.

2. (W2+Q2) The paper is quite heavily limited to the linear self-attention setting. The authors mention that some results can be extended to regular softmax attention, and provide an additional LEGO experiment with softmax and the scratchpads. However, the evidence (experimental or theoretical) stating that the GD phenomenon carries over to softmax in some shape or form even if not strictly is either not there or does not receive enough emphasis. More direct experiments addressing the softmax setting are either not present or not appropriately emphasized. Do the authors expect the GD phenomena to carry over to regular softmax transformers? Can you provide direct experiments on weight and query prediction with softmax attention to support this?

A2: We conduct softmax attention experiments on all eight combinations of task, model formulation, and embedding. We find that under our controlled architecture and training protocol, none of the eight configurations recovers the multi-step GD trajectory. These results suggest that the behavior observed with linear attention does not transfer directly to softmax attention in our setting. We will include these experiments in the revised work.

[1] Huang et al. Transformers learn to implement multi-step gradient descent with chain of thought. ICLR 2025.

[2] Gatmiry et al. Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.

[3] Cao, Choose a transformer: Fourier or galerkin. NeurIPS 2021.

Reviewer_kjV5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares recurrent implicit CoT and autoregressive explicit CoT for reproducing gradient-descent trajectories in linear regression. It attributes different outcomes to context overwriting and insufficient writable state, and gives scratchpad constructions that overcome these bottlenecks. It also presents training and OOD analyses, supported by synthetic experiments.

Critical strengths

I like the distinction between preserved context and changing computational state, it provides a useful framework for studying iterative computation. It's also nice to see how additional representation space enables algorithm execution. The experiments examine these settings.

Critical weaknesses

W1 The implicit-query comparison changes the input encoding. Section 4.1 suggests that an output sign change suffices, whereas Appendix C.4 supplies the residualized context y−Xβ0y-X\beta_0. For arbitrary β0\beta_0, this additional information is essential; the advertised comparison requires qualification.

W2 The unconditional Gaussian OOD bounds cannot hold as stated. Even noiseless scalar GD has error proportional to E(1−ηS)2L\mathbb{E}(1-\eta S)^{2L}, with S∼χN2S\sim\chi_N^2. Unbounded spectral tails produce unstable contexts and eventual growth, contradicting a uniformly decaying depth-dependent bound.

W3 Theorem C.24’s invariant-subspace claim fails at initialization. For one supervised step in the noiseless setting, direct differentiation gives ∇W11L=−ηN(N+d+1)Id≠0\nabla_{W_{11}}\mathcal{L}=-\eta N(N+d+1)I_d\neq0, although this block is claimed to remain zero. The associated training proof therefore needs correction.

W4 The broader comparison is narrow. Generated history is masked away, and scratchpads change representation width, encoding, and sometimes auxiliary supervision. The results do not characterize general implicit versus explicit reasoning.

Questions for authors

Please see weaknesses above. I am in particular concerned about the limited evaluation and empirical evidence. More data and tasks, more and larger models, and more steps would be informative.

1 comment

1. (W1) The implicit-query comparison changes the input encoding. Section 4.1 suggests that an output sign change suffices, whereas Appendix C.4 supplies the residualized context y−Xβ0y-X\beta_0. For arbitrary β0\beta_0, this additional information is essential; the advertised comparison requires qualification.

A1: Thank you for pointing this out. We agree that the description in Section 4.1 is inaccurate. The construction in Appendix C.4 makes a modification to the input encoding. Specifically, the original label block y⊤y^\top is replaced by y⊤−β0⊤X⊤y^\top - \beta_0^\top X^\top, so that the modified input embedding becomes

X^\top & x_q \\\\ y^\top - \beta_0^\top X^\top & -\beta_0^\top x_q \end{bmatrix}.$$ We will correct Section 4.1 and clarify that the result requires both this modified input embedding and the output sign change. >**2. (W2)** The unconditional Gaussian OOD bounds cannot hold as stated. Even noiseless scalar GD has error proportional to $E(1-\eta S)^{2L}$, with $S \sim \chi_N^2$. Unbounded spectral tails produce unstable contexts and eventual growth, contradicting a uniformly decaying depth-dependent bound. **A2:** We agree that the unconditional Gaussian OOD bounds are incorrect. We will replace these results with high-probability statements. Specifically, on the event $$\frac{1}{2}\lambda_0N \leq \lambda_{\min}(X^\top X) \leq \lambda_{\max}(X^\top X) \leq 2\lambda_1N,$$ which holds with probability at least $1-e^{-cN}$, the stated OOD bounds hold. We will revise the corresponding theorems and proofs. >**3. (W3)** Theorem C.24’s invariant-subspace claim fails at initialization. For one supervised step in the noiseless setting, direct differentiation gives $\nabla_{W_11} L = -\eta N(N+d+1)I_d \not= 0$, although this block is claimed to remain zero. The associated training proof therefore needs correction. **A3:** Thank you for identifying this error. We acknowledge that the proof of Theorem C.24 is incorrect by omitting some nonzero terms. In the revised theorem, we will explicitly constrain $W_{11}$ and $w_{24}$ to remain zero during training and recompute the remaining gradients. >**4. (W4)** The broader comparison is narrow. Generated history is masked away, and scratchpads change representation width, encoding, and sometimes auxiliary supervision. The results do not characterize general implicit versus explicit reasoning. **A4:** We agree that our results do not characterize implicit versus explicit CoT in general. As a theoretical study, our analysis relies on controlled assumptions, including the model architecture, mask, encoding, and supervision method. These restrictions make a rigorous comparison possible, but directly extending the results of upper and lower bounds to the broader concepts of implicit and explicit CoT overstates their scope. We believe that our results are more appropriately framed as upper- and lower-bound results for looped transformers and transformers with CoT under different embedding dimensions. In fact, we believe our analysis can lead to matching upper and lower bounds in each case. We will revise the paper accordingly. >**5. (Q1)** I am in particular concerned about the limited evaluation and empirical evidence. More data and tasks, more and larger models, and more steps would be informative. **A5:** We agree that the empirical evaluation in the current manuscript is limited. We have conducted additional experiments with real data and deeper models, which we will include in the revised work. First, we construct an in-context regression task from MNIST. Digits 0 and 1 are mapped to $-1$ and $1$, the images are represented by 64 PCA components, and the same LSA model is trained to follow the teacher GD trajectory. We find that explicit CoT reaches small error for multiple steps with and without a scratchpad, while implicit CoT without a scratchpad does not accurately recover the multi-step GD trajectory. This provides a real-data case study consistent with our synthetic experiment. Second, we evaluate all eight settings using a stack of two linear self-attention layers to represent one GD step, and then iterate the trained stack for multiple steps. The two negative cases identified in the paper continue to exhibit large multi-step errors. Among the other six cases, the three implicit settings achieve small one-step errors but become inaccurate during updating, while the three explicit settings maintain relatively small multi-step errors. This suggests that, for implicit CoT, fitting the one-step objective does not necessarily produce an update that can be applied repeatedly, since the intermediate layer may alter the original context, and the errors increase over multiple steps.
Reviewer_uZymOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a comparative study of in context linear regression by transformers with explicit/implicit cot. It shows that those two different kind of cot behaves differently and identifies the corresponding failure modes. The paper suggests additional writable state can alleviate these failures.

Critical strengths

S1. This work provides a novel theoretical perspective by comparing two tasks on in context linear regression of explicit and implicit CoT.

S2. This paper identifies the mechanisms behind the negative results, and proposes scratchpad to overcome context overwriting.

Critical weaknesses

W1. The failure of explicit CoT appears closely tied to the imposed architectural restrictions. I am not convinced by the proposed limitation of explicit CoT.

  • The model is limited not to observe the former states of CoT. It is a huge gap with the standard real world autoregressive CoT. I think this result mainly reflects a limitation of the prescribed Markovian CoT formulation and its restricted state encoding.
  • To my understanding, this paper only considers the single-head transformer, which limits the expressiveness of transformer. For one-dimensional regression with state (xq,st)(x_q,s_t), two linear-attention heads can use scalar (key, query, value) projections (xi,−ηst,xi)(x_i,-\eta s_t,x_i) and (xi,ηxq,yi)(x_i,\eta x_q,y_i), with both outputs routed only to the prediction coordinate. Adding their outputs through the residual connection preserves xqx_q and gives st+1=(1−η∑ixi2)st+η(∑ixiyi)xqs_{t+1}=(1-\eta\sum_i x_i^2)s_t+\eta(\sum_i x_i y_i)x_q, exactly implementing the GD query-prediction update without an additional scratchpad.

W2. The analyzed implementation of implicit CoT is restrictive. The paper only focuses on repeatedly applying the same layer to the entire representation matrix. It remains unclear whether the identified context-overwriting failure persists in other implementations with different mechanisms for passing latent states, such as the Full-bandwidth Transformer. I believe this work would be more insightful if it characterized how different implementations of implicit or latent-feedback reasoning affects the learning in different tasks.

Questions for authors

Q1. If we allow the model to use multiple attention heads, does your conclusion still hold? In particular, does explicit CoT still fail on query prediction?

Q2. If we consider different implementations of implicit CoT, such as the Full-bandwidth Transformer, does context overwriting still occur?

Q3. What suggestions do your findings provide for real-world model architecture design? Specifically, what would the proposed scratchpad correspond to in a practical transformer—additional hidden dimensions or some other architectural change? Also, are there observed failure modes in real-world reasoning models that correspond to the failures identified in your analysis?

1 comment

1. (W1+Q1) The failure of explicit CoT may be caused by the restricted Markovian state, masked history, and single-head architecture. And, for one-dimensional case, two heads with projections (xi,−ηst,xi)(x_i,-\eta s_t,x_i) and (xi,ηxq,yi)(x_i,\eta x_q,y_i) exactly implement the GD update without a scratchpad. Does the failure persist with multiple heads?

A1: Thank you for your insightful comment. We agree that our results depend on the assumptions of the theoretical settings we consider, such as linear attention and the Markovian CoT formulation. As is often unavoidable in theoretical studies, some level of abstraction and restriction to controlled settings is necessary to make rigorous analysis tractable and to allow systematic comparisons across different cases. While these settings were motivated by different forms of implicit and explicit CoT, with or without scratchpads, we agree that directly mapping them to these broader concepts may overstate the scope of our theoretical results. We believe that our results are more appropriately framed as upper- and lower-bound results for looped transformers and transformers with CoT under different embedding dimensions. In fact, we believe our analysis can lead to matching upper and lower bounds in each case. We will revise the paper accordingly.

For the two-head construction you proposed, we appreciate your key observation and acknowledge that a more complex structure may overcome the failure for d=1d=1. When d=1d=1, the scalar state st=xq⊤βts_t=x_q^\top\beta_t retains all information about βt\beta_t needed for the next query prediction, and the two heads can compute the two terms in the GD update separately. However, for d>1d>1, the obstruction under our encoding is not the number of attention heads but the information retained in the generated state. At each step, the model receives only (X,y,xq,st)(X,y,x_q,s_t), where st=xq⊤βts_t=x_q^\top\beta_t. Here, sts_t loses the components of βt\beta_t orthogonal to xqx_q. These components do not affect the current prediction but can affect the next prediction after the GD update.

In fact, we can show that even the optimal predictor based only on (X,y,xq,st)(X,y,x_q,s_t) has a nonzero expected error:

inf⁡fE[(f(X,y,xq,st)−st+1)2]=ηgd2Nd(d−1),\inf_f \mathbb{E}\left[ \left( f(X,y,x_q,s_t)-s_{t+1} \right)^2 \right]=\eta_{gd}^2 N d(d-1),

where the infimum is over all measurable functions ff on (X,y,xq,st)(X,y,x_q,s_t). Since any multi-head transformer under this encoding is one such function, this lower bound is independent of the number of heads.

2. (W2+Q2) The context-overwriting result is derived for repeatedly applying the same layer to the full representation. Does this failure persist in other latent-feedback architectures, such as the Full-bandwidth Transformer?

A2: We agree that full-bandwidth transformers [1] may mitigate or avoid the context-overwriting issue. Our current result is specific to the looped model, which is motivated by recent theoretical studies [2, 3, 4]. In the revision, we will clarify this scope, incorporate and discuss related work such as [1], and try to extend our analysis beyond the looped setting.

3. (Q3) What suggestions do your findings provide for real-world model architecture design? Specifically, what would the proposed scratchpad correspond to in a practical transformer—additional hidden dimensions or some other architectural change? Also, are there observed failure modes in real-world reasoning models that correspond to the failures identified in your analysis?

A3: In our setting, the scratchpad indeed corresponds to additional hidden dimensions. Since our work is theoretical and focuses on a controlled setting, we believe its implications are most direct for areas such as AI for Science, where linear attention mechanisms are often considered [5] and where models are expected to learn tasks that may also be addressed by classical numerical algorithms. For these tasks, our work suggests that scratchpad-like hidden dimensions may serve as an important computational resource, enabling transformer models to emulate certain iterative algorithms more efficiently.

[1] Wang et al. Full-bandwidth transformer. arXiv:2608.08888, 2026.

[2] Saunshi et al. Reasoning with latent thoughts: On the power of looped transformers. ICLR 2025.

[3] Gatmiry et al. Can looped transformers learn to implement multi-step gradient descent for in-context learning? ICML 2024.

[4] Xu et al. To cot or to loop? a formal comparison between chain-of-thought and looped transformers. arXiv:2505.19245 (2025).

[5] Cao, Choose a transformer: Fourier or galerkin. NeurIPS 2021.

Comments on the paper

1 comment

We sincerely thank the reviewers for their careful reading and constructive feedback. After carefully considering all the reviewers' feedback, we acknowledge that the current manuscript requires substantial revision beyond what can be adequately addressed during the rebuttal period. We have therefore decided to withdraw the submission. We will carefully revise the paper based on the valuable feedback we receive and submit the work to a future venue.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.