acceptodds
Under review as a conference paper at ICLR 2027

Learning Dynamics of Chain-of-Thought State Tracking in a Solvable Transformer Model

Abstract

Chain-of-thought generation can turn a multi-step computation into a sequence of locally checkable state updates, but the training dynamics of such state tracking remain poorly understood. We study this question in a solvable setting: a simplified one-block transformer trained by supervised next-token prediction on state sequences generated by composing permutations. The architecture separates fixed-lag token retrieval from a specialized MLP logic module that applies the retrieved permutation to the current state. Using a statistical-physics mean-field description, we derive dynamics for three order parameters measuring correct-token retrieval in attention as well as on- and off-target logic alignment. These solutions quantitatively match simulations for the order parameters and, combined with a Gaussian approximation to the logit distribution, qualitatively predict the sharp transition in final rollout accuracy. The analysis reveals staged learning: the logic module first learns a mixed heuristic, after which attention locks onto the relevant tokens and efficient MLP alignment is enabled. The same mechanism emerges in standard one-block transformers and is missing when either subsystem is frozen at initialization. Together, these results provide a controlled mechanistic account of how attention-based retrieval and MLP-based logic co-develop during chain-of-thought state tracking.

Primary area:
learning theory
Keywords:
learning dynamics, chain of thought, mean-field theory, mechanistic interpretability, state tracking, statistical physics, transformers, permutation composition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_miCqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies joint learning of attention and an MLP logic module on permutation-based CoT state tracking. A mean-field loss and empirical attention closure yield approximate dynamics for three order parameters. These describe an initial mixed heuristic, followed by attention specialization and stronger target alignment. Predictions quantitatively match constrained-model trajectories and qualitatively capture the rollout-accuracy transition. Architectural relaxations and standard one-block transformers show similar stages.

Critical strengths

S1. The A,R,SA,R,S model gives an interpretable quantitative account of the learning stages. Eq. 15 links attention learning to the logic margin R−SR-S, while improved retrieval strengthens target alignment. Fig. 2 supports this mutual dependence.

S2. The staged behavior persists beyond the constrained architecture. In standard one-block transformers, holding attention uniform or freezing the MLP at initialization leaves rollout accuracy near chance, supporting the importance of learning in both subsystems.

Critical weaknesses

W1. The qualitative advance is limited. Yang et al. (2024) already analyze MLP-first learning followed by joint attention–MLP evolution, and Cabannes et al. (2024) identify positional retrieval followed by state updates. The quantitative model is useful, but the central staged-learning picture offers limited conceptual novelty.

W2. The closures need stronger validation. Eq. 15a is empirical; taking η→0\eta\to0 does not remove its approximation error, so Appendix B.3 should qualify its exactness claim. More importantly, marginal prediction symmetry does not justify Eq. 96's recovery approximation: conditioning on an incorrect state can change the states and learned-map entries encountered during rollout. The relevant conditional probabilities are not directly checked.

Questions for authors

Q1. How closely does LcωA(1−A)Lc_{\omega}A(1-A) match the measured attention-gradient factor through the learning transition? Does Appendix B.3's exactness claim as η→0\eta\to0 concern only discretization error, retaining the mean-field and empirical-closure assumptions?

Q2. Around the rollout-accuracy transition, can you measure At,BtA_t,B_t in Eq. 93 and compare BtB_t with (1−At)/(N−1)(1-A_t)/(N-1)? Separately, how well does Eq. 17's logit-based ρ\rho match measured AtA_t? This would separate recovery-closure errors from Gaussian-logit errors.

1 comment

We thank the reviewer for the constructive comments and the precise questions. Q1 and Q2 led to the two main changes of the revision.

W1: Qualitative advance. We agree that the order, MLP first and attention later, is not new. The revised Section 4.2 relates it to both papers and states what our analysis adds. In Yang et al. (2024), a linear MLP classifies all training samples correctly by the end of the first phase, with attention near its initial values. Attention and MLP then enlarge the margin, and the attention probabilities change only by the order of the inverse squared sequence length (their Corollary F.17). In our task the first stage does not solve the problem: with attention held uniform, the rollout accuracy stays at chance (Table 2), and the task is solved only when attention moves from 1/L1/L to nearly 1. Our equations describe this transition. Cabannes et al. (2024) identify the iteration head, which retrieves the current input token by its position so that the MLP can apply the state update, and study its emergence and transfer between tasks; they leave a detailed study of the training dynamics outside their scope. What we add: (i) the first stage is a mixed heuristic, in which on- and off-target alignment both grow; (ii) the margin R−SR - S between them gates attention learning and is opposed by logit noise that grows as attention sharpens; (iii) closed differential equations for three order parameters predict the timing of both stages in the baseline model (general comment); (iv) the rollout transition is a separate, later event: attention saturates at α≈3\alpha \approx 3, and the rollout accuracy rises at α≈5\alpha \approx 5, sharply because the next-token accuracy enters to the LL-th power.

W2/Q1: Gradient factor and exactness. LcωA(1−A)L c_\omega A(1-A) does not match the measured factor through the transition. It is correct at A=1/LA = 1/L, where its coefficient was derived, but too small by a factor of 1.9 at A=0.2A = 0.2 and of about 5.5 near A=0.7A = 0.7; the general comment explains why Fig. 2 agreed nonetheless. We now derive c(A)c(A) instead of fitting its form. The new Fig. 6 (Appendix B.3 of the revised manuscript) compares it with the measured factor throughout training; the two agree to within about 10%. On exactness the reviewer is right: η→0\eta \to 0 removes only the discretization error, while the mean-field evaluation of the loss and the assumptions behind c(A)c(A) remain. Appendix B.3 now says so and lists the remaining approximations.

W2/Q2: Recovery approximation. We agree that symmetry alone does not establish Eq. (96). The equation assumes that, from an incorrect state, the model errs as often as from a correct one, and that its errors land on each of the other N−1N - 1 states with equal probability. We now measure At\mathcal{A}_t and Bt\mathcal{B}_t of Eq. (93) throughout training, including the rollout transition. The measured Bt\mathcal{B}_t show no dependence on the position tt and follow (1−At)/(N−1)(1 - \mathcal{A}_t)/(N - 1) within the resolution of the new Fig. 7 (Appendix B.6).

Q2: ρ\rho versus At\mathcal{A}_t. Here, At\mathcal{A}_t is the next-token accuracy, which can be measured empirically, whereas ρ\rho of Eq. (17) is the next-token accuracy under the assumption of an explicit logit distribution. In the course of the more detailed mean-field investigation, we updated this distribution to include the effects of the uniform initialization of the logic module's matrices (see general comment). A comparison is shown in Fig. 18 (middle panel, Appendix C.5), with the empirical At\mathcal{A}_t, averaged over the positions since there is no observable position dependence, and ρ\rho using the empirical variances. Together, the two comparisons separate the errors of the recovery approximation from those of the logit distribution.

Reviewer_tePMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a simplified transformer on a task that consists in tracking the evolution of a certain state under a sequence of permutation actions. It introduces three order parameters and derives approximate equations for their learning dynamics. These describe logic learning first, followed by attention specialization. Furthermore the final accuracy is predicted using a gaussian approximation.

Critical strengths

Having a simple model that allows attention and logic to be studied together mathematically is very interesting. The relation between attention learning and logic discrimination could also be useful for understanding larger models. The description of the dynamics through only three order parameters is also particularly appealing.

The experiments support the approximate theory. The tests with standard transformer blocks show the same learning order in a less constrained architecture. This suggests that the observed mechanism may have relevance beyond the specific toy model.

Critical weaknesses

The main limitation is the empirical closure

dgN∥∇q,kA∥2≈LcωA(1−A).d_gN\|\nabla_{q,k}A\|^2\approx Lc_\omega A(1-A).

Appendix B motivates this approximation and determines its coefficient at initialization, and the agreement of the ODEs with training gives useful support. However a direct comparison of the measured gradient factor with the closure is missing. This would be helpful to separate its error from the other mean-field approximations.

The authors explicitly state that the equations depend on this closure. Still, the meaning of "solvable" is not fully clear to me, since closing the dynamics requires an empirical approximation.

Questions for authors
  • Can the measured gradient term be compared directly with the proposed closure during training?
  • What is the main difficulty in extending this analysis to deeper transformers? Do the authors have an intuition about which tasks would benefit from additional depth?
3 comments

We thank the reviewer for the positive assessment and the helpful questions.

Empirical closure and comparison with the measured gradient factor. We made this comparison (general comment): LcωA(1−A)L c_\omega A(1-A) is correct at A=1/LA = 1/L, where its coefficient was derived, but too small by a factor of up to 5.5 during the attention transition. Separating the errors, as the reviewer suggested, showed that the mean-field estimate of the loss gradient with respect to AA erred in the opposite direction, by a factor of about 3.5 at A≈0.5A \approx 0.5; the two errors largely compensated in Fig. 2. We replaced both approximations. The gradient factor is now derived, without a fit, and lies within about 10% of the measured one for 0.2≤A≤0.90.2 \le A \le 0.9, see the new figure in Appendix B.3. The loss gradient now includes the leading correction from logit fluctuations; it agrees with the measured one to within 7% for A≤0.3A \le 0.3 and falls below it by up to about 20% later in the transition.

Solvability. We use the term as in the statistical mechanics of learning (Saad & Solla, 1995): the dynamics of all weights reduce to closed differential equations for a few order parameters, here three. The objection was justified for the original submission, where closing the equation for AA required a fitted function. The closure is now derived, and all constants follow from the hyperparameters and the initialization. Unlike the classical cases, which are exact for large input dimension, our equations are not exact in any limit. They rely on (i) a mean-field evaluation of the loss, now with its leading fluctuation correction, (ii) the assumption that the remaining attention 1−A1-A is spread evenly over the other actions, and (iii) the assumption that the distribution of the fluctuations of the learnable matrix entries does not change during training. Appendix B.3 now states these explicitly. The equations are integrated numerically; a closed form exists only for the first stage (Appendix B.5).

Deeper transformers. The reduction to three order parameters uses a property of our block that a second block cannot have: both modules receive fixed inputs. With fixed embeddings and RoPE, the attention pattern depends on the key and query weights but not on the data, and the logic module receives one-hot training inputs. The loss then depends on the key and query weights only through the attention pattern, which the even-spread assumption reduces to AA. These weights therefore move along the gradient of AA, which gives the conservation law behind c(A)c(A). For a second block, the input is the output of the first and changes during training, the attention depends on content and hence on the data, and the gradient reaching the first block passes through the second. A closed description then needs further order parameters, such as overlaps between what the first block writes and what the second reads, invariant under the symmetries of the intermediate representation. We have not attempted this; whether a few such parameters suffice is open. A natural first case is the two-layer iteration head of Cabannes et al. (2024), whose learned mechanism is known.

Tasks that benefit from depth. Per generated token, one block performs one round of retrieval (in our model, at a fixed relative position) and one update of the state. Depth helps when a task demands more in one forward pass. Two cases are established in the literature; neither is a result of ours:

i) Chained retrievals. If the position of the relevant input must itself be computed, one attention layer has to prepare the query of the next. For sequences of variable length, the iteration head of Cabannes et al. (2024) uses the first layer to locate the end of the input and the second to retrieve the current input. For polynomial iteration their results favour two layers over one. Copying what followed an earlier occurrence of the current token (induction head; Elhage et al., 2021) likewise takes two layers: a single layer would have to be exponentially larger (Sanford et al., arXiv:2408.14332). For a chain of kk such references, depth of order log⁡k\log k suffices and, under a conjecture, is necessary (Sanford et al., ICML 2024).

ii) Composed updates. If the intermediate states are not written out, our task requires applying LL permutations in one pass. LL layers would do this step by step, and O(log⁡L)O(\log L) layers suffice by composing neighbouring pairs (Liu et al., 2022; Marchetti et al., 2026). A depth independent of LL suffices only if the actions generate a solvable group (Liu et al., 2022, assuming TC$^0 \neq$ NC$^1$), as for the cyclic vocabulary of Appendix C.4; the random permutations of our main experiments generate the symmetric group S32S_{32}, which is not solvable. With the states written out, one layer suffices (Huang et al., 2025; our model): chain of thought trades depth for generated tokens (Feng et al., 2023; Li et al., 2024).

AuthorsOpenReview ↗

The requested comparison in the revised manuscript

The comparison you asked for is Fig. 6 of the revised manuscript (Appendix B.3). It shows the attention-gradient factor measured during training together with the derived c(A)c(A) of Eq. (14); the two agree to within about 10% for 0.2≤A≤0.90.2 \le A \le 0.9. The approximations on which the equations rest, which bear on the meaning of "solvable", are listed in the same appendix under "Range of validity".

We would be glad to know whether this resolves the concern you raised.

I appreciate the authors' detailed feedback. Although the assumptions are now more clearly defined, they remain quite strong and their realism is hard to judge. Weakening or removing them could enhance the paper further, but for this reason I will maintain my current score.

Reviewer_StfhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a simple task over permutations: you observe a sequence of permutations σ1,⋯ ,σdg\sigma_1,\cdots,\sigma_{d_g} over [N][N] and a random first state s0s_0, and the task is to successively apply the permutations to the previous state, i.e., st+1=σt(st)s_{t+1} = \sigma_t(s_t). For this task, they introduce an architecture based on a self-attention layer and a quadratic layer (a so-called logic module) which can solve it. Then they derive the mean field approximation of the cross entropy and compute a closed-form solution for the gradient flow on that mean field approximation. They experimentally show that the closed-form dynamics match the empirical ones. Additionally, the dynamics of the last state accuracy are calculated using the solutions to the order parameters.

Critical strengths

A key strength is the reduction of learning in the constrained model to three interpretable order parameters. The mean-field loss can be expressed in terms of these parameters, and, with an empirical closure for the attention-gradient factor, their evolution is described by a closed system of differential equations. The predicted trajectories agree closely with the reported simulations. This compact description makes the coupling between attention and logic explicit and provides an explanation of the staged learning behavior within the model.

Critical weaknesses
  • Readability. The paper is difficult to read:

    • The introduction focuses strongly on explainable AI and CoT, which have only a remote connection to this work.
    • The related work seems scattered and incomplete. For example, it lacks any discussion of related mean-field approaches for estimating learning dynamics (e.g., Goldt et al., Gerbelot et al., and Arnaboldi et al.). Comparing the assumptions and closure mechanisms would help clarify what is specific to the proposed analysis and how its empirically motivated closure relates to existing asymptotically justified descriptions.
    • Many choices and claims are not explained (see below).
  • Motivation.

    • The permutation task is motivated through CoT, but this connection is not explained in the main paper. “Learning dynamics of CoT state tracking” in the title seems like an overclaim.
    • The proposed architecture seems very ad hoc and designed to solve this specific problem. The authors mention that the task can be solved directly by a one-layer Transformer, so it is not obvious how to justify introducing this new and unusual architecture. The paper shows that its logic module can be expressed by an MLP, but it gives no justification for why a Transformer should learn such a construction.
    • Overall, it is unclear how well the results transfer to a more realistic Transformer architecture. The dynamics of the relaxations shown in Appendix C.3 seem quite different from those obtained in the main paper. For example, in Figure 13 both R_probe and S_probe blow up, which is qualitatively different from the results in Figure 2.
  • Gaps in the analysis. In some cases, the paper presents interpretations of the results whose origins are not clear. For example:

    • It is not directly clear how the learning-dynamics interpretation in lines 387–394 follows from the closed-form solutions. What is the “mixed heuristic” referenced there, and does the result predict any specific properties of it?
    • In Figure 3, the mismatch between simulated and predicted accuracy is attributed to high empirical variances. To my understanding, the paper does not explain, even in the appendix, why the empirical variances are higher than the theoretical ones. The plot in Figure 6, which uses both logit means and variances, still shows a mismatch between prediction and empirical results.
  • Proof technique. The analysis technique, which could be of interest to the community, is not really explained in the main paper. For example, why was this form of the mean-field approximation chosen and, crucially, how does one arrive at the solutions in Equation 15? A reusable sketch would be helpful.

  • Overall relevance. The relevance of the paper is therefore not clear. The analysis is sound: the authors propose a small transformer-inspired model to solve the task, and for this model they derive a closed-form approximation inspired by statistical physics. But what do we learn from that solution? This is the point that does not convince me. The authors do not mention any implications of the work that would be relevant beyond this particular architecture and task.

References:

  • Sebastian Goldt, Madhu S. Advani, Andrew M. Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup
  • Cédric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborová. Rigorous Dynamical Mean-Field Theory for Stochastic Gradient Descent Methods.
  • Luca Arnaboldi, Bruno Loureiro, Ludovic Stephan, Florent Krzakala, and Lenka Zdeborová, Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks.
Questions for authors
  1. Can you motivate the choice of the architecture studied, given that a single self-attention layer can directly solve the task?

  2. How much do the theoretical results depend on the empirical closure used for the attention dynamics in Eq. (15a)? In particular, would the two-stage learning dynamics still emerge with a different functional form for c(A)c(A), such as the one suggested by Eq. (79)? It would be interesting to understand how much of the agreement with experiments depends on this particular approximation.

  3. According to you, what are the main implications of your work, beyond deriving closed-form dynamics for the mean field problem that match the experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.