acceptodds
Under review as a conference paper at ICLR 2027

Tracing the Formation of Alternative Internal Solutions in Transformers

Abstract

How do transformers arrive at alternative internal solutions to the same state-tracking task? We study cyclic state tracking, where distinct Fourier modes can each encode the complete state. Under standard training on , runs with balanced two-mode representations converge rapidly, whereas single-mode-dominant runs converge slowly. Encouraging balanced mode energy yields balanced endpoints across runs; most converge rapidly, but a minority remain slow. Tracing the modes through the final MLP reveals a difference in how these representations are formed: in slow runs, one mode depends directionally on another, as shown by removal and rotation interventions. This intervention signature, identified in discovery runs, also separates the convergence regimes in held-out runs. Extending the analysis to the three-mode setting of , we find that faster convergence is likewise associated with energy distributed across multiple modes, while differences in their causal contributions are associated with cross-mode dependence. Together, these findings show that coexisting representations need not reflect independent computational pathways, highlighting the importance of tracing how they arise within the network when interpreting a model’s internal solution.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, representation learning, training dynamics, state tracking, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ES5qOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors studies alternative solutions learned by transformers on a cyclic state tracking task. The authors find that the task can be completed with two modes. Without regularization, most seeds are dominated by one mode; with regularization, all seeds have balanced energy across both modes and accelerated learning. The authors further show that similar energy distribution across modes does not necessarily guarantee similar functional contribution to the task output. In the few regularized runs that still converges slowly, the final MLP layer builds the weaker mode from the dominant one. This dependence correctly identifies the slow runs among the new runs, and a similar pattern appears in a version of the task with three modes.

Critical strengths
  1. The directional dependence experiment is pretty neat. Changing one mode before the last layer and checking the other mode after it in both direction gives direct evidence that one mode is built from the other.
  2. I like how the authors went from descriptive (energy share) to functional test to interventional study.
  3. Nice message that a mode can carry energy and be decodable without being computed independently. This is a useful caution for interpretability work that relies only on energy or decoding.
Critical weaknesses
  1. I find the paper very hard to follow, especially when specific terms or jargons were used but never introduced. Although the results seem interesting, I don't think the paper is ready for acceptance in its current form.
  2. Insufficient context and literature review in the introduction.
  3. The paper analyzes converged solutions entirely. There is no learning dynamics analyzed at all.
  4. I find the analyses restricted to the toy setting of the cyclic state tracking task under a specific form of regularization. The authors didn't discuss why they choose this specific form of regularizer either. Without discussion or experiments on how the results here may generalize to other forms of regularization or other tasks, I find the significance of this work very limited.
Questions for authors
  1. Why do you choose this specific form of regularization?
  2. Do any unregularized two-mode runs show one-way dependence across modes?
  3. For those one-way dependence networks, could there be a single-mode phase before the second one emerges?
  4. What happens if you change regularization strength or applied it in a middle or early layer instead?
1 comment
AuthorsOpenReview ↗

Author Response

Thank you for the careful review and constructive questions. We revised the manuscript and uploaded an updated PDF. We address your concerns below and welcome any follow-up questions.

W1–W2: Presentation and context

I find the paper very hard to follow, especially when specific terms or jargons were used but never introduced.

Insufficient context and literature review in the introduction.

We rewrote the title, abstract, and Introduction around the question of whether coexisting codes for the same state are formed independently. Section 2 now introduces the task, gives the state-centroid Fourier decomposition and energy definition, and explains the functional and cross-mode assays at first use. We also revised the Results transitions and expanded Related Work to connect the study to Fourier circuits, state-tracking mechanisms, and the interpretation of interventions.

W3 & Q3: Learning dynamics

The paper analyzes converged solutions entirely. There is no learning dynamics analyzed at all.

For those one-way dependence networks, could there be a single-mode phase before the second one emerges?

We added checkpoint analyses for all 25 fast and five slow regularized C5C_5 discovery runs (Appendix Figure 12). In the slow runs, the source-mode signal develops before the target-mode signal, and the forward removal and phase-transfer responses rise before stable convergence. This is consistent with source-first formation. The archived checkpoints do not establish a sharply bounded single-mode phase or show that this sequence causes slow learning.

Q1: Choice of regularizer

Why do you choose this specific form of regularization?

We chose an objective that directly encourages the measured harmonic energies to balance. It changes code strength without directly requiring equal contributions to prediction or independent code formation. This makes the contrast between similar endpoint energies and different functional or dependence patterns testable. We clarified this rationale in Section 3.2 and Appendix J.

Q2: Ordinary-training two-mode runs

Do any unregularized two-mode runs show one-way dependence across modes?

We applied the same directional interventions to all ten ordinary-training C5C_5 runs with two-mode endpoints (Appendix Figure 11). Their mean forward-minus-reverse effects are −0.143-0.143 for removal and −0.049-0.049 for phase transfer, compared with 0.7210.721 and 0.7530.753 in the five slow regularized runs. Thus, these baseline runs do not show the comparably strong positive one-way signature seen in the slow regularized group. We do not infer that their pathways are strictly independent.

W4 & Q4: Sensitivity and scope

What happens if you change regularization strength or applied it in a middle or early layer instead?

At λ=0.25\lambda=0.25, 0.50.5, and 1.01.0, the fast-run counts are 17/30, 25/30, and 28/30, respectively; slow runs remain at every tested strength (Appendix Figure 14). Moving the objective from the endpoint to after block 1 or 4 reduces the number of two-mode endpoints to 8/30 or 12/30, versus 30/30 when applied at the endpoint (Appendix Figure 8). We also added width, depth, and C11C_{11} comparisons (Appendix Figures 7, 9, and 10). These extend the tests within the cyclic state-tracking family. Other regularization objectives, tasks, and model families remain untested.

Please let us know if these responses address your concerns.

Reviewer_4q7wOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this paper, the authors study solutions transformers learn on a synthetic task. The task is a cyclic state tracking task, which has a natural solution in terms of Fourier components. They train a suite of transformers and then study which of two solutions the transformers represent and how exactly they encode the kind of hidden states associated with the solution to this task. In particular, there are two different frequency solutions, and they measure those using centroids and other various interventions and methods. Runs that use both frequencies learn faster, and the authors add a regularizer that encourages both, but some runs remain slow. The main conclusion is that different transformers find what they call different solutions. In particular, they find that in the slow group, one of the solutions' representations is built out of the other one in the final MLP, whereas in the fast group, the two solutions kind of exist in parallel or side by side.

Critical strengths

They give it a task which has a natural solution in terms of Fourier components, this serves as a natural, concrete solution to analyze the data relative to. It is a kind of natural ground truth. It's not a ground truth in the deep sense that a neural network must learn it, but it is a natural ground truth given the task structure.

They use this setting to study which of multiple natural solutions exist in the network. This includes some nice causal interventions, and multiple experimental settings and seeds.

Critical weaknesses
  1. It's not obvious why I care about these results. They don't motivate why this particular task is interesting, other than that it's some random task that has a mathematically motivated natural solution, so it's hard to know why they're doing it in particular. The stated reason is that coexisting representations don’t need to reflect independent computational pathways. But that is already known from the probing literature, and the paper doesn't show a case where it changes an interpretation beyond a task built so that the answer is known. To be clear, the limitations section says this, so I am not claiming the authors are unaware of this.

  2. The two different solutions are defined as two solutions to humans, not the network. They say there's a natural solution, but it's a natural solution to a human. The whole point is what's intrinsic to the transformer. More concretely, actions are uniform, so relabeling every state and action by s to 2s and a to 2a (mod 5) is the same task, and maps the m=1 code onto the m=2 code. This means that mode 1 and mode 2 dominant solutions are the same. The meaningful distinction is single-mode vs. balanced. The paper concedes this at the end of the intro (the novelty is "not the existence of multiple internal solutions"). This makes the title and abstract too much and overselling.

  3. I found the paper extremely difficult to read. I was confused even by the intro paragraph. e.g. what does energy mean? "Energy" is used in the abstract and introduction but only defined in Section 2, and the paper never says plainly what the alternative solutions are. I had to figure that out myself by piecing together Sections 2, 3.1 and 3.3. This is just one specific example. There are many parts, I’m sorry to say, that feel AI written. I am of course not sure this is the case. Even if this is not the case, the paper is in many ways substantially too difficult to read in order to recommend publication.

Questions for authors

You use centroids but within-state variance is never reported. Please report per-mode within state variance of the activations. Is the weaker mode's lower usability in slow runs explained by the noise within a centroid cluster?

3 comments
AuthorsOpenReview ↗

Author Response

Thank you for the careful review. We revised the manuscript and uploaded an updated PDF. We address your concerns below and welcome any follow-up questions.

W1: Motivation and contribution

It's not obvious why I care about these results. They don't motivate why this particular task is interesting

We rewrote the title, abstract, and Introduction to make the motivation clearer. In cyclic state tracking, each Fourier harmonic can encode the same entire state. This lets us compare the strength, isolated usability, and computational dependence of coexisting codes in one trained model. In the slow regularized runs, a code represents the state and contributes to predictions, yet interventions show that its formation across the final MLP depends on another code. We limit this conclusion to the studied models and task.

W2: Symmetry of the two modes

relabeling every state and action by s to 2s and a to 2a (mod 5) is the same task, and maps the m=1 code onto the m=2 code.

We agree. The mode labels can be exchanged by this relabeling, so we do not interpret m=1m=1-dominant and m=2m=2-dominant runs as distinct kinds of solution. We clarified in the Introduction that relabeling swaps which mode is dominant, and updated Figures 1 and 2 to group both cases under “Single-mode dominant.” The comparison is with two-mode runs. For the intervention analysis, source and target are defined within each run, without assigning a special role to either mode number. We also revised the title and abstract to focus on dependence between coexisting codes.

W3: Presentation

I was confused even by the intro paragraph. e.g. what does energy mean?

We rewrote the opening sections to introduce the task and its complete harmonic state codes before interpreting their measurements. Section 2 now gives the Fourier decomposition of the state centroids, defines the energy of a real harmonic pair as Et,m=2∥ft,m∥22E_{t,m}=2\|f_{t,m}\|_2^2, and explains what the functional and cross-mode interventions test. We also revised the Results transitions to state the question addressed by each analysis and what its result supports.

Q1: Within-state variance

Please report per-mode within state variance of the activations. Is the weaker mode's lower usability in slow runs explained by the noise within a centroid cluster?

We added measurements for all 30 regularized C5C_5 discovery runs (Appendix Figure 13, page 29). For each mode, WmW_m measures the mean squared spread around its state centroids, while Sm/WmS_m/W_m compares the between-state signal with that spread. In the five slow runs, the lower-causal-share mode has median Wm=562.7W_m=562.7, versus 112.4112.4 for the higher-share mode; their median Sm/WmS_m/W_m values are 1.391.39 and 4.944.94. In the fast group, the corresponding signal-to-spread ratios are 4.914.91 and 5.095.09.

The slow group's lower-share mode therefore has a less distinct state signal in these coordinates. This is consistent with its lower isolated usability, but does not establish that within-state dispersion causes it. We added this result and qualification in Section 3.3 and alongside Appendix Figure 13.

Please let us know if these revisions address your concerns.

Thanks for the response and the new experiments. I appreciate that you agree on the relabeling point and changed the framing. I still have concerns about the motivation and how hard the paper is to read, which is why I'm not going to increase my score.

AuthorsOpenReview ↗

Further clarity revisions

Thank you for reading the revision and for your response. We understand that motivation and readability remain concerns.

We revised the progression of Section 3:

  • Section 3.1: Two-mode baseline runs converge earlier. Does encouraging balanced energy make every run fast?
  • Section 3.2: Regularization balances energy in every run, but some remain slow. Do the two codes contribute similarly to predictions?
  • Section 3.3: In the slow runs, the codes have different causal shares and differ when tested individually. How are these codes formed within the network?
  • Section 3.4: The mode with lower causal share gains energy across the final MLP. Interventions reveal directional cross-mode dependence in the slow runs.

We clarified the terminology in the Results and figure captions. The assay names causal share, mode-only recovery, and isolated rotation now appear in italics consistently. We replaced “higher-share mode” with “mode with higher causal share,” and “recovery” with mode-only recovery.

The regularization objective and its notation are now explained where they first appear. We made a small revision to the Abstract for clarity as well.

Thank you again for the time and attention you have given the paper. We realize the discussion period is nearly over and do not expect a further reply; we wanted to let you know about these additional changes.

Reviewer_61KUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors study cyclic state tracking in causal transformers. For prime cycles, each Fourier harmonic gives a complete 2D code; C_5 admits two codes, C_7 three. Under baseline C_5 training, 10 of 30 seeds reach two-mode solutions and converge faster while 20 single-mode solution seeds are slower. Adding a regularizer makes all 30 two-mode solutions, but 5 of 30 still converge slowly. Removal and rotation interventions show that in the slow runs one coexisting code is directionally synthesized from the other. A score derived from the discovery cohort separates all 61 fast and 9 slow runs on the 70 held-out seeds. A C_7 three-mode extension replicates the C_5 results.

Critical strengths
  • The work presents a thorough and clear (but not clearly presented) result that provides insight into the training time variability observed empirically using a simple task that permits theoretical analysis and empirical investigation.
  • Section 3.1 reports a clean and interesting finding on the baseline C_5 cohort.
  • Extension to the C_7 results provide a convincing extension of the primary finding.
Critical weaknesses
  • The primary weakness of the paper is presentation and the main reason I recommend a weak rejection. The work is sound and interesting, but very hard to grasp and the authors must do a better job explaining their work clearly. The abstract and introduction are impenetrable without substantial prior context. Multiple undefined terms are used in the abstract making it virtually impossible to understand the forthcoming results. The introduction continues the pattern. A general ICLR reviewer without background in Fourier analyses of modular arithmetic in transformers cannot extract what the paper claims without substantial effort.
  • Poor presentation plagues the main text as well. Several central notions are not sufficiently explained in the main text, and where they are defined elsewhere the main-text first-use often does not point to the appendix. (i) E_m, the energy of Fourier pair m, is introduced in Section 2 with no formula; the definition as the mean squared contribution of the real pair at m, n−m lives only in Appendix B.1. (ii) Energy measures, functional assays, and cross-mode assays are introduced at lines 093-099; equations 13-15 defining them live in Appendix F, and the main-text never cites it.
  • The results sections lean on confusing transition phrases that do not state in plain language how a presented result relates to the ideas at the outset of the paper or how they should be interpreted. For example, line 154: "This dissociation changes the explanatory question." I have no idea what this means. A reader is left to figure out from context what "the explanatory question" is. The paper would be substantially more accessible if each section stated plainly (a) what central question the forthcoming section was addressing, (b) how the results demonstrate each claim and (c) an interpretation of the results in context of the main claims of the paper.
  • No code or model checkpoints are released. The paper's entire empirical story depends on specific per-seed numbers that cannot be verified.
Questions for authors
  • Do the slow discovery runs converge if trained past 200k steps, or do they stay stuck?
  • Does the provenance score correlate with convergence time within the fast group, or only across the fast/slow divide?
  • How does the finding scale with model size? Would wider or deeper transformers show the same signature, or does the picture simplify (or disappear) at scale?
  • Does the finding generalize to larger primes (C_11, C_13)?
5 comments
AuthorsOpenReview ↗

Author Response (1/2)

Thank you for the careful review and constructive feedback. We have revised the manuscript and uploaded the updated PDF and anonymized code supplement. We address your concerns below and welcome any follow-up questions.

W1–W3: Presentation

“The abstract and introduction are impenetrable”; central notions are not explained at first use.

We rewrote the title, abstract, and Introduction around the question of whether coexisting codes for the same state are computed independently or whether one helps form another. Section 2 now gives the state-centroid Fourier decomposition and defines the energy of a real Fourier pair as Et,m=2∥ft,m∥22E_{t,m}=2\|f_{t,m}\|_2^2. It then defines energy shares and effective mode count. Section 2 also introduces the functional and cross-mode assays in plain language. The Results sections specify the interventions, with detailed protocols in the Appendix.

We revised the Results transitions as well. For example, the discussion following Figure 3 now asks where the two codes become available and whether they depend on one another, rather than referring to an unexplained “change in the explanatory question.”

W4: Code and per-seed results

“No code or model checkpoints are released.”

The Supplementary Material now includes an anonymized code ZIP with training, intervention, analysis, and plotting implementations; selected run-level numerical tables; checks; and execution instructions. This permits inspection of the per-seed comparisons and numerical summaries. Model checkpoints and some intermediate inputs are not included, so we do not claim that every model-based assay can be rerun from the ZIP alone. The README and Reproducibility Statement specify these limits.

AuthorsOpenReview ↗

Author Response (2/2)

Q1

Do the slow discovery runs converge if trained past 200k steps, or do they stay stuck?

Yes. We continued the C7C_7 baseline run that had not converged by 200k steps. It met the stable-hit criterion at step 300,000.

Q2

Does the provenance score correlate with convergence time within the fast group, or only across the fast/slow divide?

No meaningful correlation was found within the fast group. Among the 61 fast held-out C5C_5 runs, Pearson r=0.050r=0.050 and Spearman ρ=−0.018\rho=-0.018. We report this in Appendix J.

Q3

How does the finding scale with model size? Would wider or deeper transformers show the same signature, or does the picture simplify (or disappear) at scale?

We added width-128 and width-512 comparisons (Appendix Figure 7) and a 12-block C5C_5 comparison (Appendix Figure 9), each with 30 baseline and 30 regularized runs. As width increases, convergence generally becomes earlier and baseline two-mode endpoints become more common (1/30 at width 128; 26/30 at width 512). In the 12-block models, balanced regularized endpoints still vary in convergence time (10.2k–42.4k steps), and energy allocation can differ from causal allocation. The distinction persists in these tested architectures; we do not infer a general scaling law.

Q4

Does the finding generalize to larger primes (C_11, C_13)?

We added 30 baseline and 30 regularized C11C_{11} runs (Appendix Figure 10). Twenty-five baseline runs and all 30 regularized runs converge within 200k steps. Despite broadly distributed energy in the regularized runs, energy and causal shares can differ, and the interventions reveal cross-mode dependence. We have not tested C13C_{13}.

Thank you again for these questions. Please let us know if our responses address your concerns.

Thank you for these additional simulations. I will examine the revision to see if the clarity has increased, which was the main reason for the weaker score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.