acceptodds
Under review as a conference paper at ICLR 2027

Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does

35 views

Abstract

Looped Transformers repeatedly apply the same set of Transformer layers, giving them a recurrent architecture for latent computation. Their strong performance on iterative reasoning and length-generalization tasks suggests an appealing explanation: recurrence may provide an inductive bias that lets the model reuse a learned algorithm across loops. However, weight sharing alone does not imply that every loop performs the same operation. This raises a basic question: is each loop actually repeating the same computation, and if not, what routes the shared parameters to different operations? We study this question using graph walks as a test case. In the model’s native trajectories, decoded predictions can advance by different numbers of graph steps or remain at a reached target, showing that recurrent progress need not follow a fixed one-loop-one-step pattern. We then show that the entering hidden state can steer a frozen loop toward different transitions: a learned affine map selects the desired transition without changing the shared Transformer layers. To test how this steering works, we use activation patching and find that attention patterns can recover its effects and switch the selected transition. Across five pairs of graph models differing only in intermediate supervision, \(J\) can select among computations supported by the frozen backbone but does not create arbitrary new ones. The controllers can be composed across two successive loops, but accuracy is order-dependent and drops with longer compositions. Together, these results show that the entering state can control shared computation, with attention routing as a causal pathway.

Primary area:
interpretability and explainable AI
Keywords:
Looped Transformers, Mechanistic Interpretability, Algorithmic Reasoning, Computational Reuse, Recurrent Computation, Causal Interventions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FLjrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether different loops of a Looped Transformer perform different computations despite sharing the same weights.

  1. Different loops can correspond to different amounts of graph-walk progress (one hop, two hops, or staying at the same node).
  2. After six loops, the paper freezes the Transformer F and train an affine map J applied before the seventh loop. Different maps J_one and J_two produce one-hop and two-hop predictions with 99.6% and 98.2% accuracy.
  3. By patching attention patterns in the seventh loop, the paper shows that attention routing causally influences which transition is predicted.
  4. Intermediate supervision changes which transitions are easier to select using J.
Critical strengths

I think this paper is a good try to explain the mechanisms of looped transformers, and I thought there should be more of such paper to understand the model.

Critical weaknesses

See the question below.

Questions for authors

There is one issue that I want to ask the author: why is the problem studied non-trivial?

  1. Different graph-walk progress does not imply different algorithms.

Consider the linear Transformer F(X)=VXX⊤WX.F(X)=VXX^\top WX.

With V=W=IV=W=I and X0=AX_0=A where AA is the adj matrix, Xt=A3t.X_t=A^{3^t}.

Although the effective walk length increases exponentially, every loop performs exactly the same operation: matrix cubing.

  1. The affine map JJ may simply select existing representations.

It seems this paper is not exponential, but polynomial according to figure 2.

The first six loops may already encode information about A,A2,…A,A^2,\ldots. Then JtwoJ_{\mathrm{two}} could extract the representation needed to predict the two-hop successor, rather than inducing a new algorithm in FF.

  1. Attention pattern couldn't establish a different mechanism since your input changes (it should attend to the next two node if your embedding is actually the next node, they are consistent)
5 comments

Thank you for highlighting the distinctions between task progress, representation selection, and algorithms. We understand your central concern: a single fixed program operating on different states can produce different progress and attention patterns, so why do these observations provide new mechanistic insight? To address this, we added paired interventions from the same native state, stage-specific probes, and component interventions to test when target representations form and how the entering state changes the causal contributions of shared components.

1. Beyond differences in progress: identifying state-dependent component contributions

"Why is the problem studied non-trivial?"

"Different graph-walk progress does not imply different algorithms."

We agree: a fixed update rule can produce different amounts of graph-walk progress. One-hop and two-hop readouts alone therefore do not establish two distinct algorithms, and the finite-loop curves in Figure 2 do not establish asymptotic complexity.

Prior work has explored learning reusable intermediate steps as a route to length generalization in looped Transformers, for example Fan et al., Looped Transformers for Length Generalization, ICLR 2025. This motivates a concrete mechanistic question: in a trained model, how does the entering state organize shared components to implement different task-level transitions?

In frozen D8L8 seed5, we start from the same native state before loop 3, apply either J1J_1 or J2J_2, and execute the same frozen block FF, targeting f3(s)f^3(s) and f4(s)f^4(s), respectively. Neither target wraps around to the starting node. Each controller is shared across unseen test graphs and their different starting nodes. The setup is documented in Appendix I.3, p. 42.

We then ablate second-layer attention heads and evaluate using probes trained on the corresponding clean controlled outputs and subsequently frozen. Specifically, we zero the selected head's answer-token context before the output projection and execute the remainder of the loop. Table R5 and Figure R6 (pp. 42-43) report:

Controlled targetNo ablationAblate H0Ablate H2
J1J_1: f3(s)f^3(s)99.79%38.67%99.06%
J2J_2: f4(s)f^4(s)99.92%67.83%21.86%

Here H0 and H2 are zero-indexed heads in the second layer. H0 is more important for target formation under J1J_1, whereas H2 is more important under J2J_2; the original output head reproduces this difference (Figure R6, upper panel).

Thus, the new evidence goes beyond describing distances traveled: with identical parameters, invocation position, and initial native state, different state interventions change the causal dependence of target formation on specific heads. This provides an empirical account of the functional organization within the shared module.

2. High-accuracy linear readability of the target forms primarily in the subsequent frozen computation

"The affine map J may simply select existing representations."

"The first six loops may already encode information about A, A², ... ."

This explanation requires distinguishing whether JJ already directly produces an easily readable target from whether the subsequent FF must still retrieve, combine, or transform existing information in a state-dependent way.

To test this, we fit separate linear probes at each stage of the loop-3 experiment described in Part 1, using independent training graphs, and evaluate the answer-token representations on unseen graphs. Appendix I.3, Table R5 and Figure R5 (p. 42) report:

Controlled targetAfter JJ, before FFAfter frozen FF
J1J_1: f3(s)f^3(s)22.03%99.79%
J2J_2: f4(s)f^4(s)27.15%99.92%

These results show that the high-accuracy linear readability of the target associated with successful control is not already established immediately after JJ; its main increase occurs during the subsequent frozen computation. The head ablations in Table R5 and Figure R6 further show that this increase depends selectively on different components.

We do not interpret limited probe performance as evidence that target information is absent. Nor do we exclude higher-order relations being stored in graph-side tokens and subsequently accessed through conditional retrieval. Even if such precomputation occurs, it leaves a mechanistic question: how does the entering state determine which information the subsequent module uses, and which components turn that information into an output-ready target? Our additional experiments make this question testable through specific component dependencies.

3. Beyond attention changes: testing current-node readability and the causal role of addressing on the same examples

"Attention pattern couldn't establish a different mechanism since your input changes (it should attend to the next two node if your embedding is actually the next node, they are consistent)"

We agree that changes in attention, including their causal effects on the output, may still be explained by changes in a latent query address. To constrain this explanation further, we test whether the current node remains readable and whether intervening on addressing changes the target on the same examples.

We use the D8L6 seed6 setting with JJ inserted between the two layers, documented in Appendix I.3, Figure R8 (p. 44). The following joint-subset statistics are an additional diagnostic reported here, rather than numbers displayed in Figure R8. A node probe is trained on native states from independent graphs and then frozen. Averaged over three controller fits, 88.66% of test examples simultaneously satisfy both conditions: after either JJ, this fixed probe still decodes u=f8(s)u=f^8(s), and the subsequent computations reach their respective one-hop and two-hop targets.

On this common diagnostic subset, selected using the two unpatched runs, we retain the receiver's V and residual and exchange only the second-layer answer-query attention patterns across all four heads. Transferring the J2J_2 pattern into the J1J_1 run yields 40.74% two-hop target accuracy under the original output head; the reverse transfer yields 51.07% one-hop target accuracy. Thus, while the same node remains readable by a fixed probe, changing addressing weights is sufficient to redirect some examples toward the other target.

This does not imply that all latent addresses remain unchanged; the pattern itself may also transmit address information. It shows that the readable identity of the current node is insufficient to characterize how the entering state affects subsequent computation, and that addressing is a causal pathway through which control operates.

This pathway must also work with content. In the earlier D8L8 loop-3 experiment, transferring the J2J_2 pattern alone into J1J_1 produces only 1.15% two-hop target accuracy under the output head, whereas jointly replacing Q and V reaches 74.63%. These results are reported in Appendix I.3, Figure R7 (p. 43), where the QK and direct-pattern implementations agree. Control therefore cannot generally be reduced to independently switching a single attention pattern: it involves coordination among the query, retrieved content, and receiving state.

Accordingly, we will revise expressions such as "many algorithms" that could equate task-level transitions with independent algorithms. Our mechanistic conclusion is: the computational parameters are shared, while the entering state can change how subsequent components address, retrieve, and use information, and can change the causal contributions of different heads to target formation. This conclusion allows precomputation and conditional retrieval to participate in the implementation. Through generalizable state control and component interventions, our work explains concretely how a shared module supports different task-level transitions.

Reviewer_w3GwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how looped Transformers perform different semantic transitions despite sharing parameters across recurrent steps. Through graph-walk tasks and experiments with task-adapted language models, the authors show that learned hidden-state interventions can redirect a frozen model’s computation. Attention-pattern patching identifies routing as a causal pathway for this control. Additional experiments examine how supervision affects controllability and how reliably the controllers compose across loops.

Critical strengths

The paper addresses an interesting question about looped Transformers: how shared parameters support different computations across recurrent steps, and how the entering hidden state influences these computations.

The combination of hidden-state steering and attention-pattern patching provides an informative approach to studying computational mechanisms. These interventions offer causal evidence beyond observations of intermediate predictions.

Critical weaknesses

Limited validation on realistic reasoning tasks. Although the paper includes language-model experiments, the tasks remain synthetic and controlled. Whether the observed steering and routing mechanisms extend to more realistic, less structured reasoning tasks remains unclear.

The distinction from existing findings on standard Transformers needs clarification. Prior work has shown that standard Transformers can learn parallel graph-search procedures that progressively expand reachability across layers, without following a strict one-layer-one-hop process. [1] Consequently, the departure from one-loop-one-hop behavior alone does not establish a mechanism specific to looped Transformers. While the state-steering experiments go beyond this observation, a matched comparison with non-looped Transformers would help clarify the contribution of weight sharing. Further analysis should examine how the training distribution shapes computation in both architectures and characterize the specific computational patterns learned by looped models.

[1] Saparov, A., Pawar, S. A., Pimpalgaonkar, S., et al. Transformers Struggle to Learn to Search. ICLR, 2025.

Questions for authors

How can the effects of recurrent weight sharing be distinguished from those of the training distribution? Could the authors compare looped and non-looped Transformers under matched data, supervision, and computational budgets, while systematically varying graph structure and path depth?

What specific computational patterns are learned and selected across loops? How do these patterns relate to those previously identified in standard Transformers, and which aspects of the observed attention-routing behavior depend specifically on recurrence?

5 comments

Response to w3Gw — Part 1/3: Contribution and matched architecture–distribution comparisons

Thank you for recognizing the causal evidence provided by state steering and attention-pattern patching. Following your suggestions, we added paired experiments with two architectures × two training-depth distributions × three seeds, and completed state control, pattern exchange, and head ablation at two computational positions. These results distinguish when target information forms, whether the state can control the target, and which components implement that control.

1. Beyond progress differences: explaining how state control is implemented

“The distinction from existing findings on standard Transformers needs clarification.”

We agree that departure from one-loop-one-hop alone does not establish a mechanism specific to recurrence. Saparov et al. (ICLR 2025) study graph search with a given start and goal, finding parallel search through layerwise expansion of each node's reachable set. Our graph-walk task studies iterative composition of a given relation, enabling precise control of targets, entering states, and local interventions. This task difference motivates the experimental design; it does not itself prove that the models use different algorithms.

Our further question is how the entering state changes the way subsequent components use relational information. The new matched experiments extend this question to shared and unshared models. Both architectures support state control, but differ in target-representation timing, the layer containing the strongest component dependencies, and the transferability of particular attention patterns; training distribution modulates these differences. Our contribution is therefore a causal comparison of state control and its component implementation. Appendix J presents the complete comparison.

2. A crossed design separates architecture from training distribution

“How can the effects of recurrent weight sharing be distinguished from those of the training distribution?”

We trained twelve backbones. LT calls the same two-layer module six times; the standard Transformer (ST) uses twelve independent layers initialized by copying the same two-layer parameters. Paired models have identical width, twelve unfolded layers, final-answer supervision, optimizer and schedule, and 20,000 updates with batch size 512. Within each distribution and seed, their complete training sample streams are identical. Parameter counts are approximately 1.60M and 9.49M: the experiment compares tying versus untying parameters at fixed representation width and unfolded computation, rather than equal parameter budgets.

Backbone training uses general random permutations of N10 nodes. Both distributions retain D1–D8 coverage and the same supervision:

  • A: each depth has probability 1/8.
  • B: D8 has probability 1/2; each of D1–D7 has probability 1/14.

Graph and starting-node streams are identical across A/B; only query-depth frequencies change. All twelve backbones achieve at least 99.8% final D8 accuracy on the original task (Appendix J.1, Table R13).

Under A, we additionally probe every layer using fixed D8 queries on the same N10 single-cycle test graphs. Each position's linear probes are fitted on independent graphs. The first unfolded layer where raw-probe accuracy for f⁸(s) reaches 95% is:

ArchitectureSeed 0Seed 1Seed 2
LT121212
ST768

With matched training data and unfolded depth, ST forms a highly linearly readable endpoint earlier. LT's larger readability gains often occur after the second layer of each loop. This characterizes representation-formation timing without equating early readability with subsequent layers ceasing computation. Appendix J.2, Figures R15–R16 and Table R14 show the full layer-by-hop trajectories and endpoint comparison.

Response to w3Gw — Part 2/3: State control, component dependence, and training distribution

3. Control success, component dependence, and pattern exchange reveal different aspects of recurrence

“What specific computational patterns are learned and selected across loops?”

“…which aspects of the observed attention-routing behavior depend specifically on recurrence?”

We perform paired control before two-layer groups 3 and 6. Let u be the node decoded by the native output head immediately before intervention. Holding u fixed, we train J₁/J₂ so that the next two layers produce f(u)/f²(u). Two independent fits per condition give 96 controllers. Within each model, both branches share the same native entering state. The reference u can differ across models: this compares relative controllability under the same protocol without assuming identical internal progress.

Interventions are evaluated with a node probe jointly trained on clean J₁/J₂ outputs from independent graphs, then frozen. All main results use all 5,120 test examples, with no probe refitting after patching. Appendix J.3, Table R15 provides the protocol, native baselines, and complete control results.

First, state control and local routing are not exclusive to LT, but successful control does not guarantee transferable patterns. We exchange only the second layer's answer-query softmax attention weights across all heads, retaining receiver V and residuals, with self and cross-graph shuffle controls. For ST seed1 under A at group 3, transferring J₂'s pattern into J₁ raises two-hop target accuracy from 0.01% to 28.33%, versus 2.85% with shuffle. This demonstrates a causal role for the tested local addressing pathway in a standard Transformer.

Conversely, the three ST models under A at group 6 achieve 92.24%–100% clean control accuracy, but paired-pattern gains over shuffle are only −0.05 to +0.04 percentage points (pp). A controllable target therefore need not be transferable through the selected local pattern. Appendix J.4, Figure R17 and Table R16 report both exchange directions for every backbone and position.

Second, the strongest component dependencies can lie in different layers. Under A at group 3, the largest single-head ablation damage lies in the second layer for all three LT models. ST seeds 0/2 mainly depend on the first layer, whereas seed1 mainly depends on the second. A fixed second-layer pattern intervention therefore does not exhaust ST's effective control pathways. Appendix J.5, Figure R18 and Table R17 show all heads and their conditional damage.

Finally, training-depth frequency changes these architecture differences. Under the D8-heavy distribution B at group 3, all three LT seeds and both exchange directions show 28.57–86.79 pp paired-minus-shuffle gains, compared with 0.31–11.12 pp for ST. LT seed0 reaches 94.61%/97.93% paired target accuracy, versus 10.94%/11.14% with shuffle.

Using this same metric, the LT-minus-ST difference increases from A to B in both directions for all three paired seeds at group 3. At group 6, the direction is not consistent. Thus, training-depth frequency and call position modulate architecture differences in the measured local control pathway. ST also has lower clean control accuracy under B at group 3, so the entire patch gap there cannot be attributed to the pattern alone. Appendix J.6, Figure R19 and Table R18 give the paired architecture-by-distribution comparison.

Our conclusion is: parameter sharing affects target-representation formation and component use, while training distribution further shapes these state-control pathways. The evidence does not support attention routing as a mechanism exclusive to recurrence.

Response to w3Gw — Part 3/3: Connection to language reasoning and additional Ouro evidence

4. Controlled relation composition connects the graph task to language reasoning

“Whether the observed steering and routing mechanisms extend to more realistic, less structured reasoning tasks remains unclear.”

We agree that controlled language tasks do not replace evaluation on realistic, less structured reasoning. We choose these tasks to retain the computational substructure of multi-hop relation composition while keeping intermediate targets and intervention outcomes checkable. Graph relations supplied in a prompt correspond to retrieving and composing relations from context; the appendix KG experiment instead learns fixed relations through training, then composes them according to an entity and relation sequence (Appendix H.2). These settings explain the connection between the synthetic task and language reasoning.

We further added mechanism experiments in task-adapted Ouro. In the Letter task, applying J only once before the fifth shared-module call, from the same native entering state, raises step-6 first-answer-token accuracy from 2.81% to 95.00% (Appendix I.6, Figure R14 and Table R11).

In a separate confirmation experiment, among 147 examples where both controls succeed, ablating L34H5 causes 6.12 pp more damage under J₂ than J₁, with a 95% interval of [2.03, 10.46]. Appendix J.7, Table R19 documents the independent discovery/confirmation split, the fixed diagnostic subset, and all heads evaluated in confirmation. This extends the finding that state control changes a specific head's functional contribution to a controlled task in a language model.

Reviewer_WTQUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes looped or weight shared transformers in a specific setting of k-hop graph walks given a permutation graph in context. Their research goal is to analyze how looped transformers in their specific setting utilize the recurrence to perform the graph walk, ie how the entering state influences what the shared block computes next. Intermediate predictions, obtained using the model’s existing output head, show trajectories that do not consistently follow one graph hop per loop. The authors freeze trained backbones and train token wise adapters J that steer an additional loop toward one-hop or two-hop continuations. Attention-pattern and head-output patching then test whether this control operates through changes in routing or retrieved representations. The toy models repeat a two-layer Transformer block for eight or six loops (D8L8, D8L6) on ten-node permutation graphs. Additionally, the authors look at Ouro-2.6B fine-tuned on a text version of the graph walk and on an antonym-cancellation task.

Critical strengths

The attention-pattern versus head-output patching experiments distinguish changes in routing from transfer of retrieved representations. Interventions, head controls, and separate head-selection and evaluation data strengthen the causal evidence that attention patterns mediate the tested steering effects.

Critical weaknesses

The provided evidence is specific to this setting. The main research question, what routes the computation is unanswered for the native unsteered model. They instead answer if their additionally trained J (an external component) can cause a different transition. Attention-pattern patching identifies a causal pathway for steering, but does not fully explain how the native state encodes and changes the choice of computation. Intermediate output-head predictions also do not uniquely identify the algorithms executed within each loop. The abstract’s claim that J selects computations supported by the backbone but "does not create arbitrary new ones" is stronger than the evidence establishes. The 100% stay accuracy on stepwise backbones further illustrates that absence of explicit supervision or native execution does not establish that a transition is unsupported.

Questions for authors

You use the output head to make predictions on intermediate states. The output head was generally trained on final states, so an intermediate prediction can be misleading. What evidence do you have to support this choice? In particular, how should the large modal increments of seven or eight positions modulo ten in D8L8 seeds 0 and 2 and D8L6 seed 4 be interpreted? Given that these increments compare aggregate modes rather than individual trajectories, what evidence distinguishes changes in the underlying computation from changes in what the output head can decode?

5 comments
AuthorsOpenReview ↗

Response to WTQU (1/5): Readout reliability

Thank you for carefully reviewing our paper, recognizing the value of the attention-pattern versus head-output comparisons and the independent evaluation design, and providing specific and valuable suggestions. In response, we have added independent probing, component-level interventions, and native-model experiments that do not use JJ at all. We address your comments from two perspectives: the reliability of intermediate readouts and state-dependent computation in the native model.

1. Reliability of intermediate output-head readouts: independent probing verifies progress and reveals information missed by the original head

To test whether the final decoder head can reliably read intermediate states, we have added independent linear probing for every trajectory in Figure 2 and present the original readouts alongside probe results in Figures R1–R2 of Appendix I.1. Probes use the actual graph nodes fk(s)f^k(s) as labels, with training, validation, and test data split by graph. The comparison asks whether, at the same intermediate state HtH_t, the original output head predicts a given node and an independent probe can recover information about that node.

Figures R1–R2 show two distinct phenomena. In some models, as the number of loops increases, the original head successively reads different nodes with high accuracy, and independent probes recover these nodes at the corresponding positions. This is what we mean by “clear readout progress consistent with probing”: the specific nodes and their main emergence times are independently verified, without assuming that the original head fully describes the hidden state. For example, representative one-hop and two-hop models exhibit the corresponding highly readable frontiers.

Other models exhibit a clearly different phenomenon: the probe can already recover a node from a state at which the original head does not yet predict it correctly. Figures R1–R2 and Table R1 report the three models you specifically identified:

ModelIntermediate stateOriginal output-head accuracy for f8(s)f^8(s)Normalized-probe accuracy for f8(s)f^8(s)
D8L8 seed0H3H_30%98.81%
D8L8 seed2H6H_62.6%67.2%
D8L6 seed4H3H_319.0%74.6%

These results show that low accuracy of the original head cannot be directly interpreted as the absence of the corresponding information. Independent probing can recover intermediate representations that the original output interface does not fully read.

The output head was generally trained on final states, so an intermediate prediction can be misleading.

We acknowledge this risk, and the differences in Figures R1–R2 show that it does occur in some models. In particular, the head–probe accuracy gaps at the same states in the table above are consistent with your explanation: an output head optimized primarily for final states may not fully decode some intermediate representations. We therefore explicitly acknowledge this phenomenon and no longer infer that “the model has not formed the information” merely because “the original head does not read it out.”

What evidence do you have to support this choice?

Our evidence is the independent validation within the task provided by Figures R1–R2. For trajectories where the original readout and probing agree on nodes and the main progress boundaries, we retain readout as an indicator of task-level progress. Where they disagree, we show the additional information recovered by probes and explicitly identify the original head's decoding limitations.

Thus, our conclusion is not that “the final output head reliably decodes every intermediate state,” but that clear trajectories validated by probes provide reasonably faithful progress indicators; states that the original head cannot read clearly may still contain recoverable information, and probing can partially address this omission. Moreover, per-hop probes can recover multiple nodes from a single state, so we do not force their results into a unique “current node.”

AuthorsOpenReview ↗

Response to WTQU (2/5): Progress and causal evidence

In particular, how should the large modal increments of seven or eight positions modulo ten in D8L8 seeds 0 and 2 and D8L6 seed 4 be interpreted?

To explain these increments, we have added an entropy analysis of per-example prediction distributions in Figure R3 of Appendix I.1, and re-examined the original trajectories using the per-hop probing in Figures R1–R2. Equations (9)–(10) and Figure R4 in Appendix I.2 separately report within-example, adjacent-loop readout-displacement statistics for the backbones in the mechanism-exchange study; Appendix I.5 explains how these statistics differ from the original increments between population modes.

These comparisons show that, at some intermediate loops, the original head cannot clearly identify a node even though probes already recover path information missed by that head. A sudden change in the original readout curve therefore need not correspond to the same number of retrievals being completed internally within that single loop. Furthermore, the per-example analysis in Figure R3 shows that some uncertainty lies within individual prediction distributions, rather than arising solely from different examples each following a clear but different trajectory.

The original +7/+8 values are differences between population readout modes at adjacent loops, modulo ten, not per-example execution step counts. We will revise the relevant captions and discussion so that they are no longer interpreted directly as seven or eight retrievals performed by a single loop. The additional intermediate-loop mechanism experiments (Figures R5–R7) use targets that do not wrap around to the starting node, so the key conclusions do not depend on the interpretation of these anomalous increments.

Given that these increments compare aggregate modes rather than individual trajectories, what evidence distinguishes changes in the underlying computation from changes in what the output head can decode?

We distinguish these possibilities through complementary experiments. Figure R3 examines whether aggregate statistics conceal per-example behavior; Figures R1–R2 examine whether the original head misses existing information; Figures R4–R6 use frozen probes to re-evaluate pattern interventions and head ablations, testing whether subsequent target representations change with component interventions. Probes are not retrained on intervened states, so these comparisons do more than reinterpret existing states with a new decoder.

For example, the seed10 pattern exchange in Figure R4 and Table R3 preserves the receiver's values while increasing the native per-hop probe's accuracy for the one-hop target from 37.51% to 96.54%; shuffling donors across graphs gives only 19.35%. This shows that, at least in this condition, intervention changes target readability under the same frozen evaluator, rather than only the original output head's prediction. The next section explains how these interventions connect to functional head dependence and the native-model experiments.

Intermediate output-head predictions also do not uniquely identify the algorithms executed within each loop.

We agree. Neither the original readout nor probing uniquely identifies an internal algorithm from progress alone. In the revision, we explicitly distinguish their roles: readout describes the original output interface's immediate prediction, probing examines recoverable information, and causal interventions test how states and components affect subsequent execution. Our mechanistic conclusions rest on this combination of evidence, rather than inferring algorithm identity directly from progress curves.

AuthorsOpenReview ↗

Response to WTQU (3/5): Native computation without J

2. The native model without JJ: how entering states organize shared computation and affect subsequent execution through attention

To directly test whether the relevant mechanisms exist in the unsteered native model, we have added experiments in Figures R9–R12 of Appendix I.4 that neither train nor apply JJ. These experiments examine, in turn, whether native heads' contributions vary with state, whether states with the same current-node readout contain different information about subsequent execution, and whether changing these state components can alter subsequent behavior. The controlled experiments in Figures R4–R8 then further localize the computational pathway.

The main research question, what routes the computation is unanswered for the native unsteered model.

Figure R9 first tests whether component roles are already state-dependent during native execution. On frozen D8L6 seed6, we pair native invocations with one-hop and two-hop readout displacements within the same example, then ablate the same attention head. The final-accuracy loss caused by ablating this head at the two-hop-readout invocation exceeds that at the one-hop-readout invocation by an average of 17.03 percentage points, with a 95% graph-level confidence interval of [15.04, 18.99]. This difference is confirmed on 512 new graphs, showing that a shared component's functional contribution varies with the native execution state, rather than appearing only after introducing JJ.

Figure R10 next asks whether this difference is already present in the entering state, beyond the currently decoded node. We extract native entering states for loops 2–6, fix the current output-head readout to node 0, and label states according to whether the next native readout advances by one or two hops. PCA is fitted only on training graphs, without class labels. On 1,449 states from 470 independent test graphs, a linear classifier using only the first two principal components achieves 89.06% balanced accuracy, and one using the full hidden state achieves 96.75%. Thus, even when the current-node readout is the same, the entering state contains information about subsequent execution; the current node is not a complete description of the state.

Figure R10 provides predictive evidence, so we further add native-state direction interventions in Figure R11. We estimate a direction shared across graphs from states with the same current-node readout but drawn from different native loops, and apply it on independent graphs. The fractions that both “preserve the current-node readout” and “reach the specified next-loop target” are 56.01% and 84.40% for one-to-two-hop and two-to-one-hop changes, respectively, both exceeding the equal-norm random-direction controls. This advances the relationship between entering state and subsequent behavior from predictability to causal intervention. The direction includes execution-phase and computational-history information, so we interpret it as a control component in native states, rather than a pure algorithm switch completely separated from other semantics.

Figure R12 more strictly controls for the explanation that the intervention merely changes the current answer. After damaging a native head, we construct a directed repair while keeping all 22 immediate logits of the damaged state nearly unchanged, with a maximum change of approximately 1.05×10−51.05\times10^{-5}. Final accuracy nevertheless rises from 82.91% to 97.30%, compared with 80.44% for the matched random-direction control. Thus, even while fixing the full immediate output distribution, state changes can affect subsequent execution. This tests a separate native damage–repair process; it does not claim that all trained JJ maps preserve logits.

Together, Figures R9–R12 support the following: native states contain components beyond the current-node readout that concern subsequent execution and can be causally intervened on; the functional contributions of the shared components are already state-dependent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.