acceptodds
Under review as a conference paper at ICLR 2027

Tracing the Internal Computation of a Chess Playing Transformer

Abstract

Modern transformer neural networks achieve strong master-level performance in chess, yet a mechanistic understanding of how their internal computations produce move decisions remains limited. Building on advances in mechanistic interpretability, we adapt Low-Rank Sparse Attention (Lorsa) to LC0's policy network and combine it with Transcoders to decompose attention and MLP computations throughout the model into a unified set of sparse features. We also extend circuit tracing to LC0's bilinear policy output head. We introduce a chess-structured taxonomy and an agentic pipeline that proposes feature interpretations and, when possible, translates them into rule-based tests grounded in chess structure. On this foundation, we construct end-to-end attribution graphs that trace attributed pathways from board inputs through sparse features to move logits. Through case studies, we uncover interpretable tactical computations underlying the model's decisions for specific input positions and target moves. Moving beyond position-specific cases, we introduce quantitative metrics showing that computations are largely separated across candidate moves and highly branched within each move, while feature attribution gradually concentrates on the corresponding source and target squares across layers. Our code is available at https://anonymous.4open.science/r/Leela-SAEs-446A.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, SAE, Chess

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FaiDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses Transcoders and Lorsa to build feature-level attribution graphs for the chess model LC0 BT4. It examines several tactical decisions through case studies and analyzes how the computation is organized across candidate moves, board squares, and layers.

Critical strengths
  • The framework traces through both attention and MLP features, allowing a more connected view of how the model reaches a decision.

  • The intervention results give some confidence that the framework identifies features that matter for the model's behavior. In the g2-pawn case, ablating its detection feature raises the mating move's probability from 28.75% to 90.6%.

Critical weaknesses

W1. The framework largely combines existing Transcoder, Lorsa, and circuit-tracing methods. The chess-specific interpretation and validation pipeline is useful, but its added value over existing approaches is not clearly established. This makes the broader takeaway especially important, yet I am still left wondering what we learn beyond these particular examples. Look-ahead in LC0 has already been demonstrated, and some of the reported organizational patterns may follow from the architecture or graph construction. The conservative-error case is interesting, though its implications beyond that position remain unclear.

W2. I also have concerns about how firmly the proposed mechanisms are established. It is unclear whether the main pathways and organizational findings are robust to the attribution approximations. In particular, tracing Q and K separately does not fully capture their joint interactions. The paper does not show whether accounting for these interactions would change which features or paths appear important. The causal validation also leaves a gap. Ablating the feature detecting the g2 pawn affects the decision, but does not establish that the effect passes through the proposed pathways involving queen protection and king safety. Likewise, influential features associated with future moves do not fully validate the look-ahead chain. The general zero-patching and completeness checks do not directly verify these specific mechanisms.

Questions for authors

Q1. What is the main technical advance over prior complete replacement and circuit-tracing frameworks? Which analytical capability specifically depends on the proposed extensions?

Q2. Can the identified mechanisms generate predictions that hold on unseen positions? For example, predicting when the same conservative failure will occur? Such a test would help show that the explanations provide reusable understanding.

Q3. How sensitive are the key pathway rankings and separation/branching findings to these approximations, especially the separate treatment of Q and K?

Q4. Can path-specific interventions, with matched controls, establish whether the g2 feature's effect on move preference is actually mediated by the proposed queen-protection/king-safety pathway?

6 comments
AuthorsOpenReview ↗

Additional Response to Reviewer FaiD Part 1

We sincerely thank the reviewer for their insightful comments and suggestions regarding our methodology, particularly the separate treatment of Q and K.

The revised part is highlighted in blue in the uploaded revised manuscript. We would like to firstly respond to the issues raised in the "Weakness" section and then "Questions" section, organized by each point.

Ans W1. Technical Contributions and Broader Findings

Our contribution is not simply combining existing methods, but extending them to LC0 BT4's architecture in three aspects:

  1. Lorsa for Smolgen-biased bidirectional attention: Using token-wise z-patterns, we can identify which square sends information to another square and what information is transmitted.

  2. Circuit tracing for the bilinear policy head: We extend attribution tracing through the Q/K policy head to connect feature-level computations to individual move logits.

  3. Chess-specific feature interpretation: We develop an agentic pipeline that translates feature interpretations into executable chess rules and validates them using precision and recall.

Together, these extensions enable end-to-end, feature-level analysis of how information flows across board squares and contributes to move decisions.

Beyond individual case studies, our statistical analysis reveals three organizational patterns: (1) move-specific separation, where different candidate moves rely on largely distinct feature sets; (2) within-move branching, where multiple computational pathways jointly support a move; and (3) spatial concentration, where attribution progressively concentrates on the move's source and target squares.

We do not claim to be the first to demonstrate look-ahead in LC0. Rather, we provide a feature-level account of how future tactical information is represented and routed to influence current decisions. The conservative-error case is intended as a mechanistic diagnosis, not a claim of a general failure mechanism.

Ans W2.1. Q/K Separation and Causal Validation

For the bilinear policy head, jointly tracing Q and K introduces multiplicative interactions between Q-side and K-side features. Explicitly representing these interactions would require feature-pair interaction nodes, changing the interpretation of our first-order feature-level attribution graph.

We therefore trace Q and K separately, fixing the opposite side during each trace. This preserves the linear feature-to-feature pathways within each branch. During backpropagation, feature-to-feature attributions remain identical in both Q-side and K-side traces, as they represent direct interactions between features. A feature's logit attribution in the Q-side trace aggregates its interactions with all K-side features, and vice versa. Thus, the attribution scores remain valid without explicitly decomposing them into individual feature-pair interactions. This approach preserves a linear attribution graph of direct effects, enabling clearer analysis of computational mechanisms, since credit assignment under nonlinear interactions is inherently challenging.

Ans W2.2. Validate the look-ahead chain.

We suppress the h4-related rook-attack supernode to block the h4-to-g6 information pathway.

  1. Downstream feature effects:

    • Steering factor = −1(zero ablation): the average activation of four downstream look-ahead feedback supernodes decreases from 3.717 to 3.070 (17.4% decrease). Three decrease, while one increases slightly.
    • Steering factor = −2: the average activation decreases from 3.717 to 2.262 (39.1% decrease), with all four downstream supernodes decreasing.
  2. Target-move effects:

    • Steering factor = −1: the target-move logit decreases from 2.691 to 2.525, and its probability decreases from 78.57% to 74.52% (−4.05 percentage points).
    • Steering factor = −2: the logit decreases to 2.441, and the probability decreases to 71.97% (−6.61 percentage points).

We also jointly zero-ablate two early-layer rook-detection features (Lorsa.0.10273@d4 and Transcoder.0.13486@d4). This reduces downstream rook-to-king threat feature activations by 63.5% on average and look-ahead feature activations by 93.12%. The target-move probability drops from 78.57% to 0.36%.

AuthorsOpenReview ↗

Additional Response to Reviewer FaiD Part 2

Q1. Main Technical Advance

The main advance is an architecture-specific extension of Lorsa and circuit tracing that enables us to trace square-to-square information flow through attention and connect feature-level pathways to individual move logits through the bilinear policy head. Combined with our agentic rule-based validation framework, this provides an end-to-end analytical capability unavailable from MLP-only decomposition or conventional circuit tracing.

Q2. Generalization to Unseen Positions

We agree that the current results do not establish that the identified defensive pathway can reliably predict similar failures on unseen positions. Our case study demonstrates a mechanistic explanation for one specific decision, rather than a general predictive mechanism. We will clarify this limitation and leave systematic cross-position prediction to future work.

Q3. Sensitivity to Attribution Approximations

If by approximations you mean the sparse decomposition of the original MLP/attention modules, their faithfulness is evaluated in Appendix D.

If you refer to approximations in attribution computation, we do not approximate the QK operation itself. The current Q-side (K-side) attribution already aggregates a feature's pairwise interactions with all K-side (Q-side) features, respectively. EThis would only reveal individual contributions without changing aggregate attributions or their rankings. Enabling gradients through Q and K simultaneously would, however, break the linearity of the attribution graph, such that summed feature attributions would no longer recover the target logit or feature activation. Following Anthropic circuit tracing paper[1], we choose linear direct effects over nonlinear attributions or ablations to enable clearer mechanistic analysis, as credit assignment under nonlinear interactions is difficult. To validate our attribution rankings, we zero-ablate each feature, measure its effect on the target logit, and compare the rankings; Figure 8(c) shows a top-10% overlap of 50.2% ± 7.25%. We then construct intervention-based graphs, assigning each feature a weight based on its relative effect on the target logit:

wf=∣ℓtargetoriginal−ℓtargetintervened∣ℓtargetoriginal.w_f=\frac{|\ell_{\mathrm{target}}^{\mathrm{original}}-\ell_{\mathrm{target}}^{\mathrm{intervened}}|}{\ell_{\mathrm{target}}^{\mathrm{original}}}.

We retain the same average number of features and edges as in the attribution graphs and compare their source/target-square ratios with the MAR reported in the paper.

LayerLorsa attribution graphLorsa interventionTranscoder attribution graphTranscoder intervention
L028.90±9.3224.96±28.2823.48±8.3525.81±15.14
L363.77±15.2948.87±22.440.38±13.5833.17±16.28
L673.95±34.6165.69±38.5248.26±11.5736.30±13.81
L966.78±29.8254.69±36.6962.24±10.9432.29±16.13
L1289.35±9.0484.03±23.578.80±10.6852.89±19.71
L14100.00±0.00100.00±0.00100.00±0.00100.00±0.00

We also construct intervention-based graphs by establishing directed edges between features and measure their branching factors using feature-to-feature intervention effects, quantified by the relative change in a target feature's activation after ablating a source feature.

We define the edge weight as:

wi→j=∣aj(h)−aj(h~(i))aj(h)∣,w_{i\rightarrow j} = \left| \frac{ a_j(\mathbf h)-a_j(\tilde{\mathbf h}^{(i)}) }{ a_j(\mathbf h) } \right|,

where aj(h)a_j(\mathbf h) denotes the original activation of feature fjf_j, and aj(h~(i))a_j(\tilde{\mathbf h}^{(i)}) denotes its activation after zero-patching feature fif_i. We then prune the intervention-based graphs to match the average number of edges in the original attribution graphs.

MetricOriginal graphsIntervention effect graphs
Branching factor75.7% ± 7.58%89.01% ± 5.50%

Although this is a crude estimate, the branching factor remains high, supporting the existence of branching structures captured by our attribution graphs.

Separating Q and K is a simplification of attribution: it sums up the feature interactions between the Q and K sides, treating them as the interaction between one feature on one side and all features on the other side combined. We discuss this in Ans W2.1. We acknowledge that this loses information about which feature pairs are important. Since the attention-like policy head is nonlinear, existing QK tracing methods[2] cannot be directly applied. We also discuss this limitation in Sec. 6.1 of the paper. We will study this issue further in future work.

[1] E. Ameisen et al., “Circuit Tracing: Revealing Computational Graphs in Language Models,” Transformer Circuits Thread, Mar. 27, 2025. [Online]. Available: https://transformer-circuits.pub/2025/attribution-graphs/methods.html

[2] Kamath, H., et al. (2025). Tracing Attention Computation Through Feature Interactions. Transformer Circuits Thread (2025).

AuthorsOpenReview ↗

Additional Response to Reviewer FaiD Part 3

Q4. Path-specific Causal Interventions

After ablating the g2-pawn detection feature, the downstream protected queen exchange feature Tc.10.1958@h3 decreases by 37%, while the rook-threat target feature Lorsa.8.16275@g2 decreases by 100%. Directly ablating Tc.10.1958@h3 increases the probabilities of Qxh7+ and Qxd7 from 28.7% to 34.9% and from 26.1% to 29.3%, respectively, indicating that the model is more inclined to move the queen when it is no longer protected against exchanges. Randomly selected active features reduce them by only 0.024% and 0.124% on average, respectively. We further intervene directly on the rook-threat feature: applying a steering coefficient of -3 increases the fxg7+ logit from 2.26 to 2.42 and its probability from 35.8% to 38.5%. Conversely, transplanting its activation from g2 to g7 reduces the probability of fxg7+ from 35.78% to 32.58%. Together, these interventions provide causal evidence that the downstream defensive signals influence move preferences, with a spatially specific effect consistent with the proposed mechanism. These results support a causal influence of the g2-related defensive signal on downstream computations and move preferences.

Thanks again for the helpful comments. Should you require any further information, please do not hesitate to let us know. We would greatly appreciate your feedback.

Reviewer_vfAtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Transformer chess networks such as Leela Chess Zero (LC0) play at master level without search. How the network turns the position into that move is still largely unknown. The paper tries to trace this computation in BT4, a public 15-layer LC0 model that reads the board as 64 tokens, one per square. Each MLP block is replaced by a Transcoder and each attention block by Lorsa. Both are sparse approximations of the original block. On each square only a few units, called features, are active, and each feature should correspond to one concept. Lorsa features also record which squares they read from. For a given position and move, the authors build an attribution graph. The graph keeps the features that contribute most to the score of that move and the links between them, from the input squares up to the output. BT4 scores a move as the dot product of a vector computed at the starting square and one computed at the destination square, so the two sides are traced separately and their scores are added. Each feature then needs a meaning in chess terms. An LLM agent (OpenAI Codex) looks at the positions where a feature is most active and describes what they share, within ten categories set by the authors, such as piece detection, captures, protection and tactics. When possible, the agent writes this description as a rule in code, for example "active on squares holding a pawn of the side to move". The rule is then checked on new random positions. Precision is the share of squares where the feature is active that satisfy the rule. Recall is the share of squares satisfying the rule where the feature is active. Separately, three players rated above 2000 Elo, and Codex, rate from 1 to 5 how consistently the top positions share one pattern, for 100 Transcoder features, 100 Lorsa features and 100 neurons of the original MLP blocks. Transcoder and Lorsa features score about 4 on average, neurons about 2. The results have two parts. First, five positions (three in the main text) are read through their graphs. In the first, a knight sacrifice in a mate-in-two is supported by features tied to the rook mate that follows. In the second, several reasons support another knight sacrifice. In the third, the model misses a mate-in-one because a signal tied to its own pawn on g2 favors a safer move. Removing that pawn, or switching off that feature, raises the probability of the mating move from about 29% to about 91%. Second, the authors look at the graphs of 1,000 random positions and describe three patterns. (i) The model explains its two best candidate moves with mostly different features. (ii) Inside the graph of a single move, many pairs of features are not linked by any path, so the computation runs along several separate strands rather than one chain. (iii) The features that matter for the move change from layer to layer. The appendix measures how closely the simplified blocks reproduce the original ones, tests whether the model still picks the same moves and solves puzzles when one block is replaced, checks whether switching off the features that the graph rates as most important changes the move score more than switching off random ones, what share of the influence on the move score passes through the features which have a meaning rather than through these unexplained terms.

Critical strengths

The paper decomposes the whole policy network, attention included, and traces the computation all the way to the move score, adapting the tracing method to BT4's unusual output, which scores a move by combining a vector computed at its start square with one computed at its destination square. Earlier work on chess networks used probes, a few attention heads, or sparse features in single layers. Chess also lets the authors check feature labels in a way that is hard to do for language models. Each rule is code over board facts (pieces, attacks, legal moves). It is tested on positions the agent did not see during refinement, and the agent can decline to give a rule. The tools are tested before they are used, and the paper is clear about what the graphs cannot show. It reports reconstruction quality for every layer. It measures the effect of replacing one block at a time on move choice and puzzle accuracy, compared with mean ablation. It measures the effect of zeroing the top-attributed features, compared with random active features, and how much of each graph is carried by features rather than error terms. The limitations section states that the graphs miss the interactions between the two sides of the output and the suppression of moves. This matters because every later claim depends on the graphs being a fair picture of the model.

Critical weaknesses
  • Only one of the five case studies includes an intervention, that is, switching off features and checking how the move probability changes. In the missed-mate case, the authors switch off a first-layer feature that fires on every pawn of the side to move. Switching off this feature on g2 raises the probability of the mating move to 90.6%. Removing the g2 pawn from the board raises it to 91.7%. The two interventions have almost the same effect, as expected, since the feature does little more than signal that the pawn is there. So the test shows that the g2 pawn matters, but not that it matters through queen protection and king safety, as the text claims. The features that would carry that effect are never switched off.
  • The look-ahead claim in Section 4.1 has no intervention at all. The h4 features said to stand for the future rook mate are not switched off, and the position is not edited to remove the mate. Jenner et al. (2024) already showed look-ahead in an LC0 network with interventions over many puzzles, so one untested example adds little to that evidence.
  • The tactical labels that the case studies rely on are the least reliable ones. Their recall in Table 1 is between 7.7% and 45%. For instance, a feature labelled as a pawn break that opens an attack on the queen or king fires on only 18.6% of the squares where that description holds. So the label says much more than the feature does.
  • To name a feature, the agent looks not only at the board positions where the feature is active, but also at the features it receives input from and sends output to in the attribution graph. The resulting label can therefore describe the role the feature plays in the graph, rather than a pattern that can be seen on the board. The case studies then explain the graph using these labels. In this way the graph is used to name the features, and the names are used to interpret the graph, so part of the explanation is circular.
  • The authors choose the positions and the groups of features shown without a stated criterion.
  • The paper finds that, layer after layer, the features that matter for the move end up on its two squares: the square the piece leaves and the square it reaches. This is partly guaranteed by how the graphs are built. They start from the move score, and in BT4 this score is computed only from those two squares. The paper compares the result only with a board where all 64 squares matter equally, which is an easy comparison to beat. A better test is to build the same graph for a random legal move. If its features also end up on its own two squares, the finding says more about the method than about the model.
  • Before measuring, the authors remove the weakest links from each graph. They keep only the strongest links, up to 57% of the total. The paper does not justify the value 57% and tries no other value. Removing links can by itself make two graphs share fewer features and split each graph into more separate branches. These are exactly the two results the paper reports, so they should be shown for several values.
  • Two features count as shared between the graphs of the two best moves only if they sit on the same square. Two moves that use different squares will therefore look different, even if the model reasons about them in the same way. The paper's own numbers show this. The graphs share about 6% of their features when the moves start from different squares, and about 17% when they start from the same square. So the low overlap may mostly reflect that the moves use different squares.
  • All measurements in Section 5 use a single set of Transcoders and Lorsa, trained once. Another training run could divide the same concepts into features in a different way. The overlap and the number of branches depend directly on which features exist. Without a second run, it is not possible to say whether the patterns are stable, yet the abstract presents them as general properties of the model.
  • The human ratings in Table 2 come from three players, but the paper does not describe how the rating was done. It is not said whether each player rated every feature, whether there are 100 features per type or 100 in total, or what the "±" values measure. The paper also reports no measure of agreement between the players.
Questions for authors
  • In Table 5, at turn 3, more features pass both thresholds (2,261 Transcoder, 1,158 Lorsa) than pass the recall threshold alone (2,133 and 1,106). This is impossible if the "Both" column is a subset of the recall column. How is "Both" counted?
  • Table 1 says that Table 6 is its full version, but two features have different scores in the two tables. Which scores are correct?
  • How was the human rating in Table 2 done? Did each player rate every feature? Are there 100 features per type or 100 in total? What does the "±" measure? How often do the players agree, and did they see the agent's interpretation?
  • Does Figure 4 include only features whose rule passed the test, or also features whose label was never checked?
  • How were the five case positions chosen?
  • In Sections 4.1 and 4.3, what happens to the move probability when the intermediate features are switched off? These are the h4 features in Section 4.1, and the queen protection and king safety features in Section 4.3. For Section 4.1, what happens if the position is edited so that the rook mate on h4 is no longer possible?
  • Does the look-ahead pathway appear across many mate-in-two puzzles, not only in one? For example, how often do features on the mating square of the second move appear in the graph of the first move? Does switching them off lower the probability of the first move?
9 comments
AuthorsOpenReview ↗

Additional Response to Reviewer vfAt Part 1

We sincerely thank you for the comprehensive comments and suggestions!

The revised part is highlighted in blue in the uploaded revised manuscript. Our point-by-point responses are provided below. And we would like to firstly respond to the issues raised in the "Weakness" section and then "Questions" section, organized by each point.

W1

We would also like to clarify that gradient-based attribution is intended to measure causal influence at a local level. We do not use this method merely for computational convenience; rather, it provides a way to characterize local causal effects while avoiding the instability that can arise from intervention scaling factors.

Ans W1.1

Thank you for pointing out the limited number of intervention experiments in the original manuscript. We have added intervention experiments corresponding to the cases discussed in the paper. We also elaborate on why attribution scores provide a more appropriate measurement of feature contributions in our setting, compared with relying on interventions alone.

Interventions:

Internal Look-Ahead(Sec.4.1). We suppress the h4-related rook-attack supernode to block the h4-to-g6 information pathway.

  1. Downstream feature effects:

    • Steering factor = −1(zero ablation): the average activation of four downstream look-ahead feedback supernodes decreases from 3.717 to 3.070 (17.4% decrease). Three decrease, while one increases slightly.
    • Steering factor = −2: the average activation decreases from 3.717 to 2.262 (39.1% decrease), with all four downstream supernodes decreasing.
  2. Target-move effects:

    • Steering factor = −1: the target-move logit decreases from 2.691 to 2.525, and its probability decreases from 78.57% to 74.52% (−4.05 percentage points).
    • Steering factor = −2: the logit decreases to 2.441, and the probability decreases to 71.97% (−6.61 percentage points).

We also jointly zero-ablate two early-layer rook-detection features (Lorsa.0.10273@d4 and Transcoder.0.13486@d4). This reduces downstream rook-to-king threat feature activations by 63.5% on average and look-ahead feature activations by 93.12%. The target-move probability drops from 78.57% to 0.36%.

Multiple Strategic Cues(Sec.4.2). We intervene on features associated with three strategic factors supporting the selected move:

InterventionTarget logitTarget probability
Remove d-file-opening features2.73 → 2.3142.7% → 29.4%
Remove future Qf7+ threat features2.73 → 0.5642.7% → 5.3%
Remove Bg2 follow-up features2.73 → 2.5942.7% → 35.86%
Jointly remove all three groups2.73 → −0.0742.7% → 2.5%
Random active-feature control2.73 → 2.7442.65% → 42.69%

Diagnosing an Overly Conservative Decision(Sec.4.3).

We conduct the following interventions to examine these mechanisms:

  1. Downstream effects: Ablating the g2-pawn detection feature reduces the activation of the downstream "protected queen exchange" feature Tc.10.1958@h3 by 37%, while the "rook-threat target" feature Lorsa.8.16275@g2 decreases by 100%.

  2. "Protected queen exchange" intervention: Directly ablating Tc.10.1958@h3 increases the probabilities of Qxh7+ and Qxd7 from 28.7% to 34.9% and from 26.1% to 29.3%, respectively, indicating that the model is more inclined to move the queen when its protection against exchanges is removed.

  3. "Rook-threat target": Applying a steering coefficient of −3 to the rook-threat feature increases the fxg7+ logit from 2.26 to 2.42 and its probability from 35.8% to 38.5%. Conversely, transplanting its activation from g2 to g7 reduces the probability of fxg7+ from 35.78% to 32.58%.

Together, these interventions provide causal evidence that downstream defensive signals influence the model's move preferences, with spatially specific effects consistent with our proposed mechanism.

Ans W1.2

Ablating the g2 pawn feature reduces the activation of the downstream feature representing the queen being protected in an exchange. We added experiments that switching off downstream features in Ans W1.1, directly suppressing this "protected queen exchange" feature increases the model's tendency to move the queen and ablating the "rook-threat target" decreases the probability of the pawn check.

Ans W2

We have added intervention experiments to the case studies, as detailed in our response to W1.1 above.

While Jenner et al. (2024) showed look-ahead through interventions, our approach further identifies the more comprehensive feature-level computational pathways across the complete model and show feedback mechanisms through unsupervised attribution graph analysis.

AuthorsOpenReview ↗

Additional Response to Reviewer vfAt Part 2

Ans W3

Thank you for pointing out the limitations of existing automatic interpretability methods in producing precise feature descriptions.

Our work provides a framework built upon these existing methods. The relatively low recall indicates that current agent-based automatic interpretability methods still have substantial room for improvement, and we intend to further improve this aspect.

We regard the generated descriptions as heuristic interpretations rather than precise and exhaustive definitions of the underlying features. Nevertheless, these descriptions already capture meaningful tactical characteristics. In particular, high precision but low recall may suggest that the tactical concepts encoded by the features are more complex than what our current descriptions can capture.

The features we discuss in Section 4 therefore require more detailed justification, which we have attempted to provide through additional analyses.

More broadly, we aim to establish a benchmark for improving the reliability of automatic feature interpretation. Compared with generating explanations through a single LLM call, our agentic framework combines multi-round hypothesis refinement, domain knowledge, and executable validation. We believe this substantially improves the reliability and verifiability of feature interpretations.

Ans W4

Attribution-graph edges help the agent identify related features and refine its interpretations. However, feature validation is independent of the graph, and the validated interpretations are applied to new attribution graphs, separating semantic validation from the subsequent case studies.

Ans W5

We selected five case studies to cover diverse model behaviors and computational mechanisms:

  1. Difficult positions solved correctly(Sec.4.1): To examine how the model identifies non-obvious tactical moves.

  2. Mate-in-two positions(Sec.4.2): To investigate look-ahead and feedback from future mating configurations.

  3. Strategic sacrifices(Sec.4.2): To examine how multiple strategic features jointly support a move.

  4. Simple positions solved incorrectly(Sec.4.3): To understand failure mechanisms, including missed mates and overly conservative decisions.

  5. Non-human-like thinking(Sec.4.1 & 4.3 & Figure 20): To reveal how the model balances offensive and defensive considerations differently from human players.

These cases were selected to illustrate complementary aspects of the model's decision-making, rather than to serve as an exhaustive or statistically representative sample.

Ans W6

The observation that MAR reaches 1 at the final layer is a structural consequence of the model architecture.

Specifically, the policy head computes a move score using representations from the source and target squares (more precisely, it reads representations from all squares and performs pairwise matching in a manner analogous to attention-score computation). Consequently, the gradient with respect to the final residual stream can only be attributed to these two token positions for a given move.

However, after backpropagating through the final attention layer, the gradient can theoretically originate from all token positions. We argue that this structural convergence is also consistent with, and provides support for, the correctness of our gradient-based attribution method. To further investigate this, we construct attribution graphs for randomly selected moves from 1,000 positions. Their MAR also exhibits an increasing trend across layers, eventually reaching 1.

LayerLorsa MARTranscoder MAR
024.46%±12.16%22.64%±10.86%
357.32%±20.00%36.00%±16.93%
669.56%±31.47%42.97%±16.81%
974.52%±27.82%56.66%±16.59%
1277.56%±25.21%65.11%±17.18%
13100.00%±0.00%100.00%±0.00%

We additionally construct intervention-based graphs. Each feature is assigned a weight according to its intervention effect on the target logit:

wf=∣ℓtargetoriginal−ℓtargetintervened∣ℓtargetoriginal.w_f=\frac{|\ell_{\mathrm{target}}^{\mathrm{original}}-\ell_{\mathrm{target}}^{\mathrm{intervened}}|}{\ell_{\mathrm{target}}^{\mathrm{original}}}.

We then measure the corresponding source/target-square intervention effect ratio compared with MAR in the text.

LayerLorsa attribution graphLorsa interventionTranscoder attribution graphTranscoder intervention
L028.90±9.3224.96±28.2823.48±8.3525.81±15.14
L363.77±15.2948.87±22.440.38±13.5833.17±16.28
L673.95±34.6165.69±38.5248.26±11.5736.30±13.81
L966.78±29.8254.69±36.6962.24±10.9432.29±16.13
L1289.35±9.0484.03±23.578.80±10.6852.89±19.71
L14100.00±0.00100.00±0.00100.00±0.00100.00±0.00

The intervention-based analysis exhibits a similar convergence. This further shows that information important to a move is represented in a convergent manner within the model.

AuthorsOpenReview ↗

Additional Response to Reviewer vfAt Part 3

Ans W7

Thank you for raising this concern.

We selected an edge-retention threshold that preserves most of the major information while keeping the attribution graph sufficiently readable and interpretable.

Specifically, the edge-retention threshold is based on the cumulative influence of each edge on the target logit through all downstream paths. This influence is computed by summing the products of normalized contributions along all paths connecting the edge to the target logit. We then aggregate these contributions across edges and retain edges until their cumulative influence reaches 57% of the total edge influence (S).

Importantly, this should not be interpreted as retaining 57% of the target logit value. Rather, it means retaining edges whose cumulative influence accounts for 57% of the total edge influence across all edges.

Including too many edges introduces numerous weak connections, making the attribution graph noisy and difficult to interpret. Nevertheless, we agree that the threshold can affect structural measurements such as the branching factor and MAR. We have therefore conducted additional experiments using different edge-retention thresholds.

The results are shown below.

Edge-retention thresholdAverage number of retained featuresBranching factorDifferent-source overlapL7 Attn MAR (%)L7 MLP MAR (%)Attribution graph entropy
0.317.949.15%±30.31%0.84%±3.79%100.00±0.0081.27±38.620.070±0.202
0.473.168.29%±15.22%9.98%±10.73%88.94±31.0262.17±31.150.878±0.461
0.5386.881.63%±6.35%11.59%±6.39%79.07±14.5550.10±13.401.882±0.282
0.57632.275.53%±7.59%5.76%±3.84%78.42±12.4351.14±10.061.985±0.222
0.65774.968.93%±7.38%4.25%±2.59%80.24±12.3753.25±11.231.964±0.220
0.75967.561.87%±7.62%2.40%±2.82%81.25±10.9655.26±8.431.952±0.211
0.851150.742.36%±6.50%1.21%±2.06%82.43±10.4757.03±8.021.915±0.216

Very low thresholds retain few features, mostly concentrated on the source and target squares, resulting in high branching-factor variance and less interpretable computational pathways. Conversely, very high thresholds introduce many weakly contributing but highly activated features, reducing the branching factor. We therefore choose 0.57 as a compromise between preserving major contributions and maintaining graph interpretability.

W8

Residual streams at different token positions are inherently different computational entities. Even if the same feature is activated at two different positions, these activations occur in different residual streams and should not be treated as identical. When two candidate moves involve different squares, they necessarily involve different residual streams. This is itself consistent with our interpretation of parallel reasoning: the underlying computational mechanisms may be similar, but the corresponding computational token positions are not the same.

W9

We trained an additional set of models using a different random seed and recomputed MAR, branching factor, and feature overlap to examine the stability of our findings.

We first report the reconstruction EV of the additional seed. We then use the same pruning coefficient, namely an edge-retention threshold of 0.57, for the subsequent analyses.

Reconstruction EV:

Layer03691214
TC (Seed 42, reported in the paper)0.920.8720.8920.9570.9060.932
TC (Seed 7)0.9250.8710.8950.9490.9050.928
Lorsa (Seed 42, reported in the paper)0.8290.6830.6590.6940.6710.796
Lorsa (Seed 7)0.8260.6840.6630.6940.6650.784

Branching factor and feature overlap:

Branching factor and feature overlap:

MetricSeed 42 (reported in the paper)Seed 7
Top-1 branching63.93%±12.38%65.25%±10.62%
Top-2 branching62.84%±14.59%64.50%±12.54%
Top-1/Top-2 node overlap (Jaccard)24.07%±10.67%25.38%±10.53%

Transcoder / MLP MAR:

Transcoder / MLP MAR:

Layer03691214
Transcoder MAR (Seed 42, reported in the paper)23.48±8.3540.38±13.5848.26±11.5762.24±10.9478.80±10.68100±0.00
MLP MAR (Seed 7)24.50±9.8439.36±12.7450.22±12.4463.33±12.7478.03±10.43100±0.00

Attention MAR:

Attention MAR:

Layer03691214
Attn MAR (Seed 42, reported in the paper)28.90±9.3263.77±15.2973.95±34.6166.78±29.8289.35±9.04100±0.00
Attn MAR (Seed 7)30.22±10.5661.78±14.6375.50±20.8762.20±27.2095.68±5.45100±0.00

The results from the two independently trained seeds are highly consistent across reconstruction EV, branching factor, feature overlap, and MAR. This provides additional evidence that the structural patterns we report are not artifacts of a particular set of trained Transcoders and Lorsa.

Reviewer_UJnbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses mech interp tools (Lorsa and transccoders) to study Leela's BT4 model. They make some adjustments to circuit tracing to account for the bespoke model architecture and build a pipeline that automatically labels features. The paper makes some interpretations on how the model works.

Critical strengths

The mech interp model seems to be well done. The feature-level circuit work is appropriate and technically rigorous. The semantic validation is a great step to give the reader confidence that the rules and features discovered are meaningful.

Critical weaknesses

The paper's results and interpretations are quite thin. The entire 4th section shows 3 examples, which would be more appropriate for a figure rather than an entire case study section (or use these examples to motivate more rigorous analyses).

The fifth section discusses some findings on how move computation is organized in the attribution graphs. These results are interesting, but I have several concerns about the methodology. Mainly, it seems that important baselines are missing, which would help disentangle structure from real findings. For example, the MAR reaches 1 by construction at the last layer, and the rising trend is probably just the head reading from source and target squares. I would've been interested in a branching baseline that preserves layer-pair edge counts, because the observed gap may just be due to layering. The causal support (besides the g2 example) is kind of weak as well. Circuit effects of 0.5% and 1.2%, a minority of features passing the thresholds, and case studies being read off of attribution graphs instead of the careful interventions done by Jenner et al.

Questions for authors

What do your metrics like MAR, entropy, and branching look like for random moves, or the kth best move, etc? Try to separate whether the convergence finding is due to structure or not. What does the target logit do when you zero-patch the h4 supernode? What about in comparison with a random supernode?

6 comments
AuthorsOpenReview ↗

Additional Response to Reviewer UJnb Part 1

We sincerely thank the reviewer for the comprehensive comments and suggestions!

The revised part is highlighted in blue in the uploaded revised manuscript. Our point-by-point responses are provided below. We would like to firstly respond to the issues raised in the "Weakness" section and then "Questions" section, organized by each point.

Weaknesses

Ans W1.1:

The paper's results and interpretations are quite thin. The entire 4th section shows 3 examples, which would be more appropriate for a figure rather than an entire case study section (or use these examples to motivate more rigorous analyses).

Thanks! The three cases in the main text are representative examples. Appendix H.2 provides their detailed attribution graphs and two additional cases. Beyond illustrating individual decisions, these graphs reveal broader computational mechanisms and motivate Section 5, where we quantitatively investigate the general organization of move computation across 1,000 positions.

Regarding rigor analyses, we evaluate the EV and attribution faithfulness of our replacement models. Unlike activation interventions [1–3], gradient-based tracing captures local computational dependencies without modifying forward-pass activations or requiring a steering coefficient.

Experimentally, we conducted additional experiments to validate our findings through additional interventions (Response W2.3) and test the generality of look-ahead across 50 model-solved mate-in-two positions. Features associated with the second-move mating destination appear in all 50 first-move attribution graphs. In the 38 cases where this destination does not overlap with the first move's source or target, ablating these features reduces the first-move logit and probability by 0.542 and 11.79 percentage points on average, respectively, compared with only 0.02 and 0.54 percentage points for random features.

Ans W2.1

Mainly, it seems that important baselines are missing, which would help disentangle structure from real findings. For example, the MAR reaches 1 by construction at the last layer, and the rising trend is probably just the head reading from source and target squares.

We agree that the final-layer MAR of 1 is structurally imposed by the bilinear policy head. For a given move, its logit is computed by representations at its source and target squares. Therefore, the gradient at the final residual stream is supported only at these two positions.

However, after backpropagating through the final attention layer, attribution can propagate to all board positions. Thus, the increasing MAR in earlier layers is not directly enforced by the policy head. We interpret this trend as a learned representation pattern: early layers are dominated by single-piece detection and local capture features, whereas later layers increasingly contain complex tactical features and these features are encoded at the move source and target squares.

Ans W2.2

I would've been interested in a branching baseline that preserves layer-pair edge counts, because the observed gap may just be due to layering.

We additionally construct two constrained random baselines. Both preserve layer-pair edge counts. The first randomly rewires edges among features in the original 1000 attribution graphs while preserving the node set, node in-degrees and out-degrees, total edge count and preserves layer-pair edge counts. The second randomly samples active features while preserving the number of nodes at each layer and preserves the layer-pair edge counts. In the paper, we replaced the original degree-rewired baseline with this two baselines and also modified Figure 10.

GraphBranching factor
Original graphs75.67 ± 7.58%
Rewired graphs preserving layer-pair edges57.13 ± 6.43%
Random active-feature graphs50.76 ± 6.35%

The original graphs remain substantially more branched than both constrained baselines.

[1] Nanda, N. (2022). Attribution Patching: Activation Patching at Industrial Scale.

[2] Heimersheim, S., & Nanda, N. (2024). How to Use and Interpret Activation Patching.

[3] Jenner, E., et al. (2024). Evidence of Learned Look-Ahead in a Chess-Playing Neural Network. NeurIPS 2024.

AuthorsOpenReview ↗

Additional Response to Reviewer UJnb Part 2

Ans W2.3

The causal support (besides the g2 example) is kind of weak as well.

Thanks! We have added further interventions to validate the proposed computational pathways.

Internal Look-Ahead(Sec.4.1). We suppress the h4-related rook-attack supernode to block the h4-to-g6 information pathway.

  1. Downstream feature effects:

    • Steering factor = −1(zero ablation): the average activation of four downstream look-ahead feedback supernodes decreases from 3.717 to 3.070 (17.4% decrease). Three decrease, while one increases slightly.
    • Steering factor = −2: the average activation decreases from 3.717 to 2.262 (39.1% decrease), with all four downstream supernodes decreasing.
  2. Target-move effects:

    • Steering factor = −1: the target-move logit decreases from 2.691 to 2.525 and its probability decreases from 78.57% to 74.52%.

We also jointly zero-ablate two early-layer rook-detection features (Lorsa.0.10273@d4 and Transcoder.0.13486@d4). This reduces downstream rook-to-king threat feature activations by 63.5% on average and look-ahead feature activations by 93.12%. The target-move probability drops from 78.57% to 0.36%.

Multiple Strategic Cues(Sec.4.2). We intervene on features associated with three strategic factors supporting the selected move:

InterventionTarget logitTarget probability
Remove d-file-opening features2.73 → 2.3142.7% → 29.4%
Remove future Qf7+ threat features2.73 → 0.5642.7% → 5.3%
Remove Bg2 follow-up features2.73 → 2.5942.7% → 35.86%
Jointly remove all three groups2.73 → −0.0742.7% → 2.5%
Random active-feature control2.73 → 2.7442.65% → 42.69%

Diagnosing an Overly Conservative Decision(Sec.4.3).

  1. Downstream effects: Ablating the g2-pawn detection feature reduces the activation of the downstream "protected queen exchange" feature Tc.10.1958@h3 by 37%, while the "rook-threat target" feature Lorsa.8.16275@g2 decreases by 100%

  2. "Protected queen exchange" intervention: Directly ablating Tc.10.1958@h3 increases the probabilities of Qxh7+ and Qxd7 from 28.7% to 34.9% and from 26.1% to 29.3%, respectively, indicating that the model is more inclined to move the queen when its protection against exchanges is removed.

  3. "Rook-threat target": Applying a steering coefficient of −3 to the rook-threat feature increases the fxg7+ logit from 2.26 to 2.42 and its probability from 35.8% to 38.5%. Conversely, transplanting its activation from g2 to g7 reduces the probability of fxg7+ from 35.78% to 32.58%.

Together, these interventions provide causal evidence that downstream defensive signals influence the model's move preferences, with spatially specific effects consistent with our proposed mechanism.

These results support the causal relevance of the identified strategic cues and their joint contribution to the move decision.

Ans W2.4

Circuit effects of 0.5% and 1.2%,

The small average circuit effects are partly due to averaging over feature pairs that are not directly connected. When restricted to actual attribution-graph edges, the mean intervention effect is 18.90%, whereas randomly sampled non-edge feature pairs produce an effect of only 3.41 ± 0.71%.

This is consistent with Figure 8, which shows a positive correlation between attribution scores and zero-patching effects. We will include this distinction in the revised manuscript.

Ans W2.5

a minority of features passing the thresholds,

A total of 2,261 Transcoder features (43.67%) and 1,158 Lorsa features (31.17%) pass both precision and recall thresholds. Failure to pass these thresholds means that the current proposed interpretation is insufficiently validated; it does not imply that the underlying feature is intrinsically uninterpretable.

The mean activation-consistency scores of features in attribution graphs are 4.56 for Transcoders and 4.47 for Lorsas. We view these results as substantial progress beyond relying solely on one-shot LLM-generated explanations, which typically identify only a limited subset of interpretable features.

Our goal is to further develop the agentic interpretation pipeline. In particular, multi-round hypothesis refinement, chess-domain knowledge, and executable rule-based validation provide a more systematic way to improve the reliability and verifiability of feature interpretations. We aim to establish a reproducible evaluation benchmark for automated feature interpretation in structured domains.

Ans W2.6

and case studies being read off of attribution graphs instead of the careful interventions done by Jenner et al.

The additional careful case-study interventions are detailed in Ans W2.3 above.

AuthorsOpenReview ↗

Additional Response to Reviewer UJnb Part 3

The additional careful case-study interventions are detailed in Ans W2.3 above.

Questions

Ans Q1.1

What do your metrics like MAR, entropy, and branching look like for random moves, or the kth best move, etc? Try to separate whether the convergence finding is due to structure or not.

We extend the analysis to Top-1, Top-3, Top-5, and uniformly sampled random legal moves using the original 1,000 positions. For each move, we independently construct its attribution graph and calculate branching factor, feature-count entropy, and MAR.

Branching factor and Transcoder metrics

MoveBranching (%)L3 EntropyL3 MAR (%)L12 EntropyL12 MAR (%)
Top-173.30 ± 7.071.998 ± 0.31653.19 ± 13.501.270 ± 0.29181.13 ± 9.69
Top-378.31 ± 6.392.081 ± 0.29647.03 ± 13.711.397 ± 0.30175.25 ± 12.81
Top-579.21 ± 6.332.097 ± 0.28744.81 ± 15.101.453 ± 0.29272.47 ± 14.22
Uniform random81.50 ± 6.912.157 ± 0.32336.00 ± 16.931.570 ± 0.37165.11 ± 17.18

Lorsa metrics

MoveL3 EntropyL3 MAR (%)L12 EntropyL12 MAR (%)
Top-11.381 ± 0.31475.23 ± 12.060.502 ± 0.36491.43 ± 11.61
Top-31.475 ± 0.30668.74 ± 13.930.577 ± 0.41987.64 ± 15.59
Top-51.499 ± 0.31766.42 ± 19.310.656 ± 0.46784.63 ± 19.31
Uniform random1.577 ± 0.38657.32 ± 20.000.764 ± 0.54077.56 ± 25.21

The results show that higher-ranked moves exhibit lower attribution entropy and higher MAR, indicating stronger spatial concentration. Their branching factors are also lower than those of lower-ranked and random moves. For random moves, the move probability can be extremely low, but we can still identify features that provide some support for the move, albeit with more noise (reflected in higher entropy and lower MAR).

Intervention-based analysis. If the reviewer refers to the possibility that convergence is an artifact of our gradient-based graph construction, we additionally construct intervention-based graphs. Each feature is assigned a weight according to its intervention effect on the target logit:

\frac{ |\ell_{\mathrm{target}}^{\mathrm{original}} -\ell_{\mathrm{target}}^{\mathrm{intervened}}| }{ \ell_{\mathrm{target}}^{\mathrm{original}} }.$$ We then measure the corresponding source/target-square intervention effect ratio compared with MAR in the text. | **Layer** | **Lorsa attribution graph** | **Lorsa intervention** | **Transcoder attribution graph** | **Transcoder intervention** | | :-------: | --------------------------: | ---------------------: | -------------------------------: | --------------------------: | | L0 | 28.90 ± 9.32 | 24.96 ± 28.28 | 23.48 ± 8.35 | 25.81 ± 15.14 | | L3 | 63.77 ± 15.29 | 48.87 ± 22.4 | 40.38 ± 13.58 | 33.17 ± 16.28 | | L6 | 73.95 ± 34.61 | 65.69 ± 38.52 | 48.26 ± 11.57 | 36.30 ± 13.81 | | L9 | 66.78 ± 29.82 | 54.69 ± 36.69 | 62.24 ± 10.94 | 32.29 ± 16.13 | | L12 | 89.35 ± 9.04 | 84.03 ± 23.5 | 78.80 ± 10.68 | 52.89 ± 19.71 | | L14 | 100.00 ± 0.00 | 100.00 ± 0.00 | 100.00 ± 0.00 | 100.00 ± 0.00 | The intervention-based analysis exhibits a similar convergence. However, direct interventions can alter the forward computation and indirectly affect MAR through changes at other squares. This effect is more pronounced for Transcoders, consistent with the larger overall output norm of MLP modules compared with attention modules. This further shows that information important to a move indeed exhibits a convergence pattern. ### Ans Q1.2 > What does the target logit do when you zero-patch the h4 supernode? What about in comparison with a random supernode? The detailed causal interventions and target-logit changes are reported in Ans W2.3 above. When zero-patching the h4 supernode, the logit of Ng6+ decreases from 2.691 to 2.525, while zero-patching the random supernode decreases the logit from 2.69120 to 2.69105, which has very little impact. Thanks again for the helpful comments. Should you require any further information, please do not hesitate to let us know. We would greatly appreciate your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.