acceptodds
Under review as a conference paper at ICLR 2027

Cross-Layer Transcoders as Hypothesis Generators for Causal Circuit Analysis

Abstract

Cross-layer transcoders (CLTs) replace transformer MLP computation with sparse, interpretable feature dictionaries that support prompt-specific attribution graphs. Their graph scores, however, are derived from a local linearization and need not predict the effect of intervening on a feature in the full nonlinear model. We study this gap in \model. We train a -feature latent-mixing CLT that reconstructs all MLP outputs while using fewer decoder parameters than a direct cross-layer parameterization, then construct direct-effect and Neumann multi-hop attribution graphs for factual and mathematical prompts. Across five held-out evaluation distributions, reconstruction MSE ranges from to . The graphs surface semantically coherent recurrent candidates, but their Neumann scores do not directly recover measured single-feature effects. Direct perturbations identify \feature as a robust sentence-boundary continuation feature rather than a feature specific to However. During AIME generation, amplifying this feature reduces the frequency of newline continuations after periods from to and increases a restricted family of connective continuations led by Since and We. This local shift changes global solution form: across matched generations, mean lines per solution fall from to and mean line length rises from to tokens. Accuracy changes are modest in aggregate and heterogeneous by problem; amplification raises AIME 2024 Problem 12 accuracy from to but is not uniformly beneficial. These results support CLT attribution graphs as tools for candidate discovery while showing that causal claims require explicit intervention.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, cross-layer transcoders, attribution graphs, causal intervention, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MbcoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce a latent-mixing cross-layer transcoder (CLT) whose decoder uses an order of magnitude fewer parameters than a direct cross-layer parameterization. With this approach the paper studies CLT attribution as a method for generating mechanistic hypotheses to determine causal and behavioral meaning of identified features. The study concludes that CLT attribution may be suited to deciding where to intervene, but not to predicting what the intervention will do.

Critical strengths
  • The central questions of how CLT attribution can be used to discover causal mechanisms, is interesting and relevant.
  • The latent-mixing idea reduces computational overhead of standard CLT.
  • A limitations section is included and acknowledges the limited empirical depth of the study.
  • The results in Section 5 mirror recent findings on analyzing reasoning and the discovery of critical reasoning steps [Bog25] as well as algorithmic primitives [Ebe25, Lip26].
Critical weaknesses
  • Limited evaluation on only one model (Gemma-3-4B-IT) and only two data points (France →Paris, 42 + 7 = 49).

  • The novelty compared to [Ame25] seems primarily the latent-mixing CLT, but no direct evaluation is included.

  • The presented results in Section 4.2 depend on several implicit decisions that are not sufficiently motivated, i.e. “we replace the relevant entity, relation, or operand with matched alternatives“ (l. 252-253), “we retain candidates whose activation tracks the replaced item rather than the shared prompt frame“ (l. 254-255), or “to recover additional feature-to-feature hops” (l. 256-257). What is the relevant entity relation or operand? How is the alternative obtained? How is it decided what tracks the replaced item rather than the shared prompt? Why is it necessary to recover additional hops?

  • No evaluation of existing methods is included, i.e. attribution graphs [Ame25].

  • The paper overall lacks depth and only reports some selected samples, but it remains unclear how well the method performs overall to uncover interpretable causal mechanisms in LLMs. I acknowledge that the trained CLT transfer-well across tasks (see Table 1).

Related works

  • [Ame25] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025. https://transformer-circuits.pub/2025/attribution-graphs/methods.html
  • [Bog25] Bogdan, PC, et al. "Thought Anchors: Which LLM Reasoning Steps Matter?." arXiv:2506.19143 (2025)
  • [Ebe25] Eberle, O, et al. Position: We Need An Algorithmic Understanding of Generative AI. ICML 2025
  • [Lip26] Lippl, S, et al. "Algorithmic primitives and compositional geometry of reasoning in language models." ICML 2026. https://openreview.net/forum?id=ppirVueEj6
Questions for authors
  • What is the full distribution of SnS_n scores, and how representative is choosing the 100 largest ones?
5 comments
AuthorsOpenReview ↗

AIME ablations and a matched 270M CLT benchmark extend the evaluation beyond the two case studies

W1. "Evaluation across tasks and a second model size."

Thank you for raising the scope of the evaluation.

The pipeline is automatic, so the two case studies are demonstrations, not its limit. We train the CLT independently of the task. Given a set of prompts and a target logit for each, the pipeline selects features in three steps using fixed rules that require no human judgment.

  1. Attribution. For each prompt, build an attribution graph and compute the Neumann score SnS_{n} for every feature node. The score sums contributions to the target logit along paths with up to nn intermediate features (Eq. 8). Keep the 100 nodes with the highest positive scores.
  2. Recurrence. Retain features that appear in the top-100 lists for most prompts.
  3. Intervention. Intervene on each remaining feature and measure the change in the target logit per unit change in feature activation (Eq. 16). Select the feature with the largest measured effect.

Matched comparison with a direct CLT. A direct CLT at our 4B scale would require 153.54B decoder parameters, making it prohibitively expensive to train (Section 3.1). We therefore trained both CLTs on Gemma-3-270M-IT for 20,000 steps using the same data, encoder architecture, sparsity settings, and optimizer. Under these matched conditions, latent mixing uses a 9.4× smaller decoder and achieves comparable reconstruction and sparsity, the same attribution correlation, and 82% loss recovery.

CLTDecoder paramsMSERelative L2L_2Reconstruction errorSparsityAttribution faithfulnessDownstream (loss recovered)
Direct224.1M0.00630.1818.7%21.60.8987%
Latent mixing23.9M0.00710.19211.0%22.20.8982%

Neumann scores reliably rank ablation effects. We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by ∣S10∣|S_{10}| separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the However logit.

∣S10∣\lvert S_{10}\rvert rankMedian effectMax effect
1–1000.0530.68
101–1,0000.00960.16
1,001–10,0000.00180.026
beyond 10,000< 1e-40.016

Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, ∣S10∣|S_{10}| has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.

We leave evaluation of the full feature discovery and intervention workflow across additional models and tasks to future work.

AuthorsOpenReview ↗

Methodological innovation beyond latent mixing and matched CLT evaluation

W2. "Method contributions and direct evaluation of latent mixing."

Novelty beyond the CLT. Beyond latent mixing, we contribute a general pipeline from attribution graphs to model behavior:

  1. Automatic feature discovery. Where Ameisen et al. (2025) interpret each graph by hand, our pipeline selects features by fixed rules, from attribution scores to intervention effects (W1).
  2. Intervention during generation. It edits the selected feature throughout autoregressive generation, so its effect is measured on complete outputs (Section 6).
  3. A new setting. To our knowledge, this is the first CLT analysis of long-form reasoning, with the first CLT for Gemma-3-4B.

Direct evaluation of latent mixing. As detailed in W1, we trained direct and latent-mixing CLTs under matched conditions on Gemma-3-270M-IT. Latent mixing uses a 9.4× smaller decoder with comparable reconstruction and sparsity and the same Spearman correlation with ablation effects. We have added the matched reconstruction and sparsity results to the revised manuscript (Section 3.2, Table 2).

Comparison with published CLTs. Gemma Scope 2 (Google DeepMind, 2025) releases CLTs for Gemma-3-270M and Gemma-3-1B and per-layer transcoders for Gemma-3-4B. The table places our 4B model alongside published results. We report our normalized mean squared error averaged over layers, and published values follow their respective evaluation protocols.

ModelFeaturesTraining tokensReconstruction errorSparsity
18-layer model (Ameisen et al., 2025)10M~3B11.5%–
Claude 3.5 Haiku (Ameisen et al., 2025)30M~16B21.7%–
Gemma-3-1B, Gemma Scope 2256k–512k–≈14–21% (FVU)≈8–11%
Llama-3.2-1B (Neuronpedia)––≈50–56%–
Gemma-3-4B-IT, ours3.43M3.2M23.1%7.0%

Approximate values are read from plots. A dash indicates an unreported value.

We have added the published reconstruction comparison to the revised manuscript (Appendix B.2, Table 8). At 4B, our layer-averaged normalized mean squared error is close to the value reported for Anthropic's Haiku CLT (23.1% vs. 21.7%) with about 5,000× fewer training tokens. On the France → Paris graph, 7.0% of feature nodes carry 80% of the influence, compared with approximately 8–11% for Gemma Scope 2's CLTs. On AIME 2024 Problem 5, the 4B CLT reaches a Spearman correlation of 0.89 (W4). See Figure 4b and Appendix D.1 (Table 12) for the factual recall and arithmetic results.

AuthorsOpenReview ↗

Details of the case study protocols and evaluation of attribution graphs through ablation

W3. Matched prompts, feature selection, and indirect effects.

Thank you for asking us to clarify these steps. We explain the procedure below and have added the details to Appendix E of the revised manuscript.

(a) What is the relevant entity, relation, or operand?

  • In France → Paris, the entity is France and the relation is capital.
  • In 42 + 7 → 9, the operands are the two digits added in the final step, the 2 in 42 and 7.

(b) How is the alternative obtained? We vary one item at a time within the same prompt frame.

  • We replace France with other countries, as in The capital of Japan is.
  • We replace capital with seven other relations, as in The currency of France is (Appendix E.1).
  • We replace each of 2 and 7 with the other nine digits in turn, as in … = 43+7 = 4 (Appendix E.2.4 for the first operand).

(c) How is it decided what tracks the replaced item rather than the shared prompt? We compare the feature's activation across prompts that differ only in the item we replace. A feature that responds to the shared frame should activate at similar levels across alternatives. A feature that tracks the item activates strongly for the original and less for the alternatives.

  • France and capital. We retain a candidate if it activates strongly at the target position for the original item and less for the alternatives.
  • Operands. We retain a candidate if it is active on the original digit and inactive on all nine alternatives.

(d) Why is it necessary to recover additional hops? Intermediate features carry much of the influence of the input features on the answer. Tracing additional hops shows how that influence reaches the answer.

We ablate each input feature in the original model and measure the effect under two conditions. For the total effect, the model responds normally to the ablation. For the direct effect, we hold the MLP outputs at their unedited values before applying the ablation, blocking propagation through intermediate features.

ReadoutInput feature(s) ablatedTotal effectDirect effectDirect share
Paris logitTop 20 input features (sum of magnitudes)8.080.9412%
Paris logitL0:F81670 (capital)+0.249+0.03012%
Paris logitL0:F26582 (France)+0.131+0.06550%
9 relative marginL0:F86086 (7)+0.260+0.07228%
9 relative marginL0:F44905 (2)−0.020+0.266—

The direct effects account for 12–50% of the total in the rows where we report a share. For the operand 2, the direct and total effects have opposite signs.


W4. Validation of attribution scores against ablations.

We evaluate the attribution graphs against intervention in the original model. We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by ∣S10∣|S_{10}| separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the However logit.

∣S10∣\lvert S_{10}\rvert rankMedian effectMax effect
1–1000.0530.68
101–1,0000.00960.16
1,001–10,0000.00180.026
beyond 10,000< 1e-40.016

Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, ∣S10∣|S_{10}| has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.

Reviewer_A4r3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces latent-mixing CLT for Gemma-3-4B-IT, retaining a single shared decoder per output layer and mixing different source layers through per-feature scalar coefficients, reducing decoder parameter count by 17.4× compared with direct CLT. Based on this CLT, the authors construct attribution graphs using direct effects and multi-hop Neumann attribution, formulate circuit hypotheses for factual retrieval (France → Paris) and arithmetic (42 + 7), and test them through prompt substitution and ablation. They then identify a recurring feature, L22:F31850, in AIME reasoning prompts and demonstrate its bidirectional control over end-of-line newlines and connective selection through generation-time interventions, while its impact on final answer accuracy varies across problems.

Critical strengths
  1. Proposes a substantially more parameter-efficient CLT parameterization (approximately 8.83B decoder parameters, compared with 153.5B for direct CLT), lowering the barrier to training and using CLTs on larger models.
  2. Explicitly distinguishes attribution scores from intervention effects: Table 2 shows that the slopes predicted by Neumann attribution do not match the empirically measured slopes and can even have opposite signs. This finding offers a valuable caution for users of attribution-based methods.
  3. The end-of-line intervention experiments are relatively well designed, involving 60 problems, 128 samples per problem, bidirectional interventions that amplify and reverse the effect, and paired comparisons using the same prefixes.
Critical weaknesses
  1. The experiments lack systematicity and are vulnerable to cherry-picking. The case studies cover only two single-prompt examples. Feature discovery is based on just six AIME prefixes, and only one feature is analyzed in depth; its selection also involves subjective judgment, as its label is closest to "proof/logic." Since the goal is to develop a tool that can reliably support feature interpretation, the authors should conduct large-scale validation—for example, measuring the reproducibility of attribution circuits across many prompts and diverse tasks, and intervening on multiple features rather than analyzing just one.

  2. The quality of the explanations cannot be adequately evaluated. The appendix shows an inverted-U pattern in layer-wise reconstruction error, with the relative L2 error reaching 0.67 in some middle layers. Fourteen of the 34 layers have errors exceeding 0.5, while the intervention feature at layer 22 has an error of approximately 0.49. This suggests that roughly 45% of the MLP output energy at that intermediate layer remains unexplained. The paper does not discuss how this limitation affects the attribution graphs or intervention conclusions. Furthermore, the current analyses are too limited to determine whether similarly meaningful features and attributions can be identified in other tasks.

  3. Baseline comparisons are missing. The paper provides no controlled comparisons against direct CLT, standard transcoders, or publicly available CLT implementations. The reported 17.4× reduction reflects only a theoretical parameter-count comparison; differences in reconstruction error, sparsity, attribution faithfulness, and downstream task performance are not quantified. Consequently, it remains unclear whether the proposed parameterization offers a genuine practical advantage.

Questions for authors
  1. What are the MSE and relative L2 error of a standard CLT? How much interpretable variance is lost by latent-mixing compared with the standard CLT?

  2. Why is the assumption that "shared directions + per-feature scalar coefficients" can adequately replace independent decoders across layers justified? The write-in directions of features may vary across layers, and the resulting approximation error should be quantified in terms of reconstruction quality, sparsity, and attribution faithfulness. For example, both parameterizations could be trained and compared at a smaller scale under otherwise identical conditions.

  3. Why did the authors not compare their method against existing open-source CLTs, or reproduce the analyses using those models? Such a comparison would directly demonstrate the gains and trade-offs of the proposed method relative to existing tools.

6 comments
AuthorsOpenReview ↗

Quantitative feature selection and validation across 60 AIME problems

W1. "The experiments lack systematicity and are vulnerable to cherry-picking."

We argue that the pipeline presented is general. We train the CLT on collected datasets. To find features that are relevant to a specific model behavior (like generating logical connectives such as "However", "Since"), we first gather a set of prompts, select the target logits we want to study. The pipeline selects the most relevant and interpretable features in three steps using the following rules:

  1. Attribution. For each prompt, build an attribution graph and compute the Neumann score SnS_{n} for every feature node. The score sums contributions to the target logit along paths with up to nn intermediate features (Eq. 8). Keep the top-KKnodes with the highest positive scores.
  2. Select Common Features. Retain features that appear in the top-KK lists for most prompts.
  3. Intervention. Intervene on each remaining feature and measure the change in the target logit per unit change in feature activation (Eq. 16). Select the feature with the largest measured effect.

In our experiment, we use K=100K=100.

Feature selection follows quantitative criteria. Attribution graphs identify candidates that rank highly across prompts, and measured intervention effects determine which candidate we study during generation. An LLM generates the labels from each feature's top-activating OpenWebMath passages, independently of the discovery prompts. The labels help interpret features after selection (Bills et al., 2023, Section 3.2). Applying the same pipeline to a new task requires only new prompts and their target logits. We leave larger-scale validation across prompts, tasks, and features to future work.

Scope of the evaluation. We appreciate the suggestion to validate the method at a larger scale. Our worked examples include feature screening on 1,347 prompt–position pairs for factual recall and answer-feature selectivity tests on 26 arithmetic prompts (Appendix E). For AIME, we use six reasoning prefixes for feature discovery and report fixed-prompt intervention results for five recurrent candidates (Appendix F). We then test the selected feature during generation on all 60 problems from AIME 2024 and 2025, with 128 samples per problem per setting (Section 6). Given our compute budget, we focused the behavioral evaluation on these settings. The same workflow can support broader studies of circuit reproducibility and interventions on additional features across domains.

AuthorsOpenReview ↗

Interventions verify feature effects despite mid-layer reconstruction error

W2. "The quality of the explanations cannot be adequately evaluated."

The reconstruction residual at layer 22 only has about 24% of the MLP output energy. Relative L2L_2 error is the ratio of the reconstruction error norm to the MLP output norm. Squaring it gives the ratio of their energies, 0.492≈24%0.49^2\approx 24\%.

Absolute reconstruction error actually falls toward the middle layers. The absolute reconstruction error actually falls and reaches its lowest values in these layers. The MLP output norm, which is the denominator of relative L2L_2 error, falls even faster, so the ratio rises. The table reports root-mean-square norms across token positions.

Layer03691215182124273033
Absolute error6.709.705.952.921.741.692.112.602.853.912.387.29
MLP output norm43.530.113.14.812.632.784.074.916.2010.47.5126.5
Relative error0.150.320.460.610.660.610.520.530.460.380.320.28

A direct CLT shows a similar rise. On Gemma-3-270M-IT, we trained a direct CLT and a latent-mixing CLT for 20,000 steps with the same data, encoder architecture, sparsity settings, and optimizer. Both show a similar rise in layerwise relative L2L_2 error and peak at layer 4.

Layer0246810121416
Direct0.160.240.400.340.360.300.250.260.23
Latent mixing0.160.280.440.380.390.330.300.290.25

Features in these layers remain useful. Relative L2L_2 error measures only the fidelity of the CLT's reconstruction of the full MLP output. The features we recover in these layers are still useful for explaining specific model computations, as our interventions show. For example, we identified a feature that helps represent the answer digit in arithmetic (Appendix E.2) and another that influences line breaks in AIME solutions (Section 6.2). We leave attribution through the unreconstructed component, which Ameisen et al. (2025) represent as error nodes, to future work.

Neumann scores reliably rank ablation effects. We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by ∣S10∣|S_{10}| separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the However logit.

∣S10∣\lvert S_{10}\rvert rankMedian effectMax effect
1–1000.0530.68
101–1,0000.00960.16
1,001–10,0000.00180.026
beyond 10,000< 1e-40.016

Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, ∣S10∣|S_{10}| has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.

Interventions isolate the effect of the feature edit. The CLT supplies the direction of the edit. We record and add back the reconstruction residual to recover the original model's output in the unedited baseline. This isolates the effect of the edit, and generation continues in the original model (Appendix G).

AuthorsOpenReview ↗

Comparison with a direct CLT and published CLTs

W3. "Baseline comparisons are missing."

We thank the reviewer for raising this point. In our main experiment on Gemma-3-4B-IT, latent mixing reduces the decoder parameter count from 153.54B to 8.83B, a 17.4× reduction. Under our experimental conditions, we cannot train a direct CLT at this 153.54B scale. Therefore, in the following, we report two comparisons: a matched comparison with a direct CLT on Gemma-3-270M-IT and a comparison with published CLTs.

Matched comparison with a direct CLT. A direct CLT at our 4B scale would require 153.54B decoder parameters, making it prohibitively expensive to train (Section 3.1). We therefore trained both CLTs on Gemma-3-270M-IT for 20,000 steps using the same data, encoder architecture, sparsity settings, and optimizer. Under these matched conditions, latent mixing uses a 9.4× smaller decoder and achieves comparable reconstruction and sparsity, the same attribution correlation, and 82% loss recovery.

CLTDecoder paramsMSERelative L2L_2Reconstruction errorSparsityAttribution faithfulnessDownstream (loss recovered)
Direct224.1M0.00630.1818.7%21.60.8987%
Latent mixing23.9M0.00710.19211.0%22.20.8982%

Comparison with published CLTs. Gemma Scope 2 (Google DeepMind, 2025) releases CLTs for Gemma-3-270M and Gemma-3-1B and per-layer transcoders for Gemma-3-4B. The table places our 4B model alongside published results. We report our normalized mean squared error averaged over layers, and published values follow their respective evaluation protocols.

ModelFeaturesTraining tokensReconstruction errorSparsity
18-layer model (Ameisen et al., 2025)10M~3B11.5%–
Claude 3.5 Haiku (Ameisen et al., 2025)30M~16B21.7%–
Gemma-3-1B, Gemma Scope 2256k–512k–≈14–21% (FVU)≈8–11%
Llama-3.2-1B (Neuronpedia)––≈50–56%–
Gemma-3-4B-IT, ours3.43M3.2M23.1%7.0%

Approximate values are read from plots. A dash indicates an unreported value.

The two comparisons address the four quantities raised by the reviewer.

  • Reconstruction error. The matched CLTs achieve comparable layer-averaged normalized mean squared error (11.0% for latent mixing and 8.7% for the direct CLT). At 4B, we achieve a value close to the one reported for Anthropic's Haiku CLT (23.1% vs. 21.7%) with about 5,000× fewer training tokens.
  • Sparsity. The matched CLTs activate a similar number of features per token per layer (22.2 vs. 21.6). On the 4B France → Paris graph, 7.0% of feature nodes carry 80% of the influence, compared with the approximately 8–11% reported for Gemma Scope 2's CLTs.
  • Attribution faithfulness. Both CLTs reach a Spearman correlation of 0.89 between Neumann scores and ablation effects. The 4B CLT reaches 0.89 on the AIME 2024 Problem 5 graph (W2). See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.
  • Downstream performance. On held-out OpenWebMath, we measure how much of the increase in next-token loss from zeroing MLP outputs across all layers we can recover by replacing them with CLT reconstructions. Latent mixing recovers 82%, compared with 87% for the direct CLT.

Q1. Reconstruction error in direct and latent-mixing CLTs.

A direct CLT would require 153.54B decoder parameters on Gemma-3-4B-IT, so we compare the two parameterizations under matched conditions on Gemma-3-270M-IT (W3, Table 2). Latent mixing achieves comparable reconstruction with a 9.4× smaller decoder. Its MSE is 0.0071 vs. 0.0063 for the direct CLT, and its pooled relative L2L_2 error is 0.192 vs. 0.181. The layer-averaged normalized mean squared errors are 11.0% and 8.7%, respectively, a difference of 2.3 percentage points. At 4B, our layer-averaged error is close to the value Anthropic reports for its Haiku CLT (23.1% vs. 21.7% under the respective evaluation protocols, Appendix B.2, Table 8).

Reviewer_JRpvOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a more parameter efficient form of crosscoders through the use of a latent mixing term. They validate the architecture by training it on Gemma-3-4B-IT activations and achieve a nearly perfect reconstruction with MSE < 0.01. They then demonstrate the use of their crosscoder by identifying an explicit feature in layer 22 that mediates next sentence continuations starting with “Since” or “We”. They also show that while such crosscoders are useful tools for producing a shortlist of nodes for use in an attribution graph, causal validation of functionality is necessary, and demonstrate some examples where use of top activating crosscoder feature can be misleading.

Critical strengths
  • The authors find an elegant technique for reducing the size, and corresponding training time, of crosscoders by 17x. They validate this on five standard evaluation sets and validity sparsity empirically.
  • The paper is clearly written, with explanations of methodology and mathematical formulae consistently laid out before discussing applications and experiments. The chosen examples with small-sized attribution graphs are helpful validations of the methodology.
  • The authors’ direct effect attribution methodology in section 4.1 is a useful contribution, showing how to track attribution across compositions of edges for recurrent attribution.
Critical weaknesses
  • As the authors address in the Limitations section, L2 error rises into the middle of the architecture, particularly where Gemma has relatively low-norm activations, and most features do not offer useful explanations there. The result has only been validated on one model and only six prompt templates. A more fulsome investigation into whether mixed latent CLTs generalize to other models/tasks would help strengthen the result.
Questions for authors
  • Can the authors comment on if/how the Neumann score can be used to track more diffuse contributions across larger attribution graphs? In particular, how does the L2 error rising in the middle of the model interact with the use of such attribution tools across multiple edges? If most tasks require attribution nodes in the middle of the architecture, does that make attribution harder with CLTs alone in their current form? If so, it would be useful to add this to the Limitations discussion.
  • Are the 41.3% of dictionary entries that never fire indicative of potential training issues in the CLT producing dead neurons, such as those which required “ghost grads” to resolve in the Jermyn et al (2024)? Have the authors explored resolving this and/or fitting a more narrow dictionary with fewer never firing entries?
  • Can the authors please fix any reference issues, such as the Figure reference on L834?
3 comments
AuthorsOpenReview ↗

Mid-layer reconstruction error and Neumann score validation

W1. "Mid-layer reconstruction error and validation across 60 AIME problems."

Absolute reconstruction error actually falls toward the middle layers. The absolute reconstruction error actually falls and reaches its lowest values in these layers. The MLP output norm, which is the denominator of relative L2L_2 error, falls even faster, so the ratio rises. The table reports root-mean-square norms across token positions.

Layer03691215182124273033
Absolute error6.709.705.952.921.741.692.112.602.853.912.387.29
MLP output norm43.530.113.14.812.632.784.074.916.2010.47.5126.5
Relative error0.150.320.460.610.660.610.520.530.460.380.320.28

A direct CLT shows a similar rise. On Gemma-3-270M-IT, we trained a direct CLT and a latent-mixing CLT for 20,000 steps with the same data, encoder architecture, sparsity settings, and optimizer. Both show a similar rise in layerwise relative L2L_2 error and peak at layer 4.

Layer0246810121416
Direct0.160.240.400.340.360.300.250.260.23
Latent mixing0.160.280.440.380.390.330.300.290.25

Features in these layers remain useful. Relative L2L_2 error measures only the fidelity of the CLT's reconstruction of the full MLP output. The features we recover in these layers are still useful for explaining specific model computations, as our interventions show. For example, we identified a feature that helps represent the answer digit in arithmetic (Appendix E.2) and another that influences line breaks in AIME solutions (Section 6.2). We leave attribution through the unreconstructed component, which Ameisen et al. (2025) represent as error nodes, to future work.

Validation beyond the discovery prefixes. We use six AIME prefixes to select L22:F31850, then validate its effect during generation on all 60 problems from AIME 2024 and 2025, sampling 128 solutions per problem per setting (Section 6). We leave broader validation across models and tasks to future work.


Q1. "Neumann scores track diffuse effects across large attribution graphs."

The Neumann score tracks diffuse contributions on large graphs. The Neumann score is Sn=∑i=0nTiS_n=\sum_{i=0}^{n}T_i (Eq. 8), where TiT_i sums the contributions of all paths to the target logit with exactly ii intermediate features. The terms decay geometrically with ii, so truncating at n=10n=10 captures nearly all of the aggregate contribution (Figure 4a).

We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by ∣S10∣|S_{10}| separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the However logit.

∣S10∣\lvert S_{10}\rvert rankMedian effectMax effect
1–1000.0530.68
101–1,0000.00960.16
1,001–10,0000.00180.026
beyond 10,000< 1e-40.016

Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, ∣S10∣|S_{10}| has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.

How we limit the impact of mid-layer error. We do not rely on attribution alone: it only nominates candidates, and each candidate is confirmed by intervention in the original model (Section 5 and Appendix F, Table 20), so every feature we report has a measured effect on the model itself. Because a diffuse contribution aggregates many paths, we perform this aggregation in fp32 to limit the accumulation of floating-point rounding error. We expand the discussion of reconstruction coverage in the Limitations (Section 7). We leave tracing paths through the reconstruction error to future work (W1).

AuthorsOpenReview ↗

Inactive CLT features and corrected figure reference

Q2. "Inactive CLT features and retraining results."

The fraction depends on the evaluation sample. We count an entry as inactive if it never fires across the 2,000 held-out StackExchange conversations. In a dictionary of 3.43M entries, many features may be this specific, and the evaluation sample may miss the patterns that activate them. For instance, L0:F30117 (Table 4) fires on enterprise software names such as SharePoint.

Inactive entries are common in large dictionaries. Published SAEs report substantial fractions of inactive entries at larger dictionary sizes. We also observe a substantial fraction in the direct CLT from our matched 270M experiment (W1).

DictionaryEntriesNever fire
SAE on Claude 3 Sonnet (Templeton et al., 2024)4M35%
SAE on Claude 3 Sonnet (Templeton et al., 2024)34M65%
Direct CLT, Gemma-3-270M-IT (W1)36,86444%
Latent-mixing CLT, Gemma-3-270M-IT (W1)36,86450%
Latent-mixing CLT, Gemma-3-4B-IT (ours)3.43M41.3%

Each fraction is measured on the corresponding evaluation sample.

Inactivity persists with narrower dictionaries and after retraining. In our 270M CLTs, 44–50% of entries never fire even at a width of 2,048 entries per layer. We also retrained the 270M latent-mixing CLT with an auxiliary loss and with ghost gradients in separate runs, holding the other training settings fixed.

RemedyNever fireReconstruction error
None50%11.0%
Auxiliary loss (Gao et al., 2024)49%11.0%
Ghost grads (Jermyn et al., 2024)89%12.6%

The negative pre-activations of these entries block gradients from the auxiliary loss. Ghost gradients push their pre-activations further below zero, increasing the fraction of inactive entries.


Q3. "Figure reference corrected."

Thank you. We have fixed the reference at L834: it now points to the direct-effect figure, added as Figure 3 in the revised version. We checked all other cross-references and found no other unresolved ones.

Thank you for the additional analysis of reconstruction error within middle layers, and for correcting my remark that it rises there. Thank you also for verifying that the feature inactivity is not an artifact to be remediated with ghost gradients or other training adjustments, as I agree that inactive entries can be common in large dictionaries. My concerns have been addressed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New CLT benchmarks, ablation tests, and case study protocols

We thank all reviewers for their careful and constructive feedback. We have revised the paper accordingly, and all content added in the revision is marked in red. The main additions are:

  1. Direct-CLT baseline (Section 3.2, Table 2). On Gemma-3-270M-IT, latent mixing uses a 9.4× smaller decoder than a direct CLT under matched training conditions, with comparable reconstruction error (11.0% vs. 8.7%) and similar sparsity (22.2 vs. 21.6 active features per token per layer).
  2. Comparison with published CLTs (Appendix B.2, Table 8). Using roughly one-ninth as many features as Anthropic's Haiku CLT (3.43M vs. 30M), our Gemma-3-4B-IT CLT achieves a similar layer-averaged normalized mean squared error (23.1% vs. 21.7% under the respective evaluation protocols).
  3. Validation of attribution graphs (Section 4.3, Appendix D). Neumann scores rank feature-ablation effects in the original model with a mean Spearman correlation of 0.86. Targeted ablations also show that much of the measured input influence reaches the output through intermediate features.
  4. Case-study protocols (Appendix E). We add protocol details for both case studies, including which entities, relations, and operands we vary, how we construct matched prompts, and how we screen and test candidate features.
  5. Limitations (Section 7). We add a discussion of how reconstruction error can obscure attribution paths through the middle layers and limit what a CLT alone can explain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.