- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors introduce a latent-mixing cross-layer transcoder (CLT) whose decoder uses an order of magnitude fewer parameters than a direct cross-layer parameterization. With this approach the paper studies CLT attribution as a method for generating mechanistic hypotheses to determine causal and behavioral meaning of identified features. The study concludes that CLT attribution may be suited to deciding where to intervene, but not to predicting what the intervention will do.
Critical strengths
- The central questions of how CLT attribution can be used to discover causal mechanisms, is interesting and relevant.
- The latent-mixing idea reduces computational overhead of standard CLT.
- A limitations section is included and acknowledges the limited empirical depth of the study.
- The results in Section 5 mirror recent findings on analyzing reasoning and the discovery of critical reasoning steps [Bog25] as well as algorithmic primitives [Ebe25, Lip26].
Critical weaknesses
-
Limited evaluation on only one model (Gemma-3-4B-IT) and only two data points (France →Paris, 42 + 7 = 49).
-
The novelty compared to [Ame25] seems primarily the latent-mixing CLT, but no direct evaluation is included.
-
The presented results in Section 4.2 depend on several implicit decisions that are not sufficiently motivated, i.e. “we replace the relevant entity, relation, or operand with matched alternatives“ (l. 252-253), “we retain candidates whose activation tracks the replaced item rather than the shared prompt frame“ (l. 254-255), or “to recover additional feature-to-feature hops” (l. 256-257). What is the relevant entity relation or operand? How is the alternative obtained? How is it decided what tracks the replaced item rather than the shared prompt? Why is it necessary to recover additional hops?
-
No evaluation of existing methods is included, i.e. attribution graphs [Ame25].
-
The paper overall lacks depth and only reports some selected samples, but it remains unclear how well the method performs overall to uncover interpretable causal mechanisms in LLMs. I acknowledge that the trained CLT transfer-well across tasks (see Table 1).
Related works
- [Ame25] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025. https://transformer-circuits.pub/2025/attribution-graphs/methods.html
- [Bog25] Bogdan, PC, et al. "Thought Anchors: Which LLM Reasoning Steps Matter?." arXiv:2506.19143 (2025)
- [Ebe25] Eberle, O, et al. Position: We Need An Algorithmic Understanding of Generative AI. ICML 2025
- [Lip26] Lippl, S, et al. "Algorithmic primitives and compositional geometry of reasoning in language models." ICML 2026. https://openreview.net/forum?id=ppirVueEj6
Questions for authors
- What is the full distribution of scores, and how representative is choosing the 100 largest ones?
5 comments
Methodological innovation beyond latent mixing and matched CLT evaluation
W2. "Method contributions and direct evaluation of latent mixing."
Novelty beyond the CLT. Beyond latent mixing, we contribute a general pipeline from attribution graphs to model behavior:
- Automatic feature discovery. Where Ameisen et al. (2025) interpret each graph by hand, our pipeline selects features by fixed rules, from attribution scores to intervention effects (W1).
- Intervention during generation. It edits the selected feature throughout autoregressive generation, so its effect is measured on complete outputs (Section 6).
- A new setting. To our knowledge, this is the first CLT analysis of long-form reasoning, with the first CLT for Gemma-3-4B.
Direct evaluation of latent mixing. As detailed in W1, we trained direct and latent-mixing CLTs under matched conditions on Gemma-3-270M-IT. Latent mixing uses a 9.4× smaller decoder with comparable reconstruction and sparsity and the same Spearman correlation with ablation effects. We have added the matched reconstruction and sparsity results to the revised manuscript (Section 3.2, Table 2).
Comparison with published CLTs. Gemma Scope 2 (Google DeepMind, 2025) releases CLTs for Gemma-3-270M and Gemma-3-1B and per-layer transcoders for Gemma-3-4B. The table places our 4B model alongside published results. We report our normalized mean squared error averaged over layers, and published values follow their respective evaluation protocols.
| Model | Features | Training tokens | Reconstruction error | Sparsity |
|---|---|---|---|---|
| 18-layer model (Ameisen et al., 2025) | 10M | ~3B | 11.5% | – |
| Claude 3.5 Haiku (Ameisen et al., 2025) | 30M | ~16B | 21.7% | – |
| Gemma-3-1B, Gemma Scope 2 | 256k–512k | – | ≈14–21% (FVU) | ≈8–11% |
| Llama-3.2-1B (Neuronpedia) | – | – | ≈50–56% | – |
| Gemma-3-4B-IT, ours | 3.43M | 3.2M | 23.1% | 7.0% |
Approximate values are read from plots. A dash indicates an unreported value.
We have added the published reconstruction comparison to the revised manuscript (Appendix B.2, Table 8). At 4B, our layer-averaged normalized mean squared error is close to the value reported for Anthropic's Haiku CLT (23.1% vs. 21.7%) with about 5,000× fewer training tokens. On the France → Paris graph, 7.0% of feature nodes carry 80% of the influence, compared with approximately 8–11% for Gemma Scope 2's CLTs. On AIME 2024 Problem 5, the 4B CLT reaches a Spearman correlation of 0.89 (W4). See Figure 4b and Appendix D.1 (Table 12) for the factual recall and arithmetic results.
Details of the case study protocols and evaluation of attribution graphs through ablation
W3. Matched prompts, feature selection, and indirect effects.
Thank you for asking us to clarify these steps. We explain the procedure below and have added the details to Appendix E of the revised manuscript.
(a) What is the relevant entity, relation, or operand?
- In France → Paris, the entity is
Franceand the relation iscapital. - In 42 + 7 → 9, the operands are the two digits added in the final step, the
2in42and7.
(b) How is the alternative obtained? We vary one item at a time within the same prompt frame.
- We replace
Francewith other countries, as inThe capital of Japan is. - We replace
capitalwith seven other relations, as inThe currency of France is(Appendix E.1). - We replace each of
2and7with the other nine digits in turn, as in… = 43+7 = 4(Appendix E.2.4 for the first operand).
(c) How is it decided what tracks the replaced item rather than the shared prompt? We compare the feature's activation across prompts that differ only in the item we replace. A feature that responds to the shared frame should activate at similar levels across alternatives. A feature that tracks the item activates strongly for the original and less for the alternatives.
Franceandcapital. We retain a candidate if it activates strongly at the target position for the original item and less for the alternatives.- Operands. We retain a candidate if it is active on the original digit and inactive on all nine alternatives.
(d) Why is it necessary to recover additional hops? Intermediate features carry much of the influence of the input features on the answer. Tracing additional hops shows how that influence reaches the answer.
We ablate each input feature in the original model and measure the effect under two conditions. For the total effect, the model responds normally to the ablation. For the direct effect, we hold the MLP outputs at their unedited values before applying the ablation, blocking propagation through intermediate features.
| Readout | Input feature(s) ablated | Total effect | Direct effect | Direct share |
|---|---|---|---|---|
Paris logit | Top 20 input features (sum of magnitudes) | 8.08 | 0.94 | 12% |
Paris logit | L0:F81670 (capital) | +0.249 | +0.030 | 12% |
Paris logit | L0:F26582 (France) | +0.131 | +0.065 | 50% |
9 relative margin | L0:F86086 (7) | +0.260 | +0.072 | 28% |
9 relative margin | L0:F44905 (2) | −0.020 | +0.266 | — |
The direct effects account for 12–50% of the total in the rows where we report a share. For the operand 2, the direct and total effects have opposite signs.
W4. Validation of attribution scores against ablations.
We evaluate the attribution graphs against intervention in the original model. We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the However logit.
| rank | Median effect | Max effect |
|---|---|---|
| 1–100 | 0.053 | 0.68 |
| 101–1,000 | 0.0096 | 0.16 |
| 1,001–10,000 | 0.0018 | 0.026 |
| beyond 10,000 | < 1e-4 | 0.016 |
Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.
AIME ablations and a matched 270M CLT benchmark extend the evaluation beyond the two case studies
W1. "Evaluation across tasks and a second model size."
Thank you for raising the scope of the evaluation.
The pipeline is automatic, so the two case studies are demonstrations, not its limit. We train the CLT independently of the task. Given a set of prompts and a target logit for each, the pipeline selects features in three steps using fixed rules that require no human judgment.
Matched comparison with a direct CLT. A direct CLT at our 4B scale would require 153.54B decoder parameters, making it prohibitively expensive to train (Section 3.1). We therefore trained both CLTs on Gemma-3-270M-IT for 20,000 steps using the same data, encoder architecture, sparsity settings, and optimizer. Under these matched conditions, latent mixing uses a 9.4× smaller decoder and achieves comparable reconstruction and sparsity, the same attribution correlation, and 82% loss recovery.
Neumann scores reliably rank ablation effects. We also validate the score on the 280,569-node graph of AIME 2024 Problem 5. We rank nodes by ∣S10∣ separately for positive and negative scores and sample 200 nodes from each rank range shown below. We ablate each sampled node in the original model and measure the absolute change in the
Howeverlogit.Both median and maximum effects decrease across successively lower-ranked groups, and every sampled node with an effect above 0.2 falls in the top group. Across these 800 nodes, ∣S10∣ has a Spearman correlation of 0.89 with the measured effects, compared with 0.52 for activation magnitude. These results show that the graph captures the circuit that drives the output. See Figure 4b and Appendix D.1 (Table 12) of the revised paper for additional results on factual recall and arithmetic.
We leave evaluation of the full feature discovery and intervention workflow across additional models and tasks to future work.