- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper compares sparse autoencoders (SAEs) and transcoders (TCs) in terms of single-feature discrimination. Using pretrained Gemma Scope 2 checkpoints on Gemma 3 1B, the authors evaluate how well individual SAE and TC features discriminate shared MLP-output clusters across 26 layers and two sparsity settings. The main finding is that TCs achieve higher postactivation AUROC than SAEs, despite SAEs generally achieving higher preactivation AUROC. The authors investigate this difference through JumpReLU thresholding and encoder-direction interventions. They also conduct entity detection and sentence retrieval experiments on Few-NERD to examine the practical implications of their findings.
Critical strengths
-
Controlled comparisons. The authors construct shared MLP-output clusters as common evaluation targets and use held-out documents for evaluation. This provides a consistent setting for comparing the discrimination capabilities of SAE and TC features.
-
Additional practical evaluation. Beyond the analysis of feature discrimination, the paper includes entity detection and sentence retrieval experiments on Few-NERD to examine whether the observed differences translate into practical benefits.
Critical weaknesses
- W1. Limited experimental scope and lack of generalizability.
All experiments are conducted on a single relatively small language model, Gemma 3 1B, using the corresponding Gemma Scope 2 checkpoints (Section 3.1). Although the authors evaluate all 26 layers and two sparsity settings, these experiments remain restricted to one model family and one model scale.
This raises concerns about whether the observed differences between SAEs and TCs are general properties or specific to the selected model and checkpoints. The paper would be substantially more convincing if the conclusions were validated across multiple model families and scales, such as Qwen3, Qwen3.5, and OLMo 3.
- W2. Insufficient motivation and unclear significance of the findings.
My main concern is that the paper presents extensive technical analyses without sufficiently establishing why these analyses matter. The paper investigates single-feature discrimination using MLP-output clusters and AUROC. However, the connection between this evaluation setting and the broader objectives of mechanistic interpretability is not adequately motivated. In particular, it remains unclear why improved discrimination of these clusters should be considered an important property of SAEs or TCs, and what practical implications follow from the observed differences.
While the authors distinguish their evaluation from prior work on automated interpretability, the paper does not sufficiently explain why the proposed evaluation perspective addresses an important limitation of existing approaches.
The practical evaluation on Few-NERD also provides relatively modest improvements. As shown in Table 6, TCs retrieve only approximately 2.75 and 4.35 additional target sentences per 200 reviewed under the two sparsity settings. These results do not seem sufficient to establish the broader practical significance of the extensive analyses presented in the paper.
Overall, the paper contains considerable technical detail, but the connections among the research motivation, experimental findings, and broader research significance are not clearly established. This makes it difficult to understand the main contribution and why the findings should matter to the broader research community.
Questions for authors
-
Q1. Incomplete reporting of confidence intervals. In Tables 3, 5, and 6, the authors report 95% confidence intervals only for selected derived quantities, such as additional AUROC reduction, AUROC recovery, and the TC–SAE performance difference. However, confidence intervals are not provided for the underlying measurements, such as the Original, Random, and Targeted AUROC values in Table 3, or the individual SAE and TC results in Table 6. Could the authors clarify why uncertainty estimates are reported only for these selected quantities? Reporting confidence intervals for the other measurements would provide a more complete picture of the statistical reliability of the experimental results.
-
Q2. Clarity and motivation in the abstract. The abstract immediately introduces SAEs and TCs and begins comparing their single-feature discrimination, without first establishing the broader research problem or explaining why this comparison is important. The evaluation task itself is not clearly introduced before the experimental findings are presented. Moreover, the abstract primarily lists experimental observations, making it difficult to identify the main scientific contribution and its significance. Could the authors revise the abstract to more clearly explain the research motivation, the problem being addressed, and the key contribution of the paper?
Response to Reviewer mv3c
Thank you for asking how far the 1B results generalize and why single-feature discrimination matters for interpretability. We extended the comparison to three more model sizes and revised the paper to explain what this evaluation tells us that a final feature score alone cannot.
W1. Do the findings generalize beyond Gemma 3 1B?
What held across model sizes was that SAE features lost more discrimination through JumpReLU, not that TC features always performed better afterward. We evaluated every layer of Gemma 3 270M, 1B, 4B, and 12B under both sparsity settings. In all eight comparisons, the selected SAE features had higher mean AUROC before JumpReLU. In 270M, 1B, and 4B, SAE lost enough of that initial advantage for TC to finish ahead. In 12B, SAE retained its advantage after JumpReLU (see Table 1).
The input-direction result also held in 12B despite this reversal. For each coder, we removed the contribution of its own first input principal component (PC1) and compared the result with random edits matched for rotation angle and final norm. In the layer averages for all four models and both settings, removing PC1 reduced preactivation AUROC more than the matched edits for TC features and less for SAE features (Figure 4). These experiments extend the comparison across sizes within Gemma 3. We have not tested other model families.
W2. Why does output-cluster discrimination matter, and how does it relate to auto-interpretation and practical use?
For a feature to serve as an indicator of a pattern, it should give higher scores to tokens with that pattern than to tokens without it. If both receive equally high scores, the pattern does not explain why the feature responds. We grouped tokens by MLP output direction because the SAE reconstructs the same MLP output that the TC predicts.
Both auto-interpretation and our evaluation ask whether the relationship between a pattern and a feature’s response holds on new examples. In auto-interpretation, a description inferred from activating examples is used to predict where the feature will activate in new text (Paulo et al., 2024, 2025). We selected one feature from each coder for the same output cluster and tested whether it gave higher scores to tokens in that cluster than to other tokens in held-out documents. Comparing the same features before and after JumpReLU showed something their final scores alone could not. Where TC features distinguished the output patterns better after JumpReLU, it was because they lost less discrimination during sparsification, not because they distinguished the patterns better to begin with.
We agree that the practical gain on Few-NERD was modest. Among 200 retrieved sentences, TC found an average of 2.75 more target sentences than SAE under
smalland 4.35 more underbig(Table 4). TC features also had higher mean AUROC after JumpReLU on human-annotated SemCor noun categories in 1B. Unlike on the output clusters, however, TC was already ahead before JumpReLU in both settings (Table 3).Q1. Can you give confidence intervals for the individual conditions?
Yes. Table 26 in Appendix B.12 now reports separate estimates and 95% bootstrap confidence intervals for SAE and TC. It covers the original, random, and targeted rotations in Table 2(b), the three leading-16-PC removal conditions in Table 18(a), and Few-NERD detection and retrieval in Table 4. We kept the selected features fixed and resampled held-out documents for AUROC and test sentences for Few-NERD.
Q2. Can the abstract explain the motivation and task before presenting the results?
We revised the abstract to start with why the comparison matters. Researchers use individual SAE and TC features to interpret a model, so it matters which distinctions in the input survive in a feature’s response. The abstract now explains how we select and evaluate features against the same MLP output patterns before presenting the two findings about discrimination lost through JumpReLU and the input directions that support discrimination before it.