acceptodds
Withdrawn from ICLR 2027

Do SAE Gains Transfer? Lexical Panels and Sparse Readouts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse autoencoders (SAEs) are central to mechanistic interpretability, commonly evaluated through two primary lenses: activation reconstruction fidelity and downstream sparse readouts. However, conventional evaluations frequently rely on two unverified premises: that architectural advantages identified on a specific lexical panel generalize across independent evaluation sets, and that optimizing reconstruction fidelity directly enhances downstream semantic readouts. In this work, we present a systematic evaluation framework that rigorously investigates these assumptions across lexical panels, optimization objectives, and detector adaptation interfaces on Pythia-160M. First, by evaluating paired checkpoints across an audited panel of 120 previously unseen lexical members across 1,200 distinct articles, we show that measured architectural margins are highly sensitive to panel composition and evaluation sample budgets, demonstrating the critical need for multi-panel auditing in representation benchmarks. Second, through controlled fixed-support coefficient optimization on official-class SAEs, we achieve substantial reconstruction error reductions of 8.23% for BatchTopK and 5.48% for Matryoshka, while discovering that reconstruction gains fundamentally decouple from downstream sparse readouts. We establish both a constructive mathematical proof and continuous intervention trajectories showing that reconstruction improvements can invert linear rank orderings. Third, we reveal that detector adaptation protocols (frozen, refitted, or reselected) systematically govern observed readout gains, exhibiting divergent behaviors across architecture families. Finally, we prove that contribution-based top- gating ensures exact scale invariance under dictionary reparameterization. Our findings provide actionable methodological principles and a reproducible benchmark harness for faithful representation evaluation.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, sparse readouts, benchmark reliability, cross-panel replication, coefficient refitting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nXUSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evaluation protocol intended to make SAE evaluation more robust.

Critical strengths
  • The paper addresses an important practical problem and could contribute to fairer comparisons between SAEs.

  • The paper provides sufficient detail to reproduce the results.

Critical weaknesses
  • The design of the proposed framework seems somewhat arbitrary. The paper compares standard evaluation with evaluation by independent panels and finds that architectural differences largely disappear under the latter. However, it is unclear how this result should be interpreted. Does the framework reveal something meaningful about the nature of SAEs or their evaluation, or is this outcome a consequence of how the framework is constructed? My main concern is that the argument seems to follow this pattern: under framework A, we observe differences of magnitude N%, while under framework B, the differences are smaller, at M%, where M < N. The paper then concludes that framework B provides a better measure of the true quality of SAEs. However, I do not see a sufficient justification for this conclusion, either in the design of the proposed framework or in the paper's arguments. One could potentially construct a different framework that amplifies the differences relative to the baseline and, by the same logic, claim that it better captures the true quality of SAEs. The paper needs a stronger argument for why the proposed protocol produces more reliable evaluations, rather than simply different results.

  • The finding that reconstruction fidelity does not necessarily translate into better probing performance (RQ2) does not seem particularly novel. This relationship is not surprising, and the paper does not make clear what new insight this result provides beyond what is already known.

Questions for authors

See weaknesses

Reviewer_cN8FOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the reliability of sparse autoencoder (SAE) evaluations, focusing on the transferability of architecture comparisons across lexical panels, the relationship between reconstruction fidelity and sparse readout performance, and the effects of probe adaptation protocols.

Critical strengths

The paper includes controlled experiments across multiple seeds and provides detailed statistical analyses.

Critical weaknesses
  1. Limited novelty and scientific contributions. The main findings, including the sensitivity of SAE evaluations and the mismatch between reconstruction quality and downstream utility, have been extensively studied in prior work, e.g., SAEBench. The paper largely reproduces these observations in a narrower setting without providing substantial new insights or methodological advances.
  2. Limited experimental scope and generalizability. The experiments are restricted to Pythia-160M at a single layer.
Questions for authors

What novel and generalizable insights does this work provide beyond existing studies on SAE benchmark reliability and reconstruction–interpretability trade-offs?

Reviewer_7oCUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether two common ways of evaluating sparse autoencoders (SAEs) are reliable. First, nested and BatchTopK SAEs trained on Pythia-160M are probed for word categories on seen and on unseen words, and the advantage of the nested SAEs shrinks on the unseen words. Second, the SAE codes are re-optimised for better reconstruction while keeping the same active latents, and probing accuracy does not change, which is presented as evidence that reconstruction and probing are decoupled. A rescaling-invariant variant of top-K selection and a small counterexample are also provided.d after the codes change affects the measured gain.

Critical strengths

S1. Timely question

Whether an SAE architecture comparison transfers to a newly sampled evaluation set is a timely question, and so is building a proper evaluation for SAEs, given that SAE benchmarks have been shown to be noisier than assumed [1].

[1] D. Chanin. Are Sparse Autoencoder Benchmarks Reliable? ICML 2026 Workshop on Mechanistic Interpretability. arXiv:2605.18229.

Critical weaknesses

W1. Limited novelty

Several elements that are presented as contributions are already known. (i) The new panel yields only a handful of probing tasks, whereas a much larger suite of probing datasets already exists [1], is used in SAEBench and forms the most reliable SAE metric in a recent audit [2], so the panel is not a contribution as a probing resource. (ii) Proposition 1 is the known rescaling ambiguity of dictionary learning, and for unit-norm decoders, as in standard SAEs [3] and the official classes used here (Table 9), contribution gating equals raw gating, while unconstrained decoders already absorb the decoder norm in the sparsity penalty [4]. (iii) Fixed-support refitting of the coefficients under a positivity constraint was already applied to SAEs by Gao et al. [3, Section 5.1] and goes back to sparse coding [5], as the paper itself acknowledges in Appendix H.3. (iv) That gains in reconstruction do not reliably translate to downstream metrics is established by SAEBench [6] and SynthSAEBench [7]. (v) That SAE benchmarks are noisy and weakly discriminative has been shown in [2], and that SAEs trained with different seeds learn different features in [8].

W2. Incomplete picture of SAE evaluation

SAEs are not evaluated only by reconstruction and probing, as stated in the abstract and lines 43 to 47. SAEBench [6] alone includes LLM-judged interpretability, absorption, causal disentanglement and unlearning, and steering [9], causal interventions [10] and automated interpretability [11] are standard evaluations. This narrows the motivation and leaves untested the evaluations in which item sampling is most likely to matter.

W3. Small scale and limited variety of experiments

All experiments use a single small model (Pythia-160M), a single layer, a single task type and two architectures, which is too narrow to support claims about SAE benchmarking, since current comparisons use Gemma-2-2B [6] or larger models [3], and seed effects, which drive W4 and W5, depend on model size [8]. The SAEs are also trained on a budget far below that of SAEBench [6]. This is avoidable, because SAEBench releases trained BatchTopK and Matryoshka SAEs at several widths for both Pythia-160M and Gemma-2-2B.

W4. Conclusions not supported by the paper's own statistics

The paper's frozen rule leaves the architecture comparison unconfirmed (Table 21), and the confidence intervals of both the larger-width gap and the width contrast include zero (Table 16). The main text nevertheless states that the nested architecture "consistently outperforms BatchTopK across both width regimes" (Section 4.2), although at the smaller width the gap is indistinguishable from zero and positive in only some of the seeds (Table 12). With the number of seeds used, only gaps well above the threshold of the paper's own rule can be detected reliably. Probe interfaces are also reported selectively, since the native gain for Matryoshka in RQ3 is reported under a different interface from the one used for BatchTopK, and under the same interface it is smaller (Table 18).

W5. Panel sensitivity confounded with selection

The shrinkage of the gap on the new panel is measured against the exploratory panel on which the comparison and the gate were selected (Appendix G, H.2). A selected estimate is expected to shrink on fresh data regardless of panel content [12], and the same pattern on new ImageNet test sets [13] needed a dedicated analysis to be explained. With only two panels, panel variance cannot be estimated, and budget matching does not address selection. Moreover, Table 14 shows large family-level gaps of opposite sign, negative on plants and positive on technology, which cancel in the macro average, so the verdict is decided by which families are probed, and these are never resampled [14].

W6. Hard-to-read abstract

The abstract is difficult to follow because its terms are not standard and are defined late or not at all. "Downstream semantic readouts" means a sparse probe on a few coordinates, a "lexical panel" is a set of probing words, "multi-panel auditing" is evaluation on several word sets, "linear rank orderings" are probe score rankings, and "detector adaptation protocols" refer to whether the probe is refitted. A reader either inside or outside SAE research cannot tell from the abstract what was measured and what was found.

References

[1] S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, N. Nanda. Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. ICML 2025. arXiv:2502.16681.

[2] D. Chanin. Are Sparse Autoencoder Benchmarks Reliable? ICML 2026 Workshop on Mechanistic Interpretability. arXiv:2605.18229.

[3] L. Gao et al. Scaling and Evaluating Sparse Autoencoders. ICLR 2025. arXiv:2406.04093.

[4] T. Conerly, A. Templeton, T. Bricken, J. Marcus, T. Henighan. Update on How We Train SAEs. Transformer Circuits Thread, April 2024.

[5] M. Aharon, M. Elad, A. Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing 54(11):4311-4322, 2006.

[6] A. Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025. arXiv:2503.09532.

[7] D. Chanin, A. Garriga-Alonso. SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data. ICML 2026 Workshop on Mechanistic Interpretability. arXiv:2602.14687.

[8] G. Paulo, N. Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026. arXiv:2501.16615.

[9] Z. Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. arXiv:2501.17148.

[10] S. Marks et al. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025. arXiv:2403.19647.

[11] G. Paulo, A. Mallen, C. Juang, N. Belrose. Automatically Interpreting Millions of Features in Large Language Models. ICML 2025. arXiv:2410.13928.

[12] G. C. Cawley, N. L. C. Talbot. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR 11:2079-2107, 2010.

[13] B. Recht, R. Roelofs, L. Schmidt, V. Shankar. Do ImageNet Classifiers Generalize to ImageNet? ICML 2019. arXiv:1902.10811.

[14] H. H. Clark. The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior 12(4):335-359, 1973.

Questions for authors

Q1 (W1). Positioning against existing probing datasets. It would strengthen the paper to run the protocol on the probing datasets of [2], or to state what the lexical panel measures that they do not.

Q2 (W2). Broader evaluation. It would strengthen the paper to test whether panel sensitivity also appears in at least one non-probing evaluation, such as steering [3].

Q3 (W3). Larger scale and more variety. It would strengthen the paper to repeat the main comparison on Gemma-2-2B with the trained SAEBench SAEs [1], which requires no training, and across more layers and architectures.

Q4 (W4). Claims matched to the evidence. It would strengthen the paper to restate the main-text conclusions at the level of the frozen rule and the intervals, and to report the same probe interface for both architectures.

Q5 (W5). Selection separated from panel composition. It would strengthen the paper to report the old-panel gap under the same gate, to estimate the shrinkage expected from selection alone, for example by a split-half replication within the exploratory panel, and to draw many disjoint panels and families from the existing candidate pool so that panel and family variance can be estimated, for instance with crossed random effects [4].

References

[1] A. Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025. arXiv:2503.09532.

[2] S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, N. Nanda. Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. ICML 2025. arXiv:2502.16681.

[3] Z. Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. arXiv:2501.17148.

[4] R. H. Baayen, D. J. Davidson, D. M. Bates. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59(4):390-412, 2008.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.