W1. Limited novelty
Several elements that are presented as contributions are already known. (i) The new panel yields only a handful of probing tasks, whereas a much larger suite of probing datasets already exists [1], is used in SAEBench and forms the most reliable SAE metric in a recent audit [2], so the panel is not a contribution as a probing resource. (ii) Proposition 1 is the known rescaling ambiguity of dictionary learning, and for unit-norm decoders, as in standard SAEs [3] and the official classes used here (Table 9), contribution gating equals raw gating, while unconstrained decoders already absorb the decoder norm in the sparsity penalty [4]. (iii) Fixed-support refitting of the coefficients under a positivity constraint was already applied to SAEs by Gao et al. [3, Section 5.1] and goes back to sparse coding [5], as the paper itself acknowledges in Appendix H.3. (iv) That gains in reconstruction do not reliably translate to downstream metrics is established by SAEBench [6] and SynthSAEBench [7]. (v) That SAE benchmarks are noisy and weakly discriminative has been shown in [2], and that SAEs trained with different seeds learn different features in [8].
W2. Incomplete picture of SAE evaluation
SAEs are not evaluated only by reconstruction and probing, as stated in the abstract and lines 43 to 47. SAEBench [6] alone includes LLM-judged interpretability, absorption, causal disentanglement and unlearning, and steering [9], causal interventions [10] and automated interpretability [11] are standard evaluations. This narrows the motivation and leaves untested the evaluations in which item sampling is most likely to matter.
W3. Small scale and limited variety of experiments
All experiments use a single small model (Pythia-160M), a single layer, a single task type and two architectures, which is too narrow to support claims about SAE benchmarking, since current comparisons use Gemma-2-2B [6] or larger models [3], and seed effects, which drive W4 and W5, depend on model size [8]. The SAEs are also trained on a budget far below that of SAEBench [6]. This is avoidable, because SAEBench releases trained BatchTopK and Matryoshka SAEs at several widths for both Pythia-160M and Gemma-2-2B.
W4. Conclusions not supported by the paper's own statistics
The paper's frozen rule leaves the architecture comparison unconfirmed (Table 21), and the confidence intervals of both the larger-width gap and the width contrast include zero (Table 16). The main text nevertheless states that the nested architecture "consistently outperforms BatchTopK across both width regimes" (Section 4.2), although at the smaller width the gap is indistinguishable from zero and positive in only some of the seeds (Table 12). With the number of seeds used, only gaps well above the threshold of the paper's own rule can be detected reliably. Probe interfaces are also reported selectively, since the native gain for Matryoshka in RQ3 is reported under a different interface from the one used for BatchTopK, and under the same interface it is smaller (Table 18).
W5. Panel sensitivity confounded with selection
The shrinkage of the gap on the new panel is measured against the exploratory panel on which the comparison and the gate were selected (Appendix G, H.2). A selected estimate is expected to shrink on fresh data regardless of panel content [12], and the same pattern on new ImageNet test sets [13] needed a dedicated analysis to be explained. With only two panels, panel variance cannot be estimated, and budget matching does not address selection. Moreover, Table 14 shows large family-level gaps of opposite sign, negative on plants and positive on technology, which cancel in the macro average, so the verdict is decided by which families are probed, and these are never resampled [14].
W6. Hard-to-read abstract
The abstract is difficult to follow because its terms are not standard and are defined late or not at all. "Downstream semantic readouts" means a sparse probe on a few coordinates, a "lexical panel" is a set of probing words, "multi-panel auditing" is evaluation on several word sets, "linear rank orderings" are probe score rankings, and "detector adaptation protocols" refer to whether the probe is refitted. A reader either inside or outside SAE research cannot tell from the abstract what was measured and what was found.
References
[1] S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, N. Nanda. Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. ICML 2025. arXiv:2502.16681.
[2] D. Chanin. Are Sparse Autoencoder Benchmarks Reliable? ICML 2026 Workshop on Mechanistic Interpretability. arXiv:2605.18229.
[3] L. Gao et al. Scaling and Evaluating Sparse Autoencoders. ICLR 2025. arXiv:2406.04093.
[4] T. Conerly, A. Templeton, T. Bricken, J. Marcus, T. Henighan. Update on How We Train SAEs. Transformer Circuits Thread, April 2024.
[5] M. Aharon, M. Elad, A. Bruckstein. K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation. IEEE Transactions on Signal Processing 54(11):4311-4322, 2006.
[6] A. Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025. arXiv:2503.09532.
[7] D. Chanin, A. Garriga-Alonso. SynthSAEBench: Evaluating Sparse Autoencoders on Scalable Realistic Synthetic Data. ICML 2026 Workshop on Mechanistic Interpretability. arXiv:2602.14687.
[8] G. Paulo, N. Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026. arXiv:2501.16615.
[9] Z. Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. arXiv:2501.17148.
[10] S. Marks et al. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025. arXiv:2403.19647.
[11] G. Paulo, A. Mallen, C. Juang, N. Belrose. Automatically Interpreting Millions of Features in Large Language Models. ICML 2025. arXiv:2410.13928.
[12] G. C. Cawley, N. L. C. Talbot. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. JMLR 11:2079-2107, 2010.
[13] B. Recht, R. Roelofs, L. Schmidt, V. Shankar. Do ImageNet Classifiers Generalize to ImageNet? ICML 2019. arXiv:1902.10811.
[14] H. H. Clark. The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior 12(4):335-359, 1973.