- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a way to test whether an SAE dictionary captures a scientific property defined outside the model. For a binary annotation, the authors rank SAE latents by how strongly they correlate with it, then use the top-m latents in two ways. They fit a linear probe on them and compare it with a single-latent probe, a random-m probe, and a full-layer probe. They also project out the span of the matching decoder columns at a residual hook and measure the task-score drop against removing a random span. The protocol is applied to a LoRA-tuned ChemBERTa hERG predictor, three ESM-2 DNA-binding predictors, and pretrained ESM-2 with an target, plus a synthetic superposition grid. The three claims are: groups of latents beat single latents, recovery and intervention sensitivity diverge as ESM-2 gets bigger, and the gap to the full-layer probe depends on the target.
Critical strengths
1.Defining the annotation externally (RDKit rules, UniProt keywords, residue composition) instead of reading it off feature activations is a good idea.
2.Table 1 is the best result in the paper. Across nine chemistry configurations, reconstruction R² barely moves (0.93–0.98), but best-feature alignment ranges from 0.27 to 0.41. This shows clearly that good reconstruction does not imply good alignment with the target.
Critical weaknesses
1.Novelty is unclear and seems to be limited. As stated in the abstract, the paper's primary contribution appears to lie in its evaluation protocol and empirical findings, rather than in a novel methodology. While systematic empirical studies can be valuable, the novelty and significance of the proposed protocol are not sufficiently established. In particular, ranking SAE latents by label correlation, probing the top-k latents, and comparing their performance against a full-residual probe closely resemble existing sparse-probing approaches (Gurnee et al., 2023; Kantamneni et al., 2025). Similarly, comparing the effects of ablating SAE decoder directions against random directions is not, by itself, a novel evaluation strategy.
Although Kantamneni et al. (2025) is cited multiple times, the paper does not provide a direct empirical comparison under a shared experimental setting. More importantly, it remains unclear what additional insights are gained by combining these existing evaluation procedures, beyond what each procedure can already reveal independently.
“We ran both and reported them together” is a thin contribution for ICLR.
2.Section 4.2 and parts of the appendix look unfinished or copied from another project. Section 4.2 describes an experiment that wasn’t run (“A useful next step is...”). The circuit summary (1024/6144 neurons, 98% faithfulness, 10⁴× attribution, R² = 0.047) and Appendix F’s “population-level intervention... on-manifold null” come with no method description.
3.In Table 2, the ChemBERTa (hERG, basic-N + aromatic) and ESM-2 35M (DNA-binding, K+R) rows report exactly the same performance values: 0.842 for the top-64 probe and 0.853 for the full-layer probe. Are either rows were copied by mistakes?
Questions for authors
- I took a look at the AI Use Statement, and pretty much everything was done by AI. The authors are responsible for the final manuscript and code. Does this mean the ideas in the paper were also generated by AI? I feel like the human contribution here is way too small.
2.Why do ChemBERTa and ESM-2 35M have exactly the same top-64 (0.842) and full-layer (0.853) scores? Why does 0.853 appear again for 35M on another target at two layers?