acceptodds
Under review as a conference paper at ICLR 2027

Split, Not Sparse: Ground-Truth Feature Superposition Across Scientific Domains and Model Scale

Abstract

Molecular and protein transformers predict properties such as hERG potassium-channel blockade and DNA binding from chemical structures and amino-acid sequences. Explaining these predictions requires identifying which scientific properties the models represent and how the associated directions affect their outputs. Sparse autoencoders (SAEs) express internal activations through learned features, with only a small subset active per input. Evaluating these features requires measuring both their coverage of a property across examples and prediction sensitivity to removing their associated directions. We introduce a paired recovery-and-removal protocol that ranks features against an external annotation, decodes it from feature groups, and removes the corresponding decoder directions with a random-removal control. We report three findings. (1) Group Recovery: groups improve property decoding in a ChemBERTa hERG predictor and three ESM-2 DNA-binding predictors. For the molecular conjunction of a basic nitrogen and an aromatic system, top-64 features reach 0.842 AUROC, versus 0.740 for one feature and 0.853 for the full layer. This recovery curve supports feature-budget selection. (2) Recovery–Intervention Dissociation: ESM-2 top-64 decoding stays within 0.026 AUROC of the full-layer reference, while targeted-minus-random removal effects range from 0.041 at 35M to 0.002 at 650M under the evaluated budgets. The paired measurements distinguish property recovery from removal sensitivity. (3) Target-Dependent Coverage: an aromatic-plus-basic protein target leaves reference gaps up to 0.116 AUROC at top-64, identifying settings where the selected group provides incomplete coverage. Together, the protocol and findings support choosing feature groups for a specified scientific property and evaluating their associated prediction effects.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
sparse autoencoders, mechanistic interpretability, scientific transformers, feature recovery, representation probing, activation interventions, molecular property prediction, protein language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nu4TOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a way to test whether an SAE dictionary captures a scientific property defined outside the model. For a binary annotation, the authors rank SAE latents by how strongly they correlate with it, then use the top-m latents in two ways. They fit a linear probe on them and compare it with a single-latent probe, a random-m probe, and a full-layer probe. They also project out the span of the matching decoder columns at a residual hook and measure the task-score drop against removing a random span. The protocol is applied to a LoRA-tuned ChemBERTa hERG predictor, three ESM-2 DNA-binding predictors, and pretrained ESM-2 with an target, plus a synthetic superposition grid. The three claims are: groups of latents beat single latents, recovery and intervention sensitivity diverge as ESM-2 gets bigger, and the gap to the full-layer probe depends on the target.

Critical strengths

1.Defining the annotation externally (RDKit rules, UniProt keywords, residue composition) instead of reading it off feature activations is a good idea.

2.Table 1 is the best result in the paper. Across nine chemistry configurations, reconstruction R² barely moves (0.93–0.98), but best-feature alignment ranges from 0.27 to 0.41. This shows clearly that good reconstruction does not imply good alignment with the target.

Critical weaknesses

1.Novelty is unclear and seems to be limited. As stated in the abstract, the paper's primary contribution appears to lie in its evaluation protocol and empirical findings, rather than in a novel methodology. While systematic empirical studies can be valuable, the novelty and significance of the proposed protocol are not sufficiently established. In particular, ranking SAE latents by label correlation, probing the top-k latents, and comparing their performance against a full-residual probe closely resemble existing sparse-probing approaches (Gurnee et al., 2023; Kantamneni et al., 2025). Similarly, comparing the effects of ablating SAE decoder directions against random directions is not, by itself, a novel evaluation strategy.

Although Kantamneni et al. (2025) is cited multiple times, the paper does not provide a direct empirical comparison under a shared experimental setting. More importantly, it remains unclear what additional insights are gained by combining these existing evaluation procedures, beyond what each procedure can already reveal independently.

“We ran both and reported them together” is a thin contribution for ICLR.

2.Section 4.2 and parts of the appendix look unfinished or copied from another project. Section 4.2 describes an experiment that wasn’t run (“A useful next step is...”). The circuit summary (1024/6144 neurons, 98% faithfulness, 10⁴× attribution, R² = 0.047) and Appendix F’s “population-level intervention... on-manifold null” come with no method description.

3.In Table 2, the ChemBERTa (hERG, basic-N + aromatic) and ESM-2 35M (DNA-binding, K+R) rows report exactly the same performance values: 0.842 for the top-64 probe and 0.853 for the full-layer probe. Are either rows were copied by mistakes?

Questions for authors
  1. I took a look at the AI Use Statement, and pretty much everything was done by AI. The authors are responsible for the final manuscript and code. Does this mean the ideas in the paper were also generated by AI? I feel like the human contribution here is way too small.

2.Why do ChemBERTa and ESM-2 35M have exactly the same top-64 (0.842) and full-layer (0.853) scores? Why does 0.853 appear again for 35M on another target at two layers?

Reviewer_gxPSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper seems to introduce a method to investigate sparse autoencoder-based explanations for molecular and protein transformers. The method is validated on the TDC hERG and AMES datasets using ChemBERTa and ESM-2 models.

Critical strengths

None, due to the presentation issues.

Critical weaknesses

The paper is very difficult to follow, with new and ambiguous terms introduced without sufficient definition or context, for example:

(L049) RDKit indicator. (L051) basic-residue composition. (L052) decoder directions. (L081) recovery profile. (Figure 1) model-specific hook, schematic depth. (Equation 1) top_m.

Many of the sentences are grammatically correct, yet hard to parse. Combined with the undefined terms, I could not get the precise meaning of many of the sentences, for example:

(L049) We study properties defined independently of the SAE. -> what are the "properties" here and what does defined independently of the SAE mean?

(L054) We pair linear probes on selected SAE features with removal of their decoder directions, comparing each readout with its own controls. -> what is a decoder direction? what kind of readouts are compared with what kind of controls?

(L127) An externally specified binary annotation is f∗(x)f^*(x) -> what does this binary annotation represent? how is it obtained?

(L131) The alignment score and selected indices are sj=∣corr(zj(aℓ(x)),f∗(x))∣,Tm=topm{sj}s_j = \lvert \text{corr}(z_j(a_\ell(x)), f^*(x)) \rvert, T_m = top_m\{s_j\} -> what is x? How is aℓ(x)a_\ell(x) pooled, across which dimension? What does ℓ\ell represent?

(L153) The intervention removes the selected span at a residual-stream hook and resumes the downstream computation.

(L227) The reference and tolerance therefore determine the required decoding score before comparing feature budgets.

(L236) The hERG case supplies the detailed feature.-budget curve; the DNA-binding models repeat the gain at three sizes.

Some sentences are odd, using the names of general fields of study as task labels without proper definitions, for example:

(L087) Chemistry uses a ChemBERTa-based hERG predictor...

(L105) Biology uses ESM-2 at 35M, 150M, and 650M parameters...

The organization of the paper is also non-standard, many experimental setups and results are mixed along the method section (Section 2). These presentation issues appear throughout the paper.

Finally, while I cannot properly assess the technical contributions and results due to the presentation issues, I feel that the results are not sufficiently convincing, even if the presentation issues are fixed. There are no baselines to position the method in the literature, the results seem to originate from single-run experiments with no uncertainty quantification, and the text seems to just report the numbers without deeper analyses.

Questions for authors

The paper requires a major revision before I can re-assess it properly. Without major changes in the presentation, I cannot think of questions that can materially affect my assessment.

Reviewer_ZdrFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how sparse autoencoders (SAEs) represent scientific properties in molecular and protein Transformers, and how the corresponding feature directions affect model predictions. The authors introduce a Recovery-and-Removal evaluation protocol. They first select groups of SAE features based on their correlations with external property labels and use linear probes to measure property recovery. They then remove the decoder subspace associated with the selected features and compare the resulting prediction changes with those from random-direction removal.

The experiments cover ChemBERTa and three sizes of ESM-2. The main findings are: (1) combining multiple SAE features substantially improves property recovery; (2) in ESM-2, Top-64 features achieve decoding performance close to the full-layer representation, while the additional performance drop from targeted removal decreases with model size; and (3) different scientific properties require different feature budgets to achieve similar coverage.

I find the research question meaningful and the evaluation protocol clear. However, the baselines and statistical analysis could be strengthened, especially for the cross-model intervention results.

Critical strengths

The paper addresses a meaningful question about property recovery from SAE features and their influence on model predictions. The proposed Recovery-and-Removal protocol is clear, includes reasonable controls, and is evaluated on both molecular and protein models. The results show that combining relevant features improves property recovery and reveal differences in feature coverage across scientific properties and SAE configurations. These findings provide useful insights for SAE evaluation and practical applications.

Critical weaknesses

1. Limited comparison with other low-dimensional representations.

The paper includes full-layer probes and random SAE feature groups as baselines, but lacks comparisons with other low-dimensional representations, such as PCA or sparse probes on raw activations. The results show that the selected SAE features retain substantial information about the target properties. However, it remains unclear how much of this performance comes from the SAE representation and how much comes from supervised feature selection. Adding one or two baselines with comparable feature budgets, label usage, and probe capacity would help assess the value of the selected SAE features.

2. Insufficient statistical support for the cross-model intervention results.

The authors acknowledge that the model-scale comparison is observational and requires further uncertainty analysis. However, the decrease in Intervention Margin is a central finding, and its statistical stability remains unclear. The ESM-2 models differ in hidden dimensions, selected layers, and intervention budgets. Although the 150M and 650M models use the same nominal intervention fraction of 5%, the removed subspaces may still account for different amounts of activation energy. The Intervention Margin for the 650M model is only 0.002 (targeted: 0.006; random: 0.004), and no confidence intervals or repeated experiments are reported.

The SAEs are also trained on mean-pooled activations, while decoder-span removal is applied at a residual-stream hook. More details on this implementation would help assess the intervention results. The reported differences across model sizes are interesting, but their relationship to model scale and representation redundancy remains uncertain.

3. The intervention evaluation metric is not clearly specified.

Section 2.3 defines the intervention effect using a prediction or task score (Q), but the exact metric used in Table 4 and Figure 3 is unclear. The removal effects are reported in "score units," without specifying whether they represent changes in task AUROC or another metric.

Since these results are central to Recovery–Intervention Dissociation, the authors should clarify the metric and how it is computed for each model. This information is important for interpreting the reported differences and reproducing the experiments.

Questions for authors
  1. How does SAE Top-64 compare with PCA or sparse probes on raw activations under comparable feature budgets and probe capacity? For supervised baselines, can the authors also control the use of training labels?
  2. Could the authors report confidence intervals for the Intervention Margin, especially for the 650M model? How sensitive are the results to evaluation samples, random-direction sampling, and SAE training randomness? Could the authors also report the activation energy removed by each intervention?
  3. What exact metric is used as (Q) in Table 4 and Figure 3? Is the same metric used across all models, and how are the reported removal effects calculated?
  4. The SAEs are trained on mean-pooled activations, while decoder-span removal is applied at a residual-stream hook. Could the authors clarify the exact intervention location and how the learned decoder directions are applied to token-level activations?
  5. The paper states that feature ranking, probe fitting, and held-out evaluation are separate stages. Could the authors clarify which data subsets are used for each stage, especially whether feature ranking uses any held-out labels?

Section 5 and Appendix F report the results of the synthetic experiment, but some details are missing, including the data generation process and the definition of the recovery score. A brief description in the appendix or reproducible code would be helpful.

The exact construction of the aromatic-plus-basic protein target could also be clarified, along with the unspecified dataset information for the 150M and 650M models in Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.