acceptodds
Under review as a conference paper at ICLR 2027

Mechanistic Interpretability Findings That Generalize: Chaperone-Dependent Folding in ESMFold2

Abstract

Interpretability findings in scientific models are often supported by post hoc plausibility rather than held-out tests. We introduce a workflow that turns interpretability outputs into statistically testable predictions: feature terms discovered in examples sharing a biological property should have stronger effects in unseen members than in controls. We apply this workflow to an SAE over ESMFold2 representations and to GroEL/GroES clients: proteins that require this chaperone system for folding in vivo. Perturbations in 25 discovery clients identify 67 stable SAE terms. Their ablations have larger effects in eight held-out clients than in 19 matched controls for 56 out of 67 terms (), and their selected-to-random effect ratio is % higher in clients than in controls. When multiple selected terms are ablated together, client-control separation increases from approximately 3.8- to 7.3-fold across a set of randomly subsampled terms. After these tests, interpretations of the selected terms connect to known client properties and motivate new hypotheses about client-cavity charge patterning, compensatory core packing, and delayed interface closure. Our contribution is a cohort-conditioned discovery-and-evaluation workflow that turns latent feature interactions into held-out tests of biological selectivity. The framework accommodates multiple discovery methods and uses the selected interventions to nominate biological hypotheses.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
mechanistic interpretability, sparse autoencoders, protein structure prediction, interaction discovery, protein language models, scientific discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R8sLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a workflow for identifying individual SAE features and feature combinations whose ablation preferentially affects a biological category of interest. Applied to GroEL/GroES clients, it identifies features in ESMC representations and evaluates their structural effects in held-out clients and matched controls.

Critical strengths
  • The protocol is well motivated and methodologically reasonable.
  • Its use of held-out evaluation, matched controls, and stability screening strengthens the evidence beyond post hoc biological interpretation.
  • Considering combinations of features alongside individual features is valuable.
  • GroEL/GroES dependence provides a biologically well-motivated case study.
Critical weaknesses

Enthusiasm is tempered by the limited scope of evaluation: a single biological category (mechanism) and perturbation (structural change) are examined. The results therefore provide limited evidence for the broader applicability of the workflow.

The analysis also begins with a substantial prefilter, retaining only 78 features from a 16,384-feature(?) SAE. Although restricting attention to commonly activated, interpretable features is reasonable, the contribution of this filtering step and the sensitivity of the results to its criteria remain unclear. Broader evaluation would clarify the method’s value and generality.

Questions for authors
  • How many distinct SAE features are activated across the discovery cohort before filtering, and how many remain after each filtering criterion?
  • How sensitive are the findings to the activation thresholds and the exclusion of features with generic descriptions?
  • Can the workflow demonstrate comparable selectivity for another biological category or using a different intervention or output measure?
2 comments

Dear Reviewer R8sL,

We appreciate the reviewer’s thoughtful feedback on SAE feature selection for discovery and about the empirical/domain scope, and are glad that they found our workflow and application of interaction discovery to SAEs and biology well motivated and having the potential to strengthen discovery and biological interpretation. Please find our response below.

SAE features

Filtering down SAE features keeps the model interpretable and the discoveries to be more impactful. The SAE begins with 16,384 features, where 10,245 activate on at least one residue of the 25 discovery clients. For the discovery to find meaningful terms/interactions that generalize to held-out proteins, ablating features must be impactful enough. Hence, we filter out SAE features that do not activate in at least 5% of a discovery proteins’ residues, and also are shared across at least 20 (80%) of discovery proteins. This leaves us with 86 SAE features. Finally, we remove 8 SAE features that do not have an interpretable description, for a total of 78 SAE features to do discovery over. Additionally, there are severe computational and algorithmic constraints on the discovery SAE feature set: ProxySPEX [1] recommends scaling the number of masks at a rate of O(nlogn), where n is the number of features for discovery, which means expanding the size of the set requires substantially more masks. Due to these computational constraints, we felt that these reasonable thresholds (5%, 80%) created a generalizable, and computationally reasonable, discovery set (and size).

For other discoveries that are more focused on individual/localized discovery (e.g. understanding a targeted portion of a protein of interest) which is less about holdout generalization, expanding the SAE feature set could be useful. We will include a discussion about this in the final draft.

Activation thresholds

There is no threshold for activation strength for SAE feature selection. We consider an SAE feature active on a residue if it is >0 and we only select features for discovery based on coverage with respect to how prominent it is activated across a protein (5% threshold), and if it is shared across proteins (appears in >= 20 discovery proteins). The sparsity property of SAEs is particularly appealing for ProxySPEX due to its masking nature (zero-ing out features) in interaction discovery, which provides a nice inductive bias that does not push the ESMC representations off manifold when we do masked ablations for discovery. Analyzing the terms/interactions after including generic SAE features, and manually providing an interpretation from 1) the resulting structural effects and 2) the descriptions of the labeled SAE features is potential for future work, and will be discussed in the final draft.

Another biological category, intervention, or outcome measure

Please see our response to Reviewer tg98 (A second setting: thermostability.) for an additional domain/application. For GroEL/ES, we also evaluate our selected terms on 10 Class III- proteins (Fujiwara et al., 2010 [2]), which bind GroEL but do not require it to fold, a stricter control than the SAE-matched set. Effects are larger in clients than in both control groups (Figure 7, Appendix A.2); please see our response to Reviewer tg98 for the details.

We focus on intervening on SAE features as this paper is motivated by interpretability and scientific (hypothesis) discovery, so the interpretability of the features is compelling. SAE features have broadly been understood and interpreted marginally (one at a time), so discovering interactions between SAE features in biological folding models is novel, and, as we propose, can provide nuanced insights into scientific behavior for proteins, in this case. Although our workflow is principled and generalizable to non-SAE/interaction discovery, we do not focus on it here. Regarding output measure, Figures 9-11 and Table 2 (Appendix A.4) report the client contrast on raw TM-score change and two RMSD metrics. It is positive on all four in both settings, despite the terms not being discovered with RMSD.

[1] Butler et al. “ProxySPEX: Inference-Efficient Interpretability via Sparse Feature Interactions in LLMs.” NeurIPS, 2025.

[2] Fujiwara et al. “A systematic survey of in vivo obligate chaperonin-dependent substrates.” EMBO J., 2010.

While I think the method is reasonable and possibly useful, even considering the thermostability evaluation, I still do not think this is appropriately scoped for an ICLR main conference paper, although reasonable people may disagree. I will be maintaining my score.

Reviewer_jEbYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a workflow for testing whether mechanistic interpretability findings generalize to held-out examples. The main idea is: features discovered from proteins sharing an external biological property should have stronger effects on unseen proteins with the same property than on matched controls.

The authors test this idea using GroEL/GroES clients and ESMFold2. They apply a sparse autoencoder to layer-60 ESMC-6B representations, sample different SAE ablation masks across different discovery clients. After bootstrap filtering, 67 terms are retained, and most of them have strong effects on clients.

Critical strengths
  1. Many interpretability evaluations are based on quantities generated by the model itself. Here, GroE dependence comes from in vivo experiments and is independent of the structure prediction model and SAE.

  2. A common concern with ablation experiments is that almost any ablation may hurt model output. Comparing the selected terms with order-matched random panels directly addresses this issue.

  3. The appendices clearly state several limitations, e.g., the term-level tests are dependent, the p-value is exploratory.

Critical weaknesses
  1. The paper states that GroE clients have known sequence and structural biases, including Ala/Gly enrichment, hydrophobic or amphipathic regions, α,β\alpha , \beta architectures, and acidic composition (in the background / related work). At the same time, several selected SAE features correspond to similar properties. This leaves a simpler explanation for the result: the SAE features may encode sequence or structural composition, and the clients and controls may differ in these properties. Ablating composition-related features could therefore affect clients more strongly even without any specific representation of GroE dependence. Can authors do more analysis: 1). Regress the per-term effect against the number of residues on which the ablated features are active, or against total activation mass. Then test whether client status still explains additional variance. 2). Add some simple baselines, such as a previously proposed GroE structural discriminator.

  2. The headroom normalization may strongly affect the main result. The effect is defined in Eq.(1). The denominator can be very small and varies strongly across proteins. The same absolute change in TM-score can produce very different normalized effects. The authors also mention this issue at several points in the experiments (especially in Figure 8), suggesting that they were aware of it. However, they seem to have chosen not to address it further. I would like to understand the reasoning behind this decision.

  3. The false-positive behavior of the workflow is not tested. The paper argues that the workflow distinguishes reproducible mechanistic findings from post hoc explanations. To support this claim, I think the pipeline needs a negative control. A direct test would be to define a cohort unrelated to folding, such as randomly selected cytosolic E. coli proteins, and run the same pipeline. If a random cohort also produces a large panel with strong held-out separation, the method is not specific enough. If it does not, this would provide strong evidence for the proposed workflow. (A cheaper test would be to swap the cohorts: use controls for discovery and clients as holdout. If similar separation appears in the reverse direction, the claimed biological asymmetry becomes difficult to interpret.)

  4. One experiment is not enough to support the broad generality claim. The framework is presented as method-agnostic and potentially applicable to other domains. However, the evidence currently comes from: one model, one SAE, one system. A useful robustness test would be to repeat the experiment with a second SAE or a second biological setting

Questions for authors

The same as weaknesses.

If the authors can address my concerns, I would consider raising my score.

3 comments

Dear Reviewer jEbY,

Thank you for the feedback on whether our terms capture chaperone dependence or reflect composition and normalization choices, and on validating the workflow beyond a single cohort.

Evaluation and Composition

Please see our response to Reviewer tg98 (Evaluation) for a composition comparison between clients and controls in GroEL/ES.

We also ran the suggested regression of per-term effect against the fraction of residues on which the ablated features are active and against total activation mass. Neither predicts the effect, and the client coefficient is essentially unchanged when they are added (0.93 to 0.91 on the Eq. 1 scale). As a simple baseline, a logistic regression fit on the six composition measures in the table in our response to reviewer tg98 cannot separate evaluation clients from controls under leave-one-out cross-validation (AUC 0.17 with permutation p≈0.9p \approx 0.9). Fold type does not distinguish the groups either: 7 of 8 clients and 13 of 19 controls are α/β\alpha/\beta, and 1 of 8 clients and 2 of 19 controls are TIM barrels. We therefore do not believe the larger effects in clients are explained by composition, fold class, or how broadly the ablated features are active.

We additionally evaluate the selected terms in a stricter control set: 10 Class III- proteins (Fujiwara et al., 2010) that bind GroEL but do not require it. Effects remain larger in clients. Details are in our response to Reviewer tg98.

Finally, for an additional domain, please see our response to Reviewer tg98 (A second setting: thermostability).

Headroom Normalization

ESMFold2 is a strong predictor, so its predictions for the discovery proteins have very high TM-scores (0.93 to 0.995, upper bounded by 1). We therefore wanted to measure an ablation's effect relative to the error the model still makes: "how much did the error grow, for this protein, under this ablation."

First, we re-ran discovery with an unnormalized value function. This yielded 57 stable terms, 40 overlapping the 67 in the paper, so only 17 required fresh evaluation ablations. Scoring all 57 on the evaluation proteins, 49 (86%) have larger effects in clients than controls when measured without normalization and 51 (89%) when measured with Eq. 1, against 56 of 67 (84%) for the panel in the paper. The selected-to-random ratio is 51% larger in clients than controls, compared with 24% for the normalized panel in Section 4.2. We use the headroom-normalized value function for the interpretive reason above, not because it produces a larger separation.

Second, Table 2 (Appendix A.4) re-evaluates the screened panel on raw TM-score change and on two RMSD metrics. The client contrast is positive on all four metrics in both settings, and raw TM-score change has no denominator at all, so the conclusion does not rest on the normalization.

These runs predate the screening step of Appendix A.1 (below), so both panels compared in the first check are unscreened and still comparable.

False-positive Behavior

Due to computational constraints, we focused on applying our workflow to a new biological mechanism (thermostability) as noted earlier. We would, however, point to three components of the workflow that address this.

Every result is stated against order-matched random terms from the same basis, ablated in the same proteins: an arbitrary or null panel gives a selected-to-random ratio of 1. Drawing them from the same cohort also normalizes how strongly that cohort responds to ablation at all.

Prompted by this discussion, we added a screen before evaluation or interpretation (Figure 6, Appendix A.1): using the discovery surrogate model only, we keep terms whose predicted solo-ablation effect in the discovery clients exceeds the basis average. This drops 19 of the 67 GroEL/ES terms and 19 of the 29 thermostability terms. The screen tightens the panel without manufacturing the result: against the same cohorts, the unscreened 67-term panel gives a client contrast of +13% and +24%, and the screened 48-term panel +12% and +26% (Appendix A.2).

When clients have natural partners, the control moves inside the value function. Thermostability discovery runs on the paired difference (median over ten ortholog pairs of thermophile minus ortholog), so a term whose effect is shared with its control contributes nothing to the discovery target and cannot be selected. The screen then additionally requires a larger predicted effect in clients than controls. Rather than testing afterwards whether a control cohort yields a panel, this stops control-driven terms from being selected at all. Selected/random is 1.89 in evaluation heat-stable proteins but 1.18 in their orthologs (Figure 8, Appendix A.3), so in the cohort whose effect was subtracted, selected terms are nearly indistinguishable from random ones. In other words, the orthologs behave the way a negative control should.

I would like to thank authors for the detailed rebuttal. I appreciate the additional analysis, which address most of my concerns. But the workflow is still evaluated with one biological setting, which provides limited value for the broader applicability. I maintain my score.

Dear Reviewer jEbY,

Thank you for the response and recognition of the additional analysis. We would like to kindly point out that since the review process, we study an additional biological study (on top of GroEL/ES). This additional biological study was detailed in our response to Reviewer tg98, but we paste it below for convenience:

A second setting: thermostability

We repeat the full workflow on a second property: heat-stable thermophile proteins [1]. Thermophiles grow at temperatures where many E. coli proteins denature, yet share many of the same enzymes, so we ask what makes the same fold more heat-stable. Each thermophile protein has a paired E. coli ortholog, giving a direct contrast for isolating that property. Applied unchanged, the workflow yields a panel that passes the same generalization test, separating heat-stable proteins from their controls on 80 evaluation proteins outside discovery.

For discovery we ablate 1000 masks over 10 Thermus thermophilus proteins and their E. coli orthologs (87-feature basis, as in GroEL/ES), using a value function of median of paired difference in structural effect (Eq. 1). We evaluate on 37 thermophiles (clients), their 37 orthologs (controls), and 6 proteins matched on fold class, length and baseline TM-score, none used in discovery.

This gives a screened panel of 10 terms. Selected terms beat order-matched random terms by more in clients than controls (Figure 8, Appendix A.3): +60% against orthologs (95% of random panels between +24% and +110%) and +201% against the non-ortholog controls (+51% to +401%). RMSD instead of TM-score tells a similar story (+33%).

The panel converges on arginine chemistry, the signature most consistently reported for thermophiles: it recovers arginine enrichment and ion pairing, packing against hydrophobic helices, and helix-stabilising i, i+4 salt bridges, each 40–60% stronger in heat-stable proteins than in their orthologs. The terms also localise that arginine, pinning turns and helix ends and tethering the N-terminal segment to the fold, which we offer as testable hypotheses (Figure 13, Appendix A.5).

[1] Jarzab et al. Nat. Methods, 2020.

Reviewer_tg98OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies SAE-based mechanistic interpretability for GroEL/GroES chaperone systems. Using ProxySPEX, it identifies significant and generalizable features related to structure folding in the chaperone conditions. These selected sensitive features are evaluated on curated held-out clients and control sets. The analytical results validate the ESMFold2 embeddings encode structural dependencies of GroEL/GroES client folding without chaperone supervision.

Critical strengths
  1. The paper presents an investigation of GroEL/GroES-dependent proteins. The proposed way to select chaperone-dependant features is a practical experimental design. The further application in biology is promising.
  2. The study examines individual selected terms, order-matched random panels, and joint ablations, and includes comparisons with several alternative discovery methods. The additional protein-specific and unnormalized results help characterize the heterogeneity and scale of the observed effects.
Critical weaknesses
  1. The main evaluation tests whether selected SAE ablations produce larger changes in prediction TM-score in chaperone-dependent clients than in controls. It remains unclear whether the observed separation reflects information specific to chaperone dependence. The matched controls and order-matched random terms provide useful comparisons, but do not fully distinguish these explanations. More targeted controls matched on relevant structural properties, ideally with experimentally characterized chaperone requirements, would strengthen the biological interpretation of the evaluation.
  2. The study uses a single ESMFold2 model, one pretrained SAE at a fixed layer, and one biological setting, with only eight held-out clients. This supports a focused test of transfer within the chosen setting. But I am not sure if it is the strength of the SAE model or the pipeline that leads to the success. I would suggest testing more SAE models as ablations. For the comparison between held-out clients and control set, I would recommend augmenting the control set with more types of random proteins. The current control set is relatively narrow.
  3. The main technical components—SAE, k-SII, and bootstrap screening—are established tools. The distinctive contribution appears to be their integration into a cohort-conditioned discovery-and-evaluation workflow. The contribution is restricted to analytical findings.
Questions for authors

The consistency is weak in the paper’s presentation. The choice of words like “held-out members”, “held-out clients”, and “control cohort” is too casual. Their meanings are ambiguous. Please also clarify whether broader expressions such as “holdout proteins” refer to clients alone or to both evaluation cohorts in each context.

1 comment

Dear reviewer tg98,

Thank you for the feedback on our controls and the scope of the study.

Evaluation

Clients and controls are constructed as in Appendix B.1: controls were matched to the discovery clients on sequence length and SAE-feature profiles. We also verified that the two evaluation cohorts are similar in sequence composition, which was not a matching criterion:

Metric (median, range)Clients (n=8)Controls (n=19)
Length (residues)384 (291–500)386 (225–599)
Ala + Gly (%)18.7 (15.9–25.2)17.4 (13.7–23.3)
Asp + Glu (%)12.0 (10.2–13.8)12.4 (8.9–14.9)
Lys + Arg (%)9.6 (8.2–10.8)9.6 (7.1–13.3)
Hydrophobic (%)41.9 (39.6–44.4)41.7 (37.7–47.8)
Mean hydropathy−0.113 (−0.221–0.011)−0.155 (−0.434–0.306)

Our clients are Class IV proteins from Fujiwara et al. (2010) [1], experimentally validated as requiring GroEL. The same study identifies Class III- proteins, which bind GroEL but do not require it to fold, a sharper comparison cohort than our matched controls. We therefore also evaluate our selected terms in 10 Class III- proteins (Figure 7, Appendix A.2). Absolute effects are larger in clients than in both control groups, and larger in Class III- than in the SAE-matched controls. The selected-to-random effect ratio in clients is 12% bigger than in Class III- controls and 26% bigger in the SAE-matched controls.

A second setting: thermostability

We repeat the full workflow on a second property: heat-stable thermophile proteins [2]. Thermophiles grow at temperatures where many E. coli proteins denature, yet share many of the same enzymes, so we ask what makes the same fold more heat-stable. Each thermophile protein has a paired E. coli ortholog, giving a direct contrast for isolating that property. Applied unchanged, the workflow yields a panel that passes the same generalization test, separating heat-stable proteins from their controls on 80 evaluation proteins outside discovery.

For discovery we ablate 1000 masks over 10 Thermus thermophilus proteins and their E. coli orthologs (87-feature basis, as in GroEL/ES), using a value function of median of paired difference in structural effect (Eq. 1). We evaluate on 37 thermophiles (clients), their 37 orthologs (controls), and 6 proteins matched on fold class, length and baseline TM-score, none used in discovery.

This gives a screened panel of 10 terms. Selected terms beat order-matched random terms by more in clients than controls (Figure 8, Appendix A.3): +60% against orthologs (95% of random panels between +24% and +110%) and +201% against the non-ortholog controls (+51% to +401%). RMSD instead of TM-score tells a similar story (+33%).

The panel converges on arginine chemistry, the signature most consistently reported for thermophiles: it recovers arginine enrichment and ion pairing, packing against hydrophobic helices, and helix-stabilising i, i+4 salt bridges, each 40–60% stronger in heat-stable proteins than in their orthologs. The terms also localise that arginine, pinning turns and helix ends and tethering the N-terminal segment to the fold, which we offer as testable hypotheses (Figure 13, Appendix A.5).

Scope

Of the 33 SAEs Biohub released for the ESMFold2 encoder, only one has labeled feature descriptions, which our hypothesis generation needs. Stability across SAEs is important future work. We now also evaluate on a second setting and wider controls in both.

Contributions

We agree the individual components are established tools and that our results are not wet-lab validated. Our contribution is a workflow that uses them to generate interpretable hypotheses from foundation models, in which each candidate must pass a generalization test before it is interpreted. An effect shown only in the proteins that produced it is no evidence it reflects anything the cohort shares. Biological causality can only be determined in the wet lab, which is expensive, so we use out-of-sample generalization, the same invariance criterion that underwrites causal discovery in statistics [3], to decide which candidates are worth that cost.

Second, SAE features have largely been interpreted one at a time. We apply interaction discovery to this interpretable basis: 39 of our 67 selected terms are pairs or triples, and client–control separation grows when they are ablated together (3.8- to 7.3-fold). The workflow can therefore address properties for which no direct reward exists, unlike binding affinity or catalytic activity, but are defined implicitly by a cohort sharing a latent property.

Consistency

Thank you for noting our inconsistent terminology. The final draft will use discovery clients (25), evaluation clients (8), evaluation controls (19), and evaluation proteins (8+19), and drop "holdout," "held-out" (unless a verb), and "unseen."

[1] Fujiwara et al. EMBO J., 2010.

[2] Jarzab et al. Nat. Methods, 2020.

[3] Peters et al. J. R. Stat. Soc. B, 2016.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.