acceptodds
Under review as a conference paper at ICLR 2027

Iterative distillation of Matryoshka sparse autoencoders improves hierarchical feature organization in ESMFold2

Abstract

Matryoshka sparse autoencoders (SAEs) can decompose model representations into interpretable features, but reconstruction quality alone does not indicate how strongly those features influence model behavior. Furthermore, learned features can also vary across runs, complicating their characterization and reproducibility. These challenges are particularly difficult to study using large language models (LLMs), since SAE dictionary sizes capture only a fraction of an LLM’s feature space. In contrast, protein folding models may provide a better test case because protein structure is organized across multiple scales, from individual amino acids and secondary structure to domains and global structure. Here, we apply Distilled Matryoshka SAEs (DMSAEs) to study activations from ESMC-6B in ESMFold2. We assign each latent an attribution score using gradient activation with respect to ESMFold2’s predicted local distance difference test (pLDDT) and predicted template modeling (pTM) scores. These metrics reflect the model's confidence in the local and global structure, respectively. We find that distillation improves the recovery and hierarchical placement of features associated with individual amino acids across independent training runs. We also find that DMSAE distillation produces more reproducible and clearly separated rankings of amino acid features than existing SAE baselines. These results show that Distilled Matryoshka SAEs can refine the coarse ordering imposed by nested Matryoshka prefixes and improve the consistency of learned feature hierarchies.

Primary area:
interpretability and explainable AI
Keywords:
Sparse autoencoders, Matryoshka SAEs, mechanistic interpretability, protein folding models, ESMFold2, feature attribution, representation learning, feature reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_H6CyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As I understand it, the paper trains Distilled Matryoshka SAEs (DMSAEs) on ESMC-6B activations used by ESMFold2. During training, stable latents get frozen into a “core,” and the proposed version picks which latent to freeze using pLDDT/pTM attribution. The paper then checks whether latents line up with amino acid directions from linear probes, and where in the Matryoshka hierarchy they end up.

After reading the paper, I can’t identify its main contribution. The proposed attribution-guided promotion doesn’t outperform random promotion. The only feature studied is amino acid identity, so I don’t see a feature hierarchy. The comparison with baselines may favor DMSAE through extra capacity. The figures and writing don’t give me enough to resolve these questions myself.

Critical strengths

None, I think.

Critical weaknesses

I had a hard time working out what this paper is trying to show, and the results did not help me figure it out.

  1. I don’t see what the pLDDT/pTM attribution contributes. The paper trains three DMSAE variants that, as far as I can tell (Sec. 3.5, App. E), differ only in how a latent is chosen for promotion: by pLDDT/pTM attribution (the proposed method), by reconstruction, or at random. If attribution is the key idea, I’d expect the guided variant to clearly win, but that doesn't quite work out. So I’m left unsure what role ESMFold2’s confidence plays in any of the results.

  2. I don’t understand what “feature hierarchy” means here. The title says the paper reveals feature hierarchies in ESMFold2, but the only feature studied is amino acid identity. As far as I can tell, that is the input token at each position. I’d expect it to be easy to decode, though probe accuracy is never reported, so I can’t check. The intro argues proteins are a good testbed because they have structure at many levels, from residues up to domains and folds. But only the lowest level is tested, which the limitations section acknowledges. I also couldn’t find an argument for why amino acid features should sit in the innermost level, which is what the paper treats as the sign of a better hierarchy.

  3. I’m not sure the comparison with the baselines is fair. According to Eq. 9 (App. D.1), core latents skip the sparsity limit, and the core can grow to 512 latents (App. D.4). The baselines are held to the k-sparsity budget. If I’m reading this right, DMSAE has extra latents that are always available, which on its own could explain the better recovery in Fig. 5. The authors note in the limitations that sparsity isn’t matched across methods. No reconstruction error or actual L0 is reported for any model, so I can’t tell how big this effect is.

  4. I can’t tell what is new. The paper says it builds on DMSAE (Martin-Linares & Ling, 2026) and “adapts this framework into an iterative training procedure.” But the cited paper is itself titled “Iterative distillation for feature-consistent sparse autoencoders.” The one clearly new piece seems to be the pLDDT/pTM promotion rule, which brings me back to point 1.

  5. The writing and figures made the paper very hard to follow.

    ○ The paper is motivated by the isoleucine/leucine latents landing in different prefixes (Fig. 1). The authors ask whether that is meaningful or an artifact, but I couldn’t find where this question is answered. ○ Most plots have small dots scattered over the bars and markers (Figs. 2, 4, 6, 7), and none of the captions say what they are: seeds, probes, or latents? Fig. 4 does not say what its error bars are. Figs. 6 and 7 don’t say what the large marker and whiskers represent. ○ Fig. 7 places “(More Important)” at the low end of the attribution score axis, while its caption says DMSAEs have higher scores. I couldn’t reconcile the two. ○ The colored strip labeled “dictionary size = 16,384” in Fig. 3 is not explained. ○ The section headings read as a list of separate observations. Sec. 4.1 is about validating the probes, and Sec. 4.3 is about a baseline. I couldn’t see how they build to a claim. ○ Basic setup is missing: which ESMC-6B layer is used, and which proteins the training activations and attribution scores come from. ○ There is a typo in Sec. 3.5: “Eq. equation 5.”

Questions for authors

What does the pLDDT/pTM attribution add, given that random and reconstruction promotion do as well or better? How do the results change when sparsity is matched across methods? What is the main takeaway about ESMFold2 itself?

Reviewer_NVt8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains distilled Matryoshka sparse autoencoders (DMSAE) on ESMC-6B embeddings. The paper shows DMSAE improved amino acid feature recovery and more consistently learn amino acid identity, compared to vanilla MSAE which place amino acid latents across multiple prefixes.

Critical strengths

None

Critical weaknesses

The "hierarchical placement" claim is misleading. The paper does not establish a meaningful biological feature hierarchy beyond amino acid identity. The experiments only show that amino acid features are more consistently concentrated. If there's only 1 concept that is being tested then this is not hierarchy.

The reason why fragmentation is harmful is not established convincingly. Authors mention "feature splitting makes it harder to identify a unique relevant latent" but is not connected to any qualitative and quantitative results: interpretability results, steering effectiveness, or downstream performance.

Questions for authors

See weaknesses.

Reviewer_c3g3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors ask the question whether Matryoshka sparse autoencoders (SAE) place interpretable features at consistent levels of their nested hierarchy across independent runs, and whether progressive distillation improves that consistency. Instead of general LLMs, the authors pick ESMC-6B residue activations as used inside ESMFold2 as testbed. The method, DMSAE, adapts Martin-Linares & Ling (2026) to an iterative schedule: the dictionary splits into a frozen core and a trainable non-core, and stable encoder directions are promoted into the core one at a time. The primary variant ranks candidates by a gradient-times-activation attribution to ESMFold2's pLDDT and pTM outputs, with reconstruction-guided and random promotion as ablations. Evaluation uses 100 independently trained probes per amino acid as reference directions, across six configurations at 10 seeds each. DMSAE variants recover amino-acid directions more often than the Matryoshka, Ordered and MP-SAE baselines.

Critical strengths
  • The paper trains ten seeds for each of six configurations at matched dictionary size and compute budget, which makes the seed-to-seed variability in Fig. 3 visible.
  • The probe construction is sound. The authors train 100 independent probes per amino acid whose directions agree closely with one another, and the probe labels never enter SAE training or promotion.
Critical weaknesses
  • The multi-scale premise that justifies choosing a protein model as the test bed is never exercised, because the evaluation covers only amino-acid identity. Importantly, no folding result is reported either: all four recorded quantities describe the geometry of the SAE dictionary relative to probe directions, so nothing in the paper tests whether the recovered features bear on the structures ESMFold2 actually predicts, which is one of most important downstream application of ESMC's representations.
  • Reconstruction-guided promotion is task-agnostic and needs no access to ESMFold2's confidence heads, yet it matches or beats the pLDDT/pTM machinery on every measure that is not itself an attribution score. This leaves the component that distinguishes the paper with little to do.
Questions for authors
  1. Can you report the performance change of ESMFold2 with DMSAE features and original ESMC features?
  2. Figure 5 shows pLDDT/pTM promotion recovering 86% of amino-acid observations against 92% for the other two policies. Is that gap stable across seeds, and what does pLDDT/pTM buy over reconstruction-guided promotion on any measure that is not itself an attribution score?
  3. Amino-acid identity is close to linearly decodable from ESMC-6B activations. How much of the recovery result survives if the reference directions come from a feature that is not recoverable by a linear probe on the raw activations?
  4. Does any of this transfer to a second protein model, for example a different ESMC scale or a non-ESM folding model? The reproducibility claim is currently established across seeds of one model rather than across models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.