- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
As I understand it, the paper trains Distilled Matryoshka SAEs (DMSAEs) on ESMC-6B activations used by ESMFold2. During training, stable latents get frozen into a “core,” and the proposed version picks which latent to freeze using pLDDT/pTM attribution. The paper then checks whether latents line up with amino acid directions from linear probes, and where in the Matryoshka hierarchy they end up.
After reading the paper, I can’t identify its main contribution. The proposed attribution-guided promotion doesn’t outperform random promotion. The only feature studied is amino acid identity, so I don’t see a feature hierarchy. The comparison with baselines may favor DMSAE through extra capacity. The figures and writing don’t give me enough to resolve these questions myself.
Critical strengths
None, I think.
Critical weaknesses
I had a hard time working out what this paper is trying to show, and the results did not help me figure it out.
-
I don’t see what the pLDDT/pTM attribution contributes. The paper trains three DMSAE variants that, as far as I can tell (Sec. 3.5, App. E), differ only in how a latent is chosen for promotion: by pLDDT/pTM attribution (the proposed method), by reconstruction, or at random. If attribution is the key idea, I’d expect the guided variant to clearly win, but that doesn't quite work out. So I’m left unsure what role ESMFold2’s confidence plays in any of the results.
-
I don’t understand what “feature hierarchy” means here. The title says the paper reveals feature hierarchies in ESMFold2, but the only feature studied is amino acid identity. As far as I can tell, that is the input token at each position. I’d expect it to be easy to decode, though probe accuracy is never reported, so I can’t check. The intro argues proteins are a good testbed because they have structure at many levels, from residues up to domains and folds. But only the lowest level is tested, which the limitations section acknowledges. I also couldn’t find an argument for why amino acid features should sit in the innermost level, which is what the paper treats as the sign of a better hierarchy.
-
I’m not sure the comparison with the baselines is fair. According to Eq. 9 (App. D.1), core latents skip the sparsity limit, and the core can grow to 512 latents (App. D.4). The baselines are held to the k-sparsity budget. If I’m reading this right, DMSAE has extra latents that are always available, which on its own could explain the better recovery in Fig. 5. The authors note in the limitations that sparsity isn’t matched across methods. No reconstruction error or actual L0 is reported for any model, so I can’t tell how big this effect is.
-
I can’t tell what is new. The paper says it builds on DMSAE (Martin-Linares & Ling, 2026) and “adapts this framework into an iterative training procedure.” But the cited paper is itself titled “Iterative distillation for feature-consistent sparse autoencoders.” The one clearly new piece seems to be the pLDDT/pTM promotion rule, which brings me back to point 1.
-
The writing and figures made the paper very hard to follow.
○ The paper is motivated by the isoleucine/leucine latents landing in different prefixes (Fig. 1). The authors ask whether that is meaningful or an artifact, but I couldn’t find where this question is answered. ○ Most plots have small dots scattered over the bars and markers (Figs. 2, 4, 6, 7), and none of the captions say what they are: seeds, probes, or latents? Fig. 4 does not say what its error bars are. Figs. 6 and 7 don’t say what the large marker and whiskers represent. ○ Fig. 7 places “(More Important)” at the low end of the attribution score axis, while its caption says DMSAEs have higher scores. I couldn’t reconcile the two. ○ The colored strip labeled “dictionary size = 16,384” in Fig. 3 is not explained. ○ The section headings read as a list of separate observations. Sec. 4.1 is about validating the probes, and Sec. 4.3 is about a baseline. I couldn’t see how they build to a claim. ○ Basic setup is missing: which ESMC-6B layer is used, and which proteins the training activations and attribution scores come from. ○ There is a typo in Sec. 3.5: “Eq. equation 5.”
Questions for authors
What does the pLDDT/pTM attribution add, given that random and reconstruction promotion do as well or better? How do the results change when sparsity is matched across methods? What is the main takeaway about ESMFold2 itself?