- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors compare embeddings from 115 protein models on fold level classification using SCOPe40. They claim negative correlation (−0.57 ) between model size and classification results. They do their own supervised lookup based approach, ANGK, over preextracted Foldseek structural tokens, using a teacher model and distillation to finetune further. ANGK reaches fold-level AUROC 0.9555, 0.012 above the strongest baseline, CATH-supervised TEDBench-ESM2-650M-ft, and 0.0235 above the 0-shot SaProt-35M. Using positional information (similarity of quartiles of each sequence) improves ANGK further. The 39 MB index table enables fast encoding once structural tokens are available, and the authors demonstrate its use on 241 million AlphaFold structures and ECOD domains (at different levels).
Critical strengths
-
The paper shows that a small learned embedding representation with distillation and structural tokens can beat strong protein models on fold AUROC in a knn setting. Use of alternate , structural aa alphabets is relatively new at scale and has room for growth and contribution.
-
Selecting PLM layers on separate validation folds also makes the comparison more credible. (It’s an important gotchas often missed in other works and a useful reference here in how it changed the numbers when optimized to -0.37 vs -0.57).
-
The position experiment is nice for showing some feature/domain engineering of a cheap way to circumvent positional embeddings, with positional encoding, and showing that which quarter contains a structural pattern improves retrieval.
Critical weaknesses
- They don't compare to the obvious baseline which is foldseek itself, or even just basic n-grams over the foldseek tokens (1-3) (without the distilling teacher), or existing sequence alignment approaches that support alphabets like blast / mmseqs2/phhmer (or trivial word2vec on the structural alphabet). At the very least, foldseek on the tokens or ngram based is needed (on the test/eval sets in the paper) given that the whole paper is essentially knn over foldseek tokens.
- Comparing 115 PLMs does not answer what the method adds over existing simple baselines. Winning AUROC also does not mean better matches near the top; it might be worthwhile adding some sort of ranking metric. SaProt-35M beats ANGK on both AUPRC and top-1 accuracy (Table A1).
- The speed comparison does not count the same work/not apples to apples. ANGK’s isn't counting its own tokenization and embedding (i.e running on new sequences and predicting foldseek tokens from AA). Including that step would change its throughput. if we were to not count model processing, then the lookup of a models output final layer (with knn) is also comparatively as fast almost, give or take, vs 11x as claimed. The speedup needs a comparison on the same proteins with the same processing stages included.
- The negative model size correlation is confused by different model families - within most families, larger models are usually better at exactly this task (esm2, esm-c are the exceptions; Carp, protSent, ProteinCLIP etc show better results with larger models (and the same layer) within the same families, on structural tasks and even similar knn setups!). I’d correct this to be a bit less misleading (although current phrasing is maybe ok “not always the best model”; but as an abstract headline it’s a bit confusing and misleading, without more analysis.
Questions for authors
- Matched comparison: How do ANGK, Foldseek and simple 3Di counts (e.g. n-grams tfidf) compare on test set? Please include top-hit retrieval or ranking and complete processing time if claiming speed. Also, why not try with knn (without distillation, ngram mergings, did you compare?
- Teacher clarification: Which teacher trained the reported tables? Section 3.4 names TM-Vec 2. The supplied code teacher_embed.py loads tmvec-cath. (the train/test overlap is also unclear, did it train on some of the test data? Scope includes most all domains, so that could be a leak)
- What supports Section 5’s explanation that additional model capacity goes into local context?
- Is there anything to be learned from the connected components/fig5 beyond similar structural clusters (from an existing output) cluster together? 6 and 7 propellors being close is hardly novel, and I missed how that’s showing benefits from your work.
- Table 1: not bolding the best results when they are better than your model is not very friendly for the reader, i’d bold them.