acceptodds
Under review as a conference paper at ICLR 2027

NO PROTEIN LANGUAGE MODEL BEATS A LOOKUP TABLE AT FOLD-LEVEL RETRIEVAL

Abstract

Remote homology search finds proteins that share a fold after their sequences have diverged past the range where alignment scores work. The field solves it by embedding every protein with a pretrained protein language model. However, whether those embeddings are what carries fold-level ranking has not been tested across the released models, because published comparisons are small and leave the readout layer, label leakage and index size uncontrolled. Here we evaluate 115 released protein language models from 2.6M to 16.1B parameters on SCOPe40 retrieval under one protocol. We compare them with the Additive Normalised Gated K-mer index (ANGK), a static index that answers a query by table lookup with no forward pass. Fold-level AUROC falls as parameter count grows, with a Spearman correlation of −0.57 across the 115 models. The 39.9 MiB index reaches 0.9555 where no model reaches 0.9500. It also encodes eleven times faster than the fastest of them and holds first place at the topology level of a second taxonomy. A four-way position bin on 3-mers supplies that margin. The same index covers 2.41 × 108 AlphaFold structures in 7.4 hours. One pass over them finds 4,931 structures that connect 53 fold pairs SCOPe keeps apart. Fold-level retrieval now has a baseline with no forward pass.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
static embeddings, dense retrieval, tokenization, protein structure

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2THKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors compare embeddings from 115 protein models on fold level classification using SCOPe40. They claim negative correlation (−0.57 ) between model size and classification results. They do their own supervised lookup based approach, ANGK, over preextracted Foldseek structural tokens, using a teacher model and distillation to finetune further. ANGK reaches fold-level AUROC 0.9555, 0.012 above the strongest baseline, CATH-supervised TEDBench-ESM2-650M-ft, and 0.0235 above the 0-shot SaProt-35M. Using positional information (similarity of quartiles of each sequence) improves ANGK further. The 39 MB index table enables fast encoding once structural tokens are available, and the authors demonstrate its use on 241 million AlphaFold structures and ECOD domains (at different levels).

Critical strengths
  • The paper shows that a small learned embedding representation with distillation and structural tokens can beat strong protein models on fold AUROC in a knn setting. Use of alternate , structural aa alphabets is relatively new at scale and has room for growth and contribution.

  • Selecting PLM layers on separate validation folds also makes the comparison more credible. (It’s an important gotchas often missed in other works and a useful reference here in how it changed the numbers when optimized to -0.37 vs -0.57).

  • The position experiment is nice for showing some feature/domain engineering of a cheap way to circumvent positional embeddings, with positional encoding, and showing that which quarter contains a structural pattern improves retrieval.

Critical weaknesses
  • They don't compare to the obvious baseline which is foldseek itself, or even just basic n-grams over the foldseek tokens (1-3) (without the distilling teacher), or existing sequence alignment approaches that support alphabets like blast / mmseqs2/phhmer (or trivial word2vec on the structural alphabet). At the very least, foldseek on the tokens or ngram based is needed (on the test/eval sets in the paper) given that the whole paper is essentially knn over foldseek tokens.
  • Comparing 115 PLMs does not answer what the method adds over existing simple baselines. Winning AUROC also does not mean better matches near the top; it might be worthwhile adding some sort of ranking metric. SaProt-35M beats ANGK on both AUPRC and top-1 accuracy (Table A1).
  • The speed comparison does not count the same work/not apples to apples. ANGK’s isn't counting its own tokenization and embedding (i.e running on new sequences and predicting foldseek tokens from AA). Including that step would change its throughput. if we were to not count model processing, then the lookup of a models output final layer (with knn) is also comparatively as fast almost, give or take, vs 11x as claimed. The speedup needs a comparison on the same proteins with the same processing stages included.
  • The negative model size correlation is confused by different model families - within most families, larger models are usually better at exactly this task (esm2, esm-c are the exceptions; Carp, protSent, ProteinCLIP etc show better results with larger models (and the same layer) within the same families, on structural tasks and even similar knn setups!). I’d correct this to be a bit less misleading (although current phrasing is maybe ok “not always the best model”; but as an abstract headline it’s a bit confusing and misleading, without more analysis.
Questions for authors
  1. Matched comparison: How do ANGK, Foldseek and simple 3Di counts (e.g. n-grams tfidf) compare on test set? Please include top-hit retrieval or ranking and complete processing time if claiming speed. Also, why not try with knn (without distillation, ngram mergings, did you compare?
  2. Teacher clarification: Which teacher trained the reported tables? Section 3.4 names TM-Vec 2. The supplied code teacher_embed.py loads tmvec-cath. (the train/test overlap is also unclear, did it train on some of the test data? Scope includes most all domains, so that could be a leak)
  3. What supports Section 5’s explanation that additional model capacity goes into local context?
  4. Is there anything to be learned from the connected components/fig5 beyond similar structural clusters (from an existing output) cluster together? 6 and 7 propellors being close is hardly novel, and I missed how that’s showing benefits from your work.
  5. Table 1: not bolding the best results when they are better than your model is not very friendly for the reader, i’d bold them.
Reviewer_NSE3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper evaluates 115 released protein models on protein-domain retrieval and introduces ANGK, a lightweight representation built from learned embedding tables over Foldseek 3Di k-mers. ANGK combines four token views, including position-binned 3-mers, and is trained by distillation from a frozen TM-Vec 2 teacher. The evaluation considers validation-based layer selection, structural supervision in released checkpoints, and matched embedding-storage budgets. ANGK achieves the highest SCOPe40 fold-level AUROC among the evaluated methods and ranks first at the ECOD topology level. The authors also report substantial encoding-speed advantages and demonstrate a large-scale application to approximately 241 million AlphaFold structures. The central empirical finding is that an inexpensive additive representation can outperform the tested protein embeddings on fold-level AUROC under the evaluated protocol.

Critical strengths
  1. A broad and practically relevant comparison. Evaluating 115 released models under a common retrieval protocol provides a useful resource for researchers choosing protein representations. Selecting layers on disjoint validation folds and reporting results under matched storage budgets address important sources of variation that can otherwise obscure comparisons. The finding that larger models do not reliably improve fold-level AUROC is useful within this setting.

  2. An efficient baseline with an informative positional ablation. ANGK offers a computationally inexpensive alternative once structural tokens are available. The position-bin ablation is particularly helpful: four-way positional bins improve fold AUROC from 0.9435 to 0.9555, whereas the vocabulary-capacity control does not reproduce this gain. This provides evidence that coarse positional information contributes beyond simply increasing table capacity. The AlphaFold-scale application further illustrates the computational feasibility of the representation.

Critical weaknesses

The comparison does not sufficiently separate input information, training supervision, and representation design. ANGK’s strongest result uses structure-derived 3Di tokens and approximately 42 million learned parameters distilled from TM-Vec 2. Many comparison models receive only amino-acid sequences. The sequence-only ANGK variant is useful, but it obtains its structural tokens through ProstT5, introducing another learned component. Moreover, Section 3.4 provides little information about the distillation corpus, pair construction, loss, or overlap with evaluation proteins. This is consequential because the paper emphasizes controlling training-label exposure in competing models. Without an equally explicit account of ANGK’s training and upstream components, I cannot confidently attribute its advantage to the proposed additive representation or assess the generalization claim.

The broader conclusions exceed what the evaluation establishes. ANGK leads on fold-level AUROC, but Appendix Table A1 reports that 18 models exceed it on sensitivity before the first false positive, six on AUPRC, and three on top-1 accuracy. These are consequential distinctions for retrieval. For example, TEDBench achieves AUPRC 0.5013 versus ANGK’s 0.3482. The paper acknowledges these differences, but its headline framing and conclusion that fold-level ranking does not depend on contextualization remain too broad. The predominantly mean-pooled comparison also does not isolate limitations of the pretrained encoders from limitations of their readout. Mean pooling contextualized residue vectors does not, by itself, establish that structural order has been lost. Finally, ECOD reuses largely the same domains, so it provides robustness to a different taxonomy rather than independent dataset validation. Together, these limitations leave the general representation-learning conclusion less established than the breadth of the model sweep initially suggests.

Lastly, allthough the problem is important, I find the methodological novelty limited. ANGK combines structural k-mer features, additive aggregation, coarse positional information, and teacher distillation, with the main contribution lying in the extensive empirical comparison. The evaluation remains concentrated on fold-level retrieval, and the ECOD analysis largely reuses the same protein domains. Consequently, the paper does not yet establish how broadly its findings inform protein representation learning or explain when contextual models are unnecessary. For an ML-centric conference such as ICLR, I would like a clearer, better-supported insight beyond the ranking of existing models under this particular protocol. In its current form, the contribution may be more directly relevant to a specialist bioinformatics or protein-engineering audience.

Questions for authors
  1. Training and evaluation separation. What datasets, pair-sampling procedure, loss, and model-selection criteria were used to train ANGK? Please specify overlap with the evaluation domains, superfamilies, and folds for both ANGK’s distillation data and its teacher’s training data. Please also clarify which data were used to construct the token vocabulary and fit PCA.

  2. Computational accounting. Which stages are included in the reported encoding throughput and AlphaFold encoding time? In particular, are structure-to-3Di conversion and, for the sequence-only variant, ProstT5 inference included? Separating preprocessing, embedding generation, compression, and search would clarify the scope of the efficiency advantage and the “no forward pass” claim.

  3. What does learning the table contribute? Under the same structural input and storage budget, how does ANGK compare with normalized 3Di k-mer counts or a simple projected count representation using the same token views? This would help distinguish the contribution of distillation and learned token geometry from the information already present in the structural alphabet.

  4. Encoder versus pooling limitations. Can a focused comparison with one or two strong PLM baselines using learned or position-aware pooling determine whether the observed gap persists beyond mean pooling? This would materially affect my interpretation of the claim about contextualization. A small controlled experiment would be sufficient; I am not requesting retraining all 115 models.

Reviewer_89TwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces ANGK, a protein structure representation that combines 3Di k-mer embeddings, coarse positional bins, and distillation from TM-Vec2. It evaluates ANGK against a broad set of embedding baselines under a unified retrieval protocol and reports the highest fold-level AUROC among the tested models, alongside efficient encoding. However, several baselines perform better on AUPRC and early-retrieval metrics. Ablations examine representation choices, particularly positional binning. An AlphaFold database scan additionally identifies candidate cross-fold relationships in embedding space.

Critical strengths
  • Efficient retrieval from large databases of available protein structures is a valuable problem. ANGK’s reported encoding efficiency and fold-level AUROC make it a potentially useful approach. Evaluating many publicly available models under a common protocol strengthens the empirical contribution and provides a more informative comparison than a narrowly selected baseline set.

  • Attention to PLM layer selection and storage budgets improves the comparison, while the positional-binning ablation provides concrete evidence for a specific design choice. The authors also report unfavorable retrieval results and acknowledge that apparent cross-fold “bridges” may reflect insensitivity to repeat count. These disclosures help identify both the representation’s strengths and its failure modes.

Critical weaknesses
  • A “lookup tables beat large models” framing also obscures ANGK’s structural inputs and distilled supervision. Its stronger AUROC, alongside weaker AUPRC and early retrieval, supports a specific performance-efficiency tradeoff rather than general retrieval superiority. The contribution would be more convincing with narrower claims or matched comparisons that isolate the proposed explanation.

  • Training provenance and training-evaluation separation are insufficiently specified. This is the most consequential issue because it prevents a reliable assessment of generalization and reproducibility. The training description identifies a frozen TM-Vec2 teacher, structural-similarity pairs, and 3,000 training steps, but does not clearly specify the student dataset’s source and size, pair-sampling procedure, or exact distillation objective. It also leaves unclear the overlap and homology relationships between student training examples and test domains, as well as the test set’s coverage in the teacher’s supervised training data. These omissions do not establish leakage, but they prevent the reader from assessing whether the reported advantage reflects generalization beyond the training supervision.

  • The comparisons support an efficient retrieval method more strongly than the broader claims about PLMs and contextualization. The paper combines three distinct questions: whether available structures can be searched cheaply, whether PLMs encode useful fold information, and whether contextual encoding is necessary for retrieval. The experiments address the first, but input information, training supervision, encoder architecture, and pooling vary together, preventing a controlled answer to the latter two.

  • The missing comparisons are particularly consequential here. [1] already represents structures through counts of discretized structural fragments and is directly relevant to both the novelty claim and the baseline set. On the PLM side, [2] demonstrate the importance of locality-aware pooling, while [3] uses direct comparisons of residue embeddings for distant homology detection. These methods have different computational tradeoffs, but they show why the tested pooled embeddings cannot establish the limits of PLM representations.

[1] Durairaj J, et al. Geometricus represents protein structures as shape-mers derived from moment invariants. Bioinformatics, 2020.

[2] Hoang M, Singh M. Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics, 2025.

[3] Kaminski K, et al. pLM-BLAST: distant homology detection based on direct comparison of sequence representations from protein language models. Bioinformatics, 2023.

Questions for authors
  • Will you revise the “lookup tables beat large models” framing? The full pipeline goes well beyond a lookup table: it relies on Foldseek’s learned 3Di representation, TM-Vec2 supervision, and student training through distillation. The title and conclusions should reflect these learned components and distinguish inexpensive lookup-based inference from the machinery required to construct the representation.

  • What are the student training data, pair-sampling procedure, and distillation loss? What overlap or homology exists between the evaluation domains and the student’s and teacher’s training data?

  • Will you compare ANGK with Geometricus [1] to establish its advantage over existing structural-fragment representations?

  • Will you evaluate PLM embeddings through an analogous pipeline, using local/k-mer aggregation and matched distillation supervision, or test stronger published approaches such as locality-aware pooling [2] and residue-level comparison with pLM-BLAST [3]? This would help determine whether ANGK’s advantage reflects its representation or limitations of the tested PLM pooling strategies.

[1] Durairaj J, et al. Geometricus represents protein structures as shape-mers derived from moment invariants. Bioinformatics, 2020.

[2] Hoang M, Singh M. Locality-aware pooling enhances protein language model performance across varied applications. Bioinformatics, 2025.

[3] Kaminski K, et al. pLM-BLAST: distant homology detection based on direct comparison of sequence representations from protein language models. Bioinformatics, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.