acceptodds
Withdrawn from ICLR 2027

RETHINKING PROTEIN SIMILARITY: ONE REPRESENTATION, MANY RELATIONS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Protein retrieval depends on the biological question: finding proteins with a similar fold and finding proteins that bind the same ligand can require different rankings of the same candidates. A fixed similarity score cannot adjust its ranking to the requested criterion, limiting retrieval tailored to different biological questions. We introduce ReaPO (Relation-adaptive Protein Operators), which uses a natural-language instruction and a few directed example comparisons to adapt the similarity rule over frozen protein language model representations. Its signed variant, ReaPO-Signed, assigns relation-dependent positive or negative weights to individual feature directions. We evaluate the approach on crossed ranking, which requires opposite correct rankings for the same query and candidates under two relations, and on ordinary protein retrieval using internal and external datasets. With both relation families withheld from readout training, ReaPO-Signed achieves 0.548 joint accuracy, compared with 0.514 for a conditional positive-semidefinite scorer augmented with a learned global signed scalar that can reverse the whole ranking. When trained separately using only a target-ranking objective, ReaPO-Signed improves nDCG@10 over matched condition-independent signed readouts by 0.0236 on Ordinary20 and 0.0309 on external CATH–BioLiP2 retrieval in the Remote setting. These results support adapting comparison rules to biological relations while reusing shared frozen protein representations.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein retrieval, Relation-conditioned similarity, Protein language models, Signed similarity operators, Few-shot learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3XJTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

RETHINKING PROTEIN SIMILARITY shows an approach for combining task specific/natural language instructions with protein similarity search (reapo). They test it on a dataset of their own of some 20 targets, and show different results with different instructions, as well as BioLiP2. It uses frozen protein embeddings (e.g. from esm2-650M). Another network takes the text and four (few shot) examples and chooses how to weight the protein embedding for ranking matches. They show that using examples and instructions mildly improves performance, e.g. "0.548 vs 0.539" when wording removed. They also test some different variants (e.g. negative and positive weights variant)

Critical strengths
  1. Few shot learning and combining nlp and sequence embeddings is relatively understudied, and improvements are useful given how structure and function (for example) can often be completely orthogonal.
  2. Relatively computationally parsimonious (putting aside embedding cost and overhead), given few shot and combinations.
Critical weaknesses
  1. Hard to understand the writing, the writing is LLM slop in many parts. Draining to read.
  2. Many methods are arguably excess-complicated and would benefit from the simpler approach first, before adding on variants. E.g. what are results just from a stacked embedding, without the reapo, ReaPO-Signed, etc? How does it compare to doing a text search on uniprot and filtered by a keyword where exists?
  3. Practical gains seem low. e.g. ablating or swapping instructions, and lacks comparisons to more common approaches. re the former: In CATH–BioLiP2 retrieval, the best model using instructions and examples got 0.644 vs 0.638 without them. That's a small difference (double so given it's got examples overhead).
Questions for authors
  1. Did you try simpler approaches, like just concatenating the embeddings? Or sentence transformers + learned embeddings?
  2. How does this compare with search on uniprot or using mmseqs2 or foldseek or even blast +- keeping by a keyword?
Reviewer_oDDYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Paper summary

This paper addresses an often overlooked issue with existing retrieval mechanisms for protein modeling: namely that utility of a given similarity score will depend on the biological problem that is under consideration. To this effect, ReaPO (and its variants) are introduced. ReaPO is trained using a dataset of predefined relations. For each relation, a support triple is defined which is comprised of a positive sample, a negative sample, and an anchor, which makes the comparisons relative. The triples are fed through a frozen ESM-2 encoder model which alongside a natural text instruction fuses into a condition vector. During retrieval, a query q and candidates x and y are similarly embedded with ESM-2. The condition vector is further processed by a signed operator network that joins with the frozen embeddings and results in a normalized readout. For a new condition (such as conditions 1 and 2 in Figure 2), the relation evidence changes and requires separate forward passes through the conditioner. A large number of evaluation protocols are introduced and the model is compared against various versions of itself as well as existing sequence and retrieval models in table 3. Amongst the suite of ReaPO models, there isn’t a consistent top performer, but the cohort improves a joint accuracy on hel-out-family crossed ranking by 0.034 as well as improvements of 0.0236 and 0.0309 in nDCG@10 in other experiments.

Review

I feel a little mixed about this paper. I think that both the abstract and introduction do an excellent job of framing the problem setting and contributions of the paper, so we’re off to a very good start.

My first issue is with the related works section (Section 2): I think it is much too short and superficial. Retrieval is a core mechanism in bioinformatics and protein modeling but much of this is glanced over (e.g., I consider multiple sequence alignments a foundational part of retrieval). Focusing only on machine learning and PLM-driven retrieval, most literature is still glossed over. Important works here are Tranception, the MSA Transformer, PoET, E1, AIDO.RAG and so on. This similarly applies to the other paragraphs in the related works. The purpose of a related works section is to show what work came before to accurately and honestly position the contributions of a paper in relation to the massive body of prior work. I do not think this is adequately achieved. I am not interested in providing a list of papers that I believe should be cited - that is not the point here.

A second and more general issue I have the paper is its presentation in sections 3 and 4. The main text becomes very information dense. It reads as if the paper used to be much longer but had to be heavily compacted to fit the template. This means that the flow of various paragraphs is very choppy. This is particularly evident with the very generous use of semicolons and lists (particular in Sections 3 and 4). I like the paper and I think it would benefit greatly from a revision that seeks to make the main text more digestible and easier to read.

The information density also results, in my opinion, in a very descriptive paper. Apart from the abstract and introduction, few mechanisms or operations are really motivated. They’re mostly listed as “We do a. a does b. b is c.”, without describing why “a” is applied in the first place, what alternatives are, or what impact it has on “b” and later “c”. I hope this example makes sense; I am seeking a greater understanding and appreciation of why your various design choices were made. In extension of this, I would very much like to see a stronger finish in the discussion and conclusion, arguing for strengths and weaknesses of the proposed ReaPO approaches. What are the impacts of using ReaPO over other methods? E.g., how can this accelerate downstream tasks, and which? Purely hill climbing is rarely interesting but I know that accurate retrieval can be crucial downstream. I would like to see this argued more.

A last point I will bring up here is the is performance gains from ReaPO. Are accuracy gains of 0.034 an nDCG@10 gains of 0.0236 and 0.039 going to move the needle? These seem like very minor gains. I also generally don’t see few error bars and confidence intervals, and the ones that are present don’t seem to suggest significance. I think the magnitude and significance of the results should be analyzed in greater detail to convince the reader of the viability of the proposed methods.

Critical strengths
  • Great abstract and introduction that introduces and motivates the core contribution of the paper.
  • The retrieval problem itself is interesting, i.e., the problem with the one-similarity-score-fits-all that the authors are addressing.
  • I think Figure 2 does a great job of showing the workflow of ReaPO.
  • A comprehensive suite of evaluation protocols are applied.
Critical weaknesses
  • The related works are short and superficial and require a significant lift to adequately position the contributions of the paper in the greater literature landscape.
  • The presentation - in particular in Sections 3 and 4 - is lacking. Few mechanisms, operators, and decisiosn in Section 3 are motivated. Section 3 reads more like a mechanical description of what was done, but I think it would be greatly improved if the "why" was also included.
  • Small gains with questionable significance. The accuracy and nDCG@10 gains are small and would benefit from a significance analysis. The results would also benefit from being shown in a larger context. While the gains seem small, perhaps they are large enough to result in measurable downstream improvements. This is however not argued. I think the discussion and conclusion section would benefit from this analysis.
Questions for authors
  • L117-121: An example of what a set of anchor, positive, and negative could be would be useful to the reader.
  • I like Figure 2. Very instructive.
  • Why 7 layers of ESM? This sounds potentially excessive. Did you run experiments that showed that including all 7 resulted in substantial gains of 1-6 layers?
  • The model is trained using a dataset of 20 defined relations with canonical instructions. Since the instruction model takes text input, how resilient is it? E.g. to perturbed, contradicting, or wrong/non-sensical inputs?
  • Similarly, how adaptable is the model to a new relation, or even a new type of relation not covered by the training set? Is retraining then required?
  • L142: “The support encoder maps contrasts to tokens”. Is this entirely accurate or am I misreading Figure 2? I thought that all ESM embeddings are mean-pooled, thus averaging out the per-token signals. Are the full-length embeddings used prior to mean pooling?
  • L158-L161. This paragraph is very information dense without much explanation. What exactly is “matched PSD control”? A more thorough introduction and motivation for the various mechanisms and operators would be helpful.
  • Subsection 3.3. This paragraph is also very information dense without sufficient lead-in or motivation. What is the implication of self-quadratic forms? Why normalize in the first place? What does this subsection aim to achieve?
  • L207-210: This paragraph is very compact with multiple semicolons. I would rephrase to make it easier to parse.
  • L213: The notation is getting complex: what does ℓ.2\ell_{.2} mean? Should it be read as the softplus function, where δ=0.2\delta = 0.2? If so, this should be explained.
  • Subsection 4.1 on evaluation protocols is very comprehensive, in particular when compared to the preceding methods section. I would suggest simplifying 4.1 and moving a bulk of the details to the appendix and then spending more time on describing and motivating the method itself.
  • L375-377: As you point out, including the instruction path appears to have a non-significant impact. However, as you point out in L1461, “This tests the scoring path under retraining, not the removal of all language supervision”. Do you have an ablation that does exactly this - removes all language supervision? I’d be curious to see the full impact.
  • I think the general framing of having multiple conditions at odds with each other is interesting and it begs the question: how do we handle simultaneous (and potentially opposing) conditions? For example, what if I am retrieving sequences that both share some domain while also sharing the same EC classification? Or assuming some larger dataset, how can we rank and retrieve enzymes that both highly active and specific? Can ReaPO be extended to this form of multi-objective retrieval?
Reviewer_KSaxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the problem of protein retrieval via learned similarities. A fixed similarity score cannot be easily adapted to different relations. To address this, the authors propose ReaPO (Relation-adaptive Protein Operators), which uses natural language and examples as condition to derive the representations for protein retrieval. They also propose a signed variant, ReaPO-Signed, which distinguishes relation-dependent positive and negative weights for different feature dimensions. They evaluate the proposed methods on different kinds of protein retrieval tasks with comparison to other text-based protein similarity.

Critical strengths
  1. The motivation is clear and reasonable. Protein similarity depends on the biological question. Adapting a shared protein representation using a few examples is a sensible way to accommodate different search goals without retraining the protein encoder.
  2. The experiments provide evidence that conditioning helps. The authors carefully separate training and evaluation proteins, withhold relation families, and include useful controls. The ordinary and external retrieval results show that the improvement generalizes beyond the constructed ranking-reversal task.
Critical weaknesses
  1. The practical value of the proposed method remains unclear. The main benchmark deliberately requires opposite rankings and supplies examples illustrating the same competing relations. The ordinary retrieval experiments use small, curated candidate sets, while the external evaluation lacks comparisons with established retrieval tools, e.g., BLAST, MMseqs2 search, HMMER and Foldseek. It is difficult to tell whether this would meaningfully improve a researcher’s search workflow.
  2. The benefit of the model components is not well supported. Removing the instruction path barely changes performance, and conditional PSD scoring performs slightly better than signed scoring on ordinary and external retrieval. The held-out-family tests show adaptation across related annotation tasks, but do not answer the question of generalization to substantially new biological criteria.
Questions for authors
  1. Can you demonstrate a realistic search task where the method improves retrieval over established tools? A larger database comparison, with comparable inputs and a concrete biological search goal, would help prove its practical value.
  2. Does the method still generalize when support examples are chosen independently of the competing relation? For example, can one fixed support set for structural similarity work across different candidate populations and competing criteria?
  3. How much do language instructions and signed coefficients contribute to ordinary retrieval? Could you provide matched ablations separating instruction-only, support-only, and combined conditioning, together with signed versus PSD scoring?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.