Paper summary
This paper addresses an often overlooked issue with existing retrieval mechanisms for protein modeling: namely that utility of a given similarity score will depend on the biological problem that is under consideration. To this effect, ReaPO (and its variants) are introduced. ReaPO is trained using a dataset of predefined relations. For each relation, a support triple is defined which is comprised of a positive sample, a negative sample, and an anchor, which makes the comparisons relative. The triples are fed through a frozen ESM-2 encoder model which alongside a natural text instruction fuses into a condition vector. During retrieval, a query q and candidates x and y are similarly embedded with ESM-2. The condition vector is further processed by a signed operator network that joins with the frozen embeddings and results in a normalized readout. For a new condition (such as conditions 1 and 2 in Figure 2), the relation evidence changes and requires separate forward passes through the conditioner. A large number of evaluation protocols are introduced and the model is compared against various versions of itself as well as existing sequence and retrieval models in table 3. Amongst the suite of ReaPO models, there isn’t a consistent top performer, but the cohort improves a joint accuracy on hel-out-family crossed ranking by 0.034 as well as improvements of 0.0236 and 0.0309 in nDCG@10 in other experiments.
Review
I feel a little mixed about this paper. I think that both the abstract and introduction do an excellent job of framing the problem setting and contributions of the paper, so we’re off to a very good start.
My first issue is with the related works section (Section 2): I think it is much too short and superficial. Retrieval is a core mechanism in bioinformatics and protein modeling but much of this is glanced over (e.g., I consider multiple sequence alignments a foundational part of retrieval). Focusing only on machine learning and PLM-driven retrieval, most literature is still glossed over. Important works here are Tranception, the MSA Transformer, PoET, E1, AIDO.RAG and so on. This similarly applies to the other paragraphs in the related works. The purpose of a related works section is to show what work came before to accurately and honestly position the contributions of a paper in relation to the massive body of prior work. I do not think this is adequately achieved. I am not interested in providing a list of papers that I believe should be cited - that is not the point here.
A second and more general issue I have the paper is its presentation in sections 3 and 4. The main text becomes very information dense. It reads as if the paper used to be much longer but had to be heavily compacted to fit the template. This means that the flow of various paragraphs is very choppy. This is particularly evident with the very generous use of semicolons and lists (particular in Sections 3 and 4). I like the paper and I think it would benefit greatly from a revision that seeks to make the main text more digestible and easier to read.
The information density also results, in my opinion, in a very descriptive paper. Apart from the abstract and introduction, few mechanisms or operations are really motivated. They’re mostly listed as “We do a. a does b. b is c.”, without describing why “a” is applied in the first place, what alternatives are, or what impact it has on “b” and later “c”. I hope this example makes sense; I am seeking a greater understanding and appreciation of why your various design choices were made. In extension of this, I would very much like to see a stronger finish in the discussion and conclusion, arguing for strengths and weaknesses of the proposed ReaPO approaches. What are the impacts of using ReaPO over other methods? E.g., how can this accelerate downstream tasks, and which? Purely hill climbing is rarely interesting but I know that accurate retrieval can be crucial downstream. I would like to see this argued more.
A last point I will bring up here is the is performance gains from ReaPO. Are accuracy gains of 0.034 an nDCG@10 gains of 0.0236 and 0.039 going to move the needle? These seem like very minor gains. I also generally don’t see few error bars and confidence intervals, and the ones that are present don’t seem to suggest significance. I think the magnitude and significance of the results should be analyzed in greater detail to convince the reader of the viability of the proposed methods.