- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Summary
This paper examines a boundary effect from BOS/EOS tokens: how it relates to copying from similar in-sequence segment and how it relates to copying from multiple similar in-sequence segments in an encoder-only protein language model.
A boundary-value contribution is defined as the change in the amount that a given residue attends to the value vectors at the BOS and EOS tokens, when these are moved from their true positions at the sequence termini to positions flanking the core of examined segments. These segments are specifically defined as three separately contiguous, equal-length segments that differ only in their central residue. The first two are references A and B with observed residues a and b, while the third is a target with a masked residue.
The paper examines the effects measured by the logits for residues a and b at the masked position. The perturbation is confined to head 15.5’s positional encoding of the BOS and EOS tokens, while the exterior “flanks” are masked off from the core and the boundary tokens.
Through various experiments, the authors essentially show that the BOS/EOS tokens aren’t technical scaffolding tokens but that they actively impact residue logits and that they provide a common boundary signal which, depending on the distance between residues and the BOS/EOS positions can shift the logit margins.
The authors then propose a gradient-based perturbation estimator that seeks to predict the boundary effect perturbation for unseen proteins as means of validating the described mechanism, whereby the margin shift is driven by information flowing from the boundary through the reference keys.
Review
I found this paper a bit difficult to read. The prose itself is straightforward and accurate, but I’m failing to see the big picture. A large number of experiments are run and the results seem fairly comprehensive. However, the main text is most declarative: “We wanted to examine A. We ran method B. The results are C.” There’s not a lot of motivation as to why various design choices were made and more importantly, I don’t see the results or their significance really being discussed. To my untrained eye, some logit differences are observed and localized by examining references and boundary keys, but I’m not sure what the impact of this actually is. I don’t want to be rude because I’m not sure I’m grasping the impact of the research question or the results. How could or should this affect protein language modeling, variant effect prediction, protein design etc. moving forward? How are these results significant for the community? For the rebuttal, I would love to see the authors tackle these questions since this is mainly what’s driving my current score. While the results are somewhat interesting and potentially novel, I don't think they are impactful enough to warrant an ICLR acceptance.
Critical strengths
- Rigor. The methods are rigorously and carefully proposed on a technical level. The data and methods are handled with care and the drawn conclusions and findings are backed up by experimental results. This also results in a sincerity which I appreciate.
- I think the problem being addressed is interesting and somewhat underexplored. The mechanism for examining the boundary effects is interesting, i.e., by moving the positions only as opposed to cropping.
- The GB1 experiment is an interesting example of how segment copying manifests in real protein sequences and the examination thoroughly localizes what causes this effect, indicating that the zero-shot performance of often-used PLMs can occasionally be driven by repeated motifs in the data.
Critical weaknesses
- Impact. This is mostly describe in the review portion above. The paper is more of a scientific examination of ESM-C behavior as opposed to a model-proposing paper, whose contributions are straightforward. For that reason, I want the contributions and their impact emphasized to really distinguish this paper from others. What did we learn from these experiments and how should we use this knowledge going forward?
- I appreciate that the authors don't overgeneralize which can often be the case these days. However, reflecting on what trends might hold for other protein language models, e.g., autoregressive models or hybrid structure-sequence models would be interesting. Do the authors have any opinions or intuitions about how the copying phenomena and boundary effects appear in these other models?
- Many of the experiments appear somewhat artificial and I would like to see a greater discussion of how the findings relate to other datasets and tasks outside of a couple of ProteinGym assays. I realize that this ties in with the impact above, i.e., how it relates to examining and solving actual biological problems - but if anything, I hope it emphasizes my former point.
- The discussion and conclusion are both too concise, in particular the latter which is simply four declarative sentences. I would like to see a more honest discussion and critique of the method and results.
Questions for authors
The questions here mostly mimic the review and weaknesses from above.
- What is the scientific - or even real-world - impact of the presented results? I.e., where does this matter in practice?
- How could or should the findings here influence future development of encoder-only protein language models?
- Would you expect similar behavior from other model classes? E.g., autoregressive protein language models or hybrid sequence-structure models?
- What are the implications of these results for general protein modeling, benchmarking, protein design etc.?
- What is the implication of the gradient-based predictor? Is it mostly to validate the proposed hypotheses, or does it have a practical role to play?