- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates in-context learning in protein language models using encoder-only ESM-2 and decoder-only ProGen2. It evaluates pattern completion, secondary-structure classification, and MHC-II binder classification, reporting performance comparable to a baseline using frozen ESM-2 embeddings.
Critical strengths
The paper addresses an interesting question: whether protein language models can perform in-context learning. The experiments and mathematical formulation appear reasonably thorough.
Critical weaknesses
The evaluation is limited to two model families and three classification tasks, providing insufficient model and task diversity to support the paper’s broad claims. Merely matching trained baselines also offers limited insight. Given the novelty and significance of the proposed capability, stronger evidence is needed to establish in-context learning in protein language models.
The absence of a related-work discussion, together with the disproportionate space devoted to standard scoring methods, raises concerns about the paper’s engagement with the existing literature.
Questions for authors
- Spacer choice. Why is G used as the spacer? Please cite relevant precedent or explain the rationale in the main text.
- Overstatement of the findings, around line 410. The statement “Our results provide strong evidence that single-sequence protein language models (PLMs) are capable of in-context peptide learning” is not adequately supported by two model families and three classification tasks. Please narrow the claim to match the evidence.
- Purpose of pattern completion. What scientific question does this experiment address, and how does it support the paper’s central argument? Its motivation and intended conclusion are currently unclear.
- Typographical error, line 211. Please correct “We next ask asks whether …”.
- Related work. Please discuss relevant prior studies and situate the contribution within the existing literature. The current omission raises concerns about whether the relevant research has been adequately considered.
Request for clarification on evaluation
We would like to clarify the scope of the submitted evaluation. Firstly, in addition to the pattern completion, secondary structure classification, and MHC-II binder classification tasks, the original submission also evaluates our method across ace inhibition, antibacterial, anti-inflammatory, antimicrobial, antioxidant, neuropeptide, nonfouling, and tumor T-cell antigen tasks leveraging the PepBenchmark dataset (Main text: Section 3.5. Additional Experiments and Ablations; Appendix: Section D. PepBenchmark generalization and ablation studies, Figures 10-23). Additionally, we performed extensive mechanistic interpretability analyses on the ProGen2 model that yielded evidence for induction heads (Main text: Section 3.4 Induction Heads, Figure 6; Appendix: Section E. Induction heads in ProGen2 Table 3, Figures 24-66). These experiments were included before the submission deadline. Could the reviewer clarify whether the concern remains after accounting for these results?
1. Regarding the strength of the claim that protein language models (PLMs) are capable of in-context peptide learning
We dispute the reader’s claim around the strength of the claim. To show that PLMs are capable of ICL, it is sufficient to show that some are capable, and we are two-for-two. Our claim does not rest solely on a few matched benchmarks: our work on induction heads shows a clear mechanistic connection with in-context learning in LLMs.
2. Why glycine and whether the spacer is arbitrary
Reviewer concern: Explain the use of G as a spacer and provide a rationale or relevant precedent.
We address this in our general comment to the reviewers.
3. Purpose of pattern completion
Reviewer concern: What scientific question does the synthetic experiment address, and how does it support the central argument?
The experiment isolates a prerequisite for contextual adaptation: can additional copies of a sequence pattern change predictions at held-out positions? It separates this question from peptide-class labels and downstream biological classification. Increased completion accuracy with repeat count demonstrates sensitivity to in-context regularities, including artificial repeats over canonical amino acids.
4. Typographical correction
Reviewer concern: Correct “We next ask asks whether …” at line 211.
Thank you for catching this. The corrected wording is “We next ask whether …”. We will include this correction in the manuscript revision.
5. Related work and presentation
Reviewer concern: Add a substantive related-work discussion and rebalance space devoted to standard scoring methods.
We had incorporated discussion of related work inside the text, but have made it to a separate section in response to reviewer feedback. We are not clear on what is meant by “standard scoring methods”: for instance, whether the reviewer means the use of likelihood-based scores for PLMs or too much discussion of our benchmarks. We do not see the paper as having too long a discussion of benchmarks, although we could be persuaded otherwise. Regarding the discussion of scores for likelihoods, we are certainly not the first to sum over logits from a PLM! However, the use of the MPEP prompting strategy in PLMs is new. Consequently, even if the resulting likelihood may be obvious to a reader intimately knowledgeable of the field, it nevertheless needs to be stated precisely to avoid ambiguity.