- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates generalisation across protein mutation assays.
It investigates three scenarios: "Study 1 asks how much sequential transfer buys when the old measurements and the new plate share the same proteins, so identity itself may carry the signal. Study 2 keeps that identical plate but admits only sources with no shared wild-type sequence, so any remaining gain cannot be memorization of the target proteins. Study 3 holds a protein out of labeled training and spends the whole 200-label budget on it alone."
Critical strengths
Studies the important problem of whether protein function prediction methods are transferable in the small data, single mutation regime close to how experiments are carried out. A supplement to benchmarks such as ProteinGym and FLIP.
Critical weaknesses
The paper comes across as perhaps interesting for the a specific context, giving empirical data science results for a specific application. It lacks both general interest or methodological novelty for at specific domain that is typical of an ICLR paper.
Questions for authors
Is is possible to make the problem statement more general and the results more widely relevant?
2 comments
Appreciate the authors' reply - might change my score to weak reject pending the other reviewers' reply
I appreciate the authors' reply. But still have the same concerns as initially raised. I want to see what the other reviewers reply before possible changing my score.
We thank you for this review. The review asks whether the problem and results can be made general enough for ICLR. We answer that.
General statement: At a fixed low label budget, how the budget is spent, and which old labels are admitted, constrains performance more tightly than which transfer or meta-learning algorithm is used. Concentrating 200 labels on one new protein supports ranking; spreading the same 200 across many proteins returns ranking near chance. Cross-protein archive labels do not beat a local plate fit when the protein is new; they do help when the same mutants are already in the archive under another assay (lookup); and they can hurt when property and sequence are both disjoint. That is a claim about data regimes, not about proteins as such.
How this ties into data-regime analysis: The three studies are one controlled protocol along that axis (positive control / campaign / negative control), read with a fixed probe so gaps are about admitted data. Previous work asks when a method family helps by fixing the evaluation and varying the regime, rather than proposing a new architecture: DomainBed (Gulrajani & Lopez-Paz, ICLR 2021) for domain generalization under consistent protocols; Project and Probe (Chen et al., ICLR 2024) for sample-efficient adaptation under limited target data; and systematic probing frameworks such as An et al. (ICLR 2023) for controlled transferability tests on frozen representations. Our design is in that spirit (and also follows the control-task idea of Hewitt & Liang). Inspired by that line of work, we hope to bring the same kind of regime analysis to mutation assays: a supplement to ProteinGym and FLIP for the small-data, single-mutation campaign setting, which the revision now states in the introduction.
We would welcome your view on whether that framing addresses the generality concern.