acceptodds
Under review as a conference paper at ICLR 2027

When Does Few-Shot Transfer Learning Work Across Mutation Assays?

Abstract

Protein mutagenesis can generate large numbers of measurements across many proteins, but characterizing any single new protein-assay pair remains expensive. A typical plate has only a few hundred labeled mutants, too few to train an accurate predictor from scratch. Related measurements offer a way to reduce that burden. A model trained on how mutations shift a property in another assay, or in other proteins, may already encode information useful for the new plate. The question is whether that older data can train a predictor for a new assay from only a small target plate, and whether any gain is genuine transfer rather than protein-identity leakage. We study this with three overlap conditions at a fixed 200-label budget. Where source and target assays measure the same proteins, identity itself may carry the signal, so we compare from-scratch training on the plate to sequential transfer. Where the sources share no exact wild-type sequence with the target, a source checkpoint has no shared identity left to exploit; what may still help is a shared initialization that re-specializes from a few target labels. There we apply first-order model-agnostic meta-learning (FOMAML), with a source protein as the task, meta-trained on solubility and thermostability toward a proteolysis-stability (K50) target, beside a random-initialization control. A third condition holds protein identity out of labeled training and spends the whole budget on one protein. Its plate geometry differs, so we report it beside the matched pair rather than on the same axis. When source and target share the same proteins, sequential transfer clearly beats training from scratch on both ranking and calibration. On the identical plate with sequence-disjoint sources, that gain disappears: every arm, including transfer and FOMAML, ranks near chance despite a large source label pool. On a held-out protein with the budget concentrated on that protein alone, transfer again leads from-scratch at every seed and FOMAML clears its random-initialization control, but both remain far below simply fitting the plate. Spreading the same budget across many held-out identities returns every arm to chance. The pattern is that few-shot transfer here is contingent on shared protein identity among the admitted sources (with a remaining confound that relatedness and provenance also change) and on how the labels are spent, not which reuse method is used.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
transfer learning, meta learning, few-shot learning, domain adaptation, multi-task regression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4qqrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates generalisation across protein mutation assays.

It investigates three scenarios: "Study 1 asks how much sequential transfer buys when the old measurements and the new plate share the same proteins, so identity itself may carry the signal. Study 2 keeps that identical plate but admits only sources with no shared wild-type sequence, so any remaining gain cannot be memorization of the target proteins. Study 3 holds a protein out of labeled training and spends the whole 200-label budget on it alone."

Critical strengths

Studies the important problem of whether protein function prediction methods are transferable in the small data, single mutation regime close to how experiments are carried out. A supplement to benchmarks such as ProteinGym and FLIP.

Critical weaknesses

The paper comes across as perhaps interesting for the a specific context, giving empirical data science results for a specific application. It lacks both general interest or methodological novelty for at specific domain that is typical of an ICLR paper.

Questions for authors

Is is possible to make the problem statement more general and the results more widely relevant?

2 comments

We thank you for this review. The review asks whether the problem and results can be made general enough for ICLR. We answer that.

General statement: At a fixed low label budget, how the budget is spent, and which old labels are admitted, constrains performance more tightly than which transfer or meta-learning algorithm is used. Concentrating 200 labels on one new protein supports ranking; spreading the same 200 across many proteins returns ranking near chance. Cross-protein archive labels do not beat a local plate fit when the protein is new; they do help when the same mutants are already in the archive under another assay (lookup); and they can hurt when property and sequence are both disjoint. That is a claim about data regimes, not about proteins as such.

How this ties into data-regime analysis: The three studies are one controlled protocol along that axis (positive control / campaign / negative control), read with a fixed probe so gaps are about admitted data. Previous work asks when a method family helps by fixing the evaluation and varying the regime, rather than proposing a new architecture: DomainBed (Gulrajani & Lopez-Paz, ICLR 2021) for domain generalization under consistent protocols; Project and Probe (Chen et al., ICLR 2024) for sample-efficient adaptation under limited target data; and systematic probing frameworks such as An et al. (ICLR 2023) for controlled transferability tests on frozen representations. Our design is in that spirit (and also follows the control-task idea of Hewitt & Liang). Inspired by that line of work, we hope to bring the same kind of regime analysis to mutation assays: a supplement to ProteinGym and FLIP for the small-data, single-mutation campaign setting, which the revision now states in the introduction.

We would welcome your view on whether that framing addresses the generality concern.

Reviewer_4qqrOpenReview ↗

Appreciate the authors' reply - might change my score to weak reject pending the other reviewers' reply

I appreciate the authors' reply. But still have the same concerns as initially raised. I want to see what the other reviewers reply before possible changing my score.

Reviewer_RAXsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This is an experimental paper that investigates the transfer learning across different mutation assays. In a mutation assay the goal is to mutate the protein typically to increase/maximize some property such as binding energy or activity or stability. This is usually achieved by rounds of experiments in which the experimentalists design 10s or (small) 100s of mutations (replacing one amino acid with another), measure their success, and feed the results into machine learning methods that propose new mutations. This is an expensive and lengthy process that may take weeks. This paper investigates whether existing measured mutations for other properties on the same or different proteins could help predict the new target protein of interest. The paper uses a three-layer perceptron and optimizes mean-squared error regression on target variables that are normalized to have zero mean and unit variance. The results show three main experiments on a dataset compiled from existing mutational assays investigating the transfer in three different set-ups that control for the amount of information that can be transferred from the existing data to the new task.

Overall, this paper is makes a step into investigating an interesting problem, but has major problems in its current state:

  1. It is missing key state-of-the-art methods, 2. experiments are insufficient and 3. writing needs a major human-driven re-write. At this state it is more on the level of a ICLR workshop (after the re-write) in its significance and contribution.
Critical strengths
  • The addressed problem of ML-driven evolution is an important problem with wide practical implications.
  • The paper investigates an interesting question of how much existing labels / data could help.
  • The initial results make a step in an interesting direction (but are so far limited in scope and have major issues, see below).
Critical weaknesses
  1. Insufficient experiments: The presented studies make a step into an interesting direction, but for an experimental paper like this, the expectation is a much more comprehensive set of experiments that will thoroughly investigate the issues. The experiments should investigate the effect on several state-of-the-art methods (see above) and investigate more thoroughly how the transfer depends on the different properties. For example, stability may be less likely to help activity, but stability may be able to capture some epistatic effects that could be important for activity too. It would be interesting to measure this and quantify this effect. 
It would be important to consider separately single-point and multi-point mutations, as the effects may be different. For multi-point mutants, again, the epistasis would be very interesting to study and put a finger on.

Please also consider other metrics that measure and focus on recovery of top mutations rather than overall correlation (which can be misleading). Please see EvolvePro. These metrics are key for practical deployment.

Please also consider methods that focus on ranking rather than regression (see MultiEvolve).

  1. Writing. The paper was hard to read and understand. I tried hard to read it. I just struggled to understand what was done (despite having worked on related problems). It appears a lot of the text is AI-written or at least drafted, but it is too dense and hard to read/understand. It is important for this paper to undergo a human editing pass to explain things clearly and motivate the various choices. Papers are still (I assume) written for humans to read, and hence I think we should insist (even more so now) that they are human-readable.

Overall, this paper is makes a step into investigating an interesting problem, but has major problems in its current state:

  1. It is missing key state-of-the-art methods, 2. experiments are insufficient and 3. writing needs a major human-driven re-write. At this state it is more on the level of a ICLR workshop (after the re-write) in its significance and contribution.
Questions for authors

See above in weaknesses.

2 comments

Thank you. This review was extremely informative for how we revised the work, so we address it point by point.

Orientation note: submitted Study 3 is now Study 2, and submitted Study 2 is now Study 3.

W1 - Missing SOTA methods (EvolvePro, Kermut, MultiEvolve)

Following your recommendations, we implemented versions of these methods that fit our frozen-feature pipeline (EvolvePro-style random forest, Kermut-style GP on embeddings, MultiEvolve-style ranking head, plus ridge, k-NN, and zero-shot priors) and ran them on the same Study 2 K50 one-protein plates. They are not the published end-to-end systems (the GP is RBF-on-embeddings, not Kermut's structure/string kernel; the RF is not EvolvePro's full active-learning loop), so we label them "-style" and report them as reference scores in the appendix.

On Study 2 K50, Spearman ρ / Top-10% hit: EvolvePro-style RF 0.676 / 38.9% (plate) and 0.674 / 38.8% (+ ~20k source); Kermut-style RBF GP 0.705 ± 0.004 / 41.5%; MultiEvolve-style ranking 0.711 / 41.5% vs MSE head 0.711 / 40.7%; ridge 0.682 / 32.7% (plate) and 0.671 / 31.0% (+source); k-NN 0.681 ± 0.005 / 40.2%; position mean 0.689 ± 0.010 / 40.7%; ESM-2 / ProtBERT masked-marginal 0.507 / 25.8% and 0.087 / 12.9%.

Yes, they beat our probe (about 0.68-0.71 against 0.396). That is useful mainly because it lets us ask the transfer question again with a stronger model in hand. Give random forest or ridge the same plate plus about 20k labels from other proteins, and on Study 2 nothing moves (0.674 against 0.676 for RF; 0.671 against 0.682 for ridge). Give random forest the Study 1 setting, where those mutants were already measured in the sister assay, and it jumps from 0.679 to 0.918. So this is evidence about the data regime, not just a note that better architectures exist.

W2 - Insufficient experiments

We expanded the evaluation substantially. Study 2 now runs every eligible protein across all four properties (63 targets, 51,440 held-out rows, four seeds), and we report both plate-weighted study averages and per-property rounds. Transfer neither helps nor hurts when the protein is new. It helps sharply when the same mutants are co-measured (0.378 to 0.546; RF to 0.918). It hinders when property and sequence are both disjoint (Study 3 K50: 0.311 against 0.398).

We stay with single-point mutations. That is the regime we intended, and we state it as a limitation. The pipeline can be extended to multi-point mutants; our corpus simply does not have enough shared multi-point variants to support a thin arm without repeating the thin-solubility mistake. We also cannot answer stability-helps-activity from this corpus (no activity/fitness assay), and we say so here.

On metrics: we now report Top-10% hit rate (top-20 predictions in the true top decile) alongside Spearman, as you suggested. On Study 2 K50: GP 41.5%, position-mean 40.7%, k-NN 40.2%, RF 38.9%, ridge 32.7%, ESM-2 25.8%. On Study 1, RF+source reaches 75.2% against 39.8% plate-only. We also compared a ranking head to an MSE head on the same features; they tie on Spearman (0.711) and Top-10% hit (41.5% against 40.7%).

We keep Spearman as the primary score for a practical reason. A lab cannot afford to measure every mutant. The useful output is an ordering: rank the remaining mutants by predicted property, pick a band worth testing, and spend the next experimental round there. Spearman asks whether that ordering is right. We still train with squared error; we just do not treat calibration as the thing a campaign uses first.

W3 - Writing

We agree with this comment. The submission was too dense, and this review shaped how we rewrote it. We reorganized the studies from the submitted order into Study 1 → Study 2 → Study 3 (ceiling, then campaign, then floor), stated each study's motivation before its construction, opened from the lab workflow rather than from split mechanics, and consolidated the appendix into one master table. Could we ask you to read the introduction and the study section in the revised PDF? If anything is still unclear after that, we are very happy to respond during the discussion period, clear it up, and update the writing.

Reviewer_RAXsOpenReview ↗

Update after the rebuttal

I have read the response and looked at the updated paper.

  1. Writing. It is hard to disentangle what is new and what has been updated. In future responses, please provide a detailed list of changes and updates and highlight the updates in a different color in the revision of the paper. This way it would be possible to verify the changes (by a human).

  2. It is great to see a step towards including the state-of-the-art methods. Yet, it seems the implemented methods are simpler implementations so it is not clear if the conclusions would hold also for their full implementations.

  3. It is great to see the inclusion of the top predictions. Authors are encouraged to also consider top 1% and top 5%. The authors are discouraged from using Spearman as the headline number. The issue with Spearman is not the ranking (this is good) but the fact that ranking is considered across the entire spectrum of mutations (from good to bad), i.e. a method that has a strong rank correlation at the bottom half of the mutation pool is not practically useful. What matters are the top variants or ranking at the top.

Overall, given the remaining issues, the review score remains.

Reviewer_QnSuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors use a frozen ProtBERT model and an adapter MLP to evaluate the performance of few-shor transfer with a 200 label budget. They test different initailisations on different assay datasets and conclude that transfer only helps inside a protein family and when labels are dense, and otherwise not.

Critical strengths

The study tackles an important question and has a solid overlap-controlled design. They clearly state confounding factors and negative results and report per-seed results so that the reader has an idea about the score variability.

Critical weaknesses

[W1] In study 1, it seems like the K50 values etc are measured on the same mutants and there seems to be mutant overlap/data leakage there?

[W2] The models often seem to be worse than even constant predictors, which questions how well the baselines as well as FOMAML are tuned.

[W3] the statistics are very thin with only 3 proteins in study 3 for example and only random within-protein splits, so more data to make a solid conclusion would be helpful.

Questions for authors
  1. I do not see a zero-shot baseline for ESM2 or ESM3 or similar PLM models, how do they compare?

  2. Can the pipeline reproduce known performance on standard held-out benchmarks like FLIP or FLIP2? To make sure baselines were tuned correctly

1 comment

We thank the reviewer for the comments.

Numbering: submitted Study 3 (3 proteins, full budget on a held-out protein) is Study 2 in the revision; submitted Study 2 is now Study 3.

W1 — Mutant overlap in Study 1: that is the objective of the control. Study 1 asks whether transfer works when wild types and mutants are shared and the two assays measure different properties of those entries. The revision/rewrite labels it a positive control / measured ceiling and makes no campaign claim from it. Without this overlap, the Study 2 null is unfalsifiable. Paired-mutant correlation is ρ = 0.95 (Pearson r = 0.73). Under the fixed probe, sequential transfer rises from ρ = 0.378 ± 0.021 to 0.546 ± 0.005; random forest from 0.679 (plate) to 0.918 (+source), just under that assay agreement. With the overlap gone in Study 2, the same arm does not move (0.676 vs 0.674). The improvement appears only when the target mutants were already measured in the old data. It is not a general transfer effect. This was an intentionally designed study.

W2 — Models below constants; baseline tuning. On the plate, from-scratch beats a constant on Study 2 K50: z-MSE 0.894 ± 0.073 vs support-mean 1.027 ± 0.054. The constant predictor has no usable ranking, and ranking is a primary claim of the study. With respect to FOMAML, the parameters are indeed fixed and not tuned. In this study, we benchmark the FOMAML against a random start control, and we aim to quantify any apparent gain or loss accordingly. We see that that gap is approximately zero, which means that FOMAML does not really learn anything for this data regime. FOMAML is expected adapt quickly to a new protein, but at a realistic k = 200 budget it does not. That is a result about this admission regime, not a claim that meta-learning is impossible under every protocol.

W3 — Statistical scale. Held-out evaluation now covers 63 target wild types and 51,440 rows over 4 seeds (was 3 proteins). Main claims rest on K50 (28) and stability (31); solubility and ΔT_m have only 3 and 1 eligible proteins, so we report them and do not lean on them (solubility from-scratch ρ = 0.057 ± 0.018). The 200 labels are still drawn at random among the target’s mutants; the change is in the old data. The negative control (now Study 3) has 32 plates. On its K50 round, transfer is 0.311 ± 0.017 vs from-scratch 0.398 ± 0.032 (study average 0.288 vs 0.358): a cost, not a false positive. We conclude that distant prior data does not always help; they can actually hurt ranking.

Q1 — Zero-shot PLM baseline. We added ESM-2 and ProtBERT masked-marginal as label-free lines on Study 2 K50. ESM-2 at 0.507 outranks our fixed probe at 0.396; the limitations accept that. We accept that: a stronger label-free prior already outranks our probe on the same plates. That is fine for the paper we wrote. We are not trying to pick the best encoder. We are asking which old labels can enter training when the protein is new. The probe stays the same in every study so that a gap between studies comes from the old data, not from swapping the model. The zero-shot numbers mainly show that better representations still leave the data-admission question open.

Q2 — Standard benchmarks. FLIP and this paper ask different questions. FLIP fixes fitness-landscape splits and scores how well a model ranks variants on that landscape. We fix a small label budget and ask which old assay measurements are allowed into training when the protein is new. Matching published FLIP ridge numbers is therefore not the right success criterion for our setup. We still report a GB1 run in Appendix C3 (frozen ProtBERT [CLS] plus ridge on train∪val): ranks stay positively correlated, and we sit below FLIP’s ridge column on every split. That is expected, because we are not running FLIP’s comparison. Our claims are about Studies 1–3: when old mutation labels help a new plate, and when they do not.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.