- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper trains a transformer that fuses the sequence and structure embedding of variants (context) to predict the fitness. It is tested on two settings: real experimental labels and synthetic labels predicted from Kermut. The experiments are done in ProteinGym training/test/val split.
Critical strengths
Experiments and methodology are sound.
Critical weaknesses
- I think this is sound paper with good empirical gains. However, the novelty and impact is limited to recommend accept. The idea of encoding multiple variants and training a transformer to predict fitness already existed. (ProteinNPT and PRIMO).
- Synthetic-only setting still depends on experimental supervision. L344-345 "This allows us to generate tasks over a distribution of GP settings which are informed by experimental data." I think it is misleading to call it synthetic-only.
- I also thought about if this method can be useful for practitioners. But Figure3 suggest at context size of 32-64 Proficient performs similar to Kermut. So the case where Proficient can be useful is very narrow, low-N setting where N=4-32.
Questions for authors
- How does choice of context affects the performance? What if you split the evaluation assays to make sure, instead of random, the context is chosen from held-out positions? I think ProteinGym has supervised split provided within assay (modulo, random, ...), what's the performance if you select context only from one bucket of the split and evaluate on the rest?
- Synthetic pretraining lets you train outside the ProteinGym training assays. Is there particular reason for limiting to ProteinGym training assays?
- How's the performance if synthetic training uses zero-shot scores instead of experimentally fitted scores?
However I think these will not materially affect my rating. I think the novelty and impact is fundamentally limited. For revision and future submission I would recommend exploring methods for other synthetic pretraining (learning from stability labels, learning from AlphaFold confidence, etc) or studying deeper in "why" rather than just showing the performance. Or submitting to soundness-prioritized journals like TMLR might make sense.
5 comments
Response [2/3]
Scientific Contributions
We substantially improve accuracy and label efficiency for protein fitness prediction. Across 33 held-out assays, PROFICIENT achieves a mean Spearman correlation of 0.671 with only four labeled variants, outperforming other baselines even when those baselines are given 32 labels. It also achieves higher mean Spearman correlation than other baselines at every evaluated context size, without requiring assay-specific parameter updates.
We find that biological priors and experimental data have complementary strengths. We might expect synthetic landscapes to be an imperfect substitute for real data. Instead, we find that the two are strong in different regimes. Training on real assays yields strong predictions from very few measurements, but the performance saturates as the context grows. Biological priors, in turn, perform well for larger contexts.
We demonstrate that our framework provides an effective way to combine biological priors with empirical data. Since the two are strong for different context sizes, we might expect that combining them would improve one regime at the expense of the other. Instead, our framework outperforms both standalone models at every context size, indicating that it captures useful predictive information from both sources.
We show that even biological priors which do not lead to strong standalone predictions can improve performance over models which are only trained on experimental data. Our new ablations (Fig. 8) demonstrate the performance of models using various biological priors. Adding synthetic pretraining consistently leads to improvements over real-only training, demonstrating the benefits of our framework.
Our contributions extend beyond combining existing components: we introduce methodological innovations and establish new findings about how biological priors and experimental data contribute to protein fitness prediction.
References
- Biswas et al., Low-N Protein Engineering with Data-Efficient Deep Learning, Nature Methods 2021
- Chu et al., Accurate Top Protein Variant Discovery via Low-N Pick-and-Validate Machine Learning, Cell Systems 2024
- Gelman et al., Biophysics-Based Protein Language Models for Protein Engineering, Nature Methods 2025
- Scheuer et al., KinPFN: Bayesian Approximation of RNA Folding Kinetics using Prior-Data Fitted Networks, ICLR 2025
- Sextro et al., MapPFN: Learning Causal Perturbation Maps in Context, NeurIPS 2026
- Ojeda et al., Prior-Fitted Functional Flows: In-Context Generative Models for Pharmacokinetics, UAI 2026
Response [3/3]
I would recommend exploring methods for other synthetic pretraining
Thanks for the suggestion. Your suggestion motivated us to add new ablations to better understand the impact of the biological prior. Specifically, we have added Section 5.3, where we compare four alternative biological priors:
- RBF Kernel: Samples from a GP with an RBF kernel between ESM-2 sequence embeddings and a constant mean function
- RBF + ESM2 mean: Uses the same RBF kernel, with ESM-2 zero-shot scores as the mean function
- Kermut Kernel: Samples from a GP with the Kermut kernel and a constant mean function
- Kermut + ESM2 mean: Uses the Kermut kernel with ESM-2 zero-shot scores as the mean function, without PROFICIENT's additional zero-shot label blending described in Section 4
We see in Figure 8 that:
- Every tested prior improves downstream performance over real-only training, including priors with relatively weak standalone performance.
- Real-assay finetuning narrows the performance gap between different priors, showing that experimental learning can compensate for some of the limitations of the synthetic data distribution.
- Prior design remains important. Even though finetuning narrows the performance gap between priors, the relative ordering is broadly preserved, and stronger priors tend to remain stronger after finetuning.
These results also motivate the exploration of alternative biological models in future work, and we have highlighted these directions in our discussion.
The synthetic-only settings still depends on experimental supervision.
We use "synthetic" to distinguish training the predictor on generated labels from training it directly on experimental labels. As we describe in Section 4, experimental measurements are used to inform a small set of GP hyperparameters, which capture broad properties of fitness landscapes such as their smoothness and the relative importance of sequence and structural similarity. Our goal is to use this compact information to generate useful training tasks, not to eliminate experimental information from the pipeline. The relevant question is therefore whether these generated tasks can add value compared to directly training on experimental assays, and we see in Figure 1 that the biological priors do provide complementary sources of information. We have updated Section 4 to make this distinction explicit.
How does the choice of context affect the performance?
We evaluate sensitivity to context selection using five independently sampled context sets per assay and context size, reporting the mean and standard deviation across draws. This measures the robustness of our results across the sampled contexts.
For the rebuttal, we have also added evaluations using the different ProteinGym supervised splits, reported in Appendix A.2. We sample the context exclusively from one fold and evaluate on single-mutant variants in the remaining four folds. We find that PROFICIENT retains its advantage over Kermut and the synthetic and real-only ablations across the three splits. These results demonstrate that combining synthetic pretraining with real-assay finetuning is beneficial across many contexts.
Why restrict synthetic generation to ProteinGym?
ProteinGym provides a standardized benchmark and lets us use the same setup as existing methods such as PRIMO. This makes the comparison more informative, because we can test the benefit of including biological priors while using the same available training assays. ProteinGym also spans diverse proteins and measured properties, including stability, activity, and fluorescence, which provides a good foundation for developing our method. We agree that our framework is not restricted to this dataset, and extending generation to additional settings would be an interesting direction for future work.
How's the performance if synthetic training uses zero-shot scores instead of experimentally fitted scores?
Could you clarify whether you mean using zero-shot scores directly as training labels? We sample synthetic labels from the GP prior, not its fitted posterior, and already incorporate zero-shot scores through the GP mean function and label blending (Section 4).
We hope these clarifications and new experiments support reconsideration of the paper’s novelty and impact. Together, our biological task generator, training framework, and new findings establish how biological priors can improve learning from experimental assays.
Response [1/3]
Thank you for recognizing the soundness of our experiments and the empirical improvements. We first clarify an important distinction that affects the assessment of our method’s practical impact, then explain our methodological and scientific contributions and address your questions.
Figure 3 does not show our final method but instead evaluates our real-only ablation. The purpose of Figure 3 is explicitly to demonstrate the limitations of training a model only on real data, which motivates the synthetic pretraining that we use in Sections 4 and 5. Our full PROFICIENT method, seen in Section 5 and Figure 1, combines synthetic pretraining with real-assay finetuning and improves over Kermut across all evaluated context sizes, including N=64 and N=128.
Furthermore, low-data prediction is practically relevant to many protein engineering settings. Previous studies have engineered fluorescent proteins and enzymes using as few as 24 experimentally measured variants [1], identified high-performing protein variants by testing only 12 candidates per round [2], and designed functional fluorescent proteins using just 64 training examples [3]. These studies highlight the practical importance of learning from small experimental datasets, particularly when measurements are expensive or time-consuming.
ProteinNPT and PRIMO do establish the use of transformers to predict fitness, but train their fitness predictors on empirical data. They do not use biologically informed synthetic fitness tasks to complement learning from real assays. Our methodological contribution is a biological task generator and a training framework that encodes domain knowledge through synthetic tasks and combines it with learning from experimental assays.
Our contributions are both methodological and empirical: we develop a data generator for protein fitness landscapes and a new framework that combines synthetic pretraining with experimental data. We also establish when and how these different training sources improve protein fitness prediction.
Methodological Novelty
We combine biological priors with experimental data for in-context fitness prediction.
We construct a novel prior distribution over protein fitness landscapes.