- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses protein optimization under a limited experimental budget, in the common setting where measurements arrive in sequential batches, and each completed batch is used to train a model that selects the variants to assay next.
The authors' central premise is that a selection model for this setting should combine two components: (i) a prior-informed predictor that encodes knowledge available before any task-specific data is collected, and (ii) a task-specific predictor trained on the experimental measurements gathered so far. Crucially, they argue that the relative influence of these two components should not be fixed in advance but re-estimated at every round.
To this end, they propose Batch-Aligned Tail Arbitration (BATA), a framework that recalibrates a single mixing weight w ∈ [0, 1] to balance the prior-informed and task-specific predictors, updating w at each round as new batch measurements become available.
BATA is evaluated primarily on three datasets—GB1, PABP, and TrpB—with supplementary results also on HIS7 and GRB2. Performance is reported using two metrics, Final@480 and Query-AUC. The headline result is that BATA achieves the best mean Final@480 rank averaged across the three main tasks.
Critical strengths
-
Well-motivated problem. The paper clearly explains the motivation and establishes the importance of the problem, identifying a specific gap that the proposed method aims to address.
-
Well-structured related work. The related work places the contribution in the context of existing approaches and clearly explains how the proposed method differs from prior work.
-
Strong and up-to-date baselines. The evaluation includes recent, competitive methods (ALDE, EVOLVEpro, and REAP100), providing a relevant basis for comparison.
Critical weaknesses
-
Limited number of evaluation tasks for the headline results. The main results are evaluated on only three protein tasks, and BATA does not win on all three. This is a narrow basis for the paper's central claims. The evaluation would be substantially more convincing with additional targets — the authors already use HIS7 and GRB2 elsewhere (e.g., Figure A6), so extending the headline experiment to those seems feasible. Further targets could be drawn from ProteinGym (Notin et al., 2023) or Li et al. (2025).
-
The headline results themselves are not clearly convincing. Reading Figure 2: on GB1, BATA ranks third; on PABP, the confidence intervals appear to overlap across methods; on TrpB, BATA wins, but it is not possible to tell from the figure whether the margin over competing methods is statistically significant. As presented, the three tasks do not clearly establish an advantage for the method.
-
No statistical significance reported for the main results. Table 1 reports mean ± standard deviation, and Figure 2 reports pointwise 95% bootstrap confidence intervals without significance testing, making the comparisons hard to interpret.
-
Task-specific predictors differ across tasks, with no guidance for new targets. Per Table A1, the predictor setup varies by task: one-hot + ridge + Thompson on GB1, ESM2 + single RF + greedy on PABP, and ESM2 + five forests + Thompson on TrpB. It is unclear how a practitioner with a new protein should choose a setup — the paper does not provide a selection rule or default recommendation. This weakens the method's practical applicability.
-
(Minor) Inconsistent number of initialization runs, unexplained. The headline results use 35 initializations, while the initialization-sensitivity results in Table A3 use 35 and 70, without explanation for the difference.
-
(Minor) Reporting only top fitness underuses the measured data. Each batch measures 96 variants, but the paper reports only the top fitness, which characterizes a single variant. Additional metrics for the 96 measured variants per batch (e.g., mean, recall) would provide more information about the method's behavior.
References: Pascal Notin, Aaron W. Kollasch, Daniel Ritter, Lood Van Niekerk, Steffanie Paul, Hansen Spinner, Nathan Rollins, Ada Shaw, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Rose Orenbuch, Yarin Gal, and Debora S. Marks. ProteinGym: Large- Scale Benchmarks for Protein Design and Fitness Prediction, December 2023. URL http://biorxiv.org/lookup/doi/10.1101/2023.12.07.570727.
Francesca-Zhoufan Li, Jason Yang, Kadina E. Johnston, Emre Gürsoy, Yisong Yue, and Frances H. Arnold. Evaluation of machine learning-assisted directed evolution across diverse combinatorial landscapes. Cell Systems, 16(9):101387, September 2025. ISSN 24054712. doi: 10.1016/j.cels.2025.101387. URL https://linkinghub.elsevier.com/ retrieve/pii/S2405471225002200.
Questions for authors
-
Why is the main evaluation limited to three tasks? Can you run the headline experiment on additional targets—e.g., HIS7 and GRB2 (already used in the supplementary material) —and/or additional proteins from ProteinGym or Li et al. (2025) to make the aggregate rank more robust?
-
Can you report paired significance tests for the main results (Table 1, Figure 2)?
-
For a new protein with no established dataset, how should a user choose the representation, task-specific model, and selection rule — is there a recommended default?
-
What accounts for the different number of initializations between the headline results (35) and the initialization-sensitivity results (35 and 70)?
-
Final@480 and Query-AUC both reduce a batch to its single best variant. Can you report distributional metrics over the 96 variants selected per batch (mean/median fitness, precision@k)?
======== Thanks to the authors for providing their response. However, it does not seem the authors have updated the submission with highlighted changes. From the responses it is hard to recover if there are any updates to the paper. I tried to check it but could not easily find it. In general, it would be great if the authors provide more analysis of the obtained results. At the moment is compact and hard to disentangle what is going on.
The response addresses some of the concerns (hence the score is updated to 3), but the major issues about an overall limited number of targets/landscapes and not fully convincing/conclusive results remain.
3 comments
C4 / Q3 — Configuration for a new protein. Table A1 lists the configuration used for each task. We will add to Appendix A.2 the inputs each configuration requires, and the PABP recipe applied unchanged to GRB2 (DCA Ridge5 prior, ESM2-650M random forest with 100 trees, greedy selection, with GRB2-specific features) as a reproducible starting point. Any choice among representations, model sizes, or selectors should rely on labels already available to the practitioner under a fixed validation rule.
C5 / Q4 — Cohorts of 35 and 70. Main-35 combines 20 initialization sets with a 15-set expansion. The 70-initialization bank for GB1 and TrpB is a separate seed list, not an extension of Main-35, and PABP keeps its original 35. We report the cohorts separately; in Figure 3 the lines connect cohort-level summaries, not paired differences. We will label the cohort and sample size in every table and caption, and document both constructions in Appendix A.3 (see also our reply to Reviewer x2Jk, Q2).
C6 / Q5 — Batch-level metrics. Final@480 and Query-AUC describe discovery of the best variant; batch mean, median and top-10 mean (the mean of the ten highest-fitness variants in each batch, which we added as a tail metric following your request for distributional measures) describe the selected batch. We computed them over the 2,100 newly selected batches of the 525 Main-35 campaigns (the shared initial batch excluded; the unit of replication is the campaign). Table R2 gives all five methods and Table R3 the paired top-10 differences. Relative to ALDE, BATA's batch mean and median are comparable on PABP and TrpB and lower on GB1; its top-10 mean is higher on PABP, comparable on TrpB, and lower on GB1. We will add these tables, with the paired differences, to the appendix. We do not report precision@k, because it requires a relevance set defined for each candidate pool.
Table R2. Batch fitness over the four newly selected batches: mean / median / top-10 mean, averaged over 35 campaigns (native fitness units of each task).
| Method | GB1 | PABP | TrpB |
|---|---|---|---|
| BATA | 0.2065 / 0.1728 / 0.5421 | 0.9563 / 0.9692 / 1.3967 | 0.4011 / 0.4254 / 0.7163 |
| ALDE | 0.2898 / 0.2867 / 0.6101 | 0.9475 / 0.9644 / 1.3242 | 0.4035 / 0.4046 / 0.7050 |
| EVOLVEpro-650M | 0.1910 / 0.1640 / 0.4821 | 0.9449 / 0.9628 / 1.3815 | 0.3551 / 0.3660 / 0.6837 |
| REAP100-650M | 0.2880 / 0.2817 / 0.5892 | 0.9721 / 0.9793 / 1.4020 | 0.4073 / 0.4280 / 0.6918 |
| RF | 0.2071 / 0.1712 / 0.5302 | 0.9148 / 0.9406 / 1.3729 | 0.3136 / 0.2957 / 0.6720 |
Table R3. Top-10 batch mean, BATA minus baseline: mean paired difference [95% bootstrap CI], 35 matched initializations, 20,000 resamples, no multiplicity correction.
| Baseline | GB1 | PABP | TrpB |
|---|---|---|---|
| ALDE | −0.0680 [−0.0854, −0.0501] | +0.0725 [+0.0456, +0.0979] | +0.0114 [−0.0041, +0.0264] |
| EVOLVEpro-650M | +0.0601 [+0.0338, +0.0881] | +0.0152 [−0.0108, +0.0428] | +0.0327 [+0.0179, +0.0482] |
| REAP100-650M | −0.0470 [−0.0689, −0.0246] | −0.0054 [−0.0253, +0.0138] | +0.0245 [+0.0044, +0.0458] |
| RF | +0.0120 [−0.0039, +0.0272] | +0.0238 [+0.0023, +0.0455] | +0.0443 [+0.0234, +0.0656] |
Major issues about an overall limited number of targets/landscapes and not fully conclusive results remain
I thank the authors for providing the response. However, it does not seem the authors have updated the submission with the highlighted changes. From the responses it is hard to recover if there are any updates to the paper. I tried to check it but could not easily find it.
The response in the discussion addresses some of the concerns (hence the score is updated from strong reject to weak reject), but the major issues about an overall limited number of targets/landscapes and not fully conclusive results remain. It would be also great if the authors provide more analysis of the obtained results.
Response to Reviewer W6Qt
Thank you for separating endpoint performance, statistical uncertainty, practical configuration, and batch-level quality. Part 1 of 2 answers C1–C3; part 2 answers C4–C6.
C1 / Q1 — Additional targets. Following your suggestion, we will add HIS7 and GRB2 to the headline evaluation. Both have complete five-method campaigns (24 matched initializations per task, identical initial sets across methods, 480 measurements each). On Final@480, BATA ranks third on HIS7 and second on GRB2. We will report each with its own sample size and provenance beside the Main-35 table in the revised paper. Evaluating further proteins from ProteinGym or Li et al. (2025) requires a fixed protocol of its own.
C2 — Task-level comparisons. Overlapping pointwise bands do not determine the uncertainty of a paired difference, so we report the paired differences directly (Table R1; BATA minus baseline, n = 35 matched initializations; Holm-adjusted p). On GB1, BATA trails ALDE on Final@480 (−0.0835, p = 0.017). On PABP, BATA improves over ALDE, REAP100 and RF (p < .001 each) but not over EVOLVEpro (+0.0494, p = 0.804). On TrpB, none of the four Final@480 comparisons is significant after correction. We will add these differences to Table 1 and Figure 2, and state the aggregate result as a mean rank over the three tasks.
C3 / Q2 — Paired significance tests. Table R1 gives them for all Main-35 comparisons: 20,000-resample paired bootstrap intervals and two-sided sign-flip tests (100,000 draws) on the BATA-minus-baseline difference, with Holm correction over the 12 comparisons for each metric. We will add the table to the paper and the method to Appendix A.4. These are paired analyses of the evaluated tasks and initializations.
Table R1. Paired differences, BATA minus baseline: mean [95% bootstrap CI] and Holm-adjusted p.