- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EvolveGym, a benchmark for evaluating protein engineering under limited experimental budgets, and EvolveRank, a learning-to-rank method for identifying high-fitness protein variants. EvolveGym consists of three tasks: identifying beneficial single mutations (Ladder A), generalizing from single and double mutants to higher-order variants (Ladder B), and optimizing combinatorial libraries through active learning (Lattice). EvolveRank uses frozen protein language model embeddings, a random forest trained on pairwise fitness preferences, and Swiss-system tournaments with Bradley–Terry ranking to prioritize candidates. Experiment results across different protein fitness landscapes demonstrate improved performance on single-mutant discovery, competitive results on combinatorial optimization, and mixed results on higher-order mutation prediction.
Critical strengths
The paper addresses a practically important task in protein engineering by introducing a unified benchmark that evaluates high-fitness variant under limited experimental budgets. The benchmark covers three tasks and includes extensive evaluations across different protein fitness landscapes.
Critical weaknesses
-
While the proposed EvolveGym contains three tasks covering relevant protein-engineering scenarios, their formulations closely resemble existing evaluation paradigms. For example, ProteinGym already evaluates single substitutions, overlapping with Ladder A, FLIP2 includes generalization from lower-order to higher-order mutations, overlapping with Ladder B.
-
The proposed EvolveRank appears to combine several established components: frozen ESM-2 representations, a random forest pairwise preference classifier, Swiss-system tournament comparisons, and Bradley–Terry ranking aggregation. In addition, the transformation of (n) measurements into ({n}{2}) pairwise comparisons does not increase the number of independent experimental observations. Furthermore, pairwise preference learning has already been explored in existing works, such as PRIMO [1], RankReg[2], and FSFP.
[1] Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training, NeurIPS 2025.
[2] Rank-guided learning accelerates automated enzyme engineering, Nature Communications, 2026.
Questions for authors
-
It is unclear why pairwise random forest classification followed by tournament-based aggregation is preferable to standard learning-to-rank methods that directly learn scalar ranking scores?
-
In Table 2, EvolveRank is evaluated only in the active-learning setting, while several baselines (e.g., FSFP and Hsu) are evaluated only in the few-shot setting. This makes it difficult to distinguish improvements from the proposed ranking method versus active data acquisition. Moreover, only Kermut and EVOLVEpro are included as active-learning competitors. Can the authors explain why the protocol is set like this?
-
There is no ablation of each component of the proposed EvolveRank, thus it is hard to identify the key contribution of the method.
5 comments
Response to Reviewer GJFf (Part 2/5)
Q1. Why is pairwise random forest classification followed by tournament-based aggregation preferable to standard learning-to-rank methods that directly learn scalar ranking scores?
Thank you for raising this question. We addressed it through a controlled comparison with two scalar ranking methods, using identical frozen embeddings and the same measured variants, without adaptive acquisition. These experiments directly evaluate whether EvolveRank's joint pairwise comparison and aggregation improves shortlist recovery over models that learn a scalar ranking score for each variant.
We compared EvolveRank with XGBoost (rank:pairwise, default learning rate 0.3) and the recently published RankReg from REAP (Xu et al., 2026), using α = 0.8 and α = 1.0. Both baselines learn scalar scores with ranking supervision; RankReg with α = 0.8 additionally includes a pointwise regression term. EvolveRank instead processes two variant embeddings jointly to predict their relative order.
The comparison covers 487 Ladder A datasets and 10 random seeds, using identical frozen ESM-2 650M embeddings and the same measured sets of 80, 160, or 240 randomly selected variants. Each method performs one fitting and ranking round and selects 16 variants from the same candidate pool, including training variants. EvolveRank and XGBoost fit on all measured variants; RankReg reserves 20% for early stopping.
EvolveRank achieves the highest mean performance on all five metrics at every tested budget (Table R4.1). It significantly outperforms both RankReg configurations on all five metrics at all three budgets, and XGBoost on top-5% precision, top-1% precision, Recall@16, and FoundTop1 at every budget. With only 80 measured variants, EvolveRank increases top-5% precision from 36.50% to 43.34%, Recall@16 from 10.93% to 14.42%, and FoundTop1 from 11.70% to 16.84% compared with XGBoost. Top-10% precision also improves significantly at budgets of 80 and 240; the difference at 160 measurements is not statistically significant (88.33% versus 87.96%; adjusted p = 0.093).
The Swiss tournament scheduling reduces the number of pairwise predictions from O(N²) for exhaustive comparisons to O(N log N), where N is the number of candidate variants. Bradley–Terry aggregation then estimates ranking scores from the comparison outcomes, accounting for which opponents each candidate faces. The component ablations on 12 DMS datasets (Q3, Table R4.3) support these choices: exhaustive round-robin comparisons (instead of the Swiss tournament) do not improve any reported mean metric, while replacing Bradley–Terry aggregation with raw win counts lowers all five metrics.
Together, these experiments demonstrate improved shortlist recovery over the tested scalar ranking baselines under matched measurement access, and identify the contributions of comparison scheduling and aggregation.
Table R4.1. Controlled comparison with scalar ranking methods on Ladder A. All methods use frozen ESM-2 650M embeddings and the same randomly measured sets, without adaptive acquisition. Results are averaged first over 10 seeds within each dataset and then across 487 datasets. All values are percentages; higher is better. Bold indicates the highest mean within each budget. Statistical comparisons use two-sided paired permutation tests on dataset-level means, with Holm correction across 45 comparisons (three baselines × five metrics × three budgets). All adjusted p-values are below 0.001 except for top-10% precision against XGBoost at 240 measurements (p = 0.009) and 160 measurements (p = 0.093). P16@90, P16@95, and P16@99 denote Precision@16 for the true top 10%, 5%, and 1% of variants, respectively.
80 measured variants
| Method | Precision top 10% | Prec. top 5% | Prec. top 1% | Recall@16 | FoundTop1 |
|---|---|---|---|---|---|
| RankReg, α = 1.0 | 23.41 | 14.29 | 4.10 | 5.49 | 6.49 |
| RankReg, α = 0.8 | 30.03 | 19.61 | 6.05 | 8.07 | 10.43 |
| XGBoost | 64.57 | 36.50 | 8.34 | 10.93 | 11.70 |
| EvolveRank | 68.23 | 43.34 | 11.13 | 14.42 | 16.84 |
160 measured variants
| Method | Precision top 10% | Prec. top 5% | Prec. top 1% | Recall@16 | FoundTop1 |
|---|---|---|---|---|---|
| RankReg, α = 1.0 | 29.71 | 18.90 | 5.57 | 7.52 | 9.49 |
| RankReg, α = 0.8 | 40.66 | 28.93 | 9.81 | 13.14 | 17.35 |
| XGBoost | 87.96 | 58.12 | 13.06 | 16.96 | 18.48 |
| EvolveRank | 88.33 | 63.06 | 16.09 | 20.65 | 23.61 |
240 measured variants
| Method | Precision top 10% | Prec. top 5% | Prec. top 1% | Recall@16 | FoundTop1 |
|---|---|---|---|---|---|
| RankReg, α = 1.0 | 34.41 | 22.22 | 6.78 | 9.12 | 11.81 |
| RankReg, α = 0.8 | 47.38 | 35.06 | 12.85 | 17.09 | 22.53 |
| XGBoost | 94.63 | 77.55 | 17.81 | 23.09 | 24.66 |
| EvolveRank | 95.14 | 79.43 | 20.65 | 26.47 | 29.65 |
Response to Reviewer GJFf (Part 3/5)
Q2. In Table 2, EvolveRank is evaluated only in the active-learning setting, while several baselines (e.g., FSFP and Hsu) are evaluated only in the few-shot setting. This makes it difficult to distinguish improvements from the proposed ranking method versus active data acquisition. Moreover, only Kermut and EVOLVEpro are included as active-learning competitors. Can the authors explain why the protocol is set like this?
We agree that the contribution of the ranking method should be distinguished from the benefit of adaptive acquisition. Table R4.2 directly addresses this concern: EvolveRank and the baselines receive the same 80 randomly selected measurements and nominate 16 variants from the same candidate pool, without adaptive acquisition. We will include these fixed-data EvolveRank results in the revised main Table 2.
The original separation into fixed-data and active-learning settings reflects different experimental workflows. Measuring 80 variants in one batch requires a different procedure from measuring batches of 16 and using each round's results to choose the next batch. Both use the same total measurement budget, but only the latter allows feedback between rounds.
FSFP, Hsu et al., and Kermut were initially evaluated in the fixed-data setting because they were introduced primarily for supervised fitness prediction. We selected EVOLVEpro as an established active-learning comparator and additionally evaluated Kermut in the same active loop because it achieved the highest mean top-5% Precision@16 among the original fixed-data baselines at 80 measurements. This selection is not exhaustive; FSFP and Hsu could also be adapted to active acquisition.
EvolveRank trained on 80 randomly selected variants without active acquisition was already evaluated in the appendix (Table E.1). We agree that this setting should be included in the original Table 2, and we will include it in the main comparison, using the shared random-80 splits and an identical evaluation protocol. This allows direct comparison of EvolveRank with fixed-data baselines and assessment of the additional benefit of adaptive acquisition within EvolveRank. We will clarify that the full performance gap across settings cannot be attributed solely to the ranking model.
Table R4.2. Fixed-data comparison on Ladder A. All methods use the same random-80 measured sets without adaptive acquisition and nominate 16 variants from the full candidate pool, including training variants. Results are averaged first over 10 splits within each dataset and then across 487 datasets. All metrics are percentages except UniqPos, which counts distinct mutation positions among nominations belonging to the true top 10% by activity. Higher is better. Statistical comparisons use two-sided paired Wilcoxon signed-rank tests on dataset-level means, with Bonferroni correction across all 90 pairwise method–metric comparisons and an adjusted significance threshold of 0.05.
| Method | Precision top 10% | Prec. top 5% | Prec. top 1% | Recall@16 | FoundTop1 | UniqPos |
|---|---|---|---|---|---|---|
| EvolveRank (SaProt 650M) | 70.8 | 46.6 | 12.6 | 16.3 | 19.6 | 6.7 |
| EvolveRank (ESM-2 650M) | 68.2 | 43.3 | 11.1 | 14.4 | 16.8 | 6.7 |
| Kermut | 56.6 | 40.8 | 12.7 | 16.5 | 18.7 | 4.7 |
| EVOLVEpro (ESM-2 650M) | 60.0 | 38.5 | 10.2 | 13.5 | 15.4 | 6.0 |
| FSFP | 54.4 | 38.4 | 11.6 | 15.4 | 17.0 | 4.3 |
| Hsu et al. | 56.0 | 33.7 | 8.5 | 10.9 | 11.3 | 5.7 |
EvolveRank's advantage in shortlist precision persists without adaptive acquisition. Both EvolveRank configurations significantly outperform Kermut, EVOLVEpro, FSFP, and Hsu et al. on top-10% precision, top-5% precision, and UniqPos. With the ESM-2 650M representation matched, EvolveRank improves top-5% precision over EVOLVEpro from 38.5% to 43.3%. These results support its ability to identify more high-activity single mutants across a broader set of positions for subsequent combination. Performance on the stricter recovery metrics depends on the backbone and comparator: EvolveRank–SaProt 650M shows no statistically significant differences from Kermut on top-1% precision, Recall@16, or FoundTop1; the two methods are therefore comparable at the extreme top, while EvolveRank leads on the broader top-10% and top-5% precision bands.
Response to Reviewer GJFf (Part 1/5)
We thank the reviewer for recognizing the practical importance of evaluating protein engineering under limited experimental budgets. We address the two concerns below, supported by the additional controlled comparisons and component ablations in Q1–Q3.
EvolveGym's contribution is the combined evaluation of small final nomination sets, limited experimental measurements, and explicit acquisition protocols, reflecting campaigns where only tens of variants can be assayed per round.
ProteinGym's top-decile recall evaluates a selection whose size grows with the landscape—for example, 1,000 variants in a landscape of 10,000. Recovering this broader set does not establish whether a method identifies useful candidates within a shortlist of 16. Its standard supervised protocol uses five-fold cross-validation, with label availability determined by assay size rather than a small, fixed experimental budget.
FLIP2 already tests whether models trained on variants with fewer mutations can generalize to variants with more mutations—for example, training on single and double mutants and testing on variants with three or more mutations. It evaluates predictions using Spearman correlation and NDCG. NDCG emphasizes high-fitness variants near the top and can be evaluated at a fixed cutoff; our distinction is the combined experimental protocol, rather than top-focused evaluation alone.
Specifically, EvolveGym evaluates 16 final nominations across all three tasks. Our main Ladder A and Lattice comparisons use 80 measured variants, with active campaigns acquiring batches of 16; Ladder B evaluates held-out higher-order nominations given shared single- and double-mutant measurements. The fixed-data comparisons in Q2 (Table R4.2) further distinguish performance with identical measurements from the benefit of adaptive acquisition.
Building on preference-learning approaches such as PRIMO, RankReg, and FSFP, EvolveRank provides a design tailored to scarce target-assay measurements: a random forest learns preferences jointly from pairs of frozen embeddings, Swiss scheduling selects comparisons efficiently, and Bradley–Terry aggregation produces the ranking.
PRIMO uses a transformer pretrained across different assays, with in-context learning and test-time adaptation. EvolveRank instead fits its comparator directly to the target assay's measurements on frozen embeddings, without cross-assay comparator pretraining or protein-language-model fine-tuning.
Q1 (Table R4.1) tests this design against scalar ranking methods using identical embeddings and measured sets. EvolveRank significantly outperforms both RankReg configurations on all five metrics at all three budgets, and XGBoost on top-5% precision, top-1% precision, Recall@16, and FoundTop1 at every budget.
The ablations in Q3 (Table R4.3) support the individual design choices: Bradley–Terry aggregation outperforms raw win counts on all five mean metrics, and concatenated embeddings outperform difference features. Exhaustive round-robin comparisons do not improve the reported means over the more efficient Swiss schedule.
We agree that constructing pairs does not create additional independent experimental observations: the measurement budget remains n. The pairwise formulation learns from the relative ordering implied by those measurements. We will clarify this distinction and strengthen the discussion of the cited prior work.