acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Directed-Evolution Benchmarks: Top-Variant Recovery Under a Fixed Budget

Abstract

Machine learning-guided directed evolution improves a protein over successive rounds of experimental screening. When variants must be assayed individually, as is common for catalytic activity, each round tests only tens of variants. Existing benchmarks such as ProteinGym reward ordering an entire mutational landscape rather than recovering the best variants under such a budget. We introduce **EvolveGym**, a benchmark of measured deep mutational scanning data that scores top-variant recovery under a fixed small screening budget, over three tasks drawn from practice: identifying beneficial single mutations, assembling higher-order multi-mutants, and searching a site-saturated library. It spans 487 single-substitution assays, 96 of them functional rather than stability assays, 44 multi-mutant landscapes, and 15 combinatorially complete enzyme-activity and binding landscapes. We further argue that variant selection is better posed as ranking variants against one another than as regressing their measured values, which are noisy and condition-dependent, and propose **EvolveRank**: it turns measurements into all pairwise comparisons and aggregates a learned preference classifier into a global ranking. Benchmarking 17 methods at a matched budget of 80 measurements, EvolveRank places 77.5% of its 16 nominations in the true top 5% of each single-mutation assay, against 41.0% for the strongest few-shot supervised baseline; with representation and budget held fixed, ranking adds nine points of precision over regression. On multi-mutant landscapes the winner is predictable from the landscape itself: additive models lead under near-additivity, EvolveRank under strong epistasis, and an ensemble of the two needs no prior knowledge of the regime.

Primary area:
datasets and benchmarks
Keywords:
machine learning-guided directed evolution, benchmark, protein engineering, active learning, pairwise ranking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_GJFfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvolveGym, a benchmark for evaluating protein engineering under limited experimental budgets, and EvolveRank, a learning-to-rank method for identifying high-fitness protein variants. EvolveGym consists of three tasks: identifying beneficial single mutations (Ladder A), generalizing from single and double mutants to higher-order variants (Ladder B), and optimizing combinatorial libraries through active learning (Lattice). EvolveRank uses frozen protein language model embeddings, a random forest trained on pairwise fitness preferences, and Swiss-system tournaments with Bradley–Terry ranking to prioritize candidates. Experiment results across different protein fitness landscapes demonstrate improved performance on single-mutant discovery, competitive results on combinatorial optimization, and mixed results on higher-order mutation prediction.

Critical strengths

The paper addresses a practically important task in protein engineering by introducing a unified benchmark that evaluates high-fitness variant under limited experimental budgets. The benchmark covers three tasks and includes extensive evaluations across different protein fitness landscapes.

Critical weaknesses
  1. While the proposed EvolveGym contains three tasks covering relevant protein-engineering scenarios, their formulations closely resemble existing evaluation paradigms. For example, ProteinGym already evaluates single substitutions, overlapping with Ladder A, FLIP2 includes generalization from lower-order to higher-order mutations, overlapping with Ladder B.

  2. The proposed EvolveRank appears to combine several established components: frozen ESM-2 representations, a random forest pairwise preference classifier, Swiss-system tournament comparisons, and Bradley–Terry ranking aggregation. In addition, the transformation of (n) measurements into ({n}{2}) pairwise comparisons does not increase the number of independent experimental observations. Furthermore, pairwise preference learning has already been explored in existing works, such as PRIMO [1], RankReg[2], and FSFP.

[1] Few-shot Protein Fitness Prediction via In-context Learning and Test-time Training, NeurIPS 2025.

[2] Rank-guided learning accelerates automated enzyme engineering, Nature Communications, 2026.

Questions for authors
  1. It is unclear why pairwise random forest classification followed by tournament-based aggregation is preferable to standard learning-to-rank methods that directly learn scalar ranking scores?

  2. In Table 2, EvolveRank is evaluated only in the active-learning setting, while several baselines (e.g., FSFP and Hsu) are evaluated only in the few-shot setting. This makes it difficult to distinguish improvements from the proposed ranking method versus active data acquisition. Moreover, only Kermut and EVOLVEpro are included as active-learning competitors. Can the authors explain why the protocol is set like this?

  3. There is no ablation of each component of the proposed EvolveRank, thus it is hard to identify the key contribution of the method.

5 comments
AuthorsOpenReview ↗

Response to Reviewer GJFf (Part 1/5)

We thank the reviewer for recognizing the practical importance of evaluating protein engineering under limited experimental budgets. We address the two concerns below, supported by the additional controlled comparisons and component ablations in Q1–Q3.

W1. Overlap of EvolveGym's tasks with existing benchmarks, particularly ProteinGym and FLIP2.

EvolveGym's contribution is the combined evaluation of small final nomination sets, limited experimental measurements, and explicit acquisition protocols, reflecting campaigns where only tens of variants can be assayed per round.

ProteinGym's top-decile recall evaluates a selection whose size grows with the landscape—for example, 1,000 variants in a landscape of 10,000. Recovering this broader set does not establish whether a method identifies useful candidates within a shortlist of 16. Its standard supervised protocol uses five-fold cross-validation, with label availability determined by assay size rather than a small, fixed experimental budget.

FLIP2 already tests whether models trained on variants with fewer mutations can generalize to variants with more mutations—for example, training on single and double mutants and testing on variants with three or more mutations. It evaluates predictions using Spearman correlation and NDCG. NDCG emphasizes high-fitness variants near the top and can be evaluated at a fixed cutoff; our distinction is the combined experimental protocol, rather than top-focused evaluation alone.

Specifically, EvolveGym evaluates 16 final nominations across all three tasks. Our main Ladder A and Lattice comparisons use 80 measured variants, with active campaigns acquiring batches of 16; Ladder B evaluates held-out higher-order nominations given shared single- and double-mutant measurements. The fixed-data comparisons in Q2 (Table R4.2) further distinguish performance with identical measurements from the benefit of adaptive acquisition.

W2. Novelty of EvolveRank, its relationship to prior preference-learning methods, and the dependence between constructed pairs.

Building on preference-learning approaches such as PRIMO, RankReg, and FSFP, EvolveRank provides a design tailored to scarce target-assay measurements: a random forest learns preferences jointly from pairs of frozen embeddings, Swiss scheduling selects comparisons efficiently, and Bradley–Terry aggregation produces the ranking.

PRIMO uses a transformer pretrained across different assays, with in-context learning and test-time adaptation. EvolveRank instead fits its comparator directly to the target assay's measurements on frozen embeddings, without cross-assay comparator pretraining or protein-language-model fine-tuning.

Q1 (Table R4.1) tests this design against scalar ranking methods using identical embeddings and measured sets. EvolveRank significantly outperforms both RankReg configurations on all five metrics at all three budgets, and XGBoost on top-5% precision, top-1% precision, Recall@16, and FoundTop1 at every budget.

The ablations in Q3 (Table R4.3) support the individual design choices: Bradley–Terry aggregation outperforms raw win counts on all five mean metrics, and concatenated embeddings outperform difference features. Exhaustive round-robin comparisons do not improve the reported means over the more efficient Swiss schedule.

We agree that constructing pairs does not create additional independent experimental observations: the measurement budget remains nn. The pairwise formulation learns from the relative ordering implied by those measurements. We will clarify this distinction and strengthen the discussion of the cited prior work.

AuthorsOpenReview ↗

Response to Reviewer GJFf (Part 2/5)

Q1. Why is pairwise random forest classification followed by tournament-based aggregation preferable to standard learning-to-rank methods that directly learn scalar ranking scores?

Thank you for raising this question. We addressed it through a controlled comparison with two scalar ranking methods, using identical frozen embeddings and the same measured variants, without adaptive acquisition. These experiments directly evaluate whether EvolveRank's joint pairwise comparison and aggregation improves shortlist recovery over models that learn a scalar ranking score for each variant.

We compared EvolveRank with XGBoost (rank:pairwise, default learning rate 0.3) and the recently published RankReg from REAP (Xu et al., 2026), using α = 0.8 and α = 1.0. Both baselines learn scalar scores with ranking supervision; RankReg with α = 0.8 additionally includes a pointwise regression term. EvolveRank instead processes two variant embeddings jointly to predict their relative order.

The comparison covers 487 Ladder A datasets and 10 random seeds, using identical frozen ESM-2 650M embeddings and the same measured sets of 80, 160, or 240 randomly selected variants. Each method performs one fitting and ranking round and selects 16 variants from the same candidate pool, including training variants. EvolveRank and XGBoost fit on all measured variants; RankReg reserves 20% for early stopping.

EvolveRank achieves the highest mean performance on all five metrics at every tested budget (Table R4.1). It significantly outperforms both RankReg configurations on all five metrics at all three budgets, and XGBoost on top-5% precision, top-1% precision, Recall@16, and FoundTop1 at every budget. With only 80 measured variants, EvolveRank increases top-5% precision from 36.50% to 43.34%, Recall@16 from 10.93% to 14.42%, and FoundTop1 from 11.70% to 16.84% compared with XGBoost. Top-10% precision also improves significantly at budgets of 80 and 240; the difference at 160 measurements is not statistically significant (88.33% versus 87.96%; adjusted p = 0.093).

The Swiss tournament scheduling reduces the number of pairwise predictions from O(N²) for exhaustive comparisons to O(N log N), where N is the number of candidate variants. Bradley–Terry aggregation then estimates ranking scores from the comparison outcomes, accounting for which opponents each candidate faces. The component ablations on 12 DMS datasets (Q3, Table R4.3) support these choices: exhaustive round-robin comparisons (instead of the Swiss tournament) do not improve any reported mean metric, while replacing Bradley–Terry aggregation with raw win counts lowers all five metrics.

Together, these experiments demonstrate improved shortlist recovery over the tested scalar ranking baselines under matched measurement access, and identify the contributions of comparison scheduling and aggregation.

Table R4.1. Controlled comparison with scalar ranking methods on Ladder A. All methods use frozen ESM-2 650M embeddings and the same randomly measured sets, without adaptive acquisition. Results are averaged first over 10 seeds within each dataset and then across 487 datasets. All values are percentages; higher is better. Bold indicates the highest mean within each budget. Statistical comparisons use two-sided paired permutation tests on dataset-level means, with Holm correction across 45 comparisons (three baselines × five metrics × three budgets). All adjusted p-values are below 0.001 except for top-10% precision against XGBoost at 240 measurements (p = 0.009) and 160 measurements (p = 0.093). P16@90, P16@95, and P16@99 denote Precision@16 for the true top 10%, 5%, and 1% of variants, respectively.

80 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.023.4114.294.105.496.49
RankReg, α = 0.830.0319.616.058.0710.43
XGBoost64.5736.508.3410.9311.70
EvolveRank68.2343.3411.1314.4216.84

160 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.029.7118.905.577.529.49
RankReg, α = 0.840.6628.939.8113.1417.35
XGBoost87.9658.1213.0616.9618.48
EvolveRank88.3363.0616.0920.6523.61

240 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.034.4122.226.789.1211.81
RankReg, α = 0.847.3835.0612.8517.0922.53
XGBoost94.6377.5517.8123.0924.66
EvolveRank95.1479.4320.6526.4729.65
AuthorsOpenReview ↗

Response to Reviewer GJFf (Part 3/5)

Q2. In Table 2, EvolveRank is evaluated only in the active-learning setting, while several baselines (e.g., FSFP and Hsu) are evaluated only in the few-shot setting. This makes it difficult to distinguish improvements from the proposed ranking method versus active data acquisition. Moreover, only Kermut and EVOLVEpro are included as active-learning competitors. Can the authors explain why the protocol is set like this?

We agree that the contribution of the ranking method should be distinguished from the benefit of adaptive acquisition. Table R4.2 directly addresses this concern: EvolveRank and the baselines receive the same 80 randomly selected measurements and nominate 16 variants from the same candidate pool, without adaptive acquisition. We will include these fixed-data EvolveRank results in the revised main Table 2.

The original separation into fixed-data and active-learning settings reflects different experimental workflows. Measuring 80 variants in one batch requires a different procedure from measuring batches of 16 and using each round's results to choose the next batch. Both use the same total measurement budget, but only the latter allows feedback between rounds.

FSFP, Hsu et al., and Kermut were initially evaluated in the fixed-data setting because they were introduced primarily for supervised fitness prediction. We selected EVOLVEpro as an established active-learning comparator and additionally evaluated Kermut in the same active loop because it achieved the highest mean top-5% Precision@16 among the original fixed-data baselines at 80 measurements. This selection is not exhaustive; FSFP and Hsu could also be adapted to active acquisition.

EvolveRank trained on 80 randomly selected variants without active acquisition was already evaluated in the appendix (Table E.1). We agree that this setting should be included in the original Table 2, and we will include it in the main comparison, using the shared random-80 splits and an identical evaluation protocol. This allows direct comparison of EvolveRank with fixed-data baselines and assessment of the additional benefit of adaptive acquisition within EvolveRank. We will clarify that the full performance gap across settings cannot be attributed solely to the ranking model.

Table R4.2. Fixed-data comparison on Ladder A. All methods use the same random-80 measured sets without adaptive acquisition and nominate 16 variants from the full candidate pool, including training variants. Results are averaged first over 10 splits within each dataset and then across 487 datasets. All metrics are percentages except UniqPos, which counts distinct mutation positions among nominations belonging to the true top 10% by activity. Higher is better. Statistical comparisons use two-sided paired Wilcoxon signed-rank tests on dataset-level means, with Bonferroni correction across all 90 pairwise method–metric comparisons and an adjusted significance threshold of 0.05.

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1UniqPos
EvolveRank (SaProt 650M)70.846.612.616.319.66.7
EvolveRank (ESM-2 650M)68.243.311.114.416.86.7
Kermut56.640.812.716.518.74.7
EVOLVEpro (ESM-2 650M)60.038.510.213.515.46.0
FSFP54.438.411.615.417.04.3
Hsu et al.56.033.78.510.911.35.7

EvolveRank's advantage in shortlist precision persists without adaptive acquisition. Both EvolveRank configurations significantly outperform Kermut, EVOLVEpro, FSFP, and Hsu et al. on top-10% precision, top-5% precision, and UniqPos. With the ESM-2 650M representation matched, EvolveRank improves top-5% precision over EVOLVEpro from 38.5% to 43.3%. These results support its ability to identify more high-activity single mutants across a broader set of positions for subsequent combination. Performance on the stricter recovery metrics depends on the backbone and comparator: EvolveRank–SaProt 650M shows no statistically significant differences from Kermut on top-1% precision, Recall@16, or FoundTop1; the two methods are therefore comparable at the extreme top, while EvolveRank leads on the broader top-10% and top-5% precision bands.

Reviewer_xHEhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper introduces EvolveGym, a benchmark focused on selecting a small number of high-fitness protein variants, and EvolveRank, a method that learns pairwise preferences from frozen protein language model embeddings. EvolveGym includes 487 single-substitution assays, 44 higher-order landscapes, and 15 site-saturated landscapes. EvolveRank trains a random-forest comparator, runs a Swiss-system tournament, and aggregates the comparisons using a Bradley–Terry model.

Critical strengths
  1. The benchmark measures an outcome that matters in practice. Whole-landscape correlation does not necessarily indicate whether a method can identify a useful batch of variants. Evaluating a small nomination set under limited measurements is therefore well motivated. Combining single-substitution screening, higher-order prediction, and site-saturated search in one framework is useful, particularly given the breadth of the single-mutant collection. I consider this benchmark contribution the strongest part of the paper.

  2. The single-mutant improvement over EVOLVEpro is substantial and supported by useful controls. With ESM-2 15B embeddings and the same measurement budget, EvolveRank improves top-5% Precision@16 from 62.5% to 71.6%, and FoundTop1 from 23.2% to 28.5%. The active Kermut comparison and acquisition ablations in Appendix E.1 are valuable because they help explain where the gains come from. The category-specific and family-level analyses also make the evaluation more informative than a single aggregate result.

Critical weaknesses
  1. Some details needed to verify the fixed-budget comparison are missing. For the Lattice ensembles, the paper states that two rankings are merged, but does not explain whether the models share one 80-measurement acquisition trajectory. Independently acquired trajectories could provide the ensemble with more measured variants than an individual method. Lattice also ranks the full combinatorial library despite measured coverage being as low as 93%, without specifying how proposals with missing fitness are handled. For Ladder B, the number of training measurements is determined by the sampled mutation pool, but its distribution is not reported. These issues may be resolved through clarification. However, experimental cost is central to the benchmark’s motivation, so they affect my assessment of the main contribution. The distinction between final recommendations and variants actually measured within the budget should also be made explicit.
Questions for authors
  1. Budget and evaluation clarification: Do both members of each Lattice ensemble train on exactly the same 80 measured variants, with a shared acquisition rule? How are proposed variants without recorded fitness handled? Please also report the training-set sizes for Ladder B and the fraction of final nominations that have already been measured.

  2. Contribution clarification: What does the joint pairwise classifier and tournament add over a scalar predictor trained with a ranking loss on the same frozen embeddings? Is there a controlled comparison that holds the measured variants and representation fixed? This would help distinguish the contribution of the proposed architecture from the benefit of ranking supervision more generally.

  3. Baseline clarification: Why does Kermut trained on 80 random measurements achieve 41.0% top-5% precision in Table 2 but 39.2% in Table E.1? Are the sampled labels, representation, zero-shot prior, and fitting settings identical? Please identify the configuration difference or reconcile the two results.

7 comments
AuthorsOpenReview ↗

Response to Reviewer xHEh (Part 1/4)

We thank the reviewer for their careful and constructive feedback. The questions on budget accounting, the handling of proposals without recorded fitness, the Ladder B training-set sizes, and the Kermut baseline discrepancy helped us clarify our experimental protocol and correct an inconsistency in the manuscript. We address each point in detail below.

W1. Clarifying ensemble budgets, missing fitness values, Ladder B training sizes, and final nominations

Both members of each Lattice ensemble share the same measured variants and acquisition decisions, using one total budget of 80 measurements. We clarify how proposals without recorded fitness are handled and report the distribution of Ladder B training-set sizes (median is 88 variants). We also distinguish measurements acquired during a campaign from the final 16 nominations: these recommendations may include previously measured variants in Ladder A and Lattice, while Ladder B nominations are entirely held out from training. Q1 below provides the detailed protocols and statistics, which we will incorporate into the revised manuscript.

Q1. Budget and evaluation clarification. Do both members of each Lattice ensemble train on exactly the same 80 measured variants, with a shared acquisition rule?

The two models that make up the ensemble are trained on the same measured variants at every round. Both start with the same 16 randomly selected variants and rank all eligible candidates. For each acquisition round, we remove previously measured variants from both rankings. We then assign each remaining candidate its better rank across the two models—for example, a candidate ranked 3rd by one model and 10th by the other receives rank 3. We select the top 16 distinct candidates from this combined ranking, measure them once, and add their measurements to the training data of both models. We repeat this process four times, acquiring 64 additional measurements, for a total experimental budget of 16 + 4 × 16 = 80 measurements per ensemble. We will include this explanation in the updated manuscript.

How are proposed variants without recorded fitness handled?

Across Ladder A, Ladder B, and Lattice, all methods other than ERA and DPO select their evaluated nominations only from variants with recorded experimental fitness. Some methods, such as the zero-shot MULTI-evolve PLM ensemble baseline on Ladder A, also score variants outside this pool; however, these are excluded before selecting the top 16 for evaluation.

ERA and DPO can sometimes generate variants with no recorded fitness in the dataset. Each such proposal uses one of the 16 available slots in that round and is not replaced. During training, we assign it a penalty energy of 10, as implemented in the original ERA code, to penalize such proposals during subsequent training. During evaluation, it earns no credit toward Precision@16, whose denominator remains 16. This treatment reflects missing data, not experimental evidence that the variant is inactive.

AuthorsOpenReview ↗

Response to Reviewer xHEh (Part 2/4)

Please also report the training-set sizes for Ladder B and the fraction of final nominations that have already been measured.

Ladder B uses fixed training sets rather than an 80-measurement acquisition campaign. Models are trained on single and double mutants and nominate candidates from held-out variants with at least three mutations to test their ability to generalize from low-order mutants to higher-order ones. Across 44 datasets and 10 splits per dataset, the median training-set size is 88 variants, the mean is 181.43, the interquartile range is 63–233.25, and the range is 33–1,066. The training and nomination pools are disjoint, so 0% of final nominations belong to the training set.

In the active-learning experiments on Ladder A and Lattice, each acquisition round selects 16 new, previously unmeasured variants. After the total acquisition budget reaches 80 measurements, the final 16 nominations are selected from the full pool of eligible candidates, including variants already measured during the campaign. Final nominations can therefore contain both previously measured variants and new candidates. The budget counts acquisition slots, including the initial measurements. For ERA and DPO, a proposal without recorded fitness still consumes a slot but supplies no experimental fitness value. Scoring the final recommendations does not provide additional measurements to the model.

The fraction of final nominations already measured within that budget is:

  • Ladder A: EVOLVEpro 96.4%; Kermut 94.5%; EvolveRank [SaProt‑1.3B: 94.1%; SaProt‑650M: 94.0%; ESM‑2 15B: 94.0%; ESM‑2 650M: 93.2%], averaged over simulations within each dataset and then across datasets.
  • Lattice: EVOLVEpro 82.4%; EvolveRank–ESM‑2 15B 76.7%; Kermut 80.7%; ALDE 99.1%; and 69.2–78.2% across ensembles, with equal weight assigned to each protein family.

We will incorporate these protocol clarifications and statistics into the revised manuscript.

Q2. Contribution clarification. What does the joint pairwise classifier and tournament add over a scalar predictor trained with a ranking loss on the same frozen embeddings? Is there a controlled comparison that holds the measured variants and representation fixed? This would help distinguish the contribution of the proposed architecture from the benefit of ranking supervision more generally.

Thank you for raising this question. Our design was motivated by an enzyme-optimization project in which the assay allowed only 15 variants per round. Absolute activity values varied substantially across repeated measurements, but the experimentalists could still rank the variants. We therefore chose to learn pairwise preferences directly.

We addressed this question through a controlled comparison with XGBoost (rank:pairwise, learning rate 0.3) and RankReg from REAP (Xu et al., 2026), using α = 0.8 and α = 1.0. Both methods learn scalar ranking scores; RankReg with α = 0.8 additionally includes a pointwise regression term. EvolveRank instead processes two variant embeddings jointly to predict their relative order. The comparison tests whether this design improves shortlist recovery beyond the use of ranking supervision alone.

The experiments cover 487 Ladder A datasets and 10 random seeds, using identical frozen ESM-2 650M embeddings and the same randomly measured sets of 80, 160, or 240 variants, without adaptive acquisition. Each method performs one fitting and ranking round and nominates 16 variants from the same candidate pool, including training variants. EvolveRank and XGBoost fit on all measured variants; RankReg reserves 20% for early stopping.

AuthorsOpenReview ↗

Response to Reviewer xHEh (Part 3/4)

Table R3.1. Controlled comparison with scalar ranking methods on Ladder A. Results are averaged first over 10 seeds within each dataset and then across 487 datasets. All values are percentages; higher is better. Bold indicates the highest mean within each budget and metric. Statistical comparisons use two-sided paired permutation tests on dataset-level means, with Holm correction across 45 comparisons: three baseline configurations × five metrics × three budgets. All adjusted p-values are below 0.001 except for top-10% precision against XGBoost at 240 measurements (p = 0.009) and 160 measurements (p = 0.093).

80 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.023.4114.294.105.496.49
RankReg, α = 0.830.0319.616.058.0710.43
XGBoost64.5736.508.3410.9311.70
EvolveRank68.2343.3411.1314.4216.84

160 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.029.7118.905.577.529.49
RankReg, α = 0.840.6628.939.8113.1417.35
XGBoost87.9658.1213.0616.9618.48
EvolveRank88.3363.0616.0920.6523.61

240 measured variants

MethodPrecision top 10%Prec. top 5%Prec. top 1%Recall@16FoundTop1
RankReg, α = 1.034.4122.226.789.1211.81
RankReg, α = 0.847.3835.0612.8517.0922.53
XGBoost94.6377.5517.8123.0924.66
EvolveRank95.1479.4320.6526.4729.65

EvolveRank achieves the highest mean performance on all five metrics at every tested budget. It significantly outperforms both RankReg configurations throughout, and XGBoost on all comparisons except top-10% precision at 160 measurements. With only 80 measured variants, EvolveRank improves top-5% precision over XGBoost from 36.50% to 43.34%, Recall@16 from 10.93% to 14.42%, and FoundTop1 from 11.70% to 16.84%. These results establish an advantage over the tested scalar ranking methods with representation and measurement access held fixed.

The Swiss tournament reduces computation, requiring O(N log N) pairwise predictions instead of O(N²) exhaustive comparisons for N candidates. Bradley–Terry aggregation estimates ranking scores from these outcomes, accounting for which opponents each candidate faces.

Complementary component ablations on 12 DMS datasets, with ten simulations per dataset and 80 measurements plus wild-type (WT) as an additional reference, provide further evidence. The complete ablation definitions, results, and discussion are provided in our response to Reviewer GJFf, Q3 (Table R4.3). We summarize the main findings below.

  • Aggregation: replacing Bradley–Terry with raw win counts lowers all five reported means; top-10% precision decreases from 83.18% to 70.73%.
  • Pair representation: with both training orientations included in both configurations, concatenated embeddings achieve 82.14% top-10% precision, compared with 66.35% for difference features.
  • Comparison scheduling: exhaustive round-robin comparisons do not improve any reported mean metric over our Swiss scheduling.

These ablations start from matched initial measurements but allow subsequent acquisitions to differ, so they evaluate the effects of component changes over an active-learning campaign. They complement the fixed-data comparison above by identifying which design choices contribute to the complete pipeline.

Reviewer_6EtvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present two contributions in their paper: An ML-guided directed evolution benchmark, EvolveGym, and a method, EvolveRank, that identifies the best variants of a protein - given a small number of wet lab measurements and a pretrained protein language model. The authors argue that existing benchmarks are not representative of the iterative topk selection normally employed under a fixed screening budget, and hence the existing benchmarks that reward ordering of an entire landscape are not fit for the directed evolution-setting. For their benchmark proposal, EvolveGym, the authors combine data from seven source studies and further partition them into three tasks: (Ladder A) identifying beneficial single substitutions, (Ladder B) assembling beneficial higher-order mutants given single and double mutants, and (Lattice) searching in a site-saturated library with site positions chosen in advance. Depending on the task, the authors assess published methods in isolation and against their EvolveRank method, using various evaluation protocols: For Ladder A, their evaluation protocols are in different settings; zero-shot, supervised few-shot, and active learning. For Ladder B, they split mutational landscapes in two groups for evaluation into a more or less epistatic setting (as evaluated by an orthogonal method). They propose to use Precision@16 (at top10%, 5%, 1% of measurements), Recall@16 and a FoundTop1 metric (+UniqPos for Ladder A), as such arguing that evaluation in EvolveGym should only take into account the very top nominations. For their EvolveRank method they use a scikit-learn random forest on frozen protein language model embedding, creating a training set by constructing pairs of measured variants. Their EvolveRank classifier learns to estimate which variant is better in a pair - classifier input is a concatenation of two PLM embeddings - and its pairwise preferential output is then input to a Bradley-Terry model that brings preferences to a global ordering (by an ability scalar).

Critical strengths

EvolveGym is the primary strength of the paper, specifically its elaborate partitioning into tasks (Ladder A, B, and Lattice) as well as the further subdivision in evaluation protocols: setting (zero-shot, few-shot, active) and degree of epistasis. This may help identify in which regimes future models excel and where they break down. The specific choice of evaluation protocols and metrics that are more in line with directed evolution campaigns is also of value.

Critical weaknesses

The authors propose to evaluate directed evolution benchmarks in terms of 'top-variant recovery under a fixed budget', it however still remains an open question how the choice of budget and 'top'-definitions (nomination size) impacts the evaluation metrics. In a proposal for a novel benchmark, it would have been sensible (and consequential to my overall assessment) to also investigate the sensitivity of the evaluation metrics relative to the hyperparameters they are subject to (budget and nomination size), especially to compare to the recall@k in e.g. ProteinGym that the authors term insufficient for the directed evolution regime.

The proposed EvolveRank method is not a strong contribution as a stand-alone model. It uses a random forest classifier, that the authors rightfully identify (and address in Appendix C) is not antisymmetric to the pair, which in return raises a question: why the specific model choice given the preferential regime the authors construct as means of obtaining a global ranking? From a model perspective, the proposed EvolveRank method is not sufficiently novel, and although the authors do demonstrate that it may be of value in identifying the 16 best variants, it is not the contribution that drives the paper's informativeness.

Questions for authors

Have you assessed how the budget, nomination size, and active learning configuration affect the comparative method rankings? If so, do the main conclusions remain consistent across these choices?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6Etv (Part 1/3)

We thank the reviewer for their constructive feedback and for recognizing the value of the benchmark's tasks and evaluation protocols. Below, we address the two critical weaknesses, labeled W1 and W2, followed by the question on evaluation sensitivity.

W1: Sensitivity to evaluation choices. The benchmark should examine how the screening budget and nomination size affect the evaluation metrics and comparative method rankings, including the relationship to ProteinGym's recall metric.

We have conducted additional experiments to evaluate the benefits of EvolveRank across different screening budgets and nomination sizes. We summarize the findings here and provide the detailed results in our response to Q1 below.

Across the tested settings, EvolveRank with SaProt 650M achieves higher mean precision than EVOLVEpro and ALDE at all three thresholds, for every tested budget and nomination size. It also achieves higher means than ALDE on Recall and FoundTop1 throughout. These results support the consistency of its advantage in recovering high-fitness shortlists.

The comparison with Kermut depends on the setting. Kermut leads on precision at 80 measurements, while EvolveRank–SaProt 650M leads at all three precision thresholds for top-16 nominations at 480 measurements. For top-96 nominations, EvolveRank retains the advantage at the top-10% and top-5% thresholds, while Kermut leads on top-1% precision, recall, and FoundTop1.

Our focus on small nomination sets is motivated by practical laboratory constraints, particularly assays that permit testing only tens of variants per round. ProteinGym's top-decile recall is also top-focused, but its evaluated selection grows with the landscape size: for example, it covers 1,000 variants in a landscape of 10,000, a shortlist that would be difficult to validate experimentally in many such settings. Our metrics instead assess whether a method can concentrate high-fitness variants into a small, experimentally tractable shortlist. They therefore complement ProteinGym's metric by evaluating performance at the selection scale relevant to these experimental campaigns.

W2: Model choice and methodological contribution.** The choice of a random-forest comparator despite its lack of intrinsic antisymmetry, and EvolveRank's contribution beyond established components, require clarification.

Our methodological contribution is a ranking procedure designed for protein engineering with very few experimental measurements. EvolveRank learns preferences by processing two frozen protein embeddings jointly, then uses an efficient tournament and Bradley–Terry aggregation to rank candidates. This differs from scalar ranking models, which score each variant separately.

To test whether this design provides benefits beyond ranking supervision alone, we compared EvolveRank with XGBoost ranking and RankReg across 487 Ladder A datasets, using the same frozen embeddings and measured sets of 80, 160, or 240 variants. EvolveRank achieves higher top-5% and top-1% precision, Recall@16, and FoundTop1 than both baselines at every budget, with Holm-adjusted p < 0.001. For example, at 80 measurements, Recall@16 increases from 10.93% for XGBoost to 14.42%, and FoundTop1 from 11.70% to 16.84%.

Active-learning ablations on 12 DMS datasets with ten simulations each support the design choices. Concatenated embeddings outperform difference features. The random forest outperforms the tested logistic-regression and MLP classifiers. Replacing Bradley–Terry aggregation with raw win counts lowers all five reported mean metrics, including top-10% precision from 83.18% to 70.73%. Exhaustive comparisons do not improve the reported means over the Swiss tournament scheduling, which requires only O(N log N) pairwise predictions compared to O(N²) for the exhaustive set-up. We address the classifier's lack of intrinsic antisymmetry by evaluating both pair orientations at inference. The complete ablations appear in Table R4.3 of our response to Reviewer GJFf.

AuthorsOpenReview ↗

Response to Reviewer 6Etv (Part 2/3)

Q1. Have you assessed how the budget, nomination size, and active learning configuration affect the comparative method rankings? If so, do the main conclusions remain consistent across these choices?**

Yes. We conducted additional Lattice experiments comparing EvolveRank, EVOLVEpro, ALDE, and Kermut with and without its zero-shot prior. Table R2.1 reports three configurations: 80 measurements with batches of 16 and top-16 nominations; 480 measurements with batches of 96 and top-16 nominations; and the same budget and batch size with top-96 nominations.

As summarized in W1, EvolveRank–SaProt 650M maintains higher mean precision than EVOLVEpro and ALDE across these configurations, while its comparison with Kermut depends on the setting and metric. The 80-versus-480 comparison changes budget and batch size jointly; the two 480-measurement configurations compare nomination sizes at the same budget and batch size. Complementing these experiments, Appendix Table E.1 compares random and adaptive acquisition at a fixed budget of 80 measurements on Ladder A. ERA and DPO were omitted from this additional sensitivity study because of computational constraints. We aim to extend these experiments and the accompanying analysis to additional screening budgets and nomination sizes.

Table R2.1. Sensitivity to screening budget, acquisition batch size, and final nomination size on Lattice. All values are percentages; higher is better. Precision@k measures the fraction of the k nominations belonging to the true top 10%, 5%, or 1% of the landscape. Recall@k measures the fraction of the true top-k variants recovered among the nominations. FoundTop1 measures the percentage of campaigns in which the nominations include the best variant. Results are averaged over ten simulations per landscape, then over landscapes within each of six protein families, with equal weight assigned to each family. Brackets denote 95% bootstrap confidence intervals obtained by resampling families. Kermut's model-initialization seed is fixed to 2024, independently of the ten experimental simulations. Bold indicates the highest mean in each column within a configuration; family-level pairwise differences were not statistically significant after accounting for multiple comparisons.

Lattice — 16 variants per round · 80 measurements total · final top-16

MethodPrec. top 10%top 5%top 1%Recall@16FoundTop1
ALDE (boosting + greedy)82.7 [64.9, 99.6]77.5 [58.8, 95.9]56.5 [40.6, 72.1]17.6 [5.3, 31.4]21.8 [5.2, 38.5]
EVOLVEpro (ESM2 15B)86.8 [70.3, 99.2]82.5 [62.4, 98.5]61.6 [44.7, 78.3]18.5 [7.3, 29.0]17.7 [6.7, 31.3]
EvolveRank (ESM2 650M)91.3 [84.7, 97.3]85.6 [75.0, 95.9]68.2 [53.8, 80.4]19.8 [7.3, 32.7]15.2 [1.8, 33.3]
EvolveRank (ESMC 6B)91.7 [84.8, 98.2]82.8 [69.6, 95.0]69.2 [50.4, 86.8]28.4 [10.6, 45.4]27.0 [5.0, 51.7]
EvolveRank (SaProt 650M)89.7 [78.0, 98.6]82.7 [66.7, 96.2]70.2 [52.7, 87.6]26.4 [7.9, 45.0]23.8 [5.5, 43.3]
Kermut (seed 2024)93.9 [85.0, 100.0]91.6 [81.3, 98.4]83.0 [72.1, 92.5]25.3 [9.8, 40.0]27.2 [8.3, 48.2]
Kermut without zero-shot prior (seed 2024)82.6 [67.2, 96.8]77.3 [61.1, 91.9]63.6 [45.8, 79.8]19.8 [7.0, 32.8]23.5 [8.3, 38.3]

Lattice — 96 variants per round · 480 measurements total · final top-16

MethodPrec. top 10%top 5%top 1%Recall@16FoundTop1
ALDE (boosting + greedy)90.9 [80.1, 100.0]88.8 [75.7, 100.0]83.5 [69.3, 96.7]38.9 [18.6, 57.1]53.5 [23.5, 80.0]
EVOLVEpro (ESM2 15B)96.7 [91.5, 100.0]96.5 [91.3, 100.0]93.5 [86.8, 100.0]50.0 [26.3, 68.8]62.0 [35.0, 81.7]
EvolveRank (ESM2 650M)98.6 [96.3, 99.9]98.3 [95.4, 99.9]93.7 [86.3, 99.5]52.8 [22.3, 79.0]52.3 [18.0, 81.7]
EvolveRank (ESMC 6B)98.1 [94.5, 100.0]98.0 [94.4, 99.9]91.9 [82.8, 99.7]59.0 [29.3, 83.9]58.5 [26.7, 86.8]
EvolveRank (SaProt 650M)99.2 [98.2, 100.0]99.1 [98.1, 99.9]97.0 [93.4, 99.5]55.1 [24.4, 81.4]60.5 [30.0, 86.7]
Kermut (seed 2024)97.3 [91.8, 100.0]96.5 [89.5, 100.0]94.2 [84.6, 100.0]55.0 [26.2, 79.8]69.8 [38.2, 93.4]
Kermut without zero-shot prior (seed 2024)95.5 [89.1, 100.0]94.2 [85.8, 100.0]91.3 [80.0, 100.0]55.4 [26.8, 79.0]71.5 [38.2, 94.8]
AuthorsOpenReview ↗

Response to Reviewer 6Etv (Part 3/3)

Lattice — 96 variants per round · 480 measurements total · final top-96

MethodPrec. top 10%top 5%top 1%Recall@96FoundTop1
ALDE (boosting + greedy)87.8 [75.0, 99.9]85.0 [70.2, 99.7]58.8 [47.2, 72.0]44.0 [19.7, 65.5]57.3 [26.7, 83.3]
EVOLVEpro (ESM2 15B)92.4 [83.0, 99.8]90.1 [79.9, 99.8]67.9 [55.3, 81.6]53.4 [27.6, 74.2]64.0 [35.0, 85.7]
EvolveRank (ESM2 650M)96.0 [92.9, 99.1]91.9 [84.3, 98.6]68.6 [54.5, 80.4]51.3 [23.4, 75.9]52.8 [19.0, 81.7]
EvolveRank (ESMC 6B)95.0 [90.8, 99.1]90.8 [81.6, 98.6]69.1 [54.4, 80.6]55.1 [26.6, 79.0]58.5 [26.7, 86.8]
EvolveRank (SaProt 650M)97.5 [94.4, 99.5]94.9 [87.3, 99.3]73.2 [57.4, 87.2]52.2 [24.2, 76.9]61.0 [30.0, 87.0]
Kermut (seed 2024)94.3 [83.9, 99.8]92.8 [80.0, 99.8]77.0 [60.0, 89.3]57.6 [27.0, 84.6]72.0 [40.0, 94.0]
Kermut without zero-shot prior (seed 2024)93.0 [82.1, 100.0]91.0 [77.3, 99.9]75.1 [58.8, 88.0]57.9 [27.1, 84.2]73.7 [40.3, 95.3]

The results show that comparative method rankings depend on the screening budget, nomination size, and evaluation metric. At 80 measurements, Kermut achieves the highest mean precision across all three ground-truth thresholds. At 480 measurements, EvolveRank–SaProt 650M achieves the highest precision for top-16 nominations, while ESMC 6B achieves the highest Recall@16 at both budgets. Expanding the final nominations to top-96 changes the comparison further: SaProt 650M retains the highest precision against the top 10% and 5% of ground truth, whereas Kermut leads against the top 1%, and Kermut without the zero-shot prior achieves the highest Recall@96 and FoundTop1. ALDE and EVOLVEpro do not change these leading methods.

Thus, the additional experiments support EvolveRank's competitiveness across these settings and show that no method consistently leads across all metrics and configurations. Results are averaged over 10 simulations per landscape and then equally across six protein families, with 95% bootstrap confidence intervals.

Reference. Jiang, K., Yan, Z., Di Bernardo, M., Sgrizzi, S. R., Villiger, L., Kayabolen, A., Kim, B. J., Carscadden, J. K., Hiraizumi, M., Nishimasu, H., Gootenberg, J. S., & Abudayyeh, O. O. (2025). Rapid in silico directed evolution by a protein language model with EVOLVEpro. Science, 387(6732), eadr6006. https://doi.org/10.1126/science.adr6006

Reviewer_HPLoOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces EvolveGym, a benchmark spanning both single substitutions and higher-order mutations, assembled from experimental measurements including deep mutational scans and combinatorially complete libraries. Alongside the benchmark, the authors propose EvolveRank, a pairwise ranking method in the spirit of earlier preference-based approaches to protein engineering such as ERA. It combines pairwise preference learning with online active learning, acquiring new measurements in rounds under a fixed budget, and shows strong performance across the tasks the benchmark defines.

Critical strengths

This paper addresses an important problem: how to use sparse experimental data to decide which experiments to run next. Much of the work in this space does not treat measurements as the scarce resource, but in a real campaign that is exactly what they are. Alongside a clear and simple algorithm that appears to be quite effective, the authors introduce a benchmark the community can use, one that is well calibrated to experimental reality.

Critical weaknesses

One aspect of the paper that makes the results difficult to interpret is that the methods are benchmarked only on the authors’ own new tasks. Because no previously published metric is reported on any previously published split, there is no external anchor against which to calibrate the baselines, and no way to tell whether each was implemented correctly and tuned sensibly. This matters because the central claim rests entirely on comparisons against those implementations. The concern is sharpest on Lattice, where the fifteen landscapes come directly from SSMuLA, which published its own evaluation of MLDE, ftMLDE and ALDE on exactly that data — the most readily available check in the paper, and one the authors neither use nor acknowledge.

Questions for authors

Does EvolveRank outperform MLDE and ALDE on SSMuLA?

1 comment
AuthorsOpenReview ↗

Response to Reviewer HPLo

Thank you for highlighting this concern. We agree that evaluation under our proposed protocols should be complemented by an established external reference, and that the connection to SSMuLA’s published evaluations should be made more explicit.

As detailed in our response below, we have added two complementary comparisons: EvolveRank under the original SSMuLA protocol and metrics, and MLDE and ALDE under our Lattice protocol. The former uses the authors’ released baseline results, providing an external reference independent of our implementations; the latter connects these established methods to our evaluation setting.

Q1. Does EvolveRank outperform MLDE and ALDE on SSMuLA?

The SSMuLA paper reports its main results on 12 landscapes (those with ≥1% active variants). Our Lattice setting is built from 15 of the 16 landscapes provided by SSMuLA (T7 was excluded from Lattice because more than 90% of its variants have a raw fitness of 0).

SSMuLA's two headline metrics, "Average max fitness achieved" and "Fraction reaching the global optimum", both focus exclusively on the top-performing variant identified by each method. As a complementary perspective, Lattice also assesses high-fitness variant recovery, namely the number of high-fitness variants recovered by a method in its final batch, quantified by Precision@16 (fraction of the 16 nominations in the top 10%, 5%, or 1%) and Recall@16 (fraction of the dataset’s true top-16 variants nominated). This addresses the practical need in experimental campaigns (e.g., in drug discovery and development) to prioritize a shortlist of candidates for experimental follow-up, as not all screening hits necessarily retain their performance upon further testing.

To compare ALDE, MLDE, and EvolveRank, we add results for ALDE, MLDE on our Lattice setup and EvolveRank on the original SSMuLA evaluation:

a) ALDE, MLDE, and EvolveRank on our Lattice setup

Each method has access to 80 fitness labels: MLDE uses 80 random samples, while ALDE and EvolveRank start from the same 16 variants and acquire another 64 adaptively. Final top-16 shortlists are ranked over the full pool and evaluated retrospectively using known experimental fitness values. Results are averaged over ten campaigns per landscape, then across landscapes within each protein group, with equal weight assigned to the six groups.

Table R1.1. ALDE, MLDE and EvolveRank on our Lattice setup. Brackets indicate 95% bootstrap confidence intervals obtained by resampling the six groups (1,000 resamples; seed 42).

MethodPrecision@16: Top 10%Top 5%Top 1%Recall@16FoundTop1
MLDE71.8 [49.8–92.3]59.8 [38.8–80.4]26.8 [12.9–43.9]5.5 [0.7–11.6]3.3 [0.0–10.0]
ALDE (boosting + greedy)82.7 [64.9–99.6]77.5 [58.8–95.9]56.5 [40.6–72.1]17.6 [5.3–31.4]21.8 [5.2–38.5]
EvolveRank (ESM-2 15B)88.9 [80.4–96.8]80.1 [65.5–93.6]67.8 [47.0–87.6]27.5 [10.5–43.8]23.2 [4.8–41.5]
EvolveRank (SaProt 650M)89.7 [78.0–98.6]82.7 [66.7–96.2]70.2 [52.7–87.6]26.4 [7.9–45.0]23.8 [5.5–43.3]

Both EvolveRank variants outperform the baseline methods, with SaProt 650M achieving the strongest overall performance across four of the five metrics.

b) ALDE, MLDE and EvolveRank on the original SSMuLA evaluation

We evaluated EvolveRank with SaProt-650M on SSMuLA’s 12 main landscapes, using the released scale2max fitness labels and experimentally measured variant pools, excluding stop-codon variants and retaining WT.

EvolveRank uses the same 120 random initial observations reconstructed for standard ALDE ×4 from the authors’ released seeds and sampling procedure, including repeated samples. It then acquires three adaptive batches of 120 each, for a total of 480 measurement slots. We ran 50 campaigns per landscape and averaged results equally across the 12 landscapes.

Table R1.2. ALDE, MLDE and EvolveRank on the original SSMuLA evaluation.

MethodAverage maximum fitnessFraction reaching the global optimum
MLDE0.829736.3%
ALDE (boosting + greedy) ×40.866558.8%
EvolveRank (SaProt-650M)0.889462.7%

Baseline values are taken from the authors’ released results. Following their implementations, MLDE evaluates its predicted top-96 shortlist, whereas ALDE and EvolveRank evaluate the maximum over the measured trajectory. A cumulative rescoring of MLDE, using training sets reconstructed under the authors’ default seed settings, changes its average maximum fitness only to 0.8298 and leaves its global-optimum fraction unchanged.

EvolveRank achieves higher aggregate values than standard MLDE and ALDE ×4 on both metrics. Its advantage varies across landscapes: it exceeds ALDE ×4 in average maximum fitness on seven of the twelve landscapes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions in Response to Reviewer Feedback

We thank the reviewers for their thoughtful feedback and follow-up questions. We have revised the manuscript to incorporate the additional experiments and clarifications discussed during the review process. New and revised passages are highlighted in blue. Below, we summarize the main changes and their connection to the reviewers’ comments.

  1. Benchmark scope and contribution — Reviewer GJFf. We expanded the discussion of how EvolveGym complements ProteinGym and FLIP/FLIP2, clarifying that its contribution concerns the combined experimental protocols, measurement constraints, and small final nomination sets. We also explain the practical questions these settings allow researchers to investigate under shared evaluation conditions (Section 2 and Appendix A).
  2. Sensitivity to nomination size and experimental budget — Reviewer 6Etv. We added nomination-size sweeps across all three tasks, including the 480-measurement campaigns on Ladder A and Lattice and the additive and more epistatic subsets of Ladder B. Section 5.4 summarizes the findings, Appendix E.4 and Tables E.9–E.15 provide the detailed results, and Table E.16 explicitly distinguishes stability of the leading configuration from stability of the complete method ordering. The discussion clarifies how comparative conclusions depend on the experimental configuration and metric.
  3. Separating model performance from adaptive acquisition — Reviewer GJFf. We expanded main Table 2 with fixed-data EvolveRank results using the same 80 randomly selected measurements as the supervised baselines. This enables direct comparisons without adaptive acquisition and distinguishes the contribution of the ranking method from the additional benefit of selecting measurements over successive rounds. We updated the corresponding discussion and property-stratified Ladder A figure.
  4. Controlled comparisons with scalar ranking methods — Reviewers xHEh, GJFf, and 6Etv. We added comparisons with XGBoost’s pairwise-ranking objective and RankReg across 487 Ladder A datasets, using shared frozen embeddings and measured sets at budgets of 80, 160, and 240 variants. The main text summarizes the results, with full numerical results and statistical comparisons in Appendix E.1.1 and Table E.2. We also expanded the related-work discussion of existing ranking and preference-learning approaches.
  5. Component ablations and methodological clarification — Reviewers GJFf and 6Etv. We added ablations examining pair construction and representation, inference orientation, comparison scheduling, aggregation, vote weighting, treatment of uncertain comparisons, and the pairwise classifier (Appendix E.1.2 and Table E.3). We also explicitly clarify that constructing pairwise training examples does not increase the number of independent experimental observations (Section 4).
  6. An external evaluation reference and additional baselines — Reviewer HPLo. We added MLDE and ALDE comparisons under the 80-measurement Lattice protocol and evaluated EvolveRank under the original SSMuLA protocol and metrics, using the authors’ released baseline results as an external reference. The protocols and evaluation conventions are documented in Appendix E.3 and Tables E.7–E.8.
  7. Budget accounting and evaluation details — Reviewer xHEh. We clarified that Lattice ensemble members share one acquisition trajectory and measurement budget, with the merged ranking used at every acquisition round and for final nominations (Appendix D.1). We additionally report Ladder B training-set sizes, explain the handling of missing fitness values for ERA/DPO, and quantify how often final nominations were already measured during the campaign.
  8. Reconciliation of Kermut results — Reviewer xHEh. We resolved the discrepancy between the main and appendix results using the corrected Kermut runs and corresponding ensemble results. The affected tables, confidence intervals, statistical comparisons, and interpretations have been updated. Table 2 and Table E.1 now report the same verified random-80 Kermut baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.