acceptodds
Under review as a conference paper at ICLR 2027

The Diversity Mirage: Quality-Gated Evaluation for Biological Sequence-Design Active Learning

Abstract

Active learning for sequence design aims to find batches of strong, diverse candidates with a limited measurement budget. Common evaluations assess overall quality but measure diversity among each method's own highest-scoring candidates. These candidate sets can differ in quality. A method can therefore look diverse even when its candidates score lower than those found by other methods. We call this the diversity mirage. We introduce quality-gated evaluation to compare methods under the same measurement budget. It uses a shared quality standard to assess both the number and diversity of high-quality candidates. We run controlled experiments on RNA design tasks and experimentally measured protein fitness data from ProteinGym. On RNA tasks, the shared standard reveals a reversal in the diversity ranking. A simple local search finds more distinct groups of high-quality sequences than a generative method designed for diversity. On ProteinGym, we use ESM features from mutated positions to select candidates. We find 42.9% more high-quality candidates on average than proxy-greedy selection, while achieving similar coverage of sequence space. With a shared quality standard, we can directly compare the diversity of high-quality candidates across methods. What matters is how many strong, distinct candidates the same budget delivers.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Active Learning, Biological Sequence Design, Quality-Diversity Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vbViOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors address a concern with evaluation of sequence-design models where the so-called "diversity-mirage" appears, where comparing diversity across models is often confounded by the fact that the sets used to compute these diversity metrics are not comparable, often being based on each model's own individual Top-K sets. To address this, the authors introduce a post-hoc method for recomputing model diversities using already-collected scores, stating that their quality-gated evaluation measures both the number of high-quality candidates and the distinct sequence groups that they cover.

Critical strengths
  • The problem is very well motivated and the authors do a good job of justifying their approach. Design choices, such as reporting the qg-hits and qg-distinct scores separately, are rational and provide a clearer picture of each method's depth versus breadth
  • The authors provide all of the code and scripts used to generate their results, providing a very high degree of reproducibility that enables in-depth auditing
  • The method introduced by the authors is entirely post-hoc, meaning that it does not require expensive oracle evaluations or re-runs of past methods. This enables retrospective analysis of past campaigns in a straigthforward manner.
Critical weaknesses
  • The authors should provide more comprehensive baselines in addition to TR-BEAM, there are many other methods that could be used such as those cited in the related work, e.g. GFlowNet-AL and AdaLead
  • The statistical significance threshold is unjustified, as p95 and p90 are both presented. A domain-motivated choice of which to use should be provided in the text to encourage easy adaptation
  • Stronger statistics are needed to reinforce the diversity reversal ranking for the RNA design task, and the variance across seed should be characterized and provided to the reader
Questions for authors
  • Does the diversity ranking reversal hold for other methods? This is a key question that should be addressed and relates to the issue of including additional baselines
4 comments

Thank you for the suggestions on baseline coverage, cutoff choice, and variation across seeds.

W1. Additional search baselines

We added GFlowNet-AL, AdaLead, and surrogate-assisted MAP-Elites. The follow-up compares five methods on three RNA tasks, with five paired seeds per task and 500 new queries per run. See our reply to rewj for the full results (W1, rebuttal Table 2), the matched scoring and query protocol, and the MAP-Elites archive settings (W2). Under the archived-reference p95 cutoff on RNA3, AdaLead had the most hits on average (111.0), while MAP-Elites covered the most qualifying groups (4.0).

W2. Choosing a quality cutoff

The p95 cutoff is the 95th percentile of a fixed reference distribution and suits a selective follow-up screen. We use the archived 22-configuration reference suite for each RNA task, with p90 as a broader screen. A prespecified minimum activity or binding score can also define the common gate. On the original eight ProteinGym landscapes, LEF averaged more hits than Greedy at p50, p60, p70, p80, p90, p95, and p99 under the same pooled six-method reference suite. The means were 98.38 vs. 90.65 at p50, 14.90 vs. 10.43 at p95, and 3.23 vs. 2.25 at p99. Rebuttal Table 2B also checks a reference distribution formed within the new RNA runs. We will distinguish these screening percentiles from statistical pp-values in the revision.

W3 and Q. Ranking reversals and sensitivity

We extended the analysis to both archived-reference cutoffs, p90 and p95, and Hamming radii 1, 2, and 3. Each setting compares the ten method pairs on each of three tasks using five-seed mean group counts. A strict reversal requires opposite, nonzero differences between the unfiltered top-50 comparison and the gated comparison at the same radius; ties are excluded.

Rebuttal Table 5. Cutoff, radius, and seed sensitivity

Each cell reports strict five-seed mean reversals out of 30 task–method pairs. Parentheses give the number retained after every leave-one-seed-out check (LOO). “All 500” applies the gate to all new queries. “Fixed top-50” applies it within the same top-50 lists used before gating, holding the candidate lists fixed. Rows reuse the same query records and are descriptive sensitivity checks.

CutoffRadiusAll 500: mean reversals (LOO retained)Fixed top-50: mean reversals (LOO retained)
p9019 (2)3 (0)
p9024 (1)0 (0)
p9034 (0)0 (0)
p95112 (1)8 (1)
p9527 (1)5 (0)
p9532 (0)4 (0)

At p95 and r=2r=2, the all-500 reversals occur in zero, three, and four pairs on RNA1, RNA2, and RNA3. Five mean reversals also occur when the top-50 lists are held fixed. The table separates these changes in mean ordering from reversals preserved after every single-seed omission.

Rebuttal Table 6. Method pairs with a mean reversal at p95 and radius 2

Entries are five-seed mean differences in group counts, A minus B. Positive values mean that A covers more connected groups than B. Bold marks a strict change of sign relative to the unfiltered top-50 comparison. The table lists every pair with a reversal in either gated scope: seven in all 500 queries and five within the fixed top-50 lists. MAP-Elites is surrogate-assisted. Ties do not count as reversals; the denominator remains 30 task–method pairs.

TaskPair (A − B)Top-50 differenceAll-500 p95 differenceFixed-top-50 p95 difference
RNA2δ-CS − MAP-Elites-2.6+0.2+0.2
RNA2GFlowNet-AL − MAP-Elites-1.8+0.4+0.4
RNA2TR-BEAM − δ-CS+0.8+0.0-0.2
RNA2TR-BEAM − MAP-Elites-1.8+0.2+0.0
RNA3AdaLead − GFlowNet-AL-2.0+1.0+0.4
RNA3δ-CS − AdaLead+0.2-0.6+0.2
RNA3δ-CS − GFlowNet-AL-1.8+0.4+0.6
RNA3TR-BEAM − GFlowNet-AL-2.4+0.2-0.4

Complete paired statistics

The supplementary archive rebuttal_supplementary_data.zip provides all method pairs, including non-reversals, with per-seed counts, paired intervals, raw and Holm-adjusted pp-values, and leave-one-out checks. Its README.md indexes the complete tables and records.

Paired tests use exact two-sided sign flips, with Holm correction over the 30 task–method pairs for each fixed metric, candidate scope, cutoff, and radius. With five nonzero paired differences, the smallest attainable two-sided pp-value is 2/32=0.06252/32=0.0625; zero paired differences or fewer valid pairs make it coarser. These tests assess paired metric differences under a given condition. Ranking-change counts remain descriptive, and we report effect sizes and seed consistency alongside them.

For AdaLead and GFlowNet-AL on RNA3, the all-500 mean reversal survives dropping any one seed. Their top-50 means are 1.4 versus 3.4 groups, and their p95 means are 2.2 versus 1.2. The paired p95 difference is +1.00 group for AdaLead (sample SD 1.87; approximate paired tt 95% interval [−1.32, 3.32]; two-sided exact sign-flip p=0.375p=0.375). This paired difference is not statistically significant; we report it as descriptive and do not rely on it for the main claim. AdaLead leads at p95 in three seeds and trails in two; seed 44 has the strict within-seed reversal. The individual counts are in our reply to Anzt (W1 and Q1, rebuttal Table 4). At p90, the means are 1.8 versus 2.6. We therefore report this ordering change at its stated cutoff and radius, with the paired variation.

The yield–coverage trade-off provides a separate, more consistent example across the tested settings. As detailed in our reply to rewj (W1, rebuttal Table 2A), at the archived-reference cutoffs TR-BEAM finds more hits and MAP-Elites covers more groups on RNA3 in all five paired seeds at p90 and in four seeds at p95, with a zero-hit tie in the fifth. This holds at each of the three radii. Together, these results show why a comparison should declare its quality threshold and report qualifying yield and connected-group coverage separately.

Reviewer_vbViOpenReview ↗

Rebuttal acknowledged

I thank the authors for addressing the points raised in my review. While the review and subsequent clarifications do improve my understanding of the paper and the main arguments, and I look forward to the revised manuscript, I will maintain my score of 3.

Thank you for your careful review and for considering our responses. We appreciate your constructive feedback on the baseline comparisons, quality cutoffs, and variation across seeds. Your suggestions have helped us clarify the evaluation protocol and improve the presentation of the results. We have addressed the baseline scope in the response to rewj, the shared quality cutoff in the responses to KX3C and Anzt, and seed-level variability in revised Section 5.1 and Table 2. Thank you again for your time and thoughtful discussion.

Reviewer_AnztOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to evaluate quality and diversity in active learning for biological sequence design. Comparing diversity within each method's own top-K candidates can compare sequences at different quality levels. The proposed post-hoc protocol applies a shared task-level threshold and reports qg-hits, the number of qualifying candidates, and qg-distinct, the number of connected components in their Hamming graph.

Under matched scoring models and query budgets, TR-BEAM finds more qualifying groups than δ−CS\delta-CS on three RNA tasks. On the original eight ProteinGym landscapes, LEF, which uses ESM features at mutated positions, obtains 14.90 hits versus 10.43 for greedy selection, a 42.9% increase, with similar mean group counts at r=2: 4.93 versus 4.98. A replication following a selection rule fixed in advance retains 18 completed, in-scope tasks. The authors report a smaller hit advantage there, +1.89 compared with approximately +4.47 on the original eight, and a pooled advantage of +2.68 across 26 tasks. These findings concern surrogate-scored RNA search and selection from measured protein variant pools under the stated evaluation rules.

Critical strengths
  1. The protocol answers a practical evaluation question. Pages 3–5 specify the reference scores, shared threshold, budget, and evaluated portfolio, then separate qualifying yield from group counts. This helps distinguish more hits from more sequence groups without additional oracle queries. Table 2 and Figure 5 provide a concrete example: LEF improves yield while its mean r=2 group count remains close to greedy's.

  2. The authors examine important limits of the findings. Appendix C checks threshold, radius, and reference-set sensitivity. Appendix A explains the relative-share interpretation of pooled gates and acknowledges that the original landscapes were selected during method development. The reported replication in Appendix I shows a smaller effect, and the authors explicitly recommend carrying forward the pooled estimate. This qualification improves the interpretation of the empirical result.

Critical weaknesses
  1. The statistic underlying the earlier RNA ordering needs an explicit computational definition. Section 5.1, p. 6, line 323, through p. 7, line 367, reports 50.0 distinct top-ranked sequences for δ-CS and an average of 2.7 for TR-BEAM, before comparing shared-gate component counts at r=2. Please specify the candidate set and whether “distinct” means exact sequence uniqueness or distance-based grouping, including the treatment of repeated sequences. This statistic anchors the ranking-reversal claim, so readers need enough detail to reproduce it. The authors already describe the result as a comparison between complete evaluation rules; that claim does not require isolating the threshold's independent effect.

  2. The radius argument and the scope of the coverage summary need clarification. On p. 5, lines 251–255, and in Appendix A, the pairwise-distance bound does not establish that r must be strictly below mutational depth. If each variant has at most k substitutions from a common wild type, r≥2k forces any nonempty hit graph into one component; this does not make r<k necessary, and mean mutation counts alone do not establish that r=2 is informative. This matters because the radius determines whether qg-distinct can distinguish methods. The authors correctly acknowledge single-substitution degeneracy. For the coverage difference of −0.02 and its interval in Section 5.4/Appendix I, the exact task set and interval construction should also be stated. If the summary includes tasks with degenerate counts, near-equality on those tasks provides little evidence about comparative breadth.

Questions for authors
  1. Required clarification: How are the RNA values 50.0/2.7 computed? Please give the precise candidate set, definition of distinctness, and treatment of repeated sequences. A concise definition or reference to the relevant evaluation code would resolve this concern.

  2. Required clarification: Which task set underlies the coverage difference of −0.02 and the 95% interval [−0.18, +0.14], and how was the interval constructed? Please state how many included tasks have degenerate r=2 counts. If such tasks are included, either report the corresponding summary for informative tasks or limit the interpretation to the stated aggregate group count.

Required textual corrections: Correct the radius condition on p. 5 and in Appendix A. In the abstract, lines 24–25, identify the 42.9% gain as applying to the original eight landscapes, consistent with Section 1.

Optional suggestion: Briefly mention the smaller replication effect or the pooled estimate in the abstract to reflect the qualification already made in Appendices A and I.

5 comments

Thank you for asking us to clarify the RNA counts and the ProteinGym coverage analysis.

W1 and Q1. RNA candidates and diversity counts

The submitted sentence reported two archived summary values without specifying the candidate set, ranking rule, or treatment of repeated sequences; the released records do not determine that counting rule. We therefore replace the sentence rather than assign it a new definition, and the revision will not report those two numbers.

The original quality-gated comparison has a separate, explicit record. Under the declared shared gate, the released per-run summaries report mean radius-2 group counts of 0.0 / 0.0 / 0.2 for δ-CS and 13.2 / 15.6 / 21.6 for TR-BEAM on RNA1/2/3. These means can be reproduced by aggregating the archived per-run counts. They describe the original campaigns; the new five-method campaigns are reported separately in our reply to rewj (W1, rebuttal Table 2).

For an evaluated sequence set SS and a common cutoff τ\tau, define the qualifying set as Hτ={x∈S:f(x)≥τ}H_{\tau}=\{x\in S:f(x)\ge\tau\}. The hit count is h=∣Hτ∣h=|H_{\tau}|, and the group count is c=∣CC(Gr(Hτ))∣c=|CC(G_{r}(H_{\tau}))|. Here, GrG_{r} links qualifying sequences whose Hamming distance is at most rr, and CCCC denotes connected components. An empty qualifying set has zero hits and zero groups.

In the new five-method follow-up, each run contributes 500 unique new queries, excluding the initial labels. We rank sequences by oracle score and break ties in query order, so every top-50 list contains 50 distinct sequences. At r=2r=2, the mean top-50 group counts on RNA1/2/3 are 1.0 / 1.0 / 1.6 for δ-CS and 1.0 / 1.8 / 1.0 for TR-BEAM. The shared gate screens all 500 new queries before applying the same grouping rule. We will use these declared candidate sets and counting rules in the revised Section 5.1 and Figure 4.

Rebuttal Table 4. RNA3 group counts at radius 2

These are paired-seed results from the matched follow-up. The Top-50 columns count radius-2 groups among each method’s 50 highest-scoring new queries. The shared-p95 columns apply the same archived-reference cutoff to all 500 new queries before counting groups. The last row reports the mean and sample SD across five paired seeds.

SeedTop-50 AdaLeadTop-50 GFlowNet-ALp95 AdaLeadp95 GFlowNet-AL
01130
71120
122423
331123
4421020
Mean ± SD1.40 ± 0.553.40 ± 3.912.20 ± 0.451.20 ± 1.64

Applying p95 within the fixed top-50 lists also reverses the mean ordering. AdaLead covers 1.6 groups and GFlowNet-AL covers 1.2. The query records for all 75 runs and the evaluation scripts are included in the prepared reproducibility package. Our reply to vbVi (W3 and Q, rebuttal Table 5) reports the cutoff, radius, and seed sensitivity of these comparisons.

W2 and Q2. ProteinGym coverage

The combined analysis includes eight original landscapes and 18 completed, in-scope landscapes from the replication set selected under a rule fixed in advance. We average the five paired seeds within each landscape, compute the LEF-minus-Greedy group difference, and report the mean difference with an approximate 95% interval of ±1.96 standard errors across landscapes. At r=2r=2, the difference is −0.023 [−0.183, +0.137] across all 26 and −0.050 [−0.593, +0.493] across the original eight. The 18 replication landscapes contain single substitutions, so each nonempty hit set forms one group at r=2r=2. At r=1r=1, LEF covers 1.72 more groups on average across all 26 (95% interval [0.85, 2.58], paired tt-test p=0.0007p=0.0007) and leads on 19 landscapes.

We also checked whether the group count can distinguish multiple regions in the observed portfolios. A landscape is observed to be nondegenerate when at least one LEF or Greedy run among the five paired seeds has more than one p95 connected component. This holds for 8 of 26 landscapes at r=2r=2 (all eight original landscapes) and all 26 at r=1r=1. The criterion concerns at least one of these ten method–seed runs per landscape. The companion archive reports the per-seed counts and the landscape-level maximum in proteingym/coverage_per_seed.csv and proteingym/coverage_per_landscape.csv.

Original eight — multi-substitution landscapes

We report the original eight separately to assess coverage at r=2r=2 on landscapes that include multi-substitution variants.

A4_HUMAN_Seuma_2022, F7YBW8_MESOW_Ding_2023, GCN4_YEAST_Staller_2018, GRB2_HUMAN_Faure_2021, HIS7_YEAST_Pokusaeva_2019, PHOT_CHLRE_Chen_2023, SPG1_STRSG_Wu_2016, YAP1_HUMAN_Araya_2012.

Replication eighteen — single-substitution landscapes

We include an r=1r=1 sensitivity check across all 26 landscapes because the smaller radius can distinguish multiple qualifying groups in single-substitution landscapes.

A0A140D2T1_ZIKV_Sourisseau_2019, A0A192B1T2_9HIV1_Haddox_2018, A0A2Z5U3Z0_9INFA_Doud_2016, A4D664_9INFA_Soh_2019, C6KNH7_9INFA_Lee_2018, ENV_HV1BR_Haddox_2016, HMDH_HUMAN_Jiang_2019, HSP82_YEAST_Flynn_2019, MSH2_HUMAN_Jia_2020, MTHR_HUMAN_Weile_2021, POLG_CXB3N_Mattenberger_2021, POLG_DEN26_Suphatrakul_2023, PPARG_HUMAN_Majithia_2016, Q2N0S5_9HIV1_Haddox_2018, RDRP_I33A0_Li_2023, S22A1_HUMAN_Yee_2023_activity, SC6A4_HUMAN_Young_2021, SHOC2_HUMAN_Kwon_2022.

Radius condition and abstract

We will correct the main text and Appendix A. If each variant has at most kk substitutions from a common wild type, pairwise Hamming distance is at most 2k2k. At r≥2kr\ge 2k, each nonempty hit set forms one group, and an empty hit set has zero groups. At smaller radii, the observed links determine the number of groups. We will state that the 42.9% gain in p95 hits (14.90 vs. 10.43) applies to the original eight landscapes. LEF averages 1.89 additional hits on the 18 replication landscapes and 2.68 across all 26. We will include the pooled estimate in the abstract.

Thank you for the candid clarifications and for performing the follow-up analyses. Resolving the unverified RNA counts (W1) and clarifying the ProteinGym coverage interval and task-level degeneracy (W2) successfully addresses our methodological questions.

As the current submission PDF is still missing several of the promised revisions, could you please upload an updated version during the discussion period that incorporates the following key changes?

  1. RNA Analysis: Update Section 5.1 with the new, reproducible comparison (e.g., Table 4), explicitly stating the counting rules and noting its descriptive (seed-variable) nature.
  2. ProteinGym & Abstract: Align the abstract with the pooled 26-landscape estimates, correct the radius-degeneracy condition (r≥2kr \ge 2k) in the text and Appendix A, and distinguish the performance on the 8 non-degenerate tasks from the 26-landscape r=1r=1 aggregate results in Section 5.4/Appendix I.

I look forward to reviewing the fully revised manuscript.

Reviewer_rewjOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

In this submission, the authors propose an alternative way to evaluate the trade-off between diversity and quality of candidate sequences in iterative biological sequence design. Specifically, diversity should be reported above certain fitness threshold determined by the quality of found sequences. The authors exercise their proposed evaluation protocol on 3 RNA fitness landscapes, and 8 protein fitness landscapes.

Critical strengths

The authors share code from their experiments, which is commendable.

Critical weaknesses

In my view, the paper has following weaknesses:

W1: Limited contribution

The only contribution of this paper is proposing a different quality-diversity evaluation strategy. However, in my eyes, it fails to communicate relevance of doing it the way proposed by the authors. Take for example experiments on RNA landscapes: TR-Beam generates sequences of much higher fitness than δ\delta-CS, but δ\delta-CS returns more diverse candidates. Looking at the two metrics together (which even the authors acknowledge that prior work establishes as a standard evaluation protocol), it comes as no surprise that above certain fitness threshold, (which δ\delta-CS is barely, if at all, able to reach), TR-Beam will be more diverse that δ\delta-CS. As a result, I am not convinced that evaluation proposed by the authors---the paper's sole contribution---carries any new, relevant insights about sequences proposed by different methods

W2: Very poor baselines

I am very confused by the choice of baselines used in this study. Unfortunately, it comes across as proposed by an LLM with little consideration from the authors. For the RNA design, I can understand TR-Beam as a representative member of a local/exploitative search strategies, however δ\delta-CS makes in my view little sense as a representative member of exploratory strategies. This is a specific, concrete GFlowNet variation that explicitly reduces the usual diversity of candidates found by GFlowNets to account for proxy misspecification. In ProteinGym experiments, different search strategies (PEX) are conflated with acquisition functions (BALD, UCB). Relevant baselines for quality-diversity generation (even though some of them, like GFlowNets, are mentioned by the authors) are missing: GFlowNets, MAP-Elites (those are very relevant in Quality-Diversity optimization and lack of them is pretty shocking), or AdaLead

Questions for authors

Q1: Baseline choices

Please elaborate on the choice of the baselines? Why the decision to compare PEX with AFs, instead of comparing AFs or different search strategies alone?

Q2: Computing BALD and UCB details

Please clarify how did you compute BALD and UCB? For this, one needs a posterior/ posterior predictive distribution, how did you implement it? Was it an ensemble/BNN? For BALD, did you compute it with MC Dropout? This circles back to Q1 and making sure that same surrogate is used for PEX

Q3 What does "LEF" stand for?

Is this some method proposed by you? If so, please define it. If not, please reference it.

4 comments

Thank you for asking what the shared gate reveals and how the baselines are matched.

W1. What a common quality gate reveals

The shared gate reveals a trade-off between finding many qualifying sequences and covering many qualifying groups. On RNA3 at the archived-reference p95 cutoff and r=2r=2, TR-BEAM finds 107.8 hits in 1.4 groups, while surrogate-assisted MAP-Elites finds 25.8 hits in 4.0 groups. Both methods produce qualifying sequences, yet their yield and group-count orderings differ. This matters when choosing between many candidates from connected sequence regions and candidates from more disconnected regions.

On RNA3, the same trade-off holds at the archived-reference p90 and p95 cutoffs across Hamming radii r=1,2,3r=1,2,3, evaluated on all 500 new queries. At each radius, TR-BEAM has more hits and MAP-Elites has more groups in all five paired seeds at p90 and in four seeds at p95; both methods have zero p95 hits in the remaining seed. These checks reuse the same five paired runs. In seed 12 at p95 and r=2r=2, 122 of TR-BEAM's 124 hits lie in one group, while the 15 MAP-Elites hits occupy five groups. The supplementary archive reports the means for every cutoff and radius.

Rebuttal Table 2. Matched RNA follow-up

Each cell gives mean qualifying hits / radius-2 groups over five paired seeds, evaluated on the 500 new queries. Panel A retains the numerical p95 cutoffs listed in manuscript Table 3 and used in manuscript Table 6. Because the retrained follow-up oracles differ from the original campaign oracles, panel B also reports counts under a gate recomputed within the follow-up. These are new search campaigns: for example, the original RNA1 TR-BEAM result is 172.6 / 13.2, and the follow-up result in panel A is 114.4 / 1.0. Each result uses the scores from its own campaign.

A. Archived-reference cutoffs

MethodRNA1RNA2RNA3
TR-BEAM114.4 / 1.0108.4 / 0.8107.8 / 1.4
δ-CS (official configuration)43.6 / 0.891.6 / 0.860.8 / 1.6
AdaLead97.8 / 0.8117.2 / 0.6111.0 / 2.2
GFlowNet-AL46.8 / 0.642.6 / 1.02.4 / 1.2
Surrogate-assisted MAP-Elites17.6 / 1.815.2 / 0.625.8 / 4.0

B. Cutoffs recomputed within the follow-up

As a post-hoc sensitivity check, we pool the 12,500 saved query scores from all five methods and five seeds on each RNA task and recompute p95 using linear interpolation. Initial labels are excluded. Each run contributes 500 equally weighted observations; observations of the same sequence in different runs are retained. The resulting task-level cutoff is shared by all methods and seeds. No search is rerun.

MethodRNA1RNA2RNA3
TR-BEAM60.6 / 0.629.0 / 0.648.0 / 1.2
δ-CS3.8 / 1.036.8 / 0.421.4 / 1.4
AdaLead36.6 / 0.651.2 / 0.444.6 / 1.2
GFlowNet-AL16.0 / 0.64.8 / 0.20.2 / 0.2
Surrogate-assisted MAP-Elites8.0 / 0.43.4 / 0.610.8 / 2.6

Under this reference, TR-BEAM again finds more qualifying sequences than MAP-Elites on RNA3 (48.0 vs. 10.8), while MAP-Elites covers more qualifying groups (2.6 vs. 1.2). This supports reporting yield and group coverage separately. The two panels use different reference distributions; the remaining RNA tables retain the archived-reference cutoffs. The archive includes both numerical cutoffs and all per-seed counts. See our reply to KX3C (Q2–Q3, rebuttal Table 1) for the complementary distance and packing measures.

W2. RNA baseline choices

δ-CS controls candidate diversity with an adaptive distance radius. We use its public RNA configuration with a three-member CNN ensemble, UCB, and an LSTM generator. GFlowNet-AL is a separate baseline using trajectory balance and a dropout MLP surrogate. The RNA3 means of 60.8 and 2.4 hits compare the complete pipelines. AdaLead and surrogate-assisted MAP-Elites add local evolutionary and archive-based search. Within each task and seed, all five methods start with the same 5,000 labels, share the frozen MLP oracle, and make 500 new queries in ten batches of 50. The comparison matches the initial data, scoring model, and query budget while retaining each search pipeline.

For MAP-Elites, archive cells use two integer descriptors: GC count and Hamming distance from the highest-scoring initial sequence. Each cell keeps its highest-scoring measured sequence. In each round, one- or two-position mutations generate 1,000 unique unseen candidates, scored by the surrogate. We query the highest-predicted new archive elites and fill the batch from the remaining candidates by predicted score. We will include these settings with the campaign scripts.

Configuration selection and budgets

We use one fixed configuration per method across all three RNA tasks and five paired seeds. The reported campaigns use these settings without a hyperparameter sweep. All methods receive the same oracle-query budget; surrogate training and proposal budgets follow the method-specific settings below.

MethodSettings used in the matched RNA campaigns
TR-BEAMMLP: 200 epochs per round; top-50 frontier; beam width 16; eight single-site expansion steps.
δ-CSThree-CNN ensemble, UCB = mean + SD; two-layer, 512-unit LSTM; 1,000 generator epochs per round; adaptive radius capped at 0.5 with uncertainty coefficient 5.
AdaLeadMLP: 200 epochs per round; parent threshold 0.05; one expected mutation; recombination disabled; initial proxy-query budget 1,000 per batch.
GFlowNet-ALTrajectory balance; MLP dropout 0.1; 25 uncertainty draws; UCB coefficient 0.1; sampling temperature 2.
Surrogate-assisted MAP-ElitesMLP: 200 epochs per round; integer GC-count and Hamming-distance bins; one- or two-position mutations; 1,000 unique unseen proposals per round.

AdaLead can raise its proxy-query budget to 2,000 and then 4,000 if a batch has fewer than 50 unseen candidates; the completed runs used the initial 1,000 setting. These are surrogate calls, separate from the 500-query oracle budget. Full training limits, configuration sources, and run entry points are listed in baseline_configurations.md in the companion archive rebuttal_supplementary_data.zip.

W2 and Q1–Q2. ProteinGym acquisition

Manuscript Table 2 compares the outputs of six complete selection pipelines on one measured pool, with 50 initial labels and five rounds of 20 queries, and evaluates all 150 labels. Every cell is reproducible from the archived selections and DMS scores. This table evaluates complete-pipeline outputs rather than a controlled change of acquisition score. We therefore describe it as a comparison of complete pipelines and make no claim about which acquisition function is preferable. The controlled experiment below isolates the acquisition score: all four rules share one fitted ensemble and one set of predictions, so they differ only in how predictions become selections.

To isolate acquisition, the independent control below holds the surrogate fixed. For each of five paired seeds on GB1, A4, and GRB2, all four rules share a five-member bootstrap ridge ensemble fitted on 50 initial labels. Each rule selects 20 new variants from the same pool. Greedy uses the ensemble mean. UCB adds one ensemble standard deviation. BALD computes predictive mutual information from a Gaussian mixture with shared noise estimated from out-of-bag residuals. PEX-style adds the standardized mutation prior from the ten highest-scoring initial variants to the standardized mean.

Rebuttal Table 3. Independent one-round ProteinGym control

Each cell gives mean p95 hits / radius-2 groups over five paired seeds among the 20 new queries, using the original numerical p95 cutoff for each landscape. The control shares the fitted ensemble and its predictions across all four rules.

RuleGB1A4GRB2
Greedy2.0 / 0.63.6 / 0.41.8 / 1.0
PEX-style1.8 / 1.03.4 / 0.62.2 / 1.4
UCB1.2 / 0.83.6 / 0.62.0 / 1.2
BALD0.8 / 0.20.4 / 0.40.8 / 0.8

On GB1, PEX-style selection covered more qualifying groups than Greedy on average (1.0 vs. 0.6), with similar hit counts (1.8 vs. 2.0). Rebuttal Table 3 isolates acquisition for one batch; manuscript Table 2 evaluates the complete five-round pipelines.

Q3. LEF

Localized ESM features (LEF) pool ESM-2 embeddings at positions that differ from the wild type to fit the protein surrogate. We will define LEF at first use and cite ESM-2. See our reply to KX3C (Q1–Q2) and manuscript Appendix G, Table 9 for the pooling comparison.

Thank you for your answers. Unfortunately, I feel that they do not address my concerns, as elaborated below.

W1 Limited contribution

I do not see a substantial difference to what was previously my concern here. Specifically, one can draw the same conclusions from the additional analysis included in the authors rebuttal: greedy method finds more hits above certain fitness threshold than a method trying to optimize for fitness and diversity at the same time, with the candidates found by the latter approach being more diverse. I remain unconvinced that those observations differ from evaluating both approaches with conventional metrics (fitness and diversity together). As such, it remains unclear to me what is the advantage of assessing the candidates with the strategy proposed by the authors, or what conventional wisdom does it really challenge.

W2 Baseline choices

Thanks for the details, this information is relevant for reproducibility, however it does not justify why certain baselines were selected over others. As such, I still leaves me, and I would imagine a large group of potential readers, very confused. To be more specific, I still dont really understand why an exploration strategy (PEX) is being compared with acquisition functions (Greedy, UCB, or BALD). If the paper is revolving around active search methods (at least, that is my perception of this work), why not compare only exploration strategies, as done in the first experiment? Otherwise, why not focusing on active learning / Bayesian optimization AFs separately (EPIG / EI, etc.)? Again, to me it comes across as being conflated with little further consideration, and does require revisions for commiting to one of the two directions.

Thank you for clarifying the two points that remain unresolved. We address the information supplied by the shared gate and the choice of search baselines separately below.

1. What information does the shared gate add?

The additional information is where the high-quality candidates lie in sequence space. Separate fitness and diversity summaries can hide this association. Two candidate sets can have the same fitness distribution and the same pairwise-distance distribution, yet differ in the coverage of their qualifying candidates.

Consider the following illustrative four-sequence portfolios under one scoring function. Treat each portfolio as its method's reported top-K set, with K = 4. The column headings give the scores; this is an explanatory construction, not an additional experiment.

PortfolioScore 0.9Score 0.8Score 0.2Score 0.1
AAAAAAAACCCCACCCC
BGGGGTTTTGGGTTTTG

Both portfolios have the same score distribution and the same six pairwise Hamming distances: {1, 1, 3, 3, 4, 4}. Thus their mean fitness, best fitness, and mean pairwise distance are identical. At Hamming radius 2, both also have two connected groups before quality filtering. Applying the same cutoff of 0.5 retains two candidates in each portfolio. In A, the qualifying pair is one mutation apart and forms one group. In B, it is four mutations apart and forms two groups.

The shared gate therefore distinguishes qualifying coverage even when these separate quality and diversity summaries, and the number of qualifying candidates, agree. The difference is which sequence locations carry the high scores. This illustrates the information needed to answer our evaluation question: how many candidates meet the same quality goal, and how many qualifying groups do they cover?

The claim we challenge is that diversity within each method's own top-K set establishes comparable diversity of high-quality discoveries across methods. A within-method rank does not impose a common quality requirement. An analysis that applies a common cutoff and then measures diversity already follows the central principle of our protocol. Our contribution is to make this condition explicit and reproducible, with qualifying yield and coverage reported separately.

The matched RNA experiments apply this principle to actual search outputs. The cutoff and radius checks in our previous response assess how the observed yield-coverage pattern changes with the evaluation settings. We will use the construction above to explain the information gap before presenting those results, so that the contribution is clear without treating the familiar quality-diversity trade-off itself as a new finding.

2. Which comparison does the paper commit to?

We choose the active-search direction you suggest. The main comparison will be organized around search strategies with distinct roles:

  • TR-BEAM and AdaLead: local and evolutionary search.
  • GFlowNet-AL: generative search using trajectory balance.
  • delta-CS: generative search with explicit control of candidate diversity to account for proxy error.
  • Surrogate-assisted MAP-Elites: archive-based quality-diversity search.

This makes GFlowNet-AL the separate generative baseline, rather than using delta-CS to represent generative exploration as a whole. The five methods share the initial labels, oracle within each task and seed, and oracle-query budget. Their surrogate and proposal mechanisms remain part of their respective search strategies. These choices test the evaluation across local, generative, and archive-based search.

To make this scope concrete, we will move the original mixed PEX/Greedy/UCB/BALD ProteinGym comparison from the main results to the appendix. ProteinGym will remain a case study of applying the evaluation to experimentally measured fitness. The fixed-surrogate acquisition control will be a separately labelled supporting diagnostic. Its PEX-style rule is a control variant, not a reproduction of the original PEX pipeline. Neither comparison will be used to claim that one acquisition function is superior.

The main empirical argument will thus concern evaluation of budgeted active-search outputs. A systematic comparison of acquisition functions such as EI or EPIG would answer a different question. These changes will make the baseline rationale and the role of each experiment explicit, beyond renaming the original entries as complete pipelines.

Reviewer_KX3COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to identify target sequences within large search spaces. The particular setting is where the oracle is expensive, so there is a finite query-budget. The authors propose a new metric for evaluating search algorithms and finds that certain methods optimize better for quality and diversity.

Critical strengths

I think this is a very useful problem setting - large search spaces are a major issue in scientific domains, and ML-based approaches can help enable more efficient searches. The trade-off between novelty of samples and their quality is also something that many practicioners care about.

Critical weaknesses
  1. The paper claims to be about active learning, but the method from my understaning is entirely active search? Active learning's goal is to train a robust, high-quality model by iteratively selecting data that can held reduce the model's uncertainty. Active search focuses on identifying targets wtihin a large search space. Unless I am missing something obvious, it seems that the paper's stated claim and method are entirely different from each other.

  2. The paper is difficult to follow. Specifically, the method is not well described. I think the authors can benefit from defining the setting more carefully. For example, for equation (1), the variables being used are not well-defined, so it is hard for a reader to make sense of what the method is doing. A scoring function is mentioned, but it is not clear what that entails.

Questions for authors
  1. What is LEF? It is mentioned throughout the paper but it is not once defined.

  2. What is the main contribution of this paper? I feel that the story has been lost - from the abstract and introduction, I expected that the contribution is a better quality-diversity metric for sequence active search. But if that were the contribution, I would have expected deeper analysis into why this particular formulation (Eqn 3 + Eqn 4) is ideal over alternative formulations. Instead, the takeaway from the experiments is that ESM features are the most informative for searching over sequence spaces.

  3. Based on the answer to (2), I think the authors should consider expanding their baselines. In particular, if the focus is on metric is most informative for active search, I would be interested to learn more about what alternative formulations could look like. If the focus is on what features are useful, I would like to see more controlled studies explaining what each feature captures and why some are better than others.

4 comments

Thank you for raising the distinction between the search objective and the learning loop.

W1. Search objective

Our task is batch active search for a high-quality sequence portfolio under a fixed query budget. Active learning describes the loop that selects a batch, obtains scores, and updates the surrogate. After search, quality-gated evaluation uses the recorded scores to measure qualifying yield and sequence-space coverage without additional queries. We will clarify these roles in the introduction.

W2. Scoring and notation

We will define tt, mm, and ss as the task, method, and paired seed. StmsS_{tms} is the evaluated portfolio. It contains the 500 new queries in RNA and all 150 labels, including the 50 initial variants, in the main ProteinGym study. ft,s(x)f_{t,s}(x) is the score of sequence xx. For RNA, methods share a frozen, task- and seed-matched MLP oracle trained to approximate ViennaRNA folding scores. ProteinGym queries reveal recorded DMS measurements. Seeds change the initialization and selection trajectory; the DMS score of a given variant stays fixed. Eq. (1) takes the (1−α)(1-\alpha)-quantile of each portfolio's own scores. Eq. (2) uses a declared reference pool to set one common cutoff for all methods and seeds on each task.

Q1–Q2. Contribution and LEF

Our contribution is a post-hoc protocol for comparing qualifying yield and sequence-space coverage at a common quality threshold. It specifies the reference pool, matched query budgets, portfolio definition, and grouping radius. This makes a practical trade-off visible: one method can find more qualifying sequences while another covers more qualifying groups. The five-method analysis below tests this distinction with several diversity measures.

Localized ESM features (LEF) provide the ProteinGym case study. LEF pools ESM-2 embeddings at positions that differ from the wild type to fit the protein surrogate. On the same fixed 150-label splits, Appendix G, Table 9 reports held-out Spearman correlations of 0.349, 0.621, and 0.451 for local pooling versus 0.221, 0.310, and 0.066 for whole-sequence pooling on GB1, A4, and YAP1, respectively, averaged over three seeds. We will define LEF at first use and describe how mutation positions are identified relative to the wild type and pooled to construct the surrogate features.

Q2–Q3. Alternative diversity measures

We evaluated alternative measures on the saved queries from all 75 RNA runs: five methods, three tasks, and five paired seeds. This analysis uses the recorded scores and needs no new oracle calls. Exact uniqueness measures distinct sequences, mean pairwise Hamming distance measures average separation, connected components measure groups linked by short sequence changes, and greedy packing counts a set of separated representatives. We apply the same quality gate before computing each measure.

Rebuttal Table 1. Alternative measures on RNA3

All measures use the archived-reference p95 cutoff applied to all 500 new queries. Groups use Hamming radius r=2r=2. Packing scans sequences by decreasing oracle score, with ties in query order, and keeps a sequence when it is more than two mutations from every retained representative. It is a greedy, order-dependent count. Cells report mean ± sample SD. Hits, groups, and packing use all five seeds. Hamming distance uses the nn seeds with at least two qualifying sequences. MAP-Elites denotes the surrogate-assisted implementation.

MethodQualifying hitsConnected groupsGreedy packingMean Hamming distanceSeeds with ≥2 hits
TR-BEAM107.8 ± 73.71.40 ± 0.8911.80 ± 9.363.83 ± 0.394
δ-CS60.8 ± 60.21.60 ± 1.348.80 ± 7.984.40 ± 1.804
AdaLead111.0 ± 90.32.20 ± 0.4510.80 ± 7.334.34 ± 0.645
GFlowNet-AL2.4 ± 3.61.20 ± 1.641.60 ± 2.303.71 ± 0.062
Surrogate-assisted MAP-Elites25.8 ± 27.64.00 ± 2.359.20 ± 9.046.49 ± 1.104

Distance denominator. Mean Hamming distance is averaged over each method’s seeds with at least two qualifying hits; the last column reports this number. These means therefore use different eligible seed subsets. Hits, groups, and packing retain all five seeds.

These measures reveal different properties of the same qualifying sets. MAP-Elites covers more connected groups than TR-BEAM (4.0 vs. 1.4), while TR-BEAM supplies more representatives under greedy packing (11.8 vs. 9.2). MAP-Elites also has greater mean pairwise separation in all four seeds where both distances are defined. Reporting these measures together makes the choice of follow-up objective explicit: qualifying yield, coverage of disconnected groups, or separated representatives.

The query records also show how chaining affects group counts. On RNA2, seed 33, TR-BEAM has 234 p95 hits, two groups, and 27 packing representatives; GFlowNet-AL has 22 hits, three groups, and seven representatives. The maximum within-group Hamming distances are nine and seven, respectively, because sequences within each group are linked by paths of radius-2 edges. The radius sets the maximum edge length; group counts describe connectivity in the sampled qualifying graph. We will make this definition explicit and report the complementary measures alongside group counts.

Every run contains 500 unique new queries, so each unfiltered top-50 list has exactly 50 unique sequences. After gating, exact uniqueness equals the qualifying hit count. A common cutoff gives these counts a shared quality meaning. Our reply to rewj (W1, rebuttal Table 2) reports all three tasks and the yield–coverage comparison across cutoffs and radii.

I do not believe the author's response have addressed my concerns.

  1. This paper is indeed about active search, not active learning. This is a critical distinction. The goal of active learning is to improve the quality of the surrogate model. The goal of this paper is discovery under a constrained budget. This is important for defining the experiments and success criteria. If I am mistaken, then the paper needs to make its problem formulation signficantly more clear.

  2. The paper's stated contribution is not well-supported. The choice of baselines (as other reviewers have pointed out) is one important aspect. From the rebuttal and the paper, it is still not clear to me why the formulation proposed in this paper is ideal. I understand the rebuttal table looks at alternative distance measures - this is a good start. But, there needs to be much more analysis into this formulation.

I think this work has the potential to be quite useful for the community, but I think the current state of this paper requires major revisions for it to be read.

Thank you for the follow-up and for recognizing the potential value of this problem. We address the task definition and the rationale for the evaluation separately below.

1. The task is budgeted sequence active search.

We agree with this distinction. The objective is to discover high-quality sequences within a fixed oracle-query budget. Surrogate fitting supports the search; success is measured by the discovered sequences and their recorded oracle scores. We will use "active search" consistently in the title, abstract, problem formulation, and experimental objectives. The ESM feature comparison serves as a diagnostic within the protein case study, while discovery outcomes determine search performance.

2. The formulation follows an explicit evaluation objective.

Our central question is: under the same query budget and quality requirement, how many qualifying sequences does each method find, and how are they distributed across sequence groups? Three design choices make this question measurable:

  • A shared quality requirement. A method-local top fraction can set different score cutoffs across methods. The shared gate gives eligibility the same meaning across methods. With the reference pool and cutoff held fixed, adding a below-threshold sequence cannot improve either reported count.
  • Separate yield and coverage. Hit counts measure candidate supply; group counts describe its distribution. Reporting both preserves the distinction between many qualifying candidates concentrated in a few groups and fewer candidates spread across more groups, without imposing a weight that makes one compensate for the other.
  • An explicit meaning of a group. Connected components give an order-independent partition of the observed qualifying sequences at a declared Hamming radius. They count groups connected through short mutation steps. Greedy packing instead counts separated representatives under its specified selection order. The radius bounds an edge, not the diameter of an entire group. These summaries answer different follow-up questions.

This is why we report alternative measures alongside group counts: they test which conclusions depend on the intended notion of coverage. The design is justified by the stated search objective, rather than by treating one diversity formula as ideal for every objective.

The expanded experiments show why these distinctions matter. The RNA comparison now includes TR-BEAM, delta-CS, AdaLead, GFlowNet-AL, and surrogate-assisted MAP-Elites: three tasks and five paired seeds, with the same initial labels and frozen oracle within each task and seed, and 500 new queries per run. This provides evidence for the evaluation contribution independently of ESM features.

On RNA3 at the archived-reference p95 cutoff, TR-BEAM finds 107.8 qualifying sequences in 1.4 radius-2 groups, while MAP-Elites finds 25.8 sequences in 4.0 groups. Their greedy-packing counts are 11.8 and 9.2, respectively. All are means over the same five paired seeds (our earlier response, rebuttal Table 1). Thus, TR-BEAM supplies more qualifying candidates and separated representatives under the packing rule, while MAP-Elites covers more disconnected groups. The evaluation identifies which search output fits each follow-up objective; a single diversity ranking would hide this distinction.

The yield-versus-group ordering also holds across both archived-reference cutoffs, p90 and p95, and all three tested radii, 1, 2, and 3. At each radius, TR-BEAM has more hits and MAP-Elites more groups in all five seeds at p90 and four seeds at p95, with a zero-hit tie in the fifth. The archived-reference p95 cutoff is the primary shared gate because it was fixed before the follow-up campaigns. The cutoff recomputed from the follow-up scores is an exploratory sensitivity analysis. It changes the absolute counts but preserves the qualitative yield–coverage ordering on RNA3: 48.0 hits / 1.2 groups for TR-BEAM versus 10.8 / 2.6 for MAP-Elites (reply to rewj, rebuttal Table 2B). We do not use the recomputed cutoff to support a separate confirmatory claim. These sensitivity analyses reuse the completed campaigns and require no new oracle calls.

The contribution is therefore a shared-quality evaluation protocol, supported by controlled comparisons showing how candidate yield, group coverage, and representative selection lead to different assessments of search outputs. We will center the revised presentation on this task definition, design rationale, and supporting analyses, with LEF serving as the protein case study.

Comments on the paper

1 comment

Thank you for the detailed reviews. Our contribution is a reproducible protocol for comparing biological sequence search under a common quality standard. For each comparison, it fixes one task-level quality cutoff, declares the reference pool and evaluated candidates, and specifies the grouping radius. It then reports two outcomes: how many candidates qualify and how many connected sequence groups they cover. These outcomes support different choices for experimental follow-up and should be reported separately.

We have added matched comparisons of five search methods on three RNA tasks with five paired seeds, a ProteinGym control that holds the fitted surrogate and predictions fixed, and analyses of alternative diversity measures, cutoffs, radii, and paired variation. The RNA results show that the method with the most qualifying candidates need not cover the most qualifying groups. We also recompute the RNA gate within the follow-up to check sensitivity to the reference distribution. We will center the abstract, introduction, and contribution statement on this yield–coverage distinction.

The responses below separate the original complete-pipeline comparisons from the new controlled experiments. They also replace the earlier RNA summary comparison with explicitly defined quality-gated counts. The supplementary archive provides the saved follow-up queries, evaluation code, and complete paired statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.