- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors address a concern with evaluation of sequence-design models where the so-called "diversity-mirage" appears, where comparing diversity across models is often confounded by the fact that the sets used to compute these diversity metrics are not comparable, often being based on each model's own individual Top-K sets. To address this, the authors introduce a post-hoc method for recomputing model diversities using already-collected scores, stating that their quality-gated evaluation measures both the number of high-quality candidates and the distinct sequence groups that they cover.
Critical strengths
- The problem is very well motivated and the authors do a good job of justifying their approach. Design choices, such as reporting the qg-hits and qg-distinct scores separately, are rational and provide a clearer picture of each method's depth versus breadth
- The authors provide all of the code and scripts used to generate their results, providing a very high degree of reproducibility that enables in-depth auditing
- The method introduced by the authors is entirely post-hoc, meaning that it does not require expensive oracle evaluations or re-runs of past methods. This enables retrospective analysis of past campaigns in a straigthforward manner.
Critical weaknesses
- The authors should provide more comprehensive baselines in addition to TR-BEAM, there are many other methods that could be used such as those cited in the related work, e.g. GFlowNet-AL and AdaLead
- The statistical significance threshold is unjustified, as p95 and p90 are both presented. A domain-motivated choice of which to use should be provided in the text to encourage easy adaptation
- Stronger statistics are needed to reinforce the diversity reversal ranking for the RNA design task, and the variance across seed should be characterized and provided to the reader
Questions for authors
- Does the diversity ranking reversal hold for other methods? This is a key question that should be addressed and relates to the issue of including additional baselines
4 comments
W3 and Q. Ranking reversals and sensitivity
We extended the analysis to both archived-reference cutoffs, p90 and p95, and Hamming radii 1, 2, and 3. Each setting compares the ten method pairs on each of three tasks using five-seed mean group counts. A strict reversal requires opposite, nonzero differences between the unfiltered top-50 comparison and the gated comparison at the same radius; ties are excluded.
Rebuttal Table 5. Cutoff, radius, and seed sensitivity
Each cell reports strict five-seed mean reversals out of 30 task–method pairs. Parentheses give the number retained after every leave-one-seed-out check (LOO). “All 500” applies the gate to all new queries. “Fixed top-50” applies it within the same top-50 lists used before gating, holding the candidate lists fixed. Rows reuse the same query records and are descriptive sensitivity checks.
| Cutoff | Radius | All 500: mean reversals (LOO retained) | Fixed top-50: mean reversals (LOO retained) |
|---|---|---|---|
| p90 | 1 | 9 (2) | 3 (0) |
| p90 | 2 | 4 (1) | 0 (0) |
| p90 | 3 | 4 (0) | 0 (0) |
| p95 | 1 | 12 (1) | 8 (1) |
| p95 | 2 | 7 (1) | 5 (0) |
| p95 | 3 | 2 (0) | 4 (0) |
At p95 and , the all-500 reversals occur in zero, three, and four pairs on RNA1, RNA2, and RNA3. Five mean reversals also occur when the top-50 lists are held fixed. The table separates these changes in mean ordering from reversals preserved after every single-seed omission.
Rebuttal Table 6. Method pairs with a mean reversal at p95 and radius 2
Entries are five-seed mean differences in group counts, A minus B. Positive values mean that A covers more connected groups than B. Bold marks a strict change of sign relative to the unfiltered top-50 comparison. The table lists every pair with a reversal in either gated scope: seven in all 500 queries and five within the fixed top-50 lists. MAP-Elites is surrogate-assisted. Ties do not count as reversals; the denominator remains 30 task–method pairs.
| Task | Pair (A − B) | Top-50 difference | All-500 p95 difference | Fixed-top-50 p95 difference |
|---|---|---|---|---|
| RNA2 | δ-CS − MAP-Elites | -2.6 | +0.2 | +0.2 |
| RNA2 | GFlowNet-AL − MAP-Elites | -1.8 | +0.4 | +0.4 |
| RNA2 | TR-BEAM − δ-CS | +0.8 | +0.0 | -0.2 |
| RNA2 | TR-BEAM − MAP-Elites | -1.8 | +0.2 | +0.0 |
| RNA3 | AdaLead − GFlowNet-AL | -2.0 | +1.0 | +0.4 |
| RNA3 | δ-CS − AdaLead | +0.2 | -0.6 | +0.2 |
| RNA3 | δ-CS − GFlowNet-AL | -1.8 | +0.4 | +0.6 |
| RNA3 | TR-BEAM − GFlowNet-AL | -2.4 | +0.2 | -0.4 |
Complete paired statistics
The supplementary archive rebuttal_supplementary_data.zip provides all method pairs, including non-reversals, with per-seed counts, paired intervals, raw and Holm-adjusted -values, and leave-one-out checks. Its README.md indexes the complete tables and records.
Paired tests use exact two-sided sign flips, with Holm correction over the 30 task–method pairs for each fixed metric, candidate scope, cutoff, and radius. With five nonzero paired differences, the smallest attainable two-sided -value is ; zero paired differences or fewer valid pairs make it coarser. These tests assess paired metric differences under a given condition. Ranking-change counts remain descriptive, and we report effect sizes and seed consistency alongside them.
For AdaLead and GFlowNet-AL on RNA3, the all-500 mean reversal survives dropping any one seed. Their top-50 means are 1.4 versus 3.4 groups, and their p95 means are 2.2 versus 1.2. The paired p95 difference is +1.00 group for AdaLead (sample SD 1.87; approximate paired 95% interval [−1.32, 3.32]; two-sided exact sign-flip ). This paired difference is not statistically significant; we report it as descriptive and do not rely on it for the main claim. AdaLead leads at p95 in three seeds and trails in two; seed 44 has the strict within-seed reversal. The individual counts are in our reply to Anzt (W1 and Q1, rebuttal Table 4). At p90, the means are 1.8 versus 2.6. We therefore report this ordering change at its stated cutoff and radius, with the paired variation.
The yield–coverage trade-off provides a separate, more consistent example across the tested settings. As detailed in our reply to rewj (W1, rebuttal Table 2A), at the archived-reference cutoffs TR-BEAM finds more hits and MAP-Elites covers more groups on RNA3 in all five paired seeds at p90 and in four seeds at p95, with a zero-hit tie in the fifth. This holds at each of the three radii. Together, these results show why a comparison should declare its quality threshold and report qualifying yield and connected-group coverage separately.
Rebuttal acknowledged
I thank the authors for addressing the points raised in my review. While the review and subsequent clarifications do improve my understanding of the paper and the main arguments, and I look forward to the revised manuscript, I will maintain my score of 3.
Thank you for your careful review and for considering our responses. We appreciate your constructive feedback on the baseline comparisons, quality cutoffs, and variation across seeds. Your suggestions have helped us clarify the evaluation protocol and improve the presentation of the results. We have addressed the baseline scope in the response to rewj, the shared quality cutoff in the responses to KX3C and Anzt, and seed-level variability in revised Section 5.1 and Table 2. Thank you again for your time and thoughtful discussion.
Thank you for the suggestions on baseline coverage, cutoff choice, and variation across seeds.
W1. Additional search baselines
We added GFlowNet-AL, AdaLead, and surrogate-assisted MAP-Elites. The follow-up compares five methods on three RNA tasks, with five paired seeds per task and 500 new queries per run. See our reply to rewj for the full results (W1, rebuttal Table 2), the matched scoring and query protocol, and the MAP-Elites archive settings (W2). Under the archived-reference p95 cutoff on RNA3, AdaLead had the most hits on average (111.0), while MAP-Elites covered the most qualifying groups (4.0).
W2. Choosing a quality cutoff
The p95 cutoff is the 95th percentile of a fixed reference distribution and suits a selective follow-up screen. We use the archived 22-configuration reference suite for each RNA task, with p90 as a broader screen. A prespecified minimum activity or binding score can also define the common gate. On the original eight ProteinGym landscapes, LEF averaged more hits than Greedy at p50, p60, p70, p80, p90, p95, and p99 under the same pooled six-method reference suite. The means were 98.38 vs. 90.65 at p50, 14.90 vs. 10.43 at p95, and 3.23 vs. 2.25 at p99. Rebuttal Table 2B also checks a reference distribution formed within the new RNA runs. We will distinguish these screening percentiles from statistical p-values in the revision.