acceptodds
Under review as a conference paper at ICLR 2027

CRAFT: A Candidate-Retention Framework for Offline-to-Online Sequence Design

Abstract

Designing biological sequences under a limited experimental budget requires selecting a small number of candidates for evaluation. Even when candidate generation produces high-performing sequences, underestimation of their performance may prevent their selection for experimental evaluation. We propose CRAFT, an offline-to-online framework that separates candidate generation from experimental selection and uses accumulated measurements to guide both. Starting from an offline dataset, it constructs candidate pools via measurement-guided search, including local mutations of high-scoring observed sequences. A separate selector applies the same scoring rule to candidates from all sources, using an upper quantile of the measured property values of their nearest neighbors, and forms evaluation batches favoring high scores and sequence diversity. After each round, new measurements update candidate generation and scoring, while recent performance determines how future evaluations are allocated among candidate sources. Theoretically, for fixed candidate pools under bounded scoring error, we derive an upper bound on the gap in top-\(K\) mean objective value between the candidate pool and the evaluated batch, and relate this gap to cumulative improvement across rounds. Across six benchmarks spanning RNA, DNA, and protein design, CRAFT achieves the highest mean top-128 score on five tasks and ranks second on GFP under matched online query budgets. Our code is available at https://anonymous.4open.science/r/craft-code-private-3C88/.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Biological Sequence Design, Offline-to-online Optimization, Model-based Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QRtPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CRAFT, an offline-to-online framework for biological sequence design that separates candidate generation from experimental selection. Candidates from multiple sources are scored by a shared kNN upper-quantile rule, selected with farthest-first diversity, and allocated across sources via discounted softmax returns. The paper provides a bound on candidate-to-batch quality loss and reports the best top-128 score on five of six tasks.

Critical strengths
  • Separating generation from selection is new to me.
  • Compare with several recent baselines on biological sequence design tasks.
  • Extensive ablation studies on several components.
Critical weaknesses
  • Unfortunately, the gain between other baselines and CRAFT is too small. As far as I understand, the gap between other baselines and CRAFT should be larger when the sequence length becomes longer, but the results is the opposite. It underperforms on GFP benchmarks.
  • No hyperparameter sensitivity analysis is reported, despite several task-specific settings (τ,η\tau, \eta)
Questions for authors
  • Justify the poor performance on GFP task. Furthermore, it is better to include more experiments on longer sequences.
  • It would be better to add hyperparameter sensitivity analysis.
3 comments
AuthorsOpenReview ↗

Response to Reviewer QRtP (1/2)

1. How substantial are the performance gains, why is CRAFT weaker on GFP, and does its advantage increase on longer sequences?

We appreciate the reviewer's careful reading and thoughtful questions, which have helped us clarify our contributions and strengthen the evaluation. As discussed in our response to Reviewer zbeL, CRAFT accommodates different selectors within its candidate-generation and feedback-update workflow. We therefore include Shared Ridge as an alternative selector.

Final scores across six tasks

TaskQuantile (Table 1)Strongest other methodOther scoreQuantile − otherRidgeRidge − matched Quantile
RNA10.9397GFN-AL+δ\delta-CS0.9211+0.0186−−
RNA20.9385GFN-AL+δ\delta-CS0.9313+0.00720.9358+0.0034
RNA30.9878GFN-AL+δ\delta-CS0.9468+0.0410−−
AAV0.6768ProSpero0.6731+0.00370.7610+0.0733
GFP3.6127ProSpero3.6549−0.04223.6099-0.0028
TFBind80.9716GFN-AL+δ\delta-CS0.9713+0.00030.9741+0.0025

In the five-seed selector comparisons, Ridge improves mean final quality over matched Quantile controls on three reported tasks, including AAV (+0.0733). This supports task-dependent selector choice within CRAFT. Replacing the selector improves final quality on AAV and RNA2, demonstrating task-dependent benefits of the framework.

Why does CRAFT achieve a lower GFP endpoint than ProSpero?

The GFP gap develops mainly in the later part of the search. CRAFT discovers high-scoring candidates earlier than ProSpero, but makes smaller subsequent improvements.

Entries below show the cumulative top-128 mean among newly evaluated candidates, excluding the initial data, for the original Quantile configuration in Table 1.

Method128 queries256 queries640 queries1,280 queries
CRAFT3.18693.58143.59893.6127
ProSpero1.94983.04613.59943.6549
PEX3.51933.57753.58433.5922

At 640 queries, CRAFT and ProSpero have nearly identical scores. Over the remaining 640 queries, CRAFT improves by 0.0138, whereas ProSpero improves by 0.0555, producing the final gap of 0.0422.

A plausible explanation is concentrated exploitation under the current GFP configuration. Its candidate sources generate single-substitution neighbors around measured elites, while the shared upper-quantile score favors candidates near high-valued observations. Together, these choices may reinforce search within already promising neighborhoods. Diversity selection operates within score-based shortlists, so it does not necessarily ensure broad exploration.

The final discoveries are consistent with this interpretation: CRAFT's mean normalized pairwise Hamming distance is 0.0128, compared with 0.0395 for PEX and 0.0600 for ProSpero. CRAFT's retained sets are therefore more concentrated, although final-set diversity alone does not establish why later improvement is smaller.

Evaluation on longer-protein benchmarks

We extend the evaluation to four experimentally measured protein variant libraries: TEM-1 [1], LGK [2], Hsp82 [3], and Cas9 [4]. These datasets cover proteins ranging from 286 to 1,368 residues.

MethodTEM-1 (286 aa)LGK (439 aa)Hsp82 (709 aa)Cas9 (1,368 aa)
CRAFT1.34670.10320.42130.8983
PEX [1]1.29280.06460.44210.8801
MLDE1.34310.09480.47650.8895
AdaLead1.26130.02910.30810.8759
BO1.26830.03750.31340.8523

CRAFT achieves the highest mean endpoint among the completed methods on TEM-1, LGK, and Cas9. These results demonstrate competitive performance on longer proteins within the evaluated single-substitution libraries.The corresponding experimental code and results are available in our anonymous repository.

References

[1] Firnberg, E., Labonte, J. W., Gray, J. J., and Ostermeier, M. (2014). A Comprehensive, High-Resolution Map of a Gene’s Fitness Landscape. Molecular Biology and Evolution, 31(6), 1581–1592.

[2] Klesmith, J. R., Bacik, J.-P., Michalczyk, R., and Whitehead, T. A. (2015). Comprehensive Sequence-Flux Mapping of a Levoglucosan Utilization Pathway in E. coli. ACS Synthetic Biology, 4(11), 1235–1243.

[3] Flynn, J. M., Rossouw, A., Cote-Hammarlof, P. A., Fragata, I., Mavor, D., Hollins, C., III, Bank, C., and Bolon, D. N. A. (2020). Comprehensive fitness maps of Hsp90 show widespread environmental dependence. eLife, 9, e53810.

[4] Spencer, J. M., and Zhang, X. (2017). Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains. Scientific Reports, 7, 16836.

AuthorsOpenReview ↗

Response to Reviewer QRtP (2/2)

2. How sensitive is CRAFT to the task-specific quantile τ and allocation temperature η?

Thank you for this thoughtful question. We evaluated both parameters over a range of settings. The following tables report mean final J128J_{128} after 1,280 queries from the corresponding sensitivity experiments.

Quantile sensitivity, with k=16k=16

τ\tauRNA1RNA2RNA3AAVGFPTFBind8
0.500.93130.94080.98980.66503.61300.9434
0.700.93930.93590.98340.69473.61130.9499
0.850.93870.93580.98160.68913.61470.9624
0.950.93400.93530.97570.65713.61340.9718
0.990.93680.93900.97870.65193.61390.9679

The quantile sweep also indicates generally stable final-discovery performance across multiple settings, with limited variation on RNA1–3 and GFP and similar endpoint scores within moderate quantiles (0.700.70–0.850.85) for AAV and higher quantiles (0.850.85–0.990.99) for TFBind8.

Allocation-temperature sensitivity

We retained the reference quantile: 0.85 for AAV and 0.95 otherwise. The reference temperature is η0=0.02\eta_0=0.02 for AAV and η0=0.08\eta_0=0.08 for the other tasks.

TemperatureRNA1RNA2RNA3AAVGFPTFBind8
0.25η00.25\eta_00.93400.93530.97570.69373.61260.9725
0.50η00.50\eta_00.93400.93530.97570.69133.61280.9730
1.00η01.00\eta_00.93400.93530.97570.68363.61320.9718
2.00η02.00\eta_00.93400.93530.97560.68443.61300.9713
4.00η04.00\eta_00.94500.93760.97610.69433.61350.9711

Overall, CRAFT maintains generally stable final-discovery performance across the tested settings, particularly for allocation temperature. Across the sixteenfold temperature range, the maximum–minimum difference in mean endpoint is less than 1.6% of the corresponding η0\eta_0 score on every task. Quantile variation also produces limited endpoint changes on RNA1–3 and GFP. On AAV and TFBind8, the reference quantiles remain near the best tested means, with moderate and higher quantiles favored, respectively. These results support low sensitivity to precise temperature tuning and stable performance with task-appropriate quantile choices.

The neighborhood sensitivity analysis appears in Sec. 5.3 and Table 3. Further parameter studies are provided in Appendix F, Tables F.3–F.5; the additional AAV comparisons are reported above.

Dear Reviewer QRtP,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_Lzp3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CRAFT, an offline-to-online biological sequence design framework that explicitly decouples candidate generation, shared candidate selection, and dynamic source budget allocation. By utilizing a non-parametric nearest-neighbor upper-quantile scoring rule alongside an adaptive portfolio allocation scheme, CRAFT demonstrates strong candidate retention and competitive top-128 discovery scores across six benchmark tasks.

Critical strengths
  • The method is novel and well explained
  • The experiments and ablations are carried out well
Critical weaknesses
  • Missing Baselines: Key biological sequence optimization baselines such as Feedback GAN (Gupta & Zou, 2019) and IDEAS (Pandey et al., 2026) are missing. Could the authors include or discuss comparisons against these methods to better contextualize performance?
  • Could the authors include or discuss comparisons against these methods to better contextualize performance? Convergence Speed & Area Under the Curve (AUC): Many methods in Figure 2 eventually converge to similar maximum fitness values by round 10. To show which algorithms reach high-scoring sequences faster and better measure sample efficiency, could the authors compute and report the Area Under the Curve (AUC) for the fitness progression curves across rounds?
  • What happens to the performance if we reduce the B to 20 or 50 from 128?
  • What is the impact of the size of initial dataset?
  • Can CRAFT be used to adopt different baselines (particularly lightweight methods such as Adalead) discussed in the results as the sources in their method? What would be the impact of it?
Questions for authors

Mentioned in Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer Lzp3 (1/2)

We appreciate the reviewer's constructive feedback. We have expanded the comparisons and examined final quality, trajectory growth, and query efficiency.

1. Comparisons with Feedback GAN and IDEAS

Thank you for highlighting Feedback GAN [1] and IDEAS [2]. Their adaptations use ten rounds of 128 evaluations. Final quality is the mean score of the best 128 newly evaluated candidates after 1,280 queries.

TaskCRAFTIDEASFeedback GAN
RNA10.93970.83950.3279
RNA20.93850.88060.2877
RNA30.98780.82020.4079
AAV0.67680.00230.5213
GFP3.61273.52032.8534
TFBind80.97160.95240.9173

CRAFT exceeds both adaptations on all six tasks in this comparison. These results appear in Sec. 5, Table 1 of the revised manuscript; adaptation details appear in Appendix B.3.

2. Which methods discover high-scoring candidates sooner?

Thank you for this thoughtful question. We supplement full-trajectory AUC with growth, target attainment, and joint AUC–endpoint comparisons against PEX [3], ProSpero [4], AdaLead [5], and GFN-AL+δ\delta-CS (GFlowNets [6]; δ\delta-CS [7]).

AUC of improvement across query rounds

We summarize improvement across query rounds using a discrete AUC measure, accumulating each method’s gains above its first-round score over the remaining nine rounds. This captures how improvements develop and persist during the search. Bold marks the highest value within each task.

18:54

TaskCRAFTPEXProSperoAdaLeadGFN-AL+δ\delta-CS
RNA14.733.824.184.273.74
RNA24.993.554.264.083.93
RNA34.683.523.934.534.17
AAV1.820.701.241.090.86
GFP3.720.6014.322.192.03
TFBind83.693.973.013.542.24

CRAFT ranks first on four tasks and second on the remaining two, exceeding AdaLead and GFN-AL+δ\delta-CS on all six. These results indicate strong accumulated improvement across query rounds, with the magnitude also influenced by each method’s starting score.

Queries to a fixed target

The target is each task's final GFN-AL+δ\delta-CS score, which the reference method reaches at 1,280 queries.

TaskTargetCRAFT queriesOther methods reaching the target
RNA10.9211896—
RNA20.93131,152—
RNA30.9468896—
AAV0.64381,152ProSpero: 1,152; AdaLead: 1,280
GFP3.5770256PEX: 256; ProSpero: 640
TFBind80.97131,280—

The last column excludes the reference method. Among these five methods, CRAFT reaches these targets earliest or joint-earliest on all six tasks. On GFP, it uses 80% fewer queries than GFN-AL+δ\delta-CS, tying PEX.

Connection to the manuscript. Sec. 5.1, Figure 2 reports the discovery curves and discusses differences between endpoint and trajectory rankings. Appendix C.4, Eq. (C.3) defines the original discrete AUC. The growth, target-attainment above are supplementary analyses supplied in this response.

References

[1] Gupta and Zou (2019). Feedback GAN for DNA optimizes protein functions. Nature Machine Intelligence.

[2] Pandey et al. (2026). Interpretability Driven Evolutionary Approach for the Design of Biological Sequences. ICML.

[3] Ren et al. (2022). Proximal Exploration for Model-guided Protein Sequence Design. ICML.

[4] Kmicikiewicz et al. (2025). ProSpero: Active Learning for Robust Protein Design Beyond Wild-Type Neighborhoods. NeurIPS.

[5] Sinai et al. (2020). AdaLead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv:2010.02141.

[6] Jain et al. (2022). Biological Sequence Design with GFlowNets. ICML.

[7] Kim et al. (2025). Improved Off-policy Reinforcement Learning in Biological Sequence Design. ICML.

AuthorsOpenReview ↗

Response to Reviewer Lzp3 (2/2)

3. How does reducing the feedback batch size from 128 to 50 or 20 affect performance?

Thank you for this suggestion. In the batch-size study, we kept the full initial dataset and the total budget of 1,280 queries fixed, using eight paired seeds per task.

TaskB=20B=20B=50B=50B=128B=128 study reference
RNA10.91410.92170.9325
RNA20.91620.91740.9365
RNA30.93840.93240.9770
AAV0.78100.74920.6701
GFP3.63563.62583.6124
TFBind80.97560.97450.9721

Batch sizes of 20, 50, and 128 correspond to 64, 26, and 10 feedback rounds, respectively; the final batch at B=50B=50 contains 30 queries.

Smaller batches improve final scores on AAV, GFP, and TFBind8, while reducing them on RNA1–3. More frequent feedback also increases computation: reducing BB from 128 to 20 increased recorded online-loop time from 18.28 to 90.18 seconds on AAV and from 37.27 to 172.64 seconds on GFP, averaged over eight runs per setting. Batch size therefore introduces a task-dependent trade-off between performance and computational cost.

The batch-size comparisons appear in Sec. 5.3, Table 4, with supplementary controls in Appendix F.3, Table F.5. The runtime measurements above provide additional context for those comparisons.

4. How does the size of the initial labeled dataset affect performance?

Thank you for this thoughtful question. With B=128B=128, we evaluated nested subsets containing 25%, 50%, 75%, and 100% of the initial dataset, sampled without using label values. Every condition excludes the entire original offline dataset from new queries. This study shares the eight-seed reference used in the batch-size study.

Task25%50%75%100% study reference
RNA10.93640.94700.94410.9325
RNA20.93140.93480.93350.9365
RNA30.97880.97910.98060.9770
AAV0.66110.64840.67470.6701
GFP3.61013.61153.61323.6124
TFBind80.97350.97310.97240.9721

Final performance is not monotonic in initial-data size. GFP and TFBind8 vary little across these fractions, while AAV shows a larger decrease at 50%. Because this intervention changes the data available to both generation and selection, its effect cannot be attributed solely to the selector's reference-set size.

These comparisons appear in Sec. 5.3, Table 4, with additional results in Appendix F.3, Table F.5.

5. Can AdaLead or BO be incorporated as candidate sources, and does this improve performance?

We evaluated AdaLead and BO as additional candidate sources for CRAFT. The source study used eight seeds per task, matched offered-candidate caps, and a budget of 1,280 new evaluations.

The first three columns below reproduce the main-table results unchanged. The augmented configurations come from the additional candidate-source study.

TaskAdaLead (Table 1)BO (Table 1)CRAFT (Table 1)CRAFT + AdaLeadCRAFT + BO
RNA10.89630.55070.93970.94060.9411
RNA20.89210.54990.93850.93370.9265
RNA30.92060.53720.98780.98100.9448
AAV0.64950.01040.67680.69180.6820
GFP3.57062.13973.61273.61233.6128
TFBind80.96650.86440.97160.97220.9733

The augmented configurations show higher reported final scores on RNA1, AAV, and TFBind8, with a smaller increase on GFP when adding BO. These descriptive comparisons illustrate the interface's potential to deliver task-dependent benefits; controlled effects are assessed against the source study's matched CRAFT reference in Appendix F.2.

Additional sources also increase computation. In the matched RNA1 source study, recorded runtime rises from 44.1 seconds for the CRAFT reference to 299.3 seconds with AdaLead and 338.1 seconds with BO. These results support the flexibility of the interface while showing that each added source should be assessed for both performance and cost. We thank the reviewer again for this constructive suggestion, which highlights opportunities to extend CRAFT to a broader range of candidate-generation methods.

The candidate-source interface is described in Sec. 3.1. Main-method references appear in Table 1, while matched source comparisons and runtime measurements appear in Appendix F.2, Table F.2 and its accompanying discussion. We have also updated our anonymous repository with the corresponding implementations.

Dear Reviewer Lzp3,

We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.

Reviewer_zbeLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRAFT, an offline-to-online framework for biological sequence design under a limited evaluation budget. It addresses the problem that good candidates may be generated but overlooked during selection. CRAFT separates candidate generation from selection, scores candidates using the measured values of nearby sequences, and builds batches that balance scores and sequence diversity. New measurements update both the search process and the allocation of evaluations across candidate sources. The paper provides a theoretical analysis of candidate-quality retention and evaluates CRAFT on six benchmarks, achieving the highest score on five tasks and ranking second on GFP.

Critical strengths
  1. The separation between candidate generation and experimental selection is useful and well motivated. A generator can produce good candidates without ranking them reliably, so evaluating candidate-pool quality separately from queried-batch quality provides a clearer picture of where performance is lost. I also appreciate that the analysis distinguishes scoring error from losses introduced by screening, quotas, and diversity handling.

  2. The empirical investigation goes beyond reporting final benchmark scores. The paper includes matched feedback interventions, fixed-pool diagnostics, full-trajectory selector comparisons, and an independent ten-seed AAV replication. In particular, the AAV replication provides evidence that shared selection improves over native ordering under the evaluated protocol. The authors are also reasonably transparent about negative results and the limits of their statistical conclusions.

Critical weaknesses
  1. I am not sure that the upper-quantile selector is sufficiently justified. Shared Ridge performs substantially better on AAV (Table C.5), while the quantile selector shows no consistent final-discovery advantage over native ordering on RNA3/GFP. P3BO is also closely related but missing from the evaluation. A comparison using the same candidate sources would help establish how much improvement comes from the shared selection stage.

  2. The text and equations in Figure 1 are crowded together, making the workflow difficult to follow. A cleaner layout with clearer connections between stages would improve readability.

Questions for authors
  1. Given that Shared Ridge achieves substantially better results on AAV, what motivates the use of the upper-quantile selector as the default? Does it offer advantages in computational cost or stability when both selectors use the same reference data?

  2. Separating candidate generation from selection is a well-motivated design choice. Can you demonstrate how much improvement this separation provides over a P3BO-style portfolio using the same candidate sources and query budget?

  3. How do the neighborhood size kk and quantile level τ\tau affect the reliability of candidate scoring? Does using a small neighborhood with a high quantile make the selector overly sensitive to a few high-scoring observations, leading to less reliable candidate rankings?

  4. Typo: Full stop missing at L.1460.

6 comments
AuthorsOpenReview ↗

Response to Reviewer zbeL (1/3)

We are grateful to the reviewer for the careful reading and constructive suggestions. In response, we clarify the role of the default selector and compare its computational cost and performance variation with Shared Ridge. We also examine the contribution of shared selection, sensitivity to neighborhood size and quantile level, and the presentation issues raised below.

1. Why use the upper-quantile selector as the default, and how does it compare with Shared Ridge in computational cost and stability?

CRAFT's specific contribution here is to establish candidate selection as an independently evaluable and replaceable stage. We use the upper-quantile rule as a transparent reference instance: it scores candidates directly from measured neighboring labels without fitting an additional predictor. This provides a simple implementation, while computational efficiency and stability require empirical assessment. Other scoring rules can be substituted within the same framework, including our shared linear selector based on ridge regression.

We agree that Shared Ridge is the stronger choice in the evaluated AAV setting. We therefore position Quantile as a reference selector whose suitability should be assessed for each task. We appreciate the reviewer's observation, which helps us distinguish the framework's contribution from the merits of a particular scoring rule.

Cost and stability

Following the experimental settings in the manuscript, we use three seeds per task for fixed-input timing and five paired seeds per task for the online AAV and GFP comparisons.And J128J_{128} is the mean score of the best 128 newly evaluated candidates after 1,280 queries, excluding the initial offline data.

MeasurementQuantileShared Ridge
Fit-plus-score time, AAV (s)1.0451.079
Fit-plus-score time, GFP (s)2.1373.298
Complete online run, AAV (s)19.2915.70
Complete online run, GFP (s)41.6566.04
Final J128J_{128}, AAV0.687720.76099
Final J128J_{128}, GFP3.612273.60990
Across-seed SD of final J128J_{128}, AAV0.027090.00157
Across-seed SD of final J128J_{128}, GFP0.002120.00135

Runtime comparisons are within the same task and measurement protocol.

Shared Ridge achieves a higher final score and shorter online runtime on AAV, whereas Quantile is faster and achieves a slightly higher final score on GFP; Ridge has smaller across-seed endpoint SD on both tasks, supporting a task-dependent choice of selector. We appreciate the reviewer's insightful observation and agree that Shared Ridge is the stronger choice in the evaluated AAV setting, highlighting the importance of assessing the selector separately for each task.

In the revised manuscript, we clarify the scoring rule in Sec. 3.2, Eq. (7), and detail the selector-comparison setup in Sec. 5.2 and Appendix F.1 (Table F.1).

AuthorsOpenReview ↗

Response to Reviewer zbeL (2/3)

2. How much does shared selection contribute beyond a P3BO-style portfolio under the same candidate sources and query budget?

We are grateful to the reviewer for suggesting this comparison. We first summarize the complete-method results across all six datasets, then present matched-source controls to assess the contribution of shared selection.

Comparison across six datasets

Our main comparison includes a 15-member adaptation of the P3BO portfolio [1]. Under the same budget of 1,280 queries and 8 seeds, CRAFT achieves a higher mean final J128J_{128} than adapted P3BO on all six tasks .

TaskCRAFTP3BODifference
RNA10.93970.8907+0.0490
RNA20.93850.8570+0.0815
RNA30.98780.7935+0.1943
AAV0.67680.6647+0.0121
GFP3.61273.6039+0.0088
TFBind80.97160.9457+0.0259

These are descriptive comparisons of the reported means. Because the complete methods differ in both candidate generation and selection, we additionally conducted the following matched-source comparisons.

Matched-source comparison

We kept the three candidate-source procedures, initial archive, diversity rule, and budget of 1,280 queries unchanged. These additional experiments use five paired seeds for AAV and three paired seeds for GFP. First-batch quality is the mean measured score of the first 128 queried candidates.

TaskAllocationRankingFirst batchFinal J128J_{128}
AAVP3BO-styleNative0.331060.63264
AAVP3BO-styleShared quantile0.390090.67608
AAVCRAFTShared quantile0.394910.68772
GFPP3BO-styleNative3.040853.60929
GFPP3BO-styleShared quantile3.205313.60975
GFPCRAFTShared quantile3.206223.61386

Keeping the P3BO-style allocation rule fixed, replacing native ranking with shared quantile ranking improves first-batch quality by 0.05903 on AAV and 0.16446 on GFP, with positive gains in every seed pair. The final-score improvement is 0.04344 on AAV (5/5 positive) and 0.00046 on GFP (2/3 positive). Thus, the AAV improvement persists to the endpoint, while the clearest GFP benefit is in the first queried batch.

Combining CRAFT allocation with shared quantile ranking gives final-score gains of 0.05508 on AAV and 0.00457 on GFP relative to P3BO-style allocation with native ranking.

Fixed-pool comparison

Online histories and realized quotas can diverge after different batches are evaluated. We therefore also fixed candidate pools, history, references, quotas, and diversity, changing only the ranking rule. Shared quantile ranking improved mean selected-batch quality over native ranking by 0.05882 on AAV and 0.21747 on GFP, with positive differences in all three seeds per task.

We appreciate the reviewer's suggestion and agree that these matched comparisons provide a more direct assessment of shared selection: it improves evaluated-batch quality, while the extent of the final-score gain depends on the task.

In the revised manuscript, we report the six-task comparison in Sec. 5, Table 1, and describe the P3BO adaptation in Appendix B.3. The additional matched-source and fixed-pool results are reported above.

[1] Angermueller et al. (2020). Population-Based Black-Box Optimization for Biological Sequence Design.

AuthorsOpenReview ↗

Response to Reviewer zbeL (3/3)

3. How do neighborhood size and quantile level affect scoring reliability? Can a high quantile in a small neighborhood be overly sensitive to a few high observations?

Thank you for raising this concern. We examine how neighborhood size and quantile level affect final discovery performance, then assess ranking sensitivity through fixed-pool reference-removal experiments.

Sensitivity in small neighborhoods

We compare three evenly spaced quantiles at each neighborhood size. Entries show mean final J128J_{128} after 1,280 new queries, using three paired seeds for AAV and five for each other task. The task reference is (16,0.85)(16,0.85) for AAV and (16,0.95)(16,0.95) for the other tasks.

kkτ\tauRNA1RNA2RNA3AAVGFPTFBind8
40.850.94230.93700.99520.68193.61210.9749
40.900.94130.93020.98540.67903.61200.9766
40.950.94680.93740.98180.68543.61370.9754
80.850.93510.93180.98440.66583.61290.9664
80.900.93300.93960.98670.68113.61310.9685
80.950.92820.93590.98050.69793.61210.9726
160.850.93870.93580.98160.68913.61470.9624
160.900.93420.93930.98920.67703.61230.9680
160.950.93400.93530.97570.65713.61340.9718

The (4,0.95)(4,0.95) setting achieves a higher mean endpoint than the task reference on five of six tasks. On AAV, (8,0.95)(8,0.95) also improves on the reference. These descriptive comparisons concern final discovery performance; we examine ranking sensitivity separately below.

Ranking sensitivity to reference removal

We agree that high quantiles in small neighborhoods can amplify the influence of individual high-valued observations. Under the interpolation in Eq. (7) [1], the highest neighboring label receives weight 0.85 at (4,0.95)(4,0.95), compared with 0.25 at (16,0.95)(16,0.95).

We additionally evaluated (k,τ)=(4,0.95)(k,\tau)=(4,0.95) on fixed first-round candidate pools. Below, we run three tasks , using five candidate-pool seeds per task. For random deletion, three deletion masks were averaged within each pool seed before aggregation across seeds.

Jaccard measures top-128 overlap with cutoff ties weighted proportionally. Δ\Delta quality is the change in the mean measured score of the selected top 128 relative to the unperturbed reference set. All entries are descriptive means across the five pool seeds.

TaskReference perturbationSpearmanTop-128 JaccardΔ\Delta quality
RNA1Randomly remove 5%0.92170.8640−0.00021
RNA1Remove highest-label reference1.00001.00000.00000
RNA3Randomly remove 5%0.92700.9123−0.00094
RNA3Remove highest-label reference1.00001.00000.00000
TFBind8Randomly remove 5%0.89470.8235−0.00400
TFBind8Remove highest-label reference0.99731.00000.00000

On these three tasks, randomly removing 5% of the references preserves substantial ranking correlation and top-128 overlap. Removing the highest-labeled reference leaves top-128 membership and mean quality unchanged. These results provide direct evidence at k=4k=4 for the selected tasks under the tested deletions; they do not establish robustness to arbitrary reference or label perturbations.

Sec. 5.3 and Table 3 report paired sensitivity results, and Appendix Table F.3 provides the broader sweep for the five tasks other than AAV.

4. Presentation of Figure 1 and the missing period at line 1460.

Thank you for pointing out these issues. We revised Figure 1 in Sec. 3 to show what each stage takes as input and passes to the next stage.

  • Reach generates candidate pools from the measured history, which starts with the offline data.
  • Selection uses a shared quantile score, diversity, and source quotas to assemble a batch. The figure shows that the batch and its source attribution are fixed before evaluation.
  • Adaptation uses the new measurements to update the history and adjust source quotas. Two dashed arrows show where these updates enter the next round.

The caption explains the feedback arrows and clarifies when the batch is committed. We will also add the missing period at line 1460 of the submitted version.

[1] Hyndman and Fan (1996). Sample Quantiles in Statistical Packages. The American Statistician.

Comments on the paper

1 comment

We thank the Area Chair and all three reviewers for their time and constructive feedback. We are especially grateful to Reviewer Lzp3 for recognizing CRAFT’s novelty and clarity and for reassessing the paper after our response, raising their recommendation from 2 to 3 . We also appreciate zbeL’s observation that separating generation from selection helps identify where promising candidates are lost, and QRtP’s recognition of the breadth of our baseline comparisons and component ablations. We have responded individually to all concerns raised during the review.

CRAFT is an offline-to-online framework for deciding which candidates to evaluate under a limited experimental budget. Offline measurements both guide candidate generation and provide a common reference for scoring proposals from different sources. Each evaluated batch adds measurements that update both generation and selection. The performance of each source also informs its share of the next round’s budget. This design lets us measure how much of the quality available in a candidate pool is retained in the batch selected for evaluation.

1. Contribution of shared selection (zbeL). We held the candidate sources, P3BO-style allocation rule, diversity rule, and budget of 1,280 new queries fixed. On AAV, replacing native rankings with shared quantile ranking improved the final mean score of the best 128 newly evaluated sequences by 0.04344, with gains on all five paired seeds. On GFP, the clearest benefit came early. The first-batch mean improved by 0.16446, while the final gain was 0.00046.

2. Adapting candidate sources and selectors (zbeL, Lzp3). We integrated AdaLead and Bayesian optimization as additional candidate sources. In the matched RNA3 comparison, adding AdaLead raised the mean endpoint from 0.9728 to 0.9810. The selector can also be replaced independently. With the same candidate sources on AAV, switching from the reference quantile selector to Shared Ridge raised the mean endpoint from 0.6877 to 0.7610.

3. Performance on short and long sequences (Lzp3, QRtP). CRAFT achieved the highest mean endpoint on five of the six main tasks, including all three 14-nucleotide RNA tasks, and ranked second to ProSpero on GFP. On four measured single-substitution libraries spanning proteins of 286 to 1,368 residues, CRAFT achieved a higher mean endpoint than both AdaLead and Bayesian optimization on every library across eight seeds per method.

4. Offline data and online feedback (zbeL, Lzp3, QRtP). We varied the fraction of available offline labels from 25% to 100%, compared batch sizes of 20, 50, and 128, and examined sensitivity to neighborhood size, quantile level, and allocation temperature. At the same total query budget, smaller batches improved mean final scores on AAV, GFP, and TFBind8, while a batch size of 128 performed best on all three RNA tasks. Appendix F.4 describes the long-sequence experimental settings.

We also redrew Figure 1, corrected punctuation, documented the added baseline methods and the processing of the new datasets, and updated our anonymous repository with the corresponding code and results.

We thank the Area Chair and reviewers again for the opportunity to clarify the framework and present these additional results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.