- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CRAFT, an offline-to-online framework for biological sequence design that separates candidate generation from experimental selection. Candidates from multiple sources are scored by a shared kNN upper-quantile rule, selected with farthest-first diversity, and allocated across sources via discounted softmax returns. The paper provides a bound on candidate-to-batch quality loss and reports the best top-128 score on five of six tasks.
Critical strengths
- Separating generation from selection is new to me.
- Compare with several recent baselines on biological sequence design tasks.
- Extensive ablation studies on several components.
Critical weaknesses
- Unfortunately, the gain between other baselines and CRAFT is too small. As far as I understand, the gap between other baselines and CRAFT should be larger when the sequence length becomes longer, but the results is the opposite. It underperforms on GFP benchmarks.
- No hyperparameter sensitivity analysis is reported, despite several task-specific settings ()
Questions for authors
- Justify the poor performance on GFP task. Furthermore, it is better to include more experiments on longer sequences.
- It would be better to add hyperparameter sensitivity analysis.
3 comments
Response to Reviewer QRtP (2/2)
2. How sensitive is CRAFT to the task-specific quantile τ and allocation temperature η?
Thank you for this thoughtful question. We evaluated both parameters over a range of settings. The following tables report mean final after 1,280 queries from the corresponding sensitivity experiments.
Quantile sensitivity, with
| RNA1 | RNA2 | RNA3 | AAV | GFP | TFBind8 | |
|---|---|---|---|---|---|---|
| 0.50 | 0.9313 | 0.9408 | 0.9898 | 0.6650 | 3.6130 | 0.9434 |
| 0.70 | 0.9393 | 0.9359 | 0.9834 | 0.6947 | 3.6113 | 0.9499 |
| 0.85 | 0.9387 | 0.9358 | 0.9816 | 0.6891 | 3.6147 | 0.9624 |
| 0.95 | 0.9340 | 0.9353 | 0.9757 | 0.6571 | 3.6134 | 0.9718 |
| 0.99 | 0.9368 | 0.9390 | 0.9787 | 0.6519 | 3.6139 | 0.9679 |
The quantile sweep also indicates generally stable final-discovery performance across multiple settings, with limited variation on RNA1–3 and GFP and similar endpoint scores within moderate quantiles (–) for AAV and higher quantiles (–) for TFBind8.
Allocation-temperature sensitivity
We retained the reference quantile: 0.85 for AAV and 0.95 otherwise. The reference temperature is for AAV and for the other tasks.
| Temperature | RNA1 | RNA2 | RNA3 | AAV | GFP | TFBind8 |
|---|---|---|---|---|---|---|
| 0.9340 | 0.9353 | 0.9757 | 0.6937 | 3.6126 | 0.9725 | |
| 0.9340 | 0.9353 | 0.9757 | 0.6913 | 3.6128 | 0.9730 | |
| 0.9340 | 0.9353 | 0.9757 | 0.6836 | 3.6132 | 0.9718 | |
| 0.9340 | 0.9353 | 0.9756 | 0.6844 | 3.6130 | 0.9713 | |
| 0.9450 | 0.9376 | 0.9761 | 0.6943 | 3.6135 | 0.9711 |
Overall, CRAFT maintains generally stable final-discovery performance across the tested settings, particularly for allocation temperature. Across the sixteenfold temperature range, the maximum–minimum difference in mean endpoint is less than 1.6% of the corresponding score on every task. Quantile variation also produces limited endpoint changes on RNA1–3 and GFP. On AAV and TFBind8, the reference quantiles remain near the best tested means, with moderate and higher quantiles favored, respectively. These results support low sensitivity to precise temperature tuning and stable performance with task-appropriate quantile choices.
The neighborhood sensitivity analysis appears in Sec. 5.3 and Table 3. Further parameter studies are provided in Appendix F, Tables F.3–F.5; the additional AAV comparisons are reported above.
Dear Reviewer QRtP,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Response to Reviewer QRtP (1/2)
We appreciate the reviewer's careful reading and thoughtful questions, which have helped us clarify our contributions and strengthen the evaluation. As discussed in our response to Reviewer zbeL, CRAFT accommodates different selectors within its candidate-generation and feedback-update workflow. We therefore include Shared Ridge as an alternative selector.
Final scores across six tasks
In the five-seed selector comparisons, Ridge improves mean final quality over matched Quantile controls on three reported tasks, including AAV (+0.0733). This supports task-dependent selector choice within CRAFT. Replacing the selector improves final quality on AAV and RNA2, demonstrating task-dependent benefits of the framework.
Why does CRAFT achieve a lower GFP endpoint than ProSpero?
The GFP gap develops mainly in the later part of the search. CRAFT discovers high-scoring candidates earlier than ProSpero, but makes smaller subsequent improvements.
Entries below show the cumulative top-128 mean among newly evaluated candidates, excluding the initial data, for the original Quantile configuration in Table 1.
At 640 queries, CRAFT and ProSpero have nearly identical scores. Over the remaining 640 queries, CRAFT improves by 0.0138, whereas ProSpero improves by 0.0555, producing the final gap of 0.0422.
A plausible explanation is concentrated exploitation under the current GFP configuration. Its candidate sources generate single-substitution neighbors around measured elites, while the shared upper-quantile score favors candidates near high-valued observations. Together, these choices may reinforce search within already promising neighborhoods. Diversity selection operates within score-based shortlists, so it does not necessarily ensure broad exploration.
The final discoveries are consistent with this interpretation: CRAFT's mean normalized pairwise Hamming distance is 0.0128, compared with 0.0395 for PEX and 0.0600 for ProSpero. CRAFT's retained sets are therefore more concentrated, although final-set diversity alone does not establish why later improvement is smaller.
Evaluation on longer-protein benchmarks
We extend the evaluation to four experimentally measured protein variant libraries: TEM-1 [1], LGK [2], Hsp82 [3], and Cas9 [4]. These datasets cover proteins ranging from 286 to 1,368 residues.
CRAFT achieves the highest mean endpoint among the completed methods on TEM-1, LGK, and Cas9. These results demonstrate competitive performance on longer proteins within the evaluated single-substitution libraries.The corresponding experimental code and results are available in our anonymous repository.
References
[1] Firnberg, E., Labonte, J. W., Gray, J. J., and Ostermeier, M. (2014). A Comprehensive, High-Resolution Map of a Gene’s Fitness Landscape. Molecular Biology and Evolution, 31(6), 1581–1592.
[2] Klesmith, J. R., Bacik, J.-P., Michalczyk, R., and Whitehead, T. A. (2015). Comprehensive Sequence-Flux Mapping of a Levoglucosan Utilization Pathway in E. coli. ACS Synthetic Biology, 4(11), 1235–1243.
[3] Flynn, J. M., Rossouw, A., Cote-Hammarlof, P. A., Fragata, I., Mavor, D., Hollins, C., III, Bank, C., and Bolon, D. N. A. (2020). Comprehensive fitness maps of Hsp90 show widespread environmental dependence. eLife, 9, e53810.
[4] Spencer, J. M., and Zhang, X. (2017). Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains. Scientific Reports, 7, 16836.