- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce ChemVerse, a benchmark for docking-based Bayesian optimization in virtual screening, covering seven protein targets and three screening libraries, with 389 million docking scores. They systematically evaluate four axes: docking oracle, molecular representation used as input to the surrogate model, uncertainty quantification method, and acquisition function. They find that the docking oracle has the largest impact on optimization performance, followed by molecular representation and acquisition function, while the choice of uncertainty quantification method has relatively little effect.
Critical strengths
S1 Large-scale and comprehensive benchmark: The authors provide a valuable dataset of 389 million docking scores across multiple protein targets, chemical libraries, and docking oracles, which is a useful resource for the molecular ML community.
S2 Some interesting empirical findings: The benchmark provides useful insights into the relative importance of different components of BO-based virtual screening, particularly the strong influence of the docking oracle and the limited impact of the uncertainty estimation method.
Critical weaknesses
W1 Limited investigation of surrogate models and uncertainty quantification: Although the authors systematically compare molecular representations, uncertainty estimation methods, and acquisition functions, they use a single fixed surrogate model architecture. Given the importance of surrogate performance for BO, this limits the comprehensiveness of the benchmark and the generalizability of its conclusions, particularly regarding the limited impact of uncertainty quantification under the specific large-batch acquisition protocol.
W2 Limited ML contribution and relevance to ICLR: While the benchmark is sound and valuable for the molecular ML community, its methodological novelty is limited, and the main findings are largely unsurprising. Furthermore, as the largest effect on optimization performance comes from the choice of docking oracle rather than the ML methods, I find it difficult to view this primarily as a benchmark of AI/ML methods. In my opinion, the contribution is therefore not sufficiently significant for ICLR, and I lean toward rejection, primarily due to its limited ML contribution rather than concerns about the quality of the work.
Minor weaknesses and suggestions
- W3 Overview figure An overview figure summarizing the benchmark and its experimental dimensions would help readers understand the overall setup.
- W4 Active learning vs. Bayesian optimization The title refers to active learning (AL), but the paper actually focuses on Bayesian optimization (BO). While the two are closely related, their objectives differ: AL typically aims to improve predictive accuracy, whereas BO aims to identify high-performing candidates. The authors themselves acknowledge this distinction in Section 2.1 and explicitly state that BO better describes their setting. I therefore find the use of AL in the title, and occasionally elsewhere in the paper (e.g., the Discussion), somewhat misleading and would suggest using BO consistently.
- W5 Missing details Important methodological details, such as the surrogate architecture and the representations used in different experiments, should be more clearly stated in the main paper.
- W6 Aggregate metrics The definitions of some reported metrics, particularly the "median best-to-worst AUC spread" and "seed-subtracted recovery AUC," should be clarified. Not sure I understand either of these.
- W7 Presentation of results Understandably, many results are presented in the appendix, including comparisons across several experimental axes, which sometimes makes the results challenging to follow. At the same time, the three recovery metrics in Figures 1–3 seem to consistently tell the same story. I would therefore suggest showing only one of them in the main paper and using the freed-up space to present more of the important comparisons currently relegated to the appendix.
- W8 Random acquisition baseline A random baseline could help contextualize recovery performance, especially for difficult docking landscapes.
Questions for authors
- Q1 AI-based docking Why are modern AI-based docking methods like DiffDock excluded? Their inclusion would also strengthen the in-scope nature of this paper for an AI conference.
- Q2 Cross-oracle comparability This is mentioned in the appendix, but should probably be mentioned in the main paper too: the molecule pools between oracles vary somewhat due to different failing complexes. Docking failures are excluded independently for each oracle, resulting in potentially different effective molecular pools. Would the conclusions hold on a shared pool?
- Q3 Replicates How are the five independent replicates aggregated in the main figures, and how much variability is there between runs?
6 comments
Q1. Inclusion of AI-based docking methods
We appreciate the suggestion, but an important technical distinction is necessary. DiffDock primarily predicts protein-ligand binding poses, and its confidence score reflects confidence in the predicted pose rather than binding affinity. Consequently, DiffDock alone does not provide the scoring oracle required for our Bayesian optimization benchmark. To directly address the reviewer's suggestion, we performed additional experiments using DiffDock-generated poses on LSD-10K, followed by rescoring with Vina, Vinardo, and Glide. Importantly, the Bayesian optimization results with Vina rescoring closely resemble those obtained using conventional Vina docking, preserving the principal recovery trends and molecular representation rankings (see Figure). These observations are consistent with the independent findings of Vu et al. (J. Chem. Inf. Model., 2025), who systematically evaluated DiffDock-L and Vina pose generation with different scoring functions across 43 protein targets. Their study demonstrated that the choice of scoring function can have a greater influence on virtual-screening performance than the pose-generation method. We will include our additional results in the revised manuscript.
In addition to these results, the submitted manuscript already includes an AI-based affinity oracle: Boltz-2 co-folding and affinity prediction on LSD-10K, described in Sec 3.3 and Appendix A.1.6. Exhaustive co-folding at the million-molecule scale is computationally prohibitive, motivating this smaller-scale evaluation. We will discuss these AI-based docking oracles more explicitly in the main text.
Q2. Comparability on a shared molecular pool
We quantified the valid fraction of each library, i.e. the molecules that yield valid scores from all docking oracles for each receptor. These “failure filters” are applied in all benchmark experiments.
| Receptor | HTS-2M | LSD-5M with DOCK3 | LSD-5M without DOCK3 |
|---|---|---|---|
| 5HT2A | 92.93% | 32.92% | 93.22% |
| AmpC | 91.78% | 95.27% | 95.27% |
| D4 | 90.01% | 65.10% | 91.41% |
| EP4R | 76.50% | 56.94% | 83.05% |
| SERT | 91.39% | 68.57% | 94.19% |
| Sigma2 | 75.33% | 66.73% | 84.40% |
| Mpro | 85.29% | 59.03% | 91.94% |
The four common oracles share substantial portions of both libraries: 75.33–92.93% for HTS-2M and 83.05–95.27% for LSD-5M. Our comparisons of featurizers, BNN approximations, and acquisition functions are conducted on identical eligible pools for each oracle. Thus, differences in pool membership between oracles do not confound these within-oracle comparisons. In the main text, we acknowledge this limitation when interpreting differences across docking oracles.
Q3. Aggregation and variability across replicates.
For each experimental condition, we compute seed-subtracted recovery AUC separately for five independent runs and average these within each receptor. Each bar then averages the seven receptor means with equal weight. Error bars indicate ±1 standard error across receptor means.
To quantify run-to-run variability separately, we calculate the sample standard deviation of the five AUCs within each condition. For the featurizer comparisons using Laplace neural networks:
| Library | Median run-to-run SD (AUC percentage points) | Interquartile range |
|---|---|---|
| HTS-2M | 1.04 | 0.59–2.07 |
| LSD-5M | 0.64 | 0.37–1.12 |
| Enamine-10M | 0.71 | 0.39–1.38 |
| LSD-10K | 3.30 | 2.03–5.36 |
These summaries span receptors, docking programs, featurizers, and the displayed top-k thresholds. Run-to-run variability is typically around one AUC percentage point or less on the large libraries and higher on LSD-10K.
We appreciate the reviewer's recognition of ChemVerse as a sound and valuable benchmark. Given that the principal reservations concern its ML significance and relevance to ICLR, we hope our clarifications demonstrate that ChemVerse directly addresses fundamental questions in probabilistic modeling, representation learning, and sample-efficient optimization, all explicitly within ICLR's scope. Together with the additional experimental evidence, we respectfully ask the reviewer to reconsider the overall assessment. If concerns remain, we would welcome further clarification on what aspects of the ML contribution are considered insufficient to warrant a more favorable recommendation.
Thank you for the detailed response and additional experiments.
I acknowledge your point regarding ICLR's scope and agree that systematic empirical benchmarks can constitute valuable ML contributions without introducing new algorithms. My original characterization of the work as outside ICLR's scope was therefore too strong. My remaining concern is primarily about the significance and generalizability of the ML insights.
Regarding the surrogate architecture, I agree that keeping it fixed enables controlled comparisons between representations. However, different representations may benefit from different architectures or hyperparameter configurations. Consequently, the results establish their relative performance under the chosen surrogate configuration rather than their general suitability for BO. The additional Gaussian-process experiments are a welcome step toward addressing this limitation.
I also appreciate the clarifications regarding DiffDock, cross-oracle comparability, and replicate variability.
Finally, while the proposed revisions appear to address several of my minor comments, the manuscript has not yet been updated, so I cannot assess their implementation. Could you please upload the revised manuscript, ideally accompanied by a version with the changes highlighted as a supplementary file?
We thank the reviewer for recognizing the scale and value of ChemVerse, its empirical insights, and the soundness of the benchmark. We address the concerns below.
W1. Surrogate architecture and generalizability of the uncertainty findings
We thank the reviewer for raising this point. Holding the neural-network architecture constant was a deliberate experimental design choice, allowing us to systematically isolate the contributions of molecular representations, uncertainty estimation methods, and acquisition strategies without confounding effects from architectural variations. This controlled comparison is central to the scientific rigor of ChemVerse. Our findings demonstrate that MC Dropout, SWAG, and last-layer Laplace yield remarkably similar recovery performance across the evaluated conditions. We do not suggest that these conclusions extend universally to all surrogate architectures or acquisition protocols. Importantly, we have also expanded our experiments to include a Gaussian-process surrogate on HTS-2M, providing an additional comparison beyond neural networks (see response to Reviewer aQvD above). We will clarify the scope of our conclusions while emphasizing the value of these controlled comparisons.
W2. ML contribution and relevance to ICLR
We respectfully but strongly disagree with the assessment that ChemVerse has limited relevance to ICLR. The ICLR 2027 Call for Papers explicitly identifies "Datasets and Benchmarks," "Optimization," "Probabilistic Methods," "Representation Learning," and "Applications to Physical Sciences" as research areas within its scope. ChemVerse lies precisely at the intersection of these areas. The contribution of ChemVerse is not another neural-network architecture, but a systematic, large-scale investigation of fundamental machine-learning questions: Which molecular representations are most effective? How much does uncertainty estimation improve optimization? Which acquisition strategies are most sample-efficient? And how do these conclusions depend on the objective landscape? Addressing these questions across multiple docking oracles, protein targets, and chemical spaces provides insights that cannot be established through isolated optimization experiments. Furthermore, the observation that oracle choice can influence optimization performance more strongly than individual ML components is itself an important finding, rather than a weakness. It highlights the critical dependence of machine-learning performance on the underlying objective landscape and the necessity of benchmarking across diverse conditions. There is substantial precedent for such contributions at leading ML conferences, including Gao et al. (NeurIPS 2022), Kristiadi et al. (ICML 2024), and Xie et al. (ICLR 2026). We therefore respectfully maintain that ChemVerse is firmly within ICLR's stated scope and that its systematic evaluation of representation learning, probabilistic modeling, and sample-efficient optimization constitutes a substantive ML contribution.
Minor weaknesses and suggestions (W3-8)
We thank the reviewer for these suggestions that improve clarity of our paper.