acceptodds
Under review as a conference paper at ICLR 2027

RECOB: Reliable Benchmarking of Experimental Optimization in Chemistry and Materials Science

Abstract

Optimization methods for experimental science are often evaluated on synthetic functions that are reproducible but omit important characteristics of real experiments. We introduce RECOB (REliable Chem Optimization Benchmark), a black-box optimization benchmark constructed exclusively from data generated through physical experiments in chemistry and materials science. The suite contains 14 single-objective and two multi-objective tasks spanning chemical reactions, material formulations, electrochemical systems, continuous-flow processes, and automated laboratories. Each task provides a machine-readable specification of its decision variables, feasible domain, physical constraints, objective direction, and experimental provenance. Continuously queryable learned oracles are screened using repeated holdout validation and prespecified admission criteria, while measured-table replay enables evaluation using the original experimental responses. Under a common paired evaluation protocol, we compare ten single-objective and eight multi-objective optimization methods. HEBO achieves the best aggregate single-objective rank, while qNEHVI leads the multi-objective comparison. Model-based methods generally outperform non-adaptive baselines, although their computational overhead varies substantially. We further assess benchmark reliability using independent oracle retraining and measured-table replay. Aggregate optimizer rankings remain highly consistent across retrained oracles, while replay preserves the broad performance hierarchy using only physically measured responses. Together, these results show that RECOB can reproducibly distinguish optimizer performance as an experimentally grounded and reliability-tested benchmark for black-box optimization in chemistry and materials science.

Primary area:
datasets and benchmarks
Keywords:
Optimization, Benchmarking, AI for science, Surrogate modeling, Reproducibility

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LhFXOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents RECOB that allows benchmarking of chemical optimization through constructing datasets from >10 publications, where the data is of chemical experiment origin. The contribution includes the collated benchmarking datasets for their use in this specific paper. Evaluation protocols to compare the performance of different optimisation setups and their computational cost on these datasets.

Critical strengths

Through collating the variety of datasets, the paper's findings are more reliable, as often the best chemical optimisation approach can vary significantly by the problem of application. The protocols are comprehensive and the consideration of computational overhead is helpful.

Critical weaknesses

It is not clear to me that the assembled dataset is, or is to be made, available, which would reduce the value to the community if so. I feel the paper could be improved with data visualisation to assist in conveying key points to readers beyond the tables in the paper (and some graphs in the appendix).

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3

We thank the reviewer for the positive assessment and these helpful suggestions.

(1) Dataset availability.

The assembled dataset and the packaged RECOB benchmark are already included in the Supplementary Materials submitted with the paper. We will also make the complete dataset and benchmark package publicly available on GitHub upon acceptance, so that other researchers can directly access and use RECOB in their own work.

(2) Data visualization.

We agree that additional visualizations would help readers better understand the benchmark and its key findings. Due to the page limit, the current manuscript relies more heavily on tables, with some additional visualizations provided in the Supplementary Materials. We will improve the visual presentation in the camera-ready version by adding more intuitive summary figures to illustrate the characteristics of the benchmark tasks and the comparative performance of different optimization algorithms.

Reviewer_QW6wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RECOB, a black-box optimization benchmark built only from physical experiments in chemistry and materials science. It has 14 single-objective and 2 multi-objective tasks, each with machine-readable variables, constraints, and provenance. The paper compares 10 single-objective and 8 multi-objective optimizers and checks reliability through oracle retraining and replay.

Critical strengths
  • Well-curated tasks with unified, machine-readable specifications (constraints, mixed variables, provenance) and prespecified oracle admission criteria.
  • Careful evaluation protocol: paired initial designs, frozen normalization, separate reporting of compute cost, and released artifacts with checksums.
  • Explicitly tests benchmark fidelity beyond predictive accuracy, through retraining and replay.
Critical weaknesses
  • The oracle design may bias the rankings. The oracles use only RF/XT tree ensembles, which produce piecewise-constant surfaces with flat extrapolation, and the paper itself reports oracle plateaus on OPV and printing.
  • The scope is small for a benchmark paper. There are 14 low-dimensional tasks (3–11 dimensions) from 9 publications, some with few observations (40–73) and weak fits (R² = 0.26 for BRINE campaign 1). The multi-objective suite has only 2 tasks.
  • Experimental noise and batch evaluation are not modeled, although both are central to real lab campaigns.
Questions for authors
  • It would be better to extend model class for oracles (e.g., GP, BNN, or Transformer-based)
  • It would be better to extend benchmarks with high-dimensional settings
  • It would be better to extend benchmarks with noisy settings and batch evaluation for more practical use.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2

We thank the reviewer for your suggestions.

(W1) We thank the reviewer for identifying this important issue. We initially adopted AutoGluon, an AutoML framework, to construct the oracles. Our intention was to use a standardized model-selection procedure across different tasks, as manually selecting and tuning different model architectures may introduce a certain level of bias. However, AutoGluon selected RF/XT tree ensembles, which indeed introduces a particular inductive bias into the oracle models.

To examine whether our benchmark conclusions depend on this choice, we followed the reviewer's suggestion and introduced two additional oracle architectures: GP and BNN. Please see public comment for detailed results.

The results are highly consistent with those obtained using the original AutoGluon oracles. HEBO retains the first position under both additional oracle architectures, while the other methods maintain broadly similar performance tiers. Although the relative rankings of some closely performing methods change slightly, the overall performance hierarchy remains stable.

(W2) We have to admit that this benchmark only contains problems of relatively small number of dimension and data. This limitation is closely related to the characteristics of publicly available experimental data in chemistry and materials science, as well as the requirements for constructing reliable ML-based oracles.

In chemistry, many practically important optimization problems involve large discrete or mixed-variable design spaces. For example, COF design may involve selecting building blocks, linkers, and topologies from large candidate pools, while photocatalytic optimization may involve combinations of dyes, solvents, and catalysts. Although these problems are important, their experimental datasets are often sparse relative to the size of the combinatorial search space. Training reliable ML oracles from such data remains challenging. In contrast, many experimental campaigns focus on optimizing a relatively small number of controllable experimental parameters, such as reaction conditions and formulation compositions. These are more amenable to constructing ML oracles because their variables are explicitly defined and their experimental measurements are available in a structured form.

Therefore, RECOB focuses on experimental campaigns that satisfy several requirements: publicly accessible experimental data, clearly defined optimization variables and constraints, sufficient observations for oracle training and evaluation, and acceptable predictive performance of the resulting oracles. These requirements limit the number of eligible datasets. This is also why we do not simply include every published chemical optimization dataset: datasets for which reliable oracles cannot be constructed are excluded.

To broaden the current benchmark, we have additionally collected data from three publications, contributing five SO and one MO task. The expanded benchmark now contains 19 SO and 3 MO tasks from 12 publications, compared with the original 14 SO and 2 MO tasks from 9 publications. We applied the same oracle construction and evaluation procedures to these additional tasks. As shown in our public comment, the resulting algorithm rankings are highly consistent with the original benchmark results.

We acknowledge that the multi-objective suite remains relatively small even after this expansion. We have clarified this limitation and will avoid claiming that the current benchmark covers the full diversity of chemical optimization problems.

(W3) We agree that experimental noise is an important aspect of real-world chemical optimization. In the current benchmark, we use the experimental values reported in the original publications to construct the optimization tasks. We did not introduce an additional synthetic noise model because the characteristics of experimental noise can vary substantially across different chemical systems and measurement protocols. Moreover, replicate-level measurements and their associated noise distributions are not consistently available across the collected datasets. Imposing a uniform noise model without sufficient experimental evidence could therefore introduce another source of bias.

Regarding batch optimization, we have added experiments with batch sizes of 2 and 4 to examine the effect of parallel evaluations. As shown in our public comment, HEBO remains the top-performing method under both settings, while the relative ordering of several closely performing methods changes slightly. The overall performance hierarchy remains consistent with the main results. These additional experiments provide an initial assessment of batch optimization in RECOB and suggest that the main comparative conclusions are not substantially affected by the choice of batch size.

AuthorsOpenReview ↗

Just in case you haven't seen the results in the public comment

Just in case you haven't seen the results in the public comment, here are the new experiments and results:

We thank all reviewers for their thoughtful and constructive comments. In response to the concerns regarding benchmark coverage, oracle dependence, and realistic optimization settings, we have expanded RECOB and conducted additional experiments along three directions.

  1. Expanding the benchmark with additional experimental datasets. We added data from three papers: A high-throughput platform for efficient exploration of functional polypeptide chemical space (Nature Synthesis), Machine Learning on a Robotic Platform for the Design of Polymer–Protein Hybrids (Advanced Materials) and Flow-driven data intensification to accelerate autonomous inorganic materials discovery (Nature Chemical Engineering). 5 more SO and 1 more MO problems are included.

Now the results are:

AlgorithmMean rank (Original mean rank)Task wins (Original task wins)
HEBO1.79 (1.71)10.5 (8.5)
BoFire LogEI3.66 (3.96)2.5 (1.5)
BoTorch UCB4.29 (4.46)1.5 (0.5)
BoTorch LogEI4.53 (4.29)1.5 (1.5)
RF/SMAC-like EI4.68 (4.43)0 (0)
CMA-ES4.89 (4.57)2 (1)
Optuna TPE5.68 (5.43)1 (1)
Gryffin8.10 (8.36)0 (0)
Sobol Search8.57 (8.50)0 (0)
Random Search8.78 (9.29)0 (0)

The overall ranking remains highly consistent after expanding the benchmark. HEBO and BoFire LogEI retain their first and second positions, respectively. The only changes in the ranking order occur among the methods ranked third to fifth, while the relative ordering of all other methods remains unchanged.

  1. Evaluating different batch sizes. We have also added two batched experiments (batch_size=2, batch_size=4) to show how batch settings may affect optimization performances. Here are the results:
AlgorithmMean rank (2)Task wins (2)Mean rank (4)Task wins (4)
HEBO1.78910.51.81613.0
BoFire LogEI3.6841.53.8162.333
BoTorch LogEI3.7892.54.2371.333
BoTorch UCB4.1581.54.2370.333
RF/SMAC-like EI4.2631.04.1052.0
CMA-ES5.3681.04.6840.0
Optuna TPE5.4211.05.7370.0
Gryffin8.6840.08.1050.0
Sobol Search8.8950.09.1050.0
Random Search8.9470.09.1580.0

We can see that these two results are also pretty consistent to the main result above, with only 3, 4, 5 changed sligtly. We can somehow call these three methods forming a closely matched group: BoTorch LogEI, BoTorch UCB and RF/SMAC-like EI. The lower-performing methods also retain broadly similar rankings.

  1. Evaluating different oracle architectures. To show consistency between different oracle settings, we included two other oracle models: Gaussian process and Bayesian neural network. Here are the results:
AlgorithmMean rank (GP)Task wins (GP)Mean rank (BNN)Task wins (BNN)
HEBO2.8427.01.9479.0
BoTorch LogEI3.1844.8333.2894.0
BoFire LogEI3.2372.8333.4212.5
BoTorch UCB4.0533.3334.1322.5
RF/SMAC-like EI4.8420.04.7891.0
Optuna TPE5.4211.05.8950.0
CMA-ES5.6320.05.3680.0
Gryffin7.7370.08.1050.0
Random Search8.8950.08.8950.0
Sobol Search9.1580.09.1580.0

We can see that in these settings HEBO kept the first rank. BoTorch LogEI, BoFire LogEI, BoTorch UCB and RF/SMAC-like EI are still number 2~5, BoTorch LogEI challenged BoFire LogEI's second place, but not in a large margin. Optuna TPE and CMA-ES swapped place, also not in a large margin.

  1. Updated multi-objective benchmark results:
AlgorithmMean rank (All 3 tasks)Mean rank (Original 2 tasks)
qNEHVI2.4001.675
MESMO2.7332.200
qNParEGO3.0173.050
JES3.4673.425
TSEMO4.8334.825
NSGA-II5.6676.075
Random Search6.9337.475
Sobol Search6.9507.275

The relative ordering of the other methods remains almost unchanged, showing that the overall ranking pattern of multi-objective optimizers is also largely consistent after expanding the benchmark.

Overall, these additional experiments provide consistent evidence across different benchmark datasets, batch settings, and oracle architectures. HEBO keeps the first place in all settings, BoFire LogEI/BoTorch LogEI/BoTorch UCB/RF/SMAC-like EI are in the secondary tier, Optuna TPE and CMA-ES lay in the third tier, and Gryffin/Random Search/Sobol Search are the worst methods. These findings show robustness of the comparative capability from RECOB and the consistent performance margin between optimization algorithms.

All the extra experiment results will be added into the paper after revision.

Reviewer_Soi5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RECOB, a benchmark for black-box optimization in chemistry and materials science. It includes 14 single-objective and two multi-objective tasks, using machine learning models trained on published experimental data as oracles to simulate experimental optimization. The authors compare various black-box optimization algorithms under a standardized evaluation protocol and assess benchmark reliability through oracle retraining and measured-table replay.

Critical strengths

RECOB provides a useful benchmark grounded in real experimental data. Compared with existing benchmarks such as Olympus and Summit, it emphasizes standardized experimental constraints, oracle admission criteria, consistent algorithm evaluation, and reliability assessment through oracle retraining. These efforts improve the standardization and reproducibility of experimental optimization benchmarks.

Critical weaknesses
  1. Limited novelty and discriminative power. While the benchmark is useful, its methodological novelty and additional insights beyond existing benchmarks are relatively limited. In Figures 1–5, the variance of most methods appears sufficiently large to obscure the differences between optimization algorithms. This is also consistent with the community's existing understanding of these methods, reducing the benchmark's added value. I suspect that this large variance may also partly explain the substantial changes in algorithm rankings observed under measured-table replay.

  2. Limited representation of realistic experimental settings. Although the benchmark aims to simulate real experiments, it does not account for experimental noise, leaving robustness to noisy observations unexamined, despite its importance in practice. Furthermore, real experimental campaigns often evaluate multiple conditions in parallel rather than conducting many sequential single-point experiments. The benchmark does not investigate how batch size affects algorithm performance.

Questions for authors
  1. Given the substantial variance shown in Figures 1–5, how reliably can the benchmark distinguish between different optimization algorithms? Could this variance also explain the changes in rankings under measured-table replay?

  2. How might experimental noise and different batch sizes affect the benchmark results?

  3. Some evaluated methods appear to differ primarily in implementation rather than their underlying algorithms, such as BoTorch LogEI and BoFire LogEI. What are the substantive differences between these methods?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1

We thank the reviewer for raising these concerns. We would like to clarify two aspects of our work.

(Weakness 1.1, novelty) We have to admit that training ML models as black-box oracles is not a new idea, but in this case, another problem arises: how can we confirm these oracle models are good simulations of the real-world problems, since the ML introduces additional bias in here? We integrated three additional steps to make sure that these oracle models can serve as a good benchmark: (1) We have a filtering machenism to discard datasets that ML models cannot learn robust relationship between feature space and targets, so that the ML models will be more accurate; (2) After evaluation, we tried different seeds to see if the benchmark results are vulnerable to random seeds; (3) In addition, we performed measured-table replay method, providing a complementary assessment that does not rely on surrogate-predicted responses at the evaluated points.

Thus, our contribution is not simply to collect experimental datasets and compare optimization algorithms, but also to systematically examine the reliability of the benchmark itself. We believe this aspect is important for experimental optimization benchmarks, where the use of learned oracles introduces an additional source of uncertainty that needs to be evaluated.

(Weakness 1.2, discriminative power) We totally disagree on the reviewer of using "overlaps on figures" to prove discriminative power. A more reasonable way is using statistical tests, which we have already included in the paper: The overall differences were significant according to the Friedman test (χ2 =81.30, p = 8.92 × 10−14),Pairwise Wilcoxon signed-rank tests indicated that HEBO significantly outperformed all other methods except BoFire LogEI (p = 0.084) and BoTorch LogEI (p = 0.084). In addition, we added 3 more datasets (5 single-objective, 1 multi-objective). After that, HEBO significantly outperforms all methods other than BoFire LogEI (p = 0.0759). We acknowledge that closely performing methods cannot always be statistically distinguished. Nevertheless, the overall statistical results and the consistent performance hierarchy across the original and expanded benchmarks demonstrate that RECOB can identify meaningful differences in optimization performance. Please see the public comment for details.

(Weakness 2) We agree that experimental noise is an important aspect of real-world chemical optimization. In the current benchmark, we use the experimental values reported in the original publications to construct the optimization tasks. We did not introduce an additional synthetic noise model because the characteristics of experimental noise can vary substantially across different chemical systems and measurement protocols. Moreover, replicate-level measurements and their associated noise distributions are not consistently available across the collected datasets. Imposing a uniform noise model without sufficient experimental evidence could therefore introduce another source of bias.

We acknowledge that using reported experimental values does not fully capture run-to-run experimental variability, even when the reported values are averages over repeated measurements. We will clarify this limitation and consider the systematic evaluation of noise-aware optimization an important direction for future extensions of RECOB.

Dispite of this, we added two batched settings (batch_size=2, 4) to examine the effect of parallel evaluations (please see public comment for details). As shown in our public comment, HEBO remains the top-performing method under both settings, while the relative ordering of several closely performing methods changes slightly. The overall performance hierarchy remains consistent with the main results. These additional experiments provide an initial assessment of batch optimization in RECOB and suggest that the main comparative conclusions are not substantially affected by the choice of batch size.

(Question 3) BoTorch LogEI and BoFire LogEI for sure belong to the same BO family, but their acquisition implimentations are different. The BoTorch baseline directly uses BoTorch's analytic LogExpectedImprovement, while the BoFire baseline uses its SoboStrategy with qLogEI, which employs a Monte Carlo-based acquisition formulation. One of them is analytical calculated and the other is estimated, therefore these two baselines represent different end-to-end implementations of LogEI-based Bayesian optimization.

AuthorsOpenReview ↗

Just in case you haven't seen the results in the public comment

Just in case you haven't seen the results in the public comment, here are the new experiments and results:

We thank all reviewers for their thoughtful and constructive comments. In response to the concerns regarding benchmark coverage, oracle dependence, and realistic optimization settings, we have expanded RECOB and conducted additional experiments along three directions.

  1. Expanding the benchmark with additional experimental datasets. We added data from three papers: A high-throughput platform for efficient exploration of functional polypeptide chemical space (Nature Synthesis), Machine Learning on a Robotic Platform for the Design of Polymer–Protein Hybrids (Advanced Materials) and Flow-driven data intensification to accelerate autonomous inorganic materials discovery (Nature Chemical Engineering). 5 more SO and 1 more MO problems are included.

Now the results are:

AlgorithmMean rank (Original mean rank)Task wins (Original task wins)
HEBO1.79 (1.71)10.5 (8.5)
BoFire LogEI3.66 (3.96)2.5 (1.5)
BoTorch UCB4.29 (4.46)1.5 (0.5)
BoTorch LogEI4.53 (4.29)1.5 (1.5)
RF/SMAC-like EI4.68 (4.43)0 (0)
CMA-ES4.89 (4.57)2 (1)
Optuna TPE5.68 (5.43)1 (1)
Gryffin8.10 (8.36)0 (0)
Sobol Search8.57 (8.50)0 (0)
Random Search8.78 (9.29)0 (0)

The overall ranking remains highly consistent after expanding the benchmark. HEBO and BoFire LogEI retain their first and second positions, respectively. The only changes in the ranking order occur among the methods ranked third to fifth, while the relative ordering of all other methods remains unchanged.

  1. Evaluating different batch sizes. We have also added two batched experiments (batch_size=2, batch_size=4) to show how batch settings may affect optimization performances. Here are the results:
AlgorithmMean rank (2)Task wins (2)Mean rank (4)Task wins (4)
HEBO1.78910.51.81613.0
BoFire LogEI3.6841.53.8162.333
BoTorch LogEI3.7892.54.2371.333
BoTorch UCB4.1581.54.2370.333
RF/SMAC-like EI4.2631.04.1052.0
CMA-ES5.3681.04.6840.0
Optuna TPE5.4211.05.7370.0
Gryffin8.6840.08.1050.0
Sobol Search8.8950.09.1050.0
Random Search8.9470.09.1580.0

We can see that these two results are also pretty consistent to the main result above, with only 3, 4, 5 changed sligtly. We can somehow call these three methods forming a closely matched group: BoTorch LogEI, BoTorch UCB and RF/SMAC-like EI. The lower-performing methods also retain broadly similar rankings.

  1. Evaluating different oracle architectures. To show consistency between different oracle settings, we included two other oracle models: Gaussian process and Bayesian neural network. Here are the results:
AlgorithmMean rank (GP)Task wins (GP)Mean rank (BNN)Task wins (BNN)
HEBO2.8427.01.9479.0
BoTorch LogEI3.1844.8333.2894.0
BoFire LogEI3.2372.8333.4212.5
BoTorch UCB4.0533.3334.1322.5
RF/SMAC-like EI4.8420.04.7891.0
Optuna TPE5.4211.05.8950.0
CMA-ES5.6320.05.3680.0
Gryffin7.7370.08.1050.0
Random Search8.8950.08.8950.0
Sobol Search9.1580.09.1580.0

We can see that in these settings HEBO kept the first rank. BoTorch LogEI, BoFire LogEI, BoTorch UCB and RF/SMAC-like EI are still number 2~5, BoTorch LogEI challenged BoFire LogEI's second place, but not in a large margin. Optuna TPE and CMA-ES swapped place, also not in a large margin.

  1. Updated multi-objective benchmark results:
AlgorithmMean rank (All 3 tasks)Mean rank (Original 2 tasks)
qNEHVI2.4001.675
MESMO2.7332.200
qNParEGO3.0173.050
JES3.4673.425
TSEMO4.8334.825
NSGA-II5.6676.075
Random Search6.9337.475
Sobol Search6.9507.275

The relative ordering of the other methods remains almost unchanged, showing that the overall ranking pattern of multi-objective optimizers is also largely consistent after expanding the benchmark.

Overall, these additional experiments provide consistent evidence across different benchmark datasets, batch settings, and oracle architectures. HEBO keeps the first place in all settings, BoFire LogEI/BoTorch LogEI/BoTorch UCB/RF/SMAC-like EI are in the secondary tier, Optuna TPE and CMA-ES lay in the third tier, and Gryffin/Random Search/Sobol Search are the worst methods. These findings show robustness of the comparative capability from RECOB and the consistent performance margin between optimization algorithms.

All the extra experiment results will be added into the paper after revision.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of major revisions: extra experiment results

We thank all reviewers for their thoughtful and constructive comments. In response to the concerns regarding benchmark coverage, oracle dependence, and realistic optimization settings, we have expanded RECOB and conducted additional experiments along three directions.

  1. Expanding the benchmark with additional experimental datasets. We added data from three papers: A high-throughput platform for efficient exploration of functional polypeptide chemical space (Nature Synthesis), Machine Learning on a Robotic Platform for the Design of Polymer–Protein Hybrids (Advanced Materials) and Flow-driven data intensification to accelerate autonomous inorganic materials discovery (Nature Chemical Engineering). 5 more SO and 1 more MO problems are included.

Now the results are:

AlgorithmMean rank (Original mean rank)Task wins (Original task wins)
HEBO1.79 (1.71)10.5 (8.5)
BoFire LogEI3.66 (3.96)2.5 (1.5)
BoTorch UCB4.29 (4.46)1.5 (0.5)
BoTorch LogEI4.53 (4.29)1.5 (1.5)
RF/SMAC-like EI4.68 (4.43)0 (0)
CMA-ES4.89 (4.57)2 (1)
Optuna TPE5.68 (5.43)1 (1)
Gryffin8.10 (8.36)0 (0)
Sobol Search8.57 (8.50)0 (0)
Random Search8.78 (9.29)0 (0)

The overall ranking remains highly consistent after expanding the benchmark. HEBO and BoFire LogEI retain their first and second positions, respectively. The only changes in the ranking order occur among the methods ranked third to fifth, while the relative ordering of all other methods remains unchanged.

  1. Evaluating different batch sizes. We have also added two batched experiments (batch_size=2, batch_size=4) to show how batch settings may affect optimization performances. Here are the results:
AlgorithmMean rank (2)Task wins (2)Mean rank (4)Task wins (4)
HEBO1.78910.51.81613.0
BoFire LogEI3.6841.53.8162.333
BoTorch LogEI3.7892.54.2371.333
BoTorch UCB4.1581.54.2370.333
RF/SMAC-like EI4.2631.04.1052.0
CMA-ES5.3681.04.6840.0
Optuna TPE5.4211.05.7370.0
Gryffin8.6840.08.1050.0
Sobol Search8.8950.09.1050.0
Random Search8.9470.09.1580.0

We can see that these two results are also pretty consistent to the main result above, with only 3, 4, 5 changed sligtly. We can somehow call these three methods forming a closely matched group: BoTorch LogEI, BoTorch UCB and RF/SMAC-like EI. The lower-performing methods also retain broadly similar rankings.

  1. Evaluating different oracle architectures. To show consistency between different oracle settings, we included two other oracle models: Gaussian process and Bayesian neural network. Here are the results:
AlgorithmMean rank (GP)Task wins (GP)Mean rank (BNN)Task wins (BNN)
HEBO2.8427.01.9479.0
BoTorch LogEI3.1844.8333.2894.0
BoFire LogEI3.2372.8333.4212.5
BoTorch UCB4.0533.3334.1322.5
RF/SMAC-like EI4.8420.04.7891.0
Optuna TPE5.4211.05.8950.0
CMA-ES5.6320.05.3680.0
Gryffin7.7370.08.1050.0
Random Search8.8950.08.8950.0
Sobol Search9.1580.09.1580.0

We can see that in these settings HEBO kept the first rank. BoTorch LogEI, BoFire LogEI, BoTorch UCB and RF/SMAC-like EI are still number 2~5, BoTorch LogEI challenged BoFire LogEI's second place, but not in a large margin. Optuna TPE and CMA-ES swapped place, also not in a large margin.

  1. Updated multi-objective benchmark results:
AlgorithmMean rank (All 3 tasks)Mean rank (Original 2 tasks)
qNEHVI2.4001.675
MESMO2.7332.200
qNParEGO3.0173.050
JES3.4673.425
TSEMO4.8334.825
NSGA-II5.6676.075
Random Search6.9337.475
Sobol Search6.9507.275

The relative ordering of the other methods remains almost unchanged, showing that the overall ranking pattern of multi-objective optimizers is also largely consistent after expanding the benchmark.

Overall, these additional experiments provide consistent evidence across different benchmark datasets, batch settings, and oracle architectures. HEBO keeps the first place in all settings, BoFire LogEI/BoTorch LogEI/BoTorch UCB/RF/SMAC-like EI are in the secondary tier, Optuna TPE and CMA-ES lay in the third tier, and Gryffin/Random Search/Sobol Search are the worst methods. These findings show robustness of the comparative capability from RECOB and the consistent performance margin between optimization algorithms.

All the extra experiment results will be added into the paper after revision.

AuthorsOpenReview ↗

Public note: Revised manuscript has been uploaded

Dear Reviewers,

Thank you again for your thoughtful feedback and constructive suggestions.

We have uploaded a revised manuscript, with major revisions highlighted in yellow. The results tables and accompanying discussion have been updated to incorporate the expanded task suite, batch-evaluation experiments, and analyses using GP and BNN oracles.

In response to Reviewer 3’s suggestion, we have also added a new Figure 1 showing task-level optimizer rankings, making performance patterns across the benchmark easier to interpret.

We would appreciate your consideration of the revised manuscript and the additional evidence when updating your assessment. Please let us know if any concerns remain unresolved; we would be happy to provide further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.