acceptodds
Withdrawn from ICLR 2027

Soft-Rejection Informed Bayesian Optimization for Expensive Sequential Experiments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bayesian optimization (BO) is widely used for expensive black-box optimization, but in many settings each evaluation is itself a long sequential experiment whose final outcome is the optimization target. Early stopping can further reduce cost; however, existing methods inherit the reliability of an underlying surrogate predictor, and can fail when trajectories cross, i.e., when a configuration that looks poor early overtakes others by the endpoint. We formally define this early stopping problem for sequential black-box optimization, and propose Soft-Rejection Informed Bayesian Optimization (SRBO) to free the stopping rule from this dependence. SRBO is built around an endpoint forecaster, a Direct-Forecasting Gaussian Process (DF-GP), one GP per decision step, that maps the partial trajectory and configuration directly to the endpoint. This forecaster is used in two ways. A Calibrated Composite Predictive Sequential Test (CCPST) turns the DF-GP forecast into a stopping rule calibrated by held-out cross-validation: for the cross-validated rule, under a per-epoch independence condition on the held-out stop indicators, the rate of falsely stopped winners has a finite-sample bound that holds even when trajectories cross and the forecast is mis-specified, and for the deployed rule it is audited on every benchmark. The same forecast is also fed back into the BO surrogate as a pseudo-observation when a trial is stopped to learn from rejected configurations. On eight cost-matched surrogate benchmarks built from real experimental and learning-curve data, SRBO attains the best average rank of the seven methods run on all eight and, against up to ten baselines, the lowest mean final regret on three of the five experimental benchmarks, significantly against every baseline on each: on in vitro permeation testing (IVPT) it is below the strongest baseline and to below the others (), on UCI Concrete compressive strength below the strongest (), and on a methane-coupling catalyst screen below it (). On the two remaining experimental benchmarks SRBO shows no significant difference from the leaders on one and trails single-fidelity BO on the other (), and on three learning-curve tasks BOHB leads.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimization, early stopping, sequential experiments, false-stop rate, calibrated sequential testing, Gaussian processes, learning curve extrapolation, expensive black-box optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hSZCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Soft-Rejection Informed Bayesian Optimization (SRBO) for settings in which each function evaluation involves multiple costly sequential steps, aiming to terminate unpromising trials early while reducing the risk of stopping eventual high-performing configurations. The method trains a separate Direct-Forecasting Gaussian Process (DF-GP) for each decision time to predict the endpoint from the configuration and partial trajectory, then incorporates the predictive means of stopped trials into the optimization surrogate as pseudo-observations. Its stopping module, Calibrated Composite Predictive Sequential Test (CCPST), combines reliability weighting with cross-validation calibration to select admissible stopping times, providing a finite-sample false-stop rate (FSR) bound for the cross-validated rule under a calibration-independence assumption, while auditing the deployed rule empirically. Across eight surrogate benchmarks constructed from real experimental or learning-curve data, SRBO achieves the best average rank among methods evaluated on all benchmarks and the lowest mean regret on IVPT, Concrete, and OCM, whereas BOHB leads on all three learning-curve tasks.

Critical strengths
  1. Coordinated stopping and information reuse: The same endpoint forecaster supports early stopping and feeds information from incomplete trials into subsequent search, connecting evaluation-cost savings with the effective use of partial trajectories.

  2. Comprehensive experimental protocol and reporting: The experiments use a common cost budget, disjoint tuning and reporting seeds, and 20 repeated runs, while reporting optimization regret, runtime FSR, and tasks where SRBO does not lead, supporting assessment of both effectiveness and applicability.

Critical weaknesses
  1. Gap between the theoretical guarantee and deployed stopping: The calibration independence assumed in Section 4.2 is not strictly ensured by the implemented procedure, and Equation (7) retains an uncontrolled deployment gap Δm\Delta_m, so the theorem does not yet provide a computable FSR guarantee for actual deployment.

  2. Incomplete uncertainty treatment in pseudo-observation updates: Section 4.1 injects only DF-GP predictive means and assigns the same noise variance to exact and pseudo-observations, without propagating predictive variance or error correlations arising from shared training data, leaving unclear how posterior overconfidence under model misspecification is prevented.

  3. Application gains depend on simulator validity: Section 5.1 evaluates all methods on continuous-box simulators fitted to historical data, with optimal values outside the measured range on seven tasks, but lacks validation on independent measured trajectories or within the original feasible domains to rule out dependence of Figure 2's gains on extrapolation error.

Questions for authors
  1. FSR guarantees for the deployed rule: Could the authors specify additional conditions that address dependence among five-fold calibration indicators and control Δm\Delta_m, so that Equation (7) yields a computable and meaningful FSR upper bound for the deployed stopping rule?

  2. Probabilistic basis of pseudo-observation updates: Could the authors justify using a shared noise variance for exact observations and predictive-mean pseudo-labels, and analyze how ignoring DF-GP predictive variance and error correlations affects the augmented surrogate posterior and EI decisions?

  3. Validity on independent measured data: Could the authors evaluate on measured trajectories excluded from forecaster training and calibration, preserve the original variable constraints and match evaluation costs, and report completed-trial regret and runtime FSR to test whether Figure 2's advantage persists?

Reviewer_5hQxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the Bayesian optimization (BO) setting where each evaluation is a sequential experiment that reveals intermediate observations before providing the final objective evaluation, and introduces Soft-Rejection Informed Bayesian Optimization (SRBO), a BO approach that consists of (i) a Direct-Forecasting Gaussian Process (DF-GP) used to estimate the endpoint from the partial trajectory and configuration; and (ii) a calibrated stopping procedure, namely Calibrated Composite Predictive Sequential Test (CCPST), used to terminate non-promising experiments, while allowing estimated endpoints to be integrated into the BO surrogate as pseudo-observations. In addition to numerical evaluations on eight oracle-based benchmarks and up to ten baselines for comparison, the paper provides additional theoretical guarantees through a conditional finite-sample bound for cross-validation-based stopping rule related to CCPST.

Critical strengths

S1. The idea of properly handling early stopping when early observations are non-helpful, seems intuitive and practically relevant.

S2. The core ideas and methodological concepts are well motivated and clearly presented.

Critical weaknesses

W1. The evaluated benchmark objectives seem unrealistic and without physical meaning. Specifically, the optima seem implausible or infeasible; see e.g., an optimum of 1282.51 by concrete oracle, compared with a maximum measured endpoint of 82.60, while the LCBench oracles attain accuracies of 1.05, 1.89, and 1.96. In addition, the HEA-OER has a composition summing to 174 at % which seems meaningless.

W2. The baseline adaptations make the comparisons unfair: Freeze-Thaw is adapted to a terminal stopping criterion rather than its original pause/resume procedure. EI_Discard and EI_Virtual are untuned and never recalibrated, while SRBO recalibrates and tunes its stopping parameters.

W3. The numerical evaluation lacks comparison with important relevant baselines: The evaluation omits trace-aware multi-fidelity BO methods; see e.g., trace-aware knowledge gradient [1] , and BO-BOS [2], which is discussed but not evaluated. Direct comparisons with these methods, or a clear explanation of their incompatibility with the experimental setting, would strengthen the merits and contributions of the proposed SRBO approach.

References

[1] Wu, J., Toscano-Palmerin, S., Frazier, P. I., & Wilson, A. G. (2020). Practical multi-fidelity Bayesian optimization for hyperparameter tuning. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, Proceedings of Machine Learning Research, 115, 788–798.

[2] Dai, Z., Yu, H., Low, B. K. H., & Jaillet, P. (2019). Bayesian optimization meets Bayesian optimal stopping. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, 97, 1496–1506.

Questions for authors

Q1. Does SRBO still perform well when optimization is restricted to feasible configurations and upon removing the oracle’s out-of-range predictions?

Q2. Does SRBO still outperform the baselines when no extra adaptation is made? See e.g., Freeze-Thaw, EI_Discard and EI_Virtual.

Q3. How does SRBO compare (empirically and/or intuitively) to [1] and [2] listed in the weaknesses section?

Reviewer_FgMKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tackles early stopping in Bayesian Optimization for expensive multi-step experiments where slow-starting recipes can still finish as top winners. At its core, it trains a separate model at each time step to directly predict a trial's final outcome using both its recipe settings and its early measurements. To decide when to stop, it checks past completed experiments and disables early stopping at initial steps where the predictor would have mistakenly killed eventual winners. Once predictions are reliable, it halts any trial that has a very low probability of finishing in the top 25%. To use early-stopped observations, rather than throwing halted runs away, it feeds their predicted final outcomes into the main optimizer as "soft rejections" so the search avoids testing similar bad recipes again.

Critical strengths
  • Decoupled Prefix-to-Endpoint Translation (DF-GP): Instead of treating partial trajectories as noisy proxies, SRBO translates them into endpoint pseudo-observations via a separate GP that can be pre-trained on offline data to directly inform the main surrogate.

  • Safety Guarantees Under Weakened Assumptions: The paper formalizes early-stopping control without assuming trajectory models or well-specified surrogates, bounding the false-stop rate via held-out cross-validation even when the predictor is misspecified.

  • Extensive Empirical and Diagnostic Evaluation: The method is thoroughly tested against ten baselines across eight benchmarks, backed by comprehensive ablations, sensitivity sweeps, and failure-mode diagnostics.

Critical weaknesses
  • Incremental Novelty and Crude Imputation: The core engine is just step-wise regression from early readings to the final outcome, and its "soft rejection" ignores per-point predictive uncertainty by globally inflating noise across both exact and pseudo-observations.

  • Heavy Warm-Start Data Dependency: Calibrating the cross-validated stopping rule on top-quartile winners requires dozens of pre-existing completed runs before optimization even starts, which contradicts the small-data Bayesian optimization paradigm. In the experiment, such historical data was thus used to possibly improve the results, making it untenable for single-task BO. This method maybe better positioned as a transfer learning BO where such a data availability assumption would be met easily.

Loose and Disconnected Theoretical Bound: The theoretical false-stop bound is far too loose to be meaningful at standard Bayesian optimization sample sizes and only covers the cross-validation rule, leaving a gap in the understanding of the full method under extensive rule.

Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.