- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Soft-Rejection Informed Bayesian Optimization (SRBO) for settings in which each function evaluation involves multiple costly sequential steps, aiming to terminate unpromising trials early while reducing the risk of stopping eventual high-performing configurations. The method trains a separate Direct-Forecasting Gaussian Process (DF-GP) for each decision time to predict the endpoint from the configuration and partial trajectory, then incorporates the predictive means of stopped trials into the optimization surrogate as pseudo-observations. Its stopping module, Calibrated Composite Predictive Sequential Test (CCPST), combines reliability weighting with cross-validation calibration to select admissible stopping times, providing a finite-sample false-stop rate (FSR) bound for the cross-validated rule under a calibration-independence assumption, while auditing the deployed rule empirically. Across eight surrogate benchmarks constructed from real experimental or learning-curve data, SRBO achieves the best average rank among methods evaluated on all benchmarks and the lowest mean regret on IVPT, Concrete, and OCM, whereas BOHB leads on all three learning-curve tasks.
Critical strengths
-
Coordinated stopping and information reuse: The same endpoint forecaster supports early stopping and feeds information from incomplete trials into subsequent search, connecting evaluation-cost savings with the effective use of partial trajectories.
-
Comprehensive experimental protocol and reporting: The experiments use a common cost budget, disjoint tuning and reporting seeds, and 20 repeated runs, while reporting optimization regret, runtime FSR, and tasks where SRBO does not lead, supporting assessment of both effectiveness and applicability.
Critical weaknesses
-
Gap between the theoretical guarantee and deployed stopping: The calibration independence assumed in Section 4.2 is not strictly ensured by the implemented procedure, and Equation (7) retains an uncontrolled deployment gap , so the theorem does not yet provide a computable FSR guarantee for actual deployment.
-
Incomplete uncertainty treatment in pseudo-observation updates: Section 4.1 injects only DF-GP predictive means and assigns the same noise variance to exact and pseudo-observations, without propagating predictive variance or error correlations arising from shared training data, leaving unclear how posterior overconfidence under model misspecification is prevented.
-
Application gains depend on simulator validity: Section 5.1 evaluates all methods on continuous-box simulators fitted to historical data, with optimal values outside the measured range on seven tasks, but lacks validation on independent measured trajectories or within the original feasible domains to rule out dependence of Figure 2's gains on extrapolation error.
Questions for authors
-
FSR guarantees for the deployed rule: Could the authors specify additional conditions that address dependence among five-fold calibration indicators and control , so that Equation (7) yields a computable and meaningful FSR upper bound for the deployed stopping rule?
-
Probabilistic basis of pseudo-observation updates: Could the authors justify using a shared noise variance for exact observations and predictive-mean pseudo-labels, and analyze how ignoring DF-GP predictive variance and error correlations affects the augmented surrogate posterior and EI decisions?
-
Validity on independent measured data: Could the authors evaluate on measured trajectories excluded from forecaster training and calibration, preserve the original variable constraints and match evaluation costs, and report completed-trial regret and runtime FSR to test whether Figure 2's advantage persists?