- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies safe Bayesian optimization over configuration-intensity pairs, assuming that safety deteriorates monotonically with intensity while the objective can be nonmonotone. It introduces Boundary Unlocking Value (BUV), which averages optimistic objective gains beyond each configuration’s certified safe boundary under a sampled boundary distribution. BUV-SafeBO weights information about selected safety frontiers by these values and combines the resulting exploration score with expected improvement. Experiments on spinal cord stimulation, quadrotor control, and drug-dose optimization claim competitive optimization performance with high empirical safe-query rates.
Critical strengths
S1. A specific contribution to prioritizing safe expansion. BUV distinguishes boundary uncertainty that exposes little objective improvement from uncertainty associated with valuable extensions. The closest structural predecessor, M-SafeOpt, already uses objective information to eliminate suboptimal configurations and restrict expansion, through a binary test against the best safe LCB. BUV differs in both reference and role: it scores each configuration by a probability-weighted magnitude of prospective extension gains relative to its own certified prefix, and uses that score to select and weight frontier information targets rather than to gate which boundaries may be expanded.
Critical weaknesses
W1. Comparisons with baselines confound acquisition design with safety certification and query protocol. Methods use different safety multipliers (Table 2), ranging from 2 to 5 on SCS. The paper’s own sensitivity analysis demonstrates that this choice matters: increasing BUV-SafeBO’s from 2 to 4 raises its safe-query rate from 95.75% to 99.30% (Table 6). Query eligibility is also insufficiently specified: the no-repeat rule is stated only for BUV-SafeBO, leaving unclear whether baselines may spend their budgets re-querying evaluated actions, which yield identical responses on these deterministic tables. Consequently, the reported progress–safety trade-offs characterize the complete implementations, but cannot be attributed solely to acquisition design. A controlled comparison using shared certification and query-eligibility rules, or per-method confidence-parameter sweeps showing progress versus safety violations, would help resolve this issue. The repeat policies should also be reported.
W2. BUV’s incremental benefit needs clearer characterization against simpler alternatives. Under matched pipeline settings, EI-only, or the “BUV-EI” variant, Eq. (44), which computes no BUV, already achieves AUC₈₀₀ of 0.656 on SCS and 0.852 on quadrotor, exceeding the strongest reported baseline in each benchmark (ITL: approximately 0.624† and 0.841†). The full method provides additional gains of 0.032 and 0.034 (Table 1). The quadrotor increment is supported by the reported summaries, while the reliability of the SCS increment remains unresolved with five seeds and no paired uncertainty estimates. The entropy-weighting ablation supports objective-sensitive weighting, but does not isolate the benefit of probability-weighted extension gains over simpler objective-aware targeting, such as M-SafeOpt-style binary expander filtering. The evidence therefore supports an incremental benefit, while leaving its consistency and advantage over simpler alternatives incompletely characterized. Paired differences with confidence intervals, EI-only in the main comparison plots, and a controlled comparison with binary-filtered information targeting would strengthen the contribution assessment.
† Baseline AUCs were estimated from the figures’ vector paths because numerical values are not reported. The same extraction procedure reproduces all eight Table 1 entries to within 0.0001.
Questions for authors
Q1. How were the per-method values in Table 2 selected? Do the progress–safety rankings persist under a common per benchmark, or across per-method sweeps plotted as progress versus violation?
Q2. Were baselines permitted to re-query evaluated actions? Please report the number of unique actions actually queried by each method on each benchmark.
Q3. For SCS and quadrotor, could you report the mean AUC₈₀₀ differences between BUV-SafeBO and each of BUV-EI and BUV-Uncertainty, paired by seed and accompanied by confidence intervals? Please also include BUV-EI in the LINCS and Sim4Life comparisons.
Q4. Under the same GP models, safety rules, and repeat-query policy, does BUV outperform an M-SafeOpt-style binary filter that retains configurations only when their optimistic extension value exceeds the highest objective LCB in the current safe set? For this comparison, the binary-filter variant would assign equal weights to the selected configurations’ first uncertified points.
Q5. What fraction of configurations violates the monotonicity assumption in Eq. (1)? In particular, how often is a higher intensity safe while a lower intensity is unsafe for the same configuration? Was any preprocessing used to enforce monotonicity? Among unsafe queries, how many were certified by the current LCB rule, and how many were admitted only through historical certificates (Eq. 4)?
6 comments
Response to Reviewer UMgF
Q4) Controlled comparison with binary-filtered information targeting
We implemented an M-SafeOpt-style binary-filter variant under the same GP models, initial observations, safety certification rule, repeat-query policy, and budget as BUV-SafeBO. Both methods use , , and the same EI term and normalization procedure.
For each configuration with a nonempty, unresolved certified-safe prefix, the variant retains the configuration only if its maximum objective UCB beyond that prefix exceeds the maximum objective LCB over the current certified-safe set. It assigns equal weights to the first uncertified points of all retained configurations in the same mutual-information score.
The binary variant uses all qualifying configurations, whereas BUV-SafeBO retains its original maximum of information targets. Thus, the comparison changes target selection and weighting, while preserving the information-routing mechanism and EI component.
We evaluated this comparison on SCS using two targets and ten seeds, and 800 online queries following 200 initial observations. Results are mean ± SEM across seeds after averaging targets within each seed.
| Method | AUC800 | Final progress | Safe queries (%) | Cumulative violation |
|---|---|---|---|---|
| BUV-SafeBO | 0.8080 ± 0.0229 | 0.9077 ± 0.0337 | 99.0417 ± 0.1807 | 0.0641 ± 0.0171 |
| Binary filter + equal-weight MI + EI | 0.7634 ± 0.0297 | 0.8702 ± 0.0149 | 99.2396 ± 0.1201 | 0.0467 ± 0.0088 |
BUV-SafeBO achieves higher mean AUC and final progress, with a mean paired AUC difference of +0.0447. The binary variant has a slightly higher safe-query rate and lower cumulative violation.
Q5) Monotonicity and the source of unsafe admissions
We audited every configuration and intensity level in SCS and Sim4Life, distinguishing numerical violations of the nonincreasing safety margin from unsafe-lower/safe-higher reversals.
| Dataset | Configurations × intensities | Configurations with nonmonotone safety margins | Unsafe-lower/safe-higher pairs |
|---|---|---|---|
| SCS | 21,800 × 10 | 2/21,800 (0.0092%) | 0/981,000 |
| Sim4Life | 171 × 100 | 0/171 (0%) | 0/846,450 |
The pair counts include all lower–higher intensity pairs within each configuration, not only adjacent levels.
The two SCS violations are small adjacent increases in the safety margin, approximately and . Neither changes the safety classification. Thus, numerical monotonicity is not exact everywhere in SCS, but no configuration exhibits a lower intensity that is unsafe while a higher intensity is safe.
Our local preprocessing does not enforce monotonicity through smoothing, monotone projection, or removal of violating rows. For SCS, the safety margin is computed directly from the maximum activation across response channels. For Sim4Life, preprocessing checks monotonicity and raises an error if the check fails, rather than modifying the responses.
Our audit found that, on both SCS and Sim4Life, all unsafe online queries were admitted by the current LCB-prefix rule. The number of unsafe queries admitted solely through historical certificates was zero in both datasets.
Dear Reviewer UMgF,
Thank you very much for your help with reviewing for ICLR'27!
The authors have now responded to your reviews. As the discussion period is approaching the end, I'm wondering if you could have a look at the authors' responses and see whether they have addressed your concerns. And could you also leave a comment to let the authors know your thoughts? If any concerns remain, please explain which ones have not been addressed and why.
Warm regards, Your AC
Response to Reviewer UMgF
We thank the reviewer for the detailed assessment and constructive suggestions. We have revised the comparison protocol and further examined the incremental contribution of BUV.
W1 / Q1) Fair comparison under matched settings
We thank the reviewer for identifying the confounding effects of safety certification and query eligibility in our original comparisons. We agree that different βg values can produce different certified-safe sets, affecting both optimization progress and safety performance.
In the original experiments, we used method-specific safety confidence multipliers to obtain reasonable performance–safety trade-offs for the respective implementations. However, we acknowledge that this did not fully isolate the contribution of acquisition design, and that the original manuscript did not adequately specify the repeat-query policies of all baselines.
To address these issues, we reran the comparisons under a unified protocol. Within each benchmark, all methods now share the same GP specification, initial observations, safety confidence multiplier, safety certification rule, repeat-query policy, query budget, and random seeds. In particular, all methods restrict query selection to previously unevaluated actions in the certified-safe set, excluding both initial observations and previous online queries. Repeated queries are not permitted.
We regenerated the optimization-performance and safety comparisons in Figures 2, 3, and 4 using these matched settings. The shared safety confidence multipliers are listed below.
The revised results broadly preserve the qualitative trends of the original experiments, with BUV-SafeBO maintaining competitive optimization performance and safety under the unified protocol.
Q2) Repeat-query policy and unique actions
The original presentation did not adequately specify repeat-query policies for all methods. In our audit of the earlier SCS comparison that unified only the safety multiplier, some baselines did re-query evaluated actions. Their mean numbers of unique online actions per 800 queries were 744.71 for M-SafeOpt, 747.83 for M-SafeUCB, 588.25 for HdSafeBO, 742.71 for SafeOpt, 193.48 for ITL, and 785.29 for ISE. The four BUV variants each queried 800 unique online actions.
In the revised unified protocol, all methods exclude previously evaluated actions, including both initial observations and earlier online queries.
W2 / Q3) Paired AUC differences and the contribution beyond EI-only
We thank the reviewer for this constructive suggestion. We agree that EI-only is a strong comparator and that BUV’s contribution should be characterized as an additional benefit beyond immediate objective optimization, rather than a replacement for EI.
The two components of our acquisition serve complementary purposes. EI evaluates the expected immediate objective improvement within the current certified-safe set, accounting for both the posterior mean and uncertainty of the objective. In contrast, the BUV-weighted information score Qt(x) measures how informative a certified-safe query is about safety margins at the first uncertified intensities of selected configurations. BUV assigns greater weight to configurations whose potential safe expansions may unlock larger optimistic objective gains. Thus, EI supports immediate improvement, while BUV-weighted information acquisition guides learning about potentially valuable future safe expansions. We have added an explicit discussion of EI’s role in Section 3.3 and expanded the ablation discussion in Section 4.4 to acknowledge its contribution and explain its complementarity with BUV-weighted information acquisition.
For each seed, we first average AUC800 over the fixed tasks or targets and then subtract the comparator’s AUC from that of BUV-SafeBO. The resulting paired differences are summarized below as mean ± sample SD across seeds. Positive values favor BUV-SafeBO.
We have not yet included BUV-EI in the LINCS comparison. LINCS permits only 272 additional queries and was therefore excluded from our common-budget AUC800 ablation study. These results show positive mean increments over both alternatives in the reported settings.