acceptodds
Under review as a conference paper at ICLR 2027

When Is Low Fidelity Worth Buying? Exact Certificates and Empirical Safeguards for Multi-Fidelity Bayesian Optimization

Abstract

Multi-fidelity Bayesian optimization can reduce the cost of expensive high-fidelity evaluations, but low-fidelity information is valuable only when its benefit justifies the additional evaluation cost. Existing approaches often model low-fidelity sources or select among fidelities without explicitly accounting for this procurement decision. We introduce a cost-aware framework that treats each high-fidelity query and its optional low-fidelity purchase as a single prepaid macro-round. The framework combines an exact finite-family certificate for reliable procurement with an empirical safeguard that probes low-fidelity usefulness before activation and falls back to high-fidelity-only optimization when the evidence is insufficient. We prove expected and high-probability bounds on cost-adjusted simple regret, a finite-horizon comparison bound against matched high-fidelity-only procurement, and an exact bounded-cost fallback property. Experiments show that the method actively exploits informative low-fidelity sources, rejects unsupported sources, and achieves a substantial improvement on a standard multi-fidelity benchmark while preserving the declared cost accounting. These results demonstrate how certification and empirical safeguards can turn low-fidelity evaluation into a controlled procurement option.

Primary area:
optimization
Keywords:
multi-fidelity Bayesian optimization, cost-aware optimization, low-fidelity evaluation, simple regret

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_s9edOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks when a low-fidelity evaluation is worth buying before the high-fidelity query it is meant to guide. It defines cost-adjusted simple regret, which charges each cheap evaluation, and proposes a decision rule that only pays for a cheap evaluation when it is expected to save more than it costs. Under the assumption that the true model lies in a known finite set, the rule is shown to never do worse in expectation than skipping cheap evaluations altogether. Experiments show that cheap data helps on standard benchmarks.

Critical strengths
  • The paper addresses an important and often overlooked question in multi-fidelity BO: whether a low-fidelity evaluation is actually worth its cost rather than assuming it helps. Charging each cheap evaluation explicitly through cost-adjusted simple regret is a sensible approach.
Critical weaknesses
  • The theory does not show that buying low fidelity helps. Theorem 4.1 only states that the policy is no worse in expectation than stopping immediately, and the proof in Appendix A.3 does not appear to use the cheap evaluation, so the same bound can hold for the high-fidelity-only policy. Theorem 4.3 bounds the loss relative to high-fidelity-only search but never shows a gain.
  • The role of ηt\eta_t is not analyzed. The bound in Theorem 4.1 is identical for every ηt≥0\eta_t \ge 0. A larger ηt\eta_t stops the search earlier and forgoes improvement, but this cost is never quantified.
  • The error quantities are not computable. δt\delta_t, ηt\eta_t, κt,s\kappa_{t,s}, ξt\xi_t and αt\alpha_t are all assumed known, with no procedure for obtaining them. The alternative, that the true model lies in a known finite set, cannot be verified in practice, and the paper offers no way to do so.
  • The analyzed method and the tested method differ. No results are reported for Exact CGS (Algorithm 1); all experiments use the empirical screen, which is never formally defined and, as the paper states, does not inherit the guarantee.

Given these weaknesses, I cannot recommend the current version of the manuscript for acceptance.

Questions for authors
  • The purchase decision depends on a surrogate-based estimate of future savings, but early in the search, the surrogate is fit on too little data for this estimate to be reliable. Is there any way to handle this?
Reviewer_b3CJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies low-fidelity purchase decisions using cost-adjusted simple regret. It derives conservative continuation and value-of-information rules under explicit model/error assumptions, and separately tests a bounded paired-check screening procedure. Empirical results show that screening can exploit useful cheap information, while LCBench and BSCW mainly illustrate fallback behavior.

Critical strengths

The experimental results are performed on the multi-seed setup

Critical weaknesses

Please see the questions.

Questions for authors
  1. Could the authors discuss more about the propose for presenting the Error bound for simulated buying decisions (Theorem 4.5)? Further, the authors may want to significantly reformat the theoretical analysis section to clearly identify the main message of the analysis. It seems that the current theorems and the propositions are more like statements with discussion instead of rigorous mathematical derivation.
  2. After a source passes the screen, what exact computable quantity determines a later purchase? How does it depend on remaining budget and horizon, and what connects the paired-check statistics to that quantity rather than merely to prediction agreement?
Reviewer_S7RiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a cost-aware framework for multi-fidelity Bayesian optimization that distinguishes between the decision to purchase low-fidelity (LF) data and the decision to query high-fidelity (HF) data. The authors introduce two components: "Exact CGS," a theoretical rule providing exact expected and high-probability bounds on cost-adjusted simple regret (CASR) under a finite-family model coverage assumption, and "Safe-LF," an empirical screening procedure that probes LF-HF agreement before activation. The paper formalizes CASR, derives accounting identities for screening costs, and evaluates the methods on synthetic grids, standard benchmarks (mf2), and real-data tasks (LCBench). A key finding is that while the theoretical guarantees hold under strict assumptions, empirical results on real data show the screening method often rejects LF sources even when correlations are high, resulting in no performance gain over HF-only search in those specific contexts.

Critical strengths
  1. The paper introduces a rigorous definition of cost-adjusted simple regret (CASR) that explicitly charges LF evaluations alongside the HF queries they are intended to guide, addressing a gap where prior work often ignored the procurement cost of cheap data.
  2. The authors provide exact proofs for expected and high-probability bounds on CASR under a finite-family model coverage assumption (Theorem 4.1) and establish a pointwise accounting identity for the cost of screening rejected sources/
  3. The manuscript explicitly acknowledges that the empirical screening method does not satisfy the strict model-coverage assumptions required for the theoretical guarantees, separating the "Exact CGS" decision rule from the "Safe-LF" heuristic.
  4. On real-data benchmarks like LCBench, the authors honestly report that the screening method rejected cheap evaluations in all runs despite high correlations, demonstrating a conservative approach that limits spending without claiming false performance gains.
Critical weaknesses
  1. The paper claims the method "achieves a substantial improvement on a standard multi-fidelity benchmark", yet Table 2 shows the "Screen" method performing worse than the "R (uses cheap when reliable)" baseline on several tasks (e.g., Borehole: 33.6 vs 28.2). The text later admits on page 7 that real-data results do not show gains over accurate-only search. This requires clarification or a revision of the abstract's claims.

  2. The manuscript reports mean scores with 95% confidence intervals but does not perform formal statistical significance tests (e.g., t-tests or Wilcoxon signed-rank tests) to determine if observed differences are statistically significant. Given the high variance in some datasets (e.g., Fashion-MNIST), this limits the reliability of the conclusions regarding performance gaps

  3. The theoretical error bound for simulated buying decisions includes a model mismatch term (κt,s\kappa_{t,s}) derived from differences between true and predicted outcome distributions (Theorem 4.5). However, the authors admit that standard GP implementations typically control only the Monte Carlo simulation error, leaving the model-mismatch term uncontrolled in practice. In my read, the manuscript does not seem to quantify or bound this specific error source for the reported experiments.

  4. On page 7, the text states that on LCBench, "The screen rejected later cheap evaluations in every run," but it is unclear if this was driven by correlation values falling below a threshold or by budget constraints during the check phase. The manuscript does not explicitly state whether rejections were due to low correlation or budget exhaustion.

  5. The method relies on a "conservative 0.80 threshold" for accepting cheap data, but the manuscript provides no ablation study varying this threshold (e.g., 0.70, 0.90) to demonstrate the trade-off between false positives and false negatives.

Questions for authors
  1. Did you perform formal statistical hypothesis testing on the results in Tables 1–3 to confirm that differences between methods are statistically significant, given the reported confidence intervals and high variance in some datasets?
  2. For the LCBench experiments, was the rejection of all cheap evaluations driven primarily by correlation values falling below the 0.80 threshold, or did budget constraints during the check phase force early termination? Please clarify the exact decision logic for each run.
  3. What is the cardinality (∣X∣|X|) of the candidate set used in the real-data experiments (LCBench/BSCW), and how was this set constructed? This information is rquired for assessing the granularity of the search and the validity of the finite-family assumption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.