- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks when a low-fidelity evaluation is worth buying before the high-fidelity query it is meant to guide. It defines cost-adjusted simple regret, which charges each cheap evaluation, and proposes a decision rule that only pays for a cheap evaluation when it is expected to save more than it costs. Under the assumption that the true model lies in a known finite set, the rule is shown to never do worse in expectation than skipping cheap evaluations altogether. Experiments show that cheap data helps on standard benchmarks.
Critical strengths
- The paper addresses an important and often overlooked question in multi-fidelity BO: whether a low-fidelity evaluation is actually worth its cost rather than assuming it helps. Charging each cheap evaluation explicitly through cost-adjusted simple regret is a sensible approach.
Critical weaknesses
- The theory does not show that buying low fidelity helps. Theorem 4.1 only states that the policy is no worse in expectation than stopping immediately, and the proof in Appendix A.3 does not appear to use the cheap evaluation, so the same bound can hold for the high-fidelity-only policy. Theorem 4.3 bounds the loss relative to high-fidelity-only search but never shows a gain.
- The role of is not analyzed. The bound in Theorem 4.1 is identical for every . A larger stops the search earlier and forgoes improvement, but this cost is never quantified.
- The error quantities are not computable. , , , and are all assumed known, with no procedure for obtaining them. The alternative, that the true model lies in a known finite set, cannot be verified in practice, and the paper offers no way to do so.
- The analyzed method and the tested method differ. No results are reported for Exact CGS (Algorithm 1); all experiments use the empirical screen, which is never formally defined and, as the paper states, does not inherit the guarantee.
Given these weaknesses, I cannot recommend the current version of the manuscript for acceptance.
Questions for authors
- The purchase decision depends on a surrogate-based estimate of future savings, but early in the search, the surrogate is fit on too little data for this estimate to be reliable. Is there any way to handle this?