- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is a relatively theory-oriented paper that starts from a practical problem. Considering that model uncertainty may be overlooked when estimating GP hyperparameters, this paper uses contrast coefficients under the RBF kernel to derive posterior regret bounds over continuous hyperparameter regions. Instead of computing the regret probability for every hyperparameter setting individually, it uses loss contrasts between different designs to control how the probability changes with hyperparameters. Subsequently, the paper analyzes the distinction between predictive uncertainty arising from the hyperparameters themselves and the conservatism of the theoretical bounds, and proves that, under specific conditions, its event-specific bound can provide a tighter certification result than existing information-theoretic bounds.
Critical strengths
- Clearly defined and systematic theoretical analysis
Rather than broadly studying uncertainty in GPs, the authors focus on how changes in hyperparameters affect regret events, demonstrating a reasonable and concrete perspective compared with existing studies. Moreover, the authors not only derive a probability bound but also analyze its conservatism, failure conditions, and the scope of its statistical guarantees, demonstrating a careful and systematic theoretical investigation.
Critical weaknesses
- The advantages of the core theoretical method over existing studies remain unclear
Theorem 1 establishes a posterior regret probability bound over a continuous hyperparameter set. However, existing probability sensitivity methods can also obtain uniform probability bounds over continuous hyperparameter regions by controlling the derivatives of event probabilities with respect to hyperparameters. Although this paper provides a specific theoretical bound for quadratic composites, it does not sufficiently clarify what substantial advantages it offers over existing methods (e.g., in terms of bound tightness, computational cost, or scope of applicability).
- Lack of convincing experimental validation
The paper mainly focuses on theoretical analysis, with experiments limited to a carefully constructed numerical example and simulations of random observation histories, without validation on real Bayesian Optimization tasks. In particular, the authors' most convincing certification advantage comes from a specifically constructed example, while the random simulations do not show that their method outperforms existing information-theoretic methods. Therefore, the current experiments are insufficient to demonstrate that the proposed method offers consistent advantages in more general practical problems, nor do they adequately demonstrate its practical value.
Questions for authors
-
Please further explain the advantages of the proposed theoretical method over existing probability sensitivity bounds.
-
Please include additional experiments on real-world Bayesian Optimization tasks.
-
Please include direct experimental comparisons with existing probability sensitivity bounds.