- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a human-AI collaborative Bayesian optimization (BO) framework that incorporates expert knowledge through a spatially varying trust mechanism. Specifically, the method learns a latent utility function from human feedback and employs conformal calibration to estimate input-dependent trust weights. These weights are then incorporated into a trust-aware kernel, allowing the BO algorithm to exploit expert knowledge while accounting for its potentially heterogeneous reliability across the search space.
Critical strengths
-
Clear presentation. The paper is generally well written, and the overall methodological description is relatively clear and logically organized, making the proposed framework easy to follow.
-
Research significance and potential novelty. Incorporating expert knowledge into BO is an important research direction, and the proposed approach appears to offer an interesting perspective. Similar to Incorporating Expert Prior in Bayesian Optimisation via Space Warping, this work integrates expert knowledge into the kernel structure to influence the BO process. However, this paper adopts a potentially more interesting mechanism by using conformal calibration to estimate spatially varying trust weights before incorporating expert information into the kernel.
Critical weaknesses
-
Insufficient experimental evaluation. Although the methodological presentation is relatively clear, the experimental section is considerably weaker. The evaluation covers only a limited number of tasks and compares the proposed method against a relatively small set of baselines. Furthermore, several important experimental details are insufficiently explained, making it difficult to fully assess the effectiveness and robustness of the proposed approach.
-
Poor figure readability and presentation errors. The font sizes of the axis labels and tick labels in the experimental figures are too small, making the results difficult to read. The authors should improve the figure layouts and increase the font sizes. There are also apparent textual inconsistencies that require careful proofreading. For example, on page 8, the phrase "all three functions" appears inconsistent with the corresponding figure, which presents results on four benchmark functions.
Questions for authors
-
How is the latent utility function obtained in practice, and how are the synthetic reliability parameters determined?
The method models the latent expert utility using a posterior distribution of the form . However, in the synthetic experiments, the authors construct a simulated human latent utility function as , where and is a spatially varying coefficient designed to control expert reliability.
My main concern is how the proposed method obtains a meaningful latent utility function from an available human-feedback dataset in realistic applications, where the true objective function and the underlying expert utility are generally unknown. The paper should clarify the assumptions required for learning this utility function and explain how its quality can be assessed without access to the ground-truth expert utility.
In addition, Appendix B.2 specifies , , and , together with manually designed spatial reliability regions. How were these values and regions selected? Are the experimental conclusions robust to alternative choices? More broadly, the combination of the synthetic utility construction, conformal rank prediction, and online calibration introduces considerable methodological complexity. The authors should better justify these design choices and demonstrate that the resulting framework is practical beyond the specifically constructed synthetic settings.
-
Why is the additive kernel construction appropriate, and what justifies its effectiveness?
The proposed kernel is . This construction incorporates expert information through an additive kernel component modulated by the estimated trust weights. However, the paper should provide a clearer justification for why this particular additive construction is appropriate and explain how it enables reliable expert information to improve BO while limiting the influence of unreliable feedback. Additional theoretical discussion or ablation studies comparing alternative kernel constructions would strengthen this design choice.
-
How are the expert priors for the baselines constructed, and how should the unexpected comparative results be interpreted?
The experimental section does not sufficiently explain how expert prior information is constructed for BO and CoExBO, or how the corresponding hyperparameters are selected. These details are essential for ensuring a fair and reproducible comparison.
Moreover, in Figures 2 and 3, vanilla GP-UCB appears to outperform BO and CoExBO in some settings. This raises questions about whether the expert information provided to these baselines is appropriately specified and whether the baseline implementations and hyperparameter settings are comparable. The authors should explain these results and provide sufficient details to rule out potential implementation or experimental-design issues.
Similarly, in the appendix experiments involving adversarial expert information, the proposed method appears to outperform vanilla GP-UCB. While such an outcome is not necessarily impossible, it is unexpected and requires further investigation. The authors should explain how the proposed trust mechanism can yield this improvement when the expert feedback is adversarial, and provide additional evidence—such as ablation studies, trust-weight visualizations, or repeated experiments—to support the interpretation.
Overall, the paper would benefit substantially from a more thorough and transparent experimental evaluation. At present, the methodological development is considerably more extensive than the empirical validation, and the limited experimental coverage and insufficiently explained results make it difficult to determine how reliably the proposed method performs across different settings.