- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a method that incorporates priors for multi objective bayesian optimization. In a multi-fidelity, multi objective BO setting, it introduces a new acquisition function that leverages multiple priors similar to pi-BO.
Critical strengths
The paper formalises the setting where we have multiple priors for the multi-objective setting, and introduces a new acquisition function that leverages this information (Equation 4).
Critical weaknesses
- The paper's approach is not the most novel nor elegant. Basically, it takes the expected improvement acquisition function that applies a discount/uplift factor (see Equation 4) to a candidate based on the belief that the prior thinks that the candidate is good (for that single objective). The discount factor is formulated (kinda via heuristic) to be dependent on the prior strength, the number of iterations. When I read the introduction of the paper, this is the immediate baseline approach I think of, and it turns out to be the method the paper introduces later.
- In addition, the way the acquisition encourages exploration is also quite ad-hoc, where we just apply a random chance of applying or not applying the priors (see section 4.1). Again, this is the first baseline approach I think of.
- The writing is quite difficult to understand because it's not written in a formal way. For example: In the definition of Pareto optimality, the authors write f(lambda_1) < f(lambda_2), but f is a vectorised function. How can we compare them?
- The paper says it uses the expected improvement acquisition function because it performs the best with its prior-injection method and the performance is worse for other acquisition functions (see figure 6). This implies their method only works well with EI and not in general with other acquisition functions (and it can be worse than other baselines). I do not know why, but it might suggest some unsoundness/ad-hocness of their approach in general.
Questions for authors
- Why does your method only work well with EI and not other approaches?
- You did extensive hyperparam tuning for your method (e.g., choice of epsilon, choice of base acquisition function) and present the best combination in your method. Why do you think these are good hyperparams in general settings, and how is this fair against the baselines?
5 comments
Thank you for the responses. I still have some outstanding concerns about the paper and some responses. Please see below.
-
Like what I have mentioned previously - Overall, I am still not convinced about the novelty of the provided approach. The approach of uplifting certain regions of the acquiisition optimization landscape based on the prior is not new, and prior work such as pi-BO has done similar things. The approach of adding a random noise to either follow or not follow the prior is also ad-hoc. I do not believe the author's responses have convinced me about the novelty of their approach. I would love to give suggestions on how to improve the paper in this regard, but I don't have any ideas at the moment, sorry.
-
In addition, I do not agree with the author's response that "these studies show that PriMO is rather robust to the choice of π (Appendix K) and of the scalarization (Appendix J)." From the diagrams, the influence of the choice of these values affect the performance a lot (just look at the figures...). As such, I do not think the algorithm is robust to the choice of epsilon that is selected in this paper for greedy exploration, which might not work in the general case. This goes back to my point on the ad-hoc implementation of components algorithm provided. My suggestion to address this is to think carefully how greediness can be controlled by the algorithm itself instead of merely adding some random noise term. For instance, in GP-UCB, the exploration is controlled by the model's inherent uncertainty about certain regions. Maybe there is some uncertainty in your priors that can be used to improve the algorithm in this way?
PriMO outperforms the strongest baseline for every tested π, robustness to the acquisition function, and the first method for expert priors over multiple objectives
Dear Reviewer gWkd,
Thank you for engaging in the discussion.
Robustness to π
For every tested value of π, PriMO outperforms our strongest baseline, MO-TPE, overall at every budget, and the differences between the π values are small compared to their gap to MO-TPE (new Figure 7, Appendix K). The per-benchmark hypervolume plots (Figure 8, previously Figure 7) now include our strongest baseline as well.
Robustness to the acquisition function
With three of the five acquisition strategies, PriMO outperforms our strongest baseline, MO-TPE, overall (Appendix J, Figure 6). We attribute the lower ranks with LCB and EHVI to these functions themselves rather than to our prior integration, which is identical in all variants; EHVI, for example, is the worst ranked optimizer of our main evaluation even without priors (qLogNEHVI, Figure 2). We added this discussion to Appendix J (in red).
Novelty: the first method for priors over multiple objectives
PriMO is the first method that can use expert priors over multiple objectives (Table 1). Just as ParEGO answers how to trade off several objectives, which EI does not, PriMO answers how to use expert priors over several objectives, which the single-objective πBO does not. The acquisition function that your review notes as new is thus not only new, but the first for this setting.
We would be glad to hear whether this addresses your points, and are happy to discuss any additional feedback.
Simplicity as a strength, extended post-hoc ablations, and formal Pareto dominance
Dear Reviewer gWkd,
Thank you for recognizing that our paper formalizes a new problem setting, multi-objective HPO with expert priors, and introduces a novel acquisition function that leverages these priors (Equation 4). As the first method for this setting, PriMO also exploits cheap approximations in its initial design, and we evaluate it across 8 DL benchmarks, the single-objective setting, and a large-scale case study. Your feedback helped us improve both the clarity of the manuscript and its evaluation, and we addressed your points in the revised manuscript (changes in red).
We agree that PriMO is simple, and we argue that the simplicity of an approach that achieves state-of-the-art performance is a strength, not a weakness. As we are the first to study expert priors over multiple objectives, a simple method that works is also a natural first step for this setting. Simplicity is also a deliberate design goal: PriMO is easy to implement and understand for practical use and keeps the runtime of classic BO (Section 4.4).
The most immediate approach, a direct adaptation of πBO to multiple objectives, is part of our evaluation as the baseline πBO+RW, and it falls short of PriMO: it is unable to recover from bad priors and has the worst final performance on most benchmarks in that setting (Appendix M.4), while PriMO is significantly more robust (Appendix N). The difference stems from the components you refer to, the faster prior decay and the π-greedy exploration, together with our initial design, all of which our post-hoc ablation shows to be important (Figure 5).
Our ablation and sensitivity studies were performed post-hoc and not to tune PriMO's hyperparameters, and we did not develop PriMO with respect to our evaluation protocol either. Further, these studies show that PriMO is rather robust to the choice of π (Appendix K) and of the scalarization (Appendix J).
For the baselines, we use established implementations (e.g., from NePS, SMAC3, Optuna, and Nevergrad) with the settings detailed in Appendix F. In the revision, we additionally aligned the initial design across methods, with unchanged conclusions: PriMO remains the best ranked optimizer overall, significantly better than all other optimizers (Appendix N).
PriMO is not tied to EI: the prior weighting in Equation 4 is a multiplicative weight that can be applied to any acquisition function. We extended our post-hoc ablation accordingly and added MO-TPE, the strongest baseline of our main evaluation, as a reference (Appendix J, Figure 6). PriMO's advantage is not specific to EI with linear scalarization: with hypervolume scalarization, PriMO performs on par with our default, and with Thompson sampling in place of EI, it still outperforms MO-TPE overall. The LCB and EHVI variants rank lower, which we attribute to the base acquisition functions and not to the prior integration, as the prior weighting is identical in all variants.
We added the formal, component-wise definition of Pareto dominance to Section 2 (so far only in Appendix B.2), with f(Ξ»2) < f(Ξ»1) as its shorthand.
Overall, our evaluation shows that PriMO's simplicity is a strength: the direct adaptation of πBO falls short, and each of PriMO's components matters. In response to your review, we further extended our acquisition function ablation, aligned the initial design across all methods, and added the formal definition of Pareto dominance to Section 2. We would appreciate it if you would consider these points and improvements in your rating and your soundness and presentation scores. We are happy to discuss any additional feedback.