- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes HPI-ParEGO, which integrates hyperparameter importance estimation into ParEGO, a classical Bayesian optimization approach. The core claim of HPI-ParEGO is to identify the important hyperparamter subspace for better acquisition function optimization, using SHAP analysis. At each iteration, HPI-ParEGO first fits a surrogate to the current scalarized objective. Then, it use HyperSHAP to assign an importance score to each hyperparameter, and select the smallest but most contributable subset. HPI-ParEGO also integrates a threshold control procedure to guide the algorithmic behavior. Evaluation covers five ZDT functions and 20 YAHPO-Gym tasks, against ParEGO, MO-TPE, NSGA-II, EHVI and Nevergrad-DE, with IGD+ and HV regret.
Critical strengths
- The writing seems hand-crafted and is good. It is worth treasuring in this AI age.
- The idea of integrating SHAP analysis to dynamically configuring BBO optimizer is illustrative and interesting.
- The shown results demonstrate good performance compared to ParEGO.
Critical weaknesses
- The major weakness, from my perspective, is that the experiments are mainly conducted on chosen instances. As pointed out in Appendix A, the selected criteria seems better for HPI-ParEGO. What about the filtered cases?
- It seems that the comparison against ParEGO is also not fair. In line 170, it notes that the original ParEGO uses u=1, while in line 285, the compared ParEGO uses u=10. Can we also use u=1 for HPI-ParEGO for comparison?
- Some components are designed as heuristics. For example, the threshold update rule is heuristic, and so is the usage of HyperSHAP.
- Typos:
- line 128, "but they rely" -> "but it relies"
- line 226, "computes its Shapley values" -> "computes its Shapley value"
Questions for authors
- As it is not so east to theoretically show the benefits of using HyperSHAP, can you just show attempts for variable selection? For example, MCTS-VS [1] uses the sum of queried scores against the selected times as the criterion.
- SHAP analysis is a general technique to assign importance to variables. Why don't you integrate this idea to more backbone algorithms (even vanilla BO)?
Overall, I like the idea. I would be pleasant to increase my ratings if my concerns are solved.
References
[1] Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. NeurIPS 2022.
7 comments
Thanks for your detailed response. im currently reconsidering my ratings.
The vanilla BO i mentioned is https://arxiv.org/abs/2402.02229 that uses GP as surrogate model. In Section 4.3, instead of TPE in Optuna combined with qEHVI, what about GP-qEHVI?
Besides, could you please show more illustrative evidence showing the benefits of HyperSHAP instead of only the fitness curve? I don't think the score alone can explain the benefits of your proposed method. Obviously we can adjust / configure the running algo with multiple rules / criterion. Then why HyperSHAP?
Vanilla BO
The paper you are citing is about improving high-dimensional BO, we are focusing on small- to medium-dimensional BO, which is the common setting for hyperparameter optimization. However, combining our approach with their length-scale prior idea is definitely possible. As for BO with a GP, BO with random forest is a valid alternative, especially for mixed and conditional spaces [1]. However, the new experiments based on qNParEGO also use a GP as a surrogate model, and HPI improves the method (see Appendix I).
[1] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, Frank Hutter: HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. NeurIPS Datasets and Benchmarks 2021
Instead of TPE in Optuna combined with qEHVI, what about GP-qEHVI?
We think there is a misunderstanding. One of our baselines is MO-TPE, and another one is EHVI. EHVI is implemented in Optuna and uses a GP as a surrogate model. We will clarify this in the final version.
Why HyperSHAP?
The commonly used fANOVA models only how sensitive a performance metric is to the hyperparameters. Instead, HyperSHAP quantifies gains relative to the best configuration (i.e., tunability) found so far and helps identify the subset of hyperparameters that promises the largest still possible gain. In HPO, this is more important than simple sensitivity, as argued by Probst et al. JMLR 2019 [2]. Moreover, we show that randomly selecting hyperparameters, with the rest of the setup unchanged, performs worse (Appendix D.2).
[2] Philipp Probst, Anne-Laure Boulesteix, Bernd Bischl: Tunability: Importance of Hyperparameters of Machine Learning Algorithms. J. Mach. Learn. Res. 20: 53:1-53:32 (2019)
Dear reviewer,
We are glad that you like our idea, and we are also happy to receive a hand-crafted review :) We will now respond in detail to your comments.
The experiments are mainly conducted on chosen instances
We selected YAHPO-Gym tasks in which HPI varies across objectives to test the mechanism under the conditions it targets, a standard practice for validating a specific mechanistic claim. We now also added an experiment on 10 randomly selected tasks from both LCBench and rbv2_ranger to show our method also performs well on unbiased tasks (see Appendix H).
Can we also use u=1 for HPI-ParEGO for comparison?
In the original ParEGO paper, ParEGO uses u=1; in our experiments, we used u=10 for ParEGO (both in our HPI-variant and in the plain baseline) for a fair comparison. The reasoning for using u=10 was that when the space is reduced, that should be exploited for a few iterations before changing the scalarization again. We now included u=1 in our ablation study (see Appendix D.6). The table below shows the results for u=1 on the ablation tasks, which also show better performance for HPI-ParEGO.
As it is not so easy to theoretically show the benefits of using HyperSHAP, can you just show attempts for variable selection? For example, MCTS-VS [1] uses the sum of queried scores against the selected times as the criterion.
To the best of our understanding, MCTS-VS builds a tree over several iterations to find the relevant subset of hyperparameters. In our approach, the important hyperparameters depend on the current objective scalarization, so the tree would have to be rebuilt whenever the weights change. However, the MCTS-VS criterion can be used as HPI method in our approach. We now included that in our ablation study (see Appendix D.2). HyperSHAP outperforms the MCTS-VS criterion.
Why don't you integrate this idea to more backbone algorithms (even vanilla BO)?
We integrated this idea into single-objective “vanilla” BO in our first experiment (Section 4.2). Is there a specific algorithm that you would be interested in?
Typos
We fixed the typos.