acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Hyperparameter Importance for Efficient Multi-Objective Optimization

Abstract

Choosing a suitable ML model is a complex task that can depend on several objectives, e.g., accuracy, fairness, or energy consumption. In practice, this requires trading off multiple, often competing, objectives through multi-objective optimization (MOO). However, existing MOO methods typically treat all hyperparameters as equally important, disregarding that hyperparameter importance (HPI) can vary significantly across objectives. We propose a novel dynamic optimization approach that prioritizes the most influential hyperparameters based on varying objective trade-offs during the search, thereby accelerating empirical convergence. We advance prior work on HPI for MOO from post-analysis to direct, dynamic integration within the optimization, using HyperSHAP, a recent HPI method that attributes performance gains to hyperparameters via Shapley values. For this, we leverage the objective weightings naturally produced by the MOO algorithm ParEGO and reduce the configuration space by fixing the unimportant hyperparameters, allowing the search to focus on the important ones. We evaluate our method on synthetic tasks from PyMOO and moderate-dimensional HPO tasks from YAHPO-Gym. For HPO, integrating HPI yields up to 42% improvement in final Pareto front quality, while on synthetic data, the improvement is 24% over standard ParEGO

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Hyperparameter Optimization, Multi-Objective Optimization, AutoML, Hyperparameter Importance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aAgCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes HPI-ParEGO, which integrates hyperparameter importance estimation into ParEGO, a classical Bayesian optimization approach. The core claim of HPI-ParEGO is to identify the important hyperparamter subspace for better acquisition function optimization, using SHAP analysis. At each iteration, HPI-ParEGO first fits a surrogate to the current scalarized objective. Then, it use HyperSHAP to assign an importance score to each hyperparameter, and select the smallest but most contributable subset. HPI-ParEGO also integrates a threshold control procedure to guide the algorithmic behavior. Evaluation covers five ZDT functions and 20 YAHPO-Gym tasks, against ParEGO, MO-TPE, NSGA-II, EHVI and Nevergrad-DE, with IGD+ and HV regret.

Critical strengths
  1. The writing seems hand-crafted and is good. It is worth treasuring in this AI age.
  2. The idea of integrating SHAP analysis to dynamically configuring BBO optimizer is illustrative and interesting.
  3. The shown results demonstrate good performance compared to ParEGO.
Critical weaknesses
  1. The major weakness, from my perspective, is that the experiments are mainly conducted on chosen instances. As pointed out in Appendix A, the selected criteria seems better for HPI-ParEGO. What about the filtered cases?
  2. It seems that the comparison against ParEGO is also not fair. In line 170, it notes that the original ParEGO uses u=1, while in line 285, the compared ParEGO uses u=10. Can we also use u=1 for HPI-ParEGO for comparison?
  3. Some components are designed as heuristics. For example, the threshold update rule is heuristic, and so is the usage of HyperSHAP.
  4. Typos:
  • line 128, "but they rely" -> "but it relies"
  • line 226, "computes its Shapley values" -> "computes its Shapley value"
Questions for authors
  1. As it is not so east to theoretically show the benefits of using HyperSHAP, can you just show attempts for variable selection? For example, MCTS-VS [1] uses the sum of queried scores against the selected times as the criterion.
  2. SHAP analysis is a general technique to assign importance to variables. Why don't you integrate this idea to more backbone algorithms (even vanilla BO)?

Overall, I like the idea. I would be pleasant to increase my ratings if my concerns are solved.

References

[1] Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization. NeurIPS 2022.

7 comments

Dear reviewer,

We are glad that you like our idea, and we are also happy to receive a hand-crafted review :) We will now respond in detail to your comments.

The experiments are mainly conducted on chosen instances

We selected YAHPO-Gym tasks in which HPI varies across objectives to test the mechanism under the conditions it targets, a standard practice for validating a specific mechanistic claim. We now also added an experiment on 10 randomly selected tasks from both LCBench and rbv2_ranger to show our method also performs well on unbiased tasks (see Appendix H).

Can we also use u=1 for HPI-ParEGO for comparison?

In the original ParEGO paper, ParEGO uses u=1; in our experiments, we used u=10 for ParEGO (both in our HPI-variant and in the plain baseline) for a fair comparison. The reasoning for using u=10 was that when the space is reduced, that should be exploited for a few iterations before changing the scalarization again. We now included u=1 in our ablation study (see Appendix D.6). The table below shows the results for u=1 on the ablation tasks, which also show better performance for HPI-ParEGO.

MethodMean HV regret AUCMean HV regret final performanceMean IGD+ regret AUCMean IGD+ regret final performance
HPI-ParEGO (u=1)0.115 ± 0.0050.037 ± 0.0060.065 ± 0.0040.006 ± 0.002
ParEGO (u=1)0.15 ± 0.0030.06 ± 0.0060.072 ± 0.0020.02 ± 0.001

As it is not so easy to theoretically show the benefits of using HyperSHAP, can you just show attempts for variable selection? For example, MCTS-VS [1] uses the sum of queried scores against the selected times as the criterion.

To the best of our understanding, MCTS-VS builds a tree over several iterations to find the relevant subset of hyperparameters. In our approach, the important hyperparameters depend on the current objective scalarization, so the tree would have to be rebuilt whenever the weights change. However, the MCTS-VS criterion can be used as HPI method in our approach. We now included that in our ablation study (see Appendix D.2). HyperSHAP outperforms the MCTS-VS criterion.

Why don't you integrate this idea to more backbone algorithms (even vanilla BO)?

We integrated this idea into single-objective “vanilla” BO in our first experiment (Section 4.2). Is there a specific algorithm that you would be interested in?

Typos

We fixed the typos.

Thanks for your detailed response. im currently reconsidering my ratings.

The vanilla BO i mentioned is https://arxiv.org/abs/2402.02229 that uses GP as surrogate model. In Section 4.3, instead of TPE in Optuna combined with qEHVI, what about GP-qEHVI?

Besides, could you please show more illustrative evidence showing the benefits of HyperSHAP instead of only the fitness curve? I don't think the score alone can explain the benefits of your proposed method. Obviously we can adjust / configure the running algo with multiple rules / criterion. Then why HyperSHAP?

Vanilla BO

The paper you are citing is about improving high-dimensional BO, we are focusing on small- to medium-dimensional BO, which is the common setting for hyperparameter optimization. However, combining our approach with their length-scale prior idea is definitely possible. As for BO with a GP, BO with random forest is a valid alternative, especially for mixed and conditional spaces [1]. However, the new experiments based on qNParEGO also use a GP as a surrogate model, and HPI improves the method (see Appendix I).

[1] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor H. Awad, Marius Lindauer, Frank Hutter: HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO. NeurIPS Datasets and Benchmarks 2021

Instead of TPE in Optuna combined with qEHVI, what about GP-qEHVI?

We think there is a misunderstanding. One of our baselines is MO-TPE, and another one is EHVI. EHVI is implemented in Optuna and uses a GP as a surrogate model. We will clarify this in the final version.

Why HyperSHAP?

The commonly used fANOVA models only how sensitive a performance metric is to the hyperparameters. Instead, HyperSHAP quantifies gains relative to the best configuration (i.e., tunability) found so far and helps identify the subset of hyperparameters that promises the largest still possible gain. In HPO, this is more important than simple sensitivity, as argued by Probst et al. JMLR 2019 [2]. Moreover, we show that randomly selecting hyperparameters, with the rest of the setup unchanged, performs worse (Appendix D.2).

[2] Philipp Probst, Anne-Laure Boulesteix, Bernd Bischl: Tunability: Importance of Hyperparameters of Machine Learning Algorithms. J. Mach. Learn. Res. 20: 53:1-53:32 (2019)

Reviewer_a4MaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HPI-ParEGO, a hyperparameter importance-guided framework for efficient multi-objective optimization. The method integrates HyperSHAP into the existing ParEGO algorithm to estimate hyperparameter importance under different objective scalarizations. Based on the estimated Shapley values, the algorithm dynamically reduces the search space by retaining influential hyperparameters and fixing the remaining ones to their incumbent values. The authors further introduce a three-phase optimization schedule to balance exploration and exploitation. Experiments are conducted on synthetic PyMOO problems and YAHPO-Gym hyperparameter optimization benchmarks, with comparisons against several established multi-objective optimization algorithms.

Critical strengths
  1. The paper addresses a relevant problem in multi-objective hyperparameter optimization. Different optimization objectives may depend on different subsets of hyperparameters, and exploiting such differences to reduce the search space is a reasonable direction for improving optimization efficiency.

  2. The proposed framework is straightforward and easy to integrate into existing scalarization-based optimization methods. The combination of ParEGO and HyperSHAP provides a practical approach to incorporating hyperparameter importance into the optimization process.

  3. The paper includes experiments on synthetic and HPO benchmarks, together with ablation studies. The results show improvements over standard ParEGO in several settings, particularly on the selected LCBench tasks.

Critical weaknesses
  1. The theoretical and methodological novelty is insufficiently established. This is my main concern. The proposed approach primarily combines existing hyperparameter importance estimation techniques with an established multi-objective optimization framework, without introducing a fundamentally new optimization formulation or theoretical insight.

First, the core components are already well studied. ParEGO (Knowles, 2006) provides the scalarization-based multi-objective optimization framework, while HyperSHAP (Wever et al., 2026) already estimates hyperparameter importance using Shapley values. Furthermore, Theodorakopoulos et al. (2024), Hyperparameter Importance Analysis for Multi-Objective AutoML, have already investigated how hyperparameter importance varies across different objective trade-offs.

Second, the idea of restricting optimization to influential hyperparameters is not new. Existing approaches, including SAASBO, VS-BO, and GSOS, have already explored effective-dimensionality reduction, variable selection, and importance-guided hyperparameter optimization. Although these methods primarily focus on single-objective settings, the underlying principle of reducing the search space according to variable importance is well established.

In this context, HPI-ParEGO essentially applies an existing importance estimator to the current scalarized objective, selects hyperparameters according to their estimated importance, and fixes the remaining dimensions during acquisition-function optimization. The proposed three-phase schedule is also a heuristic design rather than a theoretically motivated optimization mechanism.

More importantly, the method does not introduce a new model of hyperparameter dependencies or interactions. It mainly relies on importance rankings to determine which dimensions should remain active, without explicitly exploiting the structural relationships among hyperparameters. Although HyperSHAP can attribute contributions arising from interactions, the proposed optimization mechanism does not explicitly model these interactions when reducing the search space.

I acknowledge that incorporating dynamic HPI into the multi-objective optimization loop is an incremental contribution. However, the paper does not provide a sufficiently clear technical distinction from existing importance-guided optimization methods, nor does it establish theoretical conditions under which the proposed reduction improves convergence or Pareto-front approximation. Therefore, I view the principal contribution as a relatively direct integration of established techniques rather than a substantial methodological advance.

  1. The experimental comparisons are insufficient to demonstrate the advantage of the proposed importance-guided optimization mechanism.

Although the paper compares HPI-ParEGO against ParEGO, MO-TPE, NSGA-II, EHVI, and DE, it does not directly compare against existing importance-guided or variable-selection-based optimization methods discussed in Section 2, such as SAASBO, VS-BO, and GSOS.

Furthermore, the ablation studies primarily compare HyperSHAP-based selection with random importance selection, which is a relatively weak baseline. A comparison against static HPI-based search-space reduction or simpler importance-estimation strategies would be more informative for establishing the necessity of dynamically recomputing HyperSHAP.

The performance improvements are also problem-dependent. Figure 14 shows that EHVI substantially outperforms HPI-ParEGO on the synthetic PyMOO benchmarks, while Figure 17 indicates that the proposed method does not consistently outperform ParEGO across individual rbv2_ranger tasks. Therefore, the current results do not establish a consistently compelling advantage in solution quality over existing optimization methods.

  1. The computational efficiency and generalizability of the proposed method are not sufficiently demonstrated.

Section 4.6 reports additional HyperSHAP computation overhead of approximately 20 seconds per trial on PyMOO and 7 seconds per trial on YAHPO-Gym. However, the experimental comparisons mainly consider the number of objective evaluations rather than actual end-to-end computational time. Consequently, it remains unclear whether the improvements in evaluation efficiency compensate for the additional importance-estimation overhead.

Moreover, Appendix A indicates that the YAHPO-Gym tasks are deliberately selected to exhibit varying important-hyperparameter subsets across different objective scalarizations. This selection criterion closely matches the assumption underlying the proposed method and may favor HPI-ParEGO. The results therefore provide limited evidence of generalization to arbitrary HPO problems where hyperparameter importance is less concentrated or more difficult to estimate.

Questions for authors
  1. Regarding the limited methodological novelty, could the authors clarify the fundamental technical contribution beyond combining ParEGO with HyperSHAP-based search-space reduction? In particular, how does the proposed method differ conceptually from existing importance-guided and variable-selection-based optimization approaches? Can the authors provide theoretical analysis or stronger experimental evidence demonstrating that the dynamic HPI mechanism offers nontrivial benefits beyond simpler search-space reduction strategies?

  2. Regarding the modeling of hyperparameter dependencies, how does the proposed approach handle strongly interacting or conditionally dependent hyperparameters? Since the method selects dimensions primarily according to individual importance scores, could this strategy exclude parameters that are individually less important but jointly essential for obtaining high-quality solutions? Additional analysis of importance-estimation reliability and hyperparameter interactions would help clarify this limitation.

  3. Regarding experimental validation and efficiency, could the authors compare HPI-ParEGO against static HPI-based reduction and other relevant importance-guided optimization methods under identical computational budgets? In particular, I would like to see end-to-end wall-clock comparisons that account for the additional HyperSHAP overhead. It would also be useful to evaluate the method on randomly selected HPO tasks without the importance-based benchmark preselection described in Appendix A.

4 comments

Dear reviewer,

We are glad that you found the problem we are tackling relevant. We will now respond in detail to your comments.

The theoretical and methodological novelty is insufficiently established

Existing space-reduction methods either reduce the configuration space without HPI estimates (i.e., with user-defined priors) or only in single-objective settings, where importance remains static throughout the search. Theodorakopoulos et al. (2024) compute HPI for MOO, but only post-hoc on a completed Pareto front. HPI-ParEGO is the first to combine HPI-based reduction with MOO and to apply it dynamically in optimization, which is necessary because, unlike in single-objective settings, HPI shifts with the scalarization of the objectives. This is not trivial or obvious, as our ablation study shows.

The idea of restricting optimization to influential hyperparameters is not new

It is true that the approaches you name explored dimensionality reduction before. However, none of them apply it to multi-objective optimization, where the important subset changes with the objective trade-off. We select the subset for each scalarization using the Shapley-value-based HPI, which has the nice property that we can easily reuse previous data. Our ablation shows that this approach outperforms a random subset of the same size, which is, besides plain ParEGO, the only available baseline in our targeted multi-objective HPO setting.

Compare against existing importance-guided methods, such as SAASBO, VS-BO, and GSOS

These methods are single-objective and would need a multi-objective variant to be comparable to HPI-ParEGO.

Comparison against static HPI-based search-space reduction or simpler importance-estimation strategies

Unfortunately, we are not aware of simpler importance-estimation strategies. For example, the commonly used fANOVA approach is equivalent to computing Moebius interactions within the HyperSHAP framework. With its flexibility in computing different HPI games, HyperSHAP is the current state-of-the-art approach.

We compared static HPI in an experiment on the ablation tasks. HPI was estimated once at the end of the initialization phase and fixed afterward. The results shown in the table below show that keeping a fixed subset of important hyperparameters throughout the search is substantially worse.

MethodMean HV regret AUCMean HV regret final performanceMean IGD+ regret AUCMean IGD+ regret final performance
HPI-ParEGO0.1 ± 0.0040.023 ± 0.0050.059 ± 0.0030.002 ± 0.0
HPI-ParEGO (static)0.117 ± 0.0050.053 ± 0.0070.071 ± 0.0030.022 ± 0.002

The performance improvements are also problem-dependent

After the bug fix, HPI-ParEGO now also substantially outperforms ParEGO on all of the rbv2_ranger tasks (see Figure 6 and Appendix G).

Computational efficiency is not sufficiently demonstrated.

The computational efficiency is dominated by the function evaluations, i.e., training and evaluating ML models with different hyperparameter configurations, which can take minutes, hours, or even days. Compared to that, the overhead of computing HPI within a few seconds is negligible.

The YAHPO-Gym tasks are deliberately selected to exhibit varying important subsets

We selected YAHPO-Gym tasks in which HPI varies across objectives to test the mechanism under the conditions it targets, a standard practice for validating a specific mechanistic claim. We now also added an experiment on 10 randomly selected tasks from both LCBench and rbv2_ranger to show our method also performs well on unbiased tasks (see Appendix H).

How does the proposed approach handle strongly interacting or conditionally dependent hyperparameters?

Unselected conditional hyperparameters are set to their defaults, and for selected conditional hyperparameters, we add their parents to the selected set. HyperSHAP with order 1 computes Shapley values that summarize the interactions among hyperparameters fairly. It is possible that a strongly interacting pair is not selected together. At a higher order, interactions are scored directly to the interacting pair. We now included an ablation with interactions in Appendix D.7, which shows that it is not better than our current approach. Furthermore, we would like to point out that commonly there are fewer interactions between hyperparameters than one would typically assume [1,2,3]. Regarding the reliability of the importance estimates, we now added an experiment comparing against ground-truth importance (see Appendix J).

[1] Hutter, F., Hoos, H., & Leyton-Brown, K. (2014). An efficient approach for assessing hyperparameter importance. ICML 2014.

[2] Pushak, Y., & Hoos, H. (2022). AutoML loss landscapes. ACM Transactions on Evolutionary Learning, 2(3), 1-30.

[3] Wever, M., Muschalik, M., Fumagalli, F., & Lindauer, M. (2026). HyperSHAP: Shapley values and interactions for explaining hyperparameter optimization. AAAI 2026.

Since there is only one day left for the rebuttal, we would really appreciate it if you could look at our comments and new results. If they satisfy your concerns, please reconsider your score.

Thank you for your response. However, I will maintain my original score, as I believe the work does not yet meet the acceptance standard for ICLR.

We would appreciate if the reviewer could elaborate on what criticism remained unaddressed such that the work does not fulfill the acceptance criteria of ICLR.

Reviewer_rXsHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript studies multi-objective hyperparameter optimization (MO-HPO) under the assumptions that a) different parameters yield different importance across tasks and b) some optimizers, such as ParEGO, can exploit such importance. To this end, they propose HPI-ParEGO, in which ParEGO is incorporated with Shapley-value-based hyperparameter importance and a dynamic configuration space reduction scheme.

Critical strengths
  1. The motivation is clear and sound. MO-HPO with varying importance is an interesting problem domain.
  2. The method is clear and easy to follow.
  3. The authors are honest about the limitations at the current stage.
Critical weaknesses
  1. The framework appears to be incremental and empirical. HPI-ParEGO combines ParEGO with the existing HyperSHAP scheme and the commonly used dynamic configuration space reduction. The actual novel elements are the design choices, such as incumbent fixing and the three-phase τ schedule, whose values are established only empirically.

  2. The choice of ParEGO, proposed twenty years ago, over modern scalarization-based multi-objective Bayesian optimization (MOBO) methods (e.g., qNParEGO) is not justified. A more thorough review of the development of ParEGO and other multi-objective optimization strategies with dynamic scalarization is needed to support this choice.

  3. The experimental results lack sufficient insight. The methods compared are not specifically designed for problems with varying importance, and the YAHPO-Gym tasks are selected to maximize the difference in importance among their hyperparameter subsets, so the baselines are supposed to be outperformed. On the other hand, more intrinsic results are missing. For instance, is it possible to calculate the ground-truth HPI in the YAHPO-Gym tasks and gauge how much HPI-ParEGO's estimation deviates from these ground truths?

  4. Other minor issues include that equation (1) uses x as the decision variable, while equation (2) switches to λ. This creates inconsistencies and redundancy in the symbols used.

Questions for authors

Please refer to the weaknesses.

3 comments

Dear reviewer,

We are glad you found our paper clearly written. We will now respond in detail to your comments.

The framework appears to be incremental

Existing space-reduction methods either reduce the configuration space without HPI estimates (i.e., with user-defined priors) and/or only in single-objective settings, where importance remains static throughout the search. Theodorakopoulos et al. (2024) compute HPI for MOO, but only post-hoc on a completed Pareto front. HPI-ParEGO is the first to combine HPI-based reduction with MOO and to apply it dynamically in optimization, which is necessary because, unlike in single-objective settings, HPI shifts with the scalarization of the objectives. This is not trivial or obvious, as our ablation study shows.

The choice of ParEGO over modern methods (e.g., qNParEGO) not justified

The qNParEGO extension to ParEGO is designed to handle batches and noisy observations. Since we did not want any confounding factors to affect our results, we chose the sequential base version of ParEGO for our experiments, which is still widely used and well established. This way, we isolate the effect of including dynamic importance estimation in the search. However, our method also improves the performance of other scalarization-based multi-objective methods, such as qNParEGO, as we now show in Appendix I.

The methods compared are not specifically designed for problems with varying importance

To our knowledge, there are no existing multi-objective optimizers that account for varying importance; this is the gap our algorithm addresses. Our main baseline is ParEGO, which we implemented identically to our method except for the HPI component, which we isolated to assess its effect.

The YAHPO-Gym tasks are selected to maximize the difference in importance

We selected YAHPO-Gym tasks in which HPI varies across objectives to test the mechanism under the conditions it targets, a standard practice for validating a specific mechanistic claim. We now also added an experiment on 10 randomly selected tasks from both LCBench and rbv2_ranger to show our method also performs well on unbiased tasks (see Appendix H).

Ground-truth HPI on the YAHPO-Gym tasks

Yes, that is approximately possible. Since the tasks we are using are based on surrogate models, we can calculate the “true importance” by very expensive sampling of the optimization problem. We have now included an experiment for that in Appendix J. HPI-ParEGO performs almost as well as the ground-truth importance.

Minor notation issue

We now use the same notation for equations 1 and 2.

Since there is only one day left for the rebuttal, we would really appreciate it if you could look at our comments and new results. If they satisfy your concerns, please reconsider your score.

I appreciate the authors' substantial efforts in addressing my concerns W2 through W4 to a significant extent. However, I still have reservations about my initial concern W1 regarding the novelty of the contribution. It appears to be a combination of ParEGO, HyperSHAP, and established space-reduction techniques, whose originality seems more empirical than methodological. Given that ICLR is a highly selective venue in machine learning, I am cautious about whether HPI-ParEGO meets the significance threshold, even when considering all reported results at face value.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers

Dear reviewers,

We thank all of you for your reviews. Due to a small bug in the implementation that we found based on your feedback, we had to make minor changes and rerun the experiments, which led to an improvement of HPI-ParEGO.

We uploaded a new version of our paper with updated results due to the following changes:

  • The schedule is now 0-08-08, so only the initialization phase is not reduced. In contrast to our initial expectations, the HyperSHAP HPIs proved sufficient to consistently focus on the most important hyperparameters.
  • HyperSHAP now uses a Gaussian Process to estimate Hyperparameter Importance (see Appendix D.8 for a comparison of a Random Forest vs. a GP).

We also made further additions based on your questions:

  • An evaluation on 10 randomly selected LCBench tasks and 10 randomly selected RBV2_Ranger tasks. Both plots show that HPI-ParEGO also outperforms ParEGO on these tasks that are not biased toward our approach, addressing your concerns that our task selection favors our method (Appendix H).
  • Including higher-order interactions estimated by HyperSHAP did not improve our approach (Appendix D.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.