acceptodds
Under review as a conference paper at ICLR 2027

Multi-objective Hyperparameter Optimization with Expert Priors

Abstract

While Deep Learning (DL) experts often have prior knowledge about which hyperparameter settings yield strong performance, only few Hyperparameter Optimization (HPO) algorithms can leverage such prior knowledge and none incorporate priors over multiple objectives. As DL practitioners often need to optimize not just one but many objectives, this is a blind spot in the algorithmic landscape of HPO. To address this shortcoming, we propose the novel problem setting of multiobjective HPO with expert priors and introduce PriMO, the first HPO algorithm that can integrate user beliefs over multiple objectives. We show PriMO achieves state-of-the-art performance across 8 DL benchmarks in the multi-objective and single-objective setting. Further, we demonstrate that PriMO remains state-of-the-art at large scales by optimizing a vision-language model over three objectives.

Primary area:
optimization
Keywords:
Hyperparameter Optimization, Multi-objective, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_gWkdOpenReview β†—
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method that incorporates priors for multi objective bayesian optimization. In a multi-fidelity, multi objective BO setting, it introduces a new acquisition function that leverages multiple priors similar to pi-BO.

Critical strengths

The paper formalises the setting where we have multiple priors for the multi-objective setting, and introduces a new acquisition function that leverages this information (Equation 4).

Critical weaknesses
  1. The paper's approach is not the most novel nor elegant. Basically, it takes the expected improvement acquisition function that applies a discount/uplift factor (see Equation 4) to a candidate based on the belief that the prior thinks that the candidate is good (for that single objective). The discount factor is formulated (kinda via heuristic) to be dependent on the prior strength, the number of iterations. When I read the introduction of the paper, this is the immediate baseline approach I think of, and it turns out to be the method the paper introduces later.
  2. In addition, the way the acquisition encourages exploration is also quite ad-hoc, where we just apply a random chance of applying or not applying the priors (see section 4.1). Again, this is the first baseline approach I think of.
  3. The writing is quite difficult to understand because it's not written in a formal way. For example: In the definition of Pareto optimality, the authors write f(lambda_1) < f(lambda_2), but f is a vectorised function. How can we compare them?
  4. The paper says it uses the expected improvement acquisition function because it performs the best with its prior-injection method and the performance is worse for other acquisition functions (see figure 6). This implies their method only works well with EI and not in general with other acquisition functions (and it can be worse than other baselines). I do not know why, but it might suggest some unsoundness/ad-hocness of their approach in general.
Questions for authors
  1. Why does your method only work well with EI and not other approaches?
  2. You did extensive hyperparam tuning for your method (e.g., choice of epsilon, choice of base acquisition function) and present the best combination in your method. Why do you think these are good hyperparams in general settings, and how is this fair against the baselines?
5 comments
AuthorsOpenReview β†—

Simplicity as a strength, extended post-hoc ablations, and formal Pareto dominance

Dear Reviewer gWkd,

Thank you for recognizing that our paper formalizes a new problem setting, multi-objective HPO with expert priors, and introduces a novel acquisition function that leverages these priors (Equation 4). As the first method for this setting, PriMO also exploits cheap approximations in its initial design, and we evaluate it across 8 DL benchmarks, the single-objective setting, and a large-scale case study. Your feedback helped us improve both the clarity of the manuscript and its evaluation, and we addressed your points in the revised manuscript (changes in red).

Simplicity of the prior-weighted acquisition function and the πœ–-greedy exploration (W1, W2)

We agree that PriMO is simple, and we argue that the simplicity of an approach that achieves state-of-the-art performance is a strength, not a weakness. As we are the first to study expert priors over multiple objectives, a simple method that works is also a natural first step for this setting. Simplicity is also a deliberate design goal: PriMO is easy to implement and understand for practical use and keeps the runtime of classic BO (Section 4.4).

The most immediate approach, a direct adaptation of πœ‹BO to multiple objectives, is part of our evaluation as the baseline πœ‹BO+RW, and it falls short of PriMO: it is unable to recover from bad priors and has the worst final performance on most benchmarks in that setting (Appendix M.4), while PriMO is significantly more robust (Appendix N). The difference stems from the components you refer to, the faster prior decay and the πœ–-greedy exploration, together with our initial design, all of which our post-hoc ablation shows to be important (Figure 5).

Were PriMO's hyperparameters tuned, and is the comparison fair to the baselines? (Q2)

Our ablation and sensitivity studies were performed post-hoc and not to tune PriMO's hyperparameters, and we did not develop PriMO with respect to our evaluation protocol either. Further, these studies show that PriMO is rather robust to the choice of πœ– (Appendix K) and of the scalarization (Appendix J).

For the baselines, we use established implementations (e.g., from NePS, SMAC3, Optuna, and Nevergrad) with the settings detailed in Appendix F. In the revision, we additionally aligned the initial design across methods, with unchanged conclusions: PriMO remains the best ranked optimizer overall, significantly better than all other optimizers (Appendix N).

PriMO with acquisition functions other than EI (W4, Q1)

PriMO is not tied to EI: the prior weighting in Equation 4 is a multiplicative weight that can be applied to any acquisition function. We extended our post-hoc ablation accordingly and added MO-TPE, the strongest baseline of our main evaluation, as a reference (Appendix J, Figure 6). PriMO's advantage is not specific to EI with linear scalarization: with hypervolume scalarization, PriMO performs on par with our default, and with Thompson sampling in place of EI, it still outperforms MO-TPE overall. The LCB and EHVI variants rank lower, which we attribute to the base acquisition functions and not to the prior integration, as the prior weighting is identical in all variants.

Formal definition of Pareto optimality (W3)

We added the formal, component-wise definition of Pareto dominance to Section 2 (so far only in Appendix B.2), with f(Ξ»2) < f(Ξ»1) as its shorthand.

Overall, our evaluation shows that PriMO's simplicity is a strength: the direct adaptation of πœ‹BO falls short, and each of PriMO's components matters. In response to your review, we further extended our acquisition function ablation, aligned the initial design across all methods, and added the formal definition of Pareto dominance to Section 2. We would appreciate it if you would consider these points and improvements in your rating and your soundness and presentation scores. We are happy to discuss any additional feedback.

Thank you for the responses. I still have some outstanding concerns about the paper and some responses. Please see below.

  1. Like what I have mentioned previously - Overall, I am still not convinced about the novelty of the provided approach. The approach of uplifting certain regions of the acquiisition optimization landscape based on the prior is not new, and prior work such as pi-BO has done similar things. The approach of adding a random noise to either follow or not follow the prior is also ad-hoc. I do not believe the author's responses have convinced me about the novelty of their approach. I would love to give suggestions on how to improve the paper in this regard, but I don't have any ideas at the moment, sorry.

  2. In addition, I do not agree with the author's response that "these studies show that PriMO is rather robust to the choice of πœ– (Appendix K) and of the scalarization (Appendix J)." From the diagrams, the influence of the choice of these values affect the performance a lot (just look at the figures...). As such, I do not think the algorithm is robust to the choice of epsilon that is selected in this paper for greedy exploration, which might not work in the general case. This goes back to my point on the ad-hoc implementation of components algorithm provided. My suggestion to address this is to think carefully how greediness can be controlled by the algorithm itself instead of merely adding some random noise term. For instance, in GP-UCB, the exploration is controlled by the model's inherent uncertainty about certain regions. Maybe there is some uncertainty in your priors that can be used to improve the algorithm in this way?

AuthorsOpenReview β†—

PriMO outperforms the strongest baseline for every tested πœ–, robustness to the acquisition function, and the first method for expert priors over multiple objectives

Dear Reviewer gWkd,

Thank you for engaging in the discussion.

Robustness to πœ–

For every tested value of πœ–, PriMO outperforms our strongest baseline, MO-TPE, overall at every budget, and the differences between the πœ– values are small compared to their gap to MO-TPE (new Figure 7, Appendix K). The per-benchmark hypervolume plots (Figure 8, previously Figure 7) now include our strongest baseline as well.

Robustness to the acquisition function

With three of the five acquisition strategies, PriMO outperforms our strongest baseline, MO-TPE, overall (Appendix J, Figure 6). We attribute the lower ranks with LCB and EHVI to these functions themselves rather than to our prior integration, which is identical in all variants; EHVI, for example, is the worst ranked optimizer of our main evaluation even without priors (qLogNEHVI, Figure 2). We added this discussion to Appendix J (in red).

Novelty: the first method for priors over multiple objectives

PriMO is the first method that can use expert priors over multiple objectives (Table 1). Just as ParEGO answers how to trade off several objectives, which EI does not, PriMO answers how to use expert priors over several objectives, which the single-objective πœ‹BO does not. The acquisition function that your review notes as new is thus not only new, but the first for this setting.

We would be glad to hear whether this addresses your points, and are happy to discuss any additional feedback.

Reviewer_6rW9OpenReview β†—
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PriMO, a multi-objective hyperparameter optimization method that leverages expert priors across multiple objectives. It defines prior distributions over the locations of the optima of individual objective functions and combines a low-fidelity initial search via MOASHA with Bayesian optimization, where the acquisition function is weighted by the prior distributions. Furthermore, it gradually attenuates the influence of the prior distributions to enhance robustness against incorrect prior knowledge. The authors report superior hypervolume and anytime performance compared to existing multi-objective HPO methods through experiments on eight deep learning benchmarks and OpenCLIP.

Critical strengths
  1. Practical Problem Formulation and Leveraging Expert Priors

The problem of incorporating expert knowledge regarding multiple objective functions into HPO is useful in real-world deep learning development. Furthermore, considering robustness to incorrect prior knowledge contributes to its practicality.

  1. Relatively Comprehensive Experimental Validation

The paper compares the proposed method with existing multi-objective optimization methods using multiple benchmarks and OpenCLIP. Furthermore, analyses based on prior distribution quality and ablation studies have been conducted, providing some experimental evidence for the effectiveness of the proposed method.

Critical weaknesses
  1. Limited Technical Novelty and Theoretical Foundation

The proposed method combines existing techniques, such as the prior-weighted acquisition function from Ο€BO, initial exploration via MOASHA, and random linear scalarization, with heuristics such as random prior selection, prior decay, and Ξ΅-greedy. Thus, its algorithmic novelty appears limited. Additionally, no theoretical convergence guarantees are provided. In particular, it is known that linear scalarization cannot recover all non-dominated solutions on non-convex Pareto fronts and may result in unevenly distributed solutions [Das and Dennis, 1997]. While a comparison with hypervolume scalarization is provided, these limitations of linear scalarization are not directly investigated.

Furthermore, the necessity of initial exploration via MOASHA remains unclear. While an ablation study removing the initial exploration was conducted, comparisons with simpler initialization methods such as random sampling or expert prior-based sampling are insufficient.

  1. Practicality of Prior Specification and Limited Experimental Setup

Specifying separate prior distributions over the locations of the optima of individual objective functions provides a limited way to represent expert knowledge, and it remains unclear whether experts can specify such distributions appropriately in a multidimensional search space. In fact, the prior distributions were generated by identifying the best and worst configurations among 100,000 evaluated configurations, which does not sufficiently reflect the imperfect expertise available in practice.

Additionally, the main benchmark experiments are limited to 4–7-dimensional search spaces, two objectives, and a budget equivalent to a maximum of 20 full-fidelity evaluations. The OpenCLIP case study considers three objectives under a separate compute budget. Since the influence of the prior distributions becomes negligible after a few BO iterations, the focus is primarily on performance improvements within a relatively short budget. It remains unclear whether the advantage is maintained in longer-term or higher-dimensional optimization.

Questions for authors
  1. In Algorithm 3, it appears that a scalarized objective value yt=wt⊀f(λt)y_t=w_t^\top f(\lambda_t) is added to the dataset using different random weights wtw_t at each iteration. However, in this case, the same hyperparameter configuration can have different scalarized values depending on the weights. Training a single GP on these observations could mix different scalarized objectives, potentially preventing the surrogate model from properly capturing the trade-offs between objectives. In practice, are you modeling each objective function separately, or incorporating the weights into the GP input?

  2. How much additional benefit does the initial exploration via MOASHA provide compared to initialization via random sampling or prior-based sampling within the same evaluation budget?

  3. Does PriMO's advantage hold for evaluation budgets larger than 20 equivalent full-fidelity evaluations (e.g., 50–100) or for high-dimensional search spaces? Furthermore, how does the early decay of the prior influence affect performance over longer optimization runs?

  4. How realistic is it for experts to specify prior distributions across all search dimensions? Can performance improvements still be expected when experts have knowledge of only some hyperparameters, or when inaccurate prior distributions with high uncertainty (even if not the worst-case scenario) are provided?

2 comments
AuthorsOpenReview β†—

New LLM benchmark, longer budgets, a non-scalarized variant, and clarified surrogate model and priors

Dear Reviewer 6rW9,

Thank you for assessing our problem formulation as practical and useful for real-world deep learning development, for highlighting that its robustness to incorrect priors contributes to PriMO's practicality, and for naming our experimental validation among the critical strengths of the paper, with multiple benchmarks, a large-scale case study on OpenCLIP, different prior qualities, and ablation studies. Your detailed feedback led to a substantially extended evaluation and a clearer manuscript. We address your points below, grouped by topic (changes in the revision are in red).

How is the surrogate model trained under changing random scalarization weights? (Q1)

The GP is always trained on one consistent objective: we store the full objective vectors and, in each BO iteration, scalarize all observations with the weight vector of that iteration before fitting the GP. Thank you for prompting this clarification, Algorithm 3 now states it explicitly.

Longer budgets (50 to 100 evaluations), higher dimensions, and the effect of the early prior decay (W2, Q3)

We added experiments on HW-GPT-Bench, an LLM benchmark with search spaces of up to 74 dimensions, where PriMO is the best ranked optimizer overall (Appendix M.2, Figure 12). We also extended our OpenCLIP case study from 10 to 15 PFLOPs, where PriMO retains the highest hypervolume (revised Figure 4). With budgets of up to 100 evaluations, PriMO remains the best ranked optimizer overall (results and analysis in Appendix M.6). The early decay of the prior is by design: the prior shapes the early phase, where it helps most, and afterwards the model takes over, so a misleading prior does not hold back longer runs.

Limitations of linear scalarization on non-convex Pareto fronts, and theory (W1)

To investigate this directly, we added a variant of PriMO with the non-scalarized acquisition function EHVI (Appendix J, Figure 6). It ranks clearly below our default in all prior settings. A variant with hypervolume scalarization (Golovin & Zhang, 2020), which has provable guarantees for non-convex Pareto fronts, performs on par with, but not better than, linear scalarization. Linear scalarization is therefore not what limits PriMO in practice. We designed PriMO around it for its simplicity, which is key for adoption, and the ablation confirms that more complex alternatives bring no gain (see also our Limitations, Section 6). PriMO also outperforms the baselines BO with qLogNEHVI and ParEGO (Chebyshev scalarization) overall (Figure 2).

How realistic are per-objective priors, and what if they are partial, uncertain, or imperfect? (W2, Q4)

Such priors are what DL experts often already have: a configuration they expect to perform well for an objective, e.g., from the paper that introduced an architecture or from experience in tuning similar models (Appendix D), with a Gaussian distribution around it, as in the single-objective literature (Hvarfner et al., 2022; Mallik et al., 2023).

  • Partial knowledge: hyperparameters without a prior get a uniform distribution (now stated in Section 4.1), which leaves the acquisition function unweighted along these dimensions.

  • Uncertainty: it is expressed through the width of the Gaussian. Our benchmark priors already use a medium confidence (Οƒ = 0.25, Appendix E).

  • Imperfect expertise: we also study bad priors, centered on the worst configuration found, and mixed settings (good for one objective, bad for the other). PriMO recovers from bad priors and is the best ranked optimizer under mixed priors (Figure 2 and Appendix N).

Technical novelty (W1)

Our main contribution is the problem setting itself: we are the first to study expert priors over multiple objectives, and PriMO is the first HPO algorithm that can use them (Table 1). PriMO builds on established components on purpose: the simplicity of an approach that achieves state-of-the-art performance is a strength, not a weakness. Further, the components that are new matter: the direct extension of πœ‹BO (πœ‹BO+RW) does not recover from misleading priors (Appendix M.4), the direct extension of Priorband (MO-Priorband) does not utilize good priors well (Figure 15), and our post-hoc ablation shows that all components of PriMO are important (Figure 5).

Benefit of the MOASHA initial design over simpler initializations (W1, Q2)

Figure 5 isolates the contribution of the initial design: all variants with the MOASHA initial design start off much stronger than PriMO without it, which uses random sampling at full fidelity instead (now stated in Section 5.4).

Overall, in response to your review, we extended our evaluation with an LLM benchmark, longer budgets, and a non-scalarized variant of PriMO, and clarified Algorithm 3 and how priors can be specified. We would appreciate it if you would consider these points and improvements in your rating and your soundness and presentation scores. We are happy to discuss any additional feedback.

AuthorsOpenReview β†—

Reminder before the end of the discussion period

Dear Reviewer 6rW9,

As the discussion period ends in less than 12 hours, we would like to make sure our reply above reaches you. In response to your review, we added an LLM benchmark with up to 74 dimensions, budgets of up to 100 evaluations, a non-scalarized variant of PriMO, and separate ablations of the prior decay and πœ–-greedy. We would appreciate it if you would consider these improvements in your scores.

Reviewer_tfD9OpenReview β†—
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes an algorithm called PriMO. The central idea is to formulate prior knowledge as a probability distribution over the location of the optimum of each individual objective, and then incorporate these distributions directly into the Bayesian optimization acquisition function.

Critical strengths

Incorporating prior knowledge into real world applications is highly valuable, as many practical optimization problems are not strictly black box problems. However, most existing acquisition functions are designed under the assumption that the objective function is entirely unknown. This study therefore helps bridge the gap between conventional black box BO and practical optimization problems where prior knowledge is available.

Critical weaknesses

In results such as those presented in Figure 4, the initialization strategy appears potentially unfair across different methods. Please ensure that all methods use the same initial random seeds for each trial to guarantee a fair comparison.

The incorporation of prior knowledge into the acquisition function would benefit from a more principled and theoretically motivated formulation. Beyond demonstrating empirical improvements, the authors should provide a theoretical explanation of why the proposed integration is reasonable and how it influences the optimization process.

I have some concerns about the authors' choice of converting the multi-objective problem into a single-objective problem through weighted sum scalarization. Although this is an established and computationally simple approach, its limitations in representing complex Pareto fronts are well known. It would be valuable to compare this strategy with established multi-objective acquisition functions, such as EHVI, to demonstrate its effectiveness and computational efficiency relative to more direct MOBO approaches.

Questions for authors

Could the authors explain why random search achieves performance comparable to some commonly used optimization strategies, such as TPE? Is this result primarily due to the high dimensional search spaces where identifying informative candidates becomes less effective? or because the objective functions have complex landscapes that cause methods such as TPE to become trapped in local optima? A deeper analysis would help clarify the limitations of existing HPO strategies in these settings.

Do the authors have any real-world application cases demonstrating the practical value of the proposed prior guided strategy? For example, in materials design, prior knowledge may indicate that certain material compositions or structures are likely to exhibit favorable performance in specific individual properties. It would be interesting to incorporate such knowledge into the real multi-objective optimization framework and investigate whether it can guide the generation of materials with multiple desirable properties. Such an application would provide a more convincing demonstration of the practical utility of the proposed approach.

Why do the authors restrict their approach to the current acquisition strategy rather than exploring other established multi-objective acquisition functions? There are several existing MOBO acquisition strategies that could potentially be incorporated into the proposed framework. Is this design choice related to the fact that the probability of being optimal is estimated separately for each individual objective?

I believe that the probability of being optimal for individual objectives could have a meaningful connection to the structure of the Pareto front. In particular, identifying regions with a high probability of being optimal for individual objectives may help characterize the boundary or backbone of the Pareto front, while a multi-objective acquisition function can further guide the selection of candidates to improve Pareto front coverage. This distinction could provide a useful theoretical perspective for understanding why the proposed prior-guided strategy works. I encourage the authors to consider the concept of Probability of Matching (POM) [1], which provides a connection between optimality related probabilities and Pareto front coverage. This perspective may help establish a more principled theoretical foundation for the proposed method, rather than relying primarily on heuristic prior incorporation and estimation strategies.

[1] https://icml.cc/virtual/2026/poster/64043

3 comments
AuthorsOpenReview β†—

Same initial design (conclusions unchanged) and a new comparison to EHVI

Dear Reviewer tfD9,

Thank you for assessing the incorporation of prior knowledge as highly valuable for real-world applications and for recognizing that our work helps bridge the gap between black-box BO and practical problems where prior knowledge is available. Your constructive feedback helped us substantially improve the fairness and breadth of our evaluation, and we addressed your points in the revised manuscript (changes in red).

Same initial design and seeds across methods for a fair comparison (e.g., Figure 4)

We re-ran the main evaluation and the OpenCLIP case study with the same initial design across methods, and our conclusions are unchanged. In the main evaluation, all baselines with a random initial design now share their initial configurations, and PriMO remains the best ranked optimizer overall, significantly better than all other optimizers (revised Figure 2, and Figure 24 in Appendix N). In the revised Figure 4, BO+RW, MO-TPE, and random search now also share theirs, as PriMO and MOASHA already did (Appendix H), and PriMO still achieves the highest hypervolume. The small differences that remain at the start stem from non-determinism in GPU training.

Comparison to direct multi-objective acquisition functions such as EHVI, and why PriMO uses scalarization

Our ablation in Appendix J compares PriMO under different acquisition functions, and we extended it with a variant that replaces the scalarized EI with EHVI (PriMO+EHVI, Figure 6). It shows two things:

  • PriMO is not restricted to its current acquisition strategy. The prior weighting in Equation 4 is multiplicative and applies to EHVI unchanged, so our choice is not forced by the priors being defined per objective.

  • The direct approach does not help here. The EHVI variant ranks clearly below our default in all prior settings (overall rank of about 4.8 vs. 2.2 after 20 evaluations). A variant with hypervolume scalarization (Golovin & Zhang, 2020) performs on par with, but not better than, our default. Likewise, PriMO outperforms the baseline BO with qLogNEHVI in our main evaluation (Figure 2).

Random linear scalarization is also the computationally efficient choice: it needs only a single surrogate model and a single-objective acquisition function, so PriMO keeps the runtime of classic BO and scales readily with the number of objectives (Section 4.4). Our runtime analysis confirms that PriMO's sampling time matches that of standard BO and is negligible compared to the cost of evaluating a configuration (Table 2). We designed PriMO around random linear scalarization for exactly these reasons, simplicity and efficiency, which are key for adoption in practice, and our ablation confirms that the more complex alternatives bring no gain.

Real-world applications of the prior-guided strategy

Yes, our large-scale case study (Section 5.3): we optimize the pretraining of the vision-language model OpenCLIP over three objectives with real, GPU-based training runs, and PriMO achieves the highest hypervolume (revised Figure 4). In the revision, we further added the hardware-aware LLM benchmark HW-GPT-Bench, where PriMO is the best ranked optimizer overall (Appendix M.2). As the algorithm itself is not specific to DL, applications such as materials design are an interesting direction for future work.

Performance of random search relative to common optimizers such as TPE

At the end of the budget, random search does not perform comparably to MO-TPE in our results: at 20 evaluations it ranks clearly behind MO-TPE and far behind PriMO, and also behind BO+RW and ParEGO (revised Figure 2, overall, and Figure 24). It is only close to the model-based optimizers during the first evaluations, as expected: they start with a random initial design (Appendix F) and need some observations before their models become informative.

Why this integration of the priors is reasonable, and how it influences the optimization

We extended Section 4.1 with this explanation. In short, the optimum of each individual objective is itself Pareto optimal (under unique optima) and an extreme point of the Pareto front. Priors over these optima therefore steer the early search toward the extremes of the front, while the random scalarization weights direct it to the trade-offs between them. As the weight of the prior decays, and with probability πœ– at any step, the search relies on the model alone, which allows PriMO to recover from misleading priors (Figure 2, right).

Overall, in response to your review, we aligned the initial design across all methods, extended our comparison of acquisition functions (Appendix J) with an EHVI variant, added an LLM benchmark next to our real-world case study on OpenCLIP (Section 5.3), and extended Section 4.1 with an explanation of how the priors act. We would appreciate it if you would consider these points and improvements in your rating and soundness score. We are happy to discuss any additional feedback.

AuthorsOpenReview β†—

Reminder before the end of the discussion period

Dear Reviewer tfD9,

As the discussion period ends in less than 12 hours, we would like to make sure our reply above reaches you. In response to your review, we aligned the initial design across all methods with unchanged conclusions, added an EHVI variant of PriMO and an LLM benchmark, and explained how the priors act (Section 4.1). We would appreciate it if you would consider these improvements in your scores.

Originally I had a positive impression of the proposed concept and the experimental results. However, following the discussion, I believe that the authors may need more time to develop a theoretically grounded and principled framework for integrating the prior distribution into existing acquisition functions, which is unlikely to be fully addressed during the rebuttal period.

Meanwhile, the EHVI results remain concerning, as EHVI would generally be expected to outperform both EI based approaches and random sampling in multi-objective optimization settings. For now, I will maintain my current score rather than lowering it and wait to see how the other reviewers assess the results. However, if the paper is accepted, I strongly encourage the authors to carefully verify their EHVI implementation or provide a clear and convincing explanation for its unexpectedly poor performance.

Reviewer_5n7FOpenReview β†—
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PriMO, the first Bayesian optimization algorithm to incorporate expert priors for multiple objectives into hyperparameter optimization. Its key designs include factorized per-objective priors, a prior-weighting exponent that decays exponentially with the squared number of BO samples, an Ξ΅-greedy prior-augmentation mechanism to aid recovery from misleading priors, and a multi-fidelity initialization strategy based on MOASHA. The evaluation covers eight deep learning surrogate benchmarks with 25 seeds and a three-objective OpenCLIP case study with 10 seeds under a budget of 15 PFLOPs.

Critical strengths

The problem setting is practically relevant: DL practitioners often optimize error and cost jointly, and expert knowledge can help guide the search.

The experiments are solid: eight surrogate benchmarks, 25 seeds, equivalent-evaluation budgets, LMEM-based significance analysis, and an OpenCLIP case study.

The ablations suggest gains beyond warm-starting: PriMO outperforms Ο€BO+RW with the same initial design overall, particularly under misleading priors.

Critical weaknesses

Although Appendix L compares the original and NePS prior-decay schedules, it remains unclear whether Ο€BO+RW’s prior-strength parameter was tuned. A broader sensitivity analysis would strengthen the claim that PriMO’s robustness advantage persists under well-tuned baseline settings.

The OpenCLIP study includes absolute metrics and a manual reference, but the latter is a single scaled-down configuration rather than a systematically tuned baseline. The practical advantage over strong manual tuning therefore remains unclear.

Questions for authors

Could you separately ablate the fast prior decay and Ξ΅-greedy augmentation? The current Ξ΅-BO ablation changes both, making their individual contributions to robustness unclear.

Could you evaluate PriMO with priors concentrated in regions that are poor across all objectives? In HW-GPT-Bench, a bad prior for one objective can benefit another, so this would provide a clearer test of recovery from misleading priors.

2 comments
AuthorsOpenReview β†—

Original πœ‹BO schedule (results unchanged), anti-correlated objectives, and an OpenCLIP reference

Dear Reviewer 5n7F,

Thank you for recognizing PriMO as the first BO algorithm to incorporate multi-objective expert priors into HPO, for assessing this problem setting as practically relevant, and for your positive assessment of our evaluation: solid experiments with statistical tests, and a convincing ablation that shows PriMO's gains go well beyond warm-starting. Your feedback helped us strengthen our evaluation even further, and we addressed all of your points in the revised manuscript, three of them with new experiments (changes in red).

πœ‹BO+RW baseline: prior-decay schedule and tuning

We re-ran πœ‹BO+RW with the original schedule of the πœ‹BO paper and now use this version throughout the manuscript. Both versions are basically indistinguishable in performance in all prior settings (new Appendix L, Figure 10). In particular, it still does not recover from bad priors and remains the worst-ranked optimizer in that setting, while PriMO recovers (Figures 15 and 24). The robustness gap is therefore not due to an implementation difference.

We tuned neither schedule: the original one uses Ξ² = 10 as set by the authors of πœ‹BO, and the previous one is the schedule implemented in the NePS library.

Stress test with strongly anti-correlated objectives

Following your recommendation of a synthetic anti-correlated stress test, we constructed one from the multi-fidelity Hartmann benchmark with the objective pair (f, -f), i.e., perfectly anti-correlated objectives (Appendix M.1). To isolate the effect of the factorized priors, we compare PriMO to PriMO without priors: the priors clearly and consistently help on all four variants of the benchmark (Figure 11).

In addition, the objectives of our new LLM benchmark HW-GPT-Bench are strongly negatively correlated (Kendall's Ο„ between FLOPS and perplexity of -0.85 to -0.92, Table 20), and PriMO is the best ranked optimizer overall there (Appendix M.2, Figure 12).

Is the single-objective πœ–-BO exactly the same code path as in the multi-objective case?

Yes, it is exactly the same code path, only without the scalarization. We now state this explicitly in Section 4.3.

Reference for the OpenCLIP case study

We agree that a reference makes the practical significance easier to judge and added a manually designed reference configuration, based on the literature, to the revised Figure 4 (dashed line, details in Appendix H). PriMO reaches a clearly higher hypervolume than this reference (about 0.57 vs. 0.37).

Overall, in response to your review, we re-ran the πœ‹BO+RW baseline with the original πœ‹BO schedule, added a stress test with perfectly anti-correlated objectives and an LLM benchmark with strongly negatively correlated objectives, and added a manually designed reference to the OpenCLIP case study. We would appreciate it if you would consider these improvements in your rating and soundness score. We are happy to discuss any additional feedback.

AuthorsOpenReview β†—

PriMO outperforms πœ‹BO+RW for every Ξ², separate ablations of prior decay and πœ–-greedy, and a second manual baseline

Dear Reviewer 5n7F,

Thank you for updating your review. Your new points helped us further strengthen our evaluation with three new experiments in the revised manuscript.

Priors that are poor for all objectives

This is the all priors bad setting of our main evaluation, where each objective's prior is centered on its worst configuration. Our main evaluation and ablations always cover all good, mixed, and all bad priors, as well as their average (e.g., Figures 2, 3, and 5). Under all bad priors, PriMO recovers, while πœ‹BO+RW does not (Figures 15 and 24). HW-GPT-Bench complements this as a stress test with strongly conflicting objectives (Appendix M.2).

Broader sensitivity analysis of πœ‹BO+RW's prior strength

We added a sensitivity study of Ξ² ∈ {1, 5, 10, 50, 100} for πœ‹BO+RW (new Appendix O, Figures 25 to 27). PriMO outperforms πœ‹BO+RW for every Ξ² overall, as well as under mixed and all bad priors (Figure 25). PriMO's robustness advantage thus does not depend on πœ‹BO+RW's prior strength.

Separate ablation of the fast prior decay and the πœ–-greedy augmentation

We added both variants to our ablation, with PriMO's initial design (revised Figure 5, Section 5.4): PriMO's prior decay without πœ–-greedy, and πœ‹BO's prior decay with πœ–-greedy. The fast decay is key to recovering from misleading priors: with it, PriMO recovers even without πœ–-greedy, while with πœ‹BO's slower decay, πœ–-greedy alone does not suffice. Together, both make PriMO the most robust variant with priors under all bad priors.

Manual baselines in the OpenCLIP case study

We added a second manual baseline, so that we now compare against manual designs at two model scales of our search space (widths 32 and 192, Figure 4). Both use the OpenCLIP defaults of Cherti et al. (2023), with learning rates scaled to our widths via Β΅P (Yang et al., 2022; details in Appendix H). PriMO's performance remains convincing.

Overall, in this second round of improvements in response to your updated review, we added a Ξ² sensitivity study for πœ‹BO+RW, separate ablations of the fast prior decay and πœ–-greedy, and a second manual baseline with Β΅P-scaled learning rates. We would appreciate it if you would consider these improvements in your soundness score and rating.

Comments on the paper

1 comment
AuthorsOpenReview β†—

Summary: a novel and relevant problem setting and a strong, substantially extended evaluation

Dear Chairs, dear Reviewers,

We thank all reviewers for their constructive feedback, which helped us substantially improve the manuscript and its evaluation. Below, we summarize their assessment and our most important improvements (marked in red in the revised manuscript). Details are in the individual replies.

Contribution and strengths noted by the reviewers

All four reviewers agree on the central strength of our paper: it addresses a new and practically relevant problem, multi-objective HPO with expert priors. In detail, they highlight:

  • New and relevant problem: PriMO is the first HPO algorithm that can use expert priors over multiple objectives. The reviewers note that the paper "formalises the setting" (gWkd) and assess it as "practically relevant" (5n7F), "useful in real-world deep learning development" (6rW9), and "highly valuable" for real-world applications (tfD9).

  • Strong evaluation: 5n7F assesses that "The experiments are solid" and "The ablation is convincing", and 6rW9 likewise lists our experimental validation as a critical strength, with multiple benchmarks, a large-scale case study on OpenCLIP, different prior qualities, and ablation studies. Following the suggestions of all four reviewers, we extended it substantially further, e.g., with an LLM benchmark (see below).

  • Robust and effective approach: PriMO, with its new acquisition function (gWkd), "helps bridge the gap between conventional black box BO and practical optimization problems where prior knowledge is available" (tfD9). Our ablation convincingly shows that its gains go well beyond warm-starting (5n7F), and its robustness to incorrect priors "contributes to its practicality" (6rW9). Our post-hoc studies further show that PriMO outperforms our strongest baseline overall for every tested value of πœ– (Appendix K) and with three of the five acquisition strategies (Appendix J).

Most important improvements in response to the reviews

  1. Substantially extended evaluation (6rW9, 5n7F). We added HW-GPT-Bench, an LLM benchmark with up to 74 dimensions and strongly negatively correlated objectives, where PriMO is the best ranked optimizer overall (Appendix M.2). In a new stress test with perfectly anti-correlated objectives, the priors still clearly help (Appendix M.1). In the OpenCLIP case study, now with a longer budget and two manually designed references, PriMO achieves the highest hypervolume (Figure 4). With budgets of up to 100 evaluations, PriMO remains the best ranked optimizer overall, and we provide an analysis of these results in Appendix M.6.

  2. Fairness of the comparison confirmed (5n7F, tfD9, gWkd). We aligned the initial design across methods in the main evaluation and the case study, and PriMO remains the best ranked optimizer overall, significantly better than all other optimizers (Appendix N). πœ‹BO+RW, the baseline we constructed by directly adapting πœ‹BO to multiple objectives (it does not exist in the literature), now uses the original πœ‹BO schedule, with basically indistinguishable results (Appendix L): it still does not recover from bad priors, and PriMO outperforms it overall for every value of its prior strength Ξ² (Appendix O). Our ablation and sensitivity studies were performed post-hoc and not to tune PriMO's hyperparameters.

  3. Novelty and simplicity (6rW9, gWkd). PriMO opens a new problem setting, and its simplicity is a strength that our results now back even more clearly: the direct adaptations of single-objective methods that we constructed as baselines (πœ‹BO+RW, MO-Priorband) fall short in our evaluation, a new variant of PriMO with the more complex acquisition function EHVI ranks clearly below our default in all prior settings (Appendix J), and our post-hoc ablation, now also separating the fast prior decay and πœ–-greedy, shows that each component of PriMO matters (Section 5.4).

All conclusions of the paper hold under the new results, and the revision addresses the main points raised. We are happy to discuss any additional feedback with the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.