acceptodds
Under review as a conference paper at ICLR 2027

Likelihood-Free Multi-Objective Bayesian Optimization under Heteroscedastic Noise

Abstract

Multi-objective Bayesian optimization (MOBO) fits a probabilistic surrogate, typically one Gaussian process (GP) per objective, so every acquisition function inherits the noise model of that surrogate. When the noise is heteroscedastic, that model cannot be identified from a single evaluation per design, and both the search and its evaluation degrade. We propose likelihood-free acquisition functions that do not require a noise model. We define a soft non-dominance indicator by nonparametric subsampling of the noisy observed objective vectors, derive it in closed form, and use it to weight a classification objective, which yields the likelihood-free counterparts of two state-of-the-art noisy acquisitions. These acquisition functions depend on the data only through the dominance relations between observations, and we prove that they preserve dominance and that their maximizers eventually lie in the true Pareto set. We further show that the standard evaluation protocols do not penalize a method for the set it would recommend, and we replace it by an exact decomposition of the decision maker's loss into sampling, set-selection and mis-selection terms, of which we observe mis-selection accounts for more than 40% at high noise. Across a comprehensive experimental campaign, GP-based baselines remain competitive when the noise is low and well specified, but as the noise level and structural complexity grow our acquisitions outperform GP baselines by margins that widen with the input dimension.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
multiobjective optimization, heteroscedastic noise, expensive black-box

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PJixOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Likelihood-free Bayesian Optimization (LFBO) to the framework of Multi-Objective Bayesian Optimization (MOBO). In conventional single-objective LFBO, the acquisition function is constructed using a classifier trained to solve a two-class discrimination problem (selected positive examples versus other samples), with an appropriate utility function serving as the weight (or selection criteria for positive examples). As an extension to the multi-objective setting, this study uses a soft non-dominance indicator as the utility in LFBO. This indicator measures the stability of non-dominance in the design while acknowledging the presence of noise. Furthermore, the authors propose an evaluation methodology that explicitly exposes the decision-maker's loss. Comprehensive experiments demonstrate that in low-noise scenarios, GP-based methods perform superiorly, while it is also shown that the proposed acquisition function significantly outperforms GP-based methods as the noise level and input dimensionality increases.

Critical strengths
  1. Simple and Computationally Tractable Formulation for Avoiding Noise Model Estimation: The concept of constructing the LFBO utility by leveraging inter-observation dominance relationships and subsampling, rather than directly estimating heteroscedastic or non-Gaussian noise, is interesting. In particular, the closed-form derivation in terms of kik_i provides a computational advantage. While LFBO itself is an existing methodology, the utility design for noisy multi-objective optimization is highly worthy of further investigation.

  2. Evaluation Design Separating Oracle Evaluation from Actual Decision-Making Quality: Traditional methods like Hypervolume (HV) and IGD+ often overlook the crucial question of whether the selected designs are truly optimal. The approach of decomposing this problem into three components—sampling, set selection, and misselection—and quantifying each is significant. Although the decomposition itself is an algebraic identity and therefore has limited theoretical novelty, the experimental finding that misselection accounts for over 40% of the total loss is valuable. This contribution can be valued independently of the proposed acquisition function.

Critical weaknesses

Unclear Mechanism of Noise Mitigation via Subsampling: It is unclear whether subsampling truly mitigates misclassification induced by noise. ζi\zeta_i measures the stability of non-dominance after removing a subset of observed designs, which is not the noise uncertainty itself. Specifically, if a remarkably good objective vector is observed purely by noise, that design will lead to ki=0k_i=0, causing ζi=1\zeta_i=1. While this might mitigate false negatives, the mechanism for suppressing false positives resulting from extreme positive noise is not clearly defined. The ability to operate without a noise model and the robustness to noise are distinct issues. Currently, the mechanism of the proposed method appears to rely more on empirical observations than on theoretical grounding.

If my comment is due to my misunderstanding, please correct it in the rebuttal.

Questions for authors
  1. Is LF-NEHVI truly a Likelihood-Free counterpart to NEHVI?: qNEHVI marginalizes over the posterior distribution of the latent objective function, whereas LF-NEHVI averages over the subsampled observed front. The uncertainties inherent in these two approaches are fundamentally different. Furthermore, since LF-NEHVI depends on the numerical value of the Hypervolume (HV), the claim that it is "invariant to the objective function's monotonic transformation and depends only on the dominance relationship" holds true for LF-NPHVI, but generally does not hold for LF-NEHVI.

  2. Is the novelty and necessity of the soft non-dominance indicator sufficiently demonstrated?: Ultimately, ζi\zeta_i is merely a monotonic function of kik_i. Therefore, it is necessary to compare it with LFBO methods that utilize weighting based on simple dominance count or Pareto rank. We need to determine whether the construction of subsampling itself contributes to the performance improvement.

  3. Can ζi\zeta_i be interpreted as a probability of belonging to the Pareto set?: ζi\zeta_i represents the survival probability under subsampling, not the true probability of belonging to the Pareto set. Therefore, the explanation that selecting based on the threshold of 1/2 minimizes misclassification loss cannot be straightforwardly justified. The final decision-making quality when using different thresholds should also be confirmed.

5 comments

We thank the reviewer for the careful review. The questions on subsampling led to new experiments and a simpler implementation (Q2). “HV gap” is the terminal log⁡10\log_{10} hypervolume gap of the returned set; W/T/L counts per-condition wins/ties/losses at the 5% level over 10 seeds.

W. Subsampling and false positives. Agreed: ζ\zeta softens false negatives, not false positives. A design dominated by one lucky observation keeps ζ≈1/2\zeta \approx 1/2 (for γ=1/2\gamma = 1/2), and ζ\zeta decays geometrically in the number kk of observations dominating it (Lemma 1), so a design is not immediately discarded based on a single potentially misleading observation. Conversely, a design with a favorable draw gets ζ=1\zeta = 1.

The classifier may further mitigate such errors by learning from observations across the search space. We have clarified these limitations in Sec. 3.1 and, in more detail, in App. A, distinguishing the theoretical properties of ζ\zeta from the empirical robustness of the resulting acquisition functions. In particular, our expanded ablation shows that soft labels outperform hard labels for LF-NPHVI (see our response to Reviewer WMfu, Q3).

Q1. LF-NEHVI and NEHVI. Agreed on both points. Invariance to monotone transformations holds only for ζ\zeta and LF-NPHVI; we have restricted the claim in the abstract, conclusion and related work. LF-NEHVI shares qNEHVI’s structure (hypervolume improvement averaged over an uncertain front), but its uncertainty comes from resampling the observed front, not from a GP posterior. Sec. 3.3 now says so and no longer uses “counterpart”.

Q2. Is subsampling necessary? To compute ζ\zeta, no. As the reviewer notes, ζ\zeta is a function of k: Lemma 1 gives ζ=(n−ks)/(ns)≈(1−γ)k\zeta = \binom{n-k}{s}\big/\binom{n}{s} \approx (1-\gamma)^k (Eq. 7), the B→∞B \to \infty limit of subsampling. Our implementation of LF-NPHVI now uses Eq. (7) (a remark in Sec. 3, the full algorithm in the appendix). Exact labels give the same results as subsampled ones (B = 128), paired by seed:

exact vs subsampled ζ\zetaW/T/L, HV gapW/T/L, oracle estimatormedian HV gap change
γ=1/2\gamma = 1/2, full design (180 conditions)5/174/14/174/20% (p=0.44p = 0.44)
γ=0.1\gamma = 0.1, App. F ablation (32)0/32/01/31/00% (p=0.80p = 0.80)
γ=0.25\gamma = 0.25 (32)0/32/01/31/00% (p=0.52p = 0.52)
γ=0.75\gamma = 0.75 (32)0/31/10/31/1+1% (p=0.10p = 0.10)

Exact labels are 100–160$\times$ cheaper to compute, and whole runs about 4% faster. We keep subsampling as the definition in Sec. 3 because it is the general one: it also defines the hypervolume-improvement labels of LF-NEHVI, which have no closed form we know of.

On counts, we compared ζ\zeta with 1/(1+k)1/(1+k) over the paper’s full design (16 problems ×\times 4 noise models ×\times 3 noise levels = 192 conditions; 180 on HV gap, which excludes DTLZ7, as it has no known maximal hypervolume), changing only the label of LF-NPHVI. The Pareto-rank label 1/r1/r matched 1/(1+k)1/(1+k) in the 32-condition ablation, so we did not rerun it.

vs γ=1/2\gamma = 1/2 (paper default), 180 conditionsγ=0.1\gamma = 0.1γ=0.25\gamma = 0.251/(1+k)1/(1+k)
W/T/L, HV gap27/140/1316/162/217/153/10
W/T/L, HV gap, oracle estimator29/130/2111/164/518/147/15
better in, median HV gap change112/180, −2-2%119/180, −2-2%115/180, −1-1%
mean rank of the four labels (γ=1/2\gamma = 1/2: 2.95)2.172.382.51

Labels that decay more slowly than γ=1/2\gamma = 1/2 do 1–2% better once η≥0.2\eta \ge 0.2. The differences among them are small, and 1/(1+k)1/(1+k) is itself ζ\zeta averaged over a uniformly random subsample size. We keep γ=1/2\gamma = 1/2 and have added both comparisons to App. F.

Q3. Is ζ\zeta a probability of Pareto membership? No. ζ\zeta is a survival frequency under subsampling, not a calibrated probability, and we have removed the misclassification claim. At γ=1/2\gamma = 1/2 the 1/21/2 threshold of the evaluation has a direct reading instead: ζ≥1/2\zeta \ge 1/2 exactly when at most one observation dominates the design (Eq. 13).

Could the reviewer clarify which threshold they would like varied (“the final decision-making quality when using different thresholds should also be confirmed”)? Our method uses no threshold during optimization: the acquisition maximizes the odds of a classifier trained on the soft labels. A threshold appears only in our evaluation protocol (Sec. 4), where the set returned to the decision maker is the designs with ζ≥1/2\zeta \ge 1/2. If this is the threshold the reviewer means, we can recompute the returned sets and their decision quality (nIGD+ and HV) at other values from our stored runs. We also ablate γ\gamma in Appendix F (Tables 7 and 10).

We hope these answers address the reviewer’s concerns.

Thank you for the clarification. My question 3 refers specifically to the threshold of 0.5 used in Eq. (12) to construct the recommended set, not to the acquisition function or the subsampling parameter γ\gamma.

My main concern is the interpretation of ζi\zeta_i as a probability of Pareto-set membership, to which the author answered "no". I understand ζi\zeta_i represents the probability that a design remains nondominated under random subsampling of the observed data, rather than the probability that it belongs to the true Pareto set. Therefore, I though the argument that a threshold of 0.5 minimizes the expected number of misclassifications requires additional justification.

A sensitivity analysis using different thresholds (e.g., 0.25, 0.5, and 0.75) would be useful, particularly for nIGD+, to assess how the recommended-set selection affects decision quality. This is separate from the ablation of γ\gamma in Appendix F. I'm guessing this is not excessive requirement, but if there are reasons that the additional experiment isn't reasonable, some discussion to justify the particular value "0.5" would suffice.

Alternatively, if the results are insensitive to the threshold, it would be sufficient to report this and clarify that 0.5 is a practical convention rather than a theoretically optimal threshold. This is just for clarification, not the critial flaw in the paper.

Thank you for the clarification and suggestion. We re-scored all stored runs of the main campaign (7 methods, 16 problems, 4 noise structures, 3 noise levels, 10 seeds, no new optimization) on the three main performance metrics, with different thresholds in Eq. (12). At γ=1/2\gamma = 1/2, ζi\zeta_i depends only on the number kik_i of observations dominating design ii, so τ=0.75\tau = 0.75, 0.50.5 and ≈0.25\approx 0.25 return the designs with ki≤0k_i \le 0, ki≤1k_i \le 1 and ki≤2k_i \le 2, respectively.

ThresholdnIGD+ LF-NPHVInIGD+ LF-NEHVInIGD+ best baselineHV gap LF-NPHVIHV gap LF-NEHVIHV gap best baselineIGD+ LF-NPHVIIGD+ LF-NEHVIIGD+ best baseline
τ=0.75\tau = 0.75 (k≤0k \le 0)3.283.773.352.523.033.243.113.933.27
τ=0.5\tau = 0.5 (k≤1k \le 1, paper)3.263.743.312.483.013.273.103.933.29
τ≈0.25\tau \approx 0.25 (k≤2k \le 2)3.253.803.352.453.013.263.133.913.27

Mean rank among the seven methods (lower is better); the best baseline is qLogNParEGO for nIGD+ and qLogNEHVI for HV gap and IGD+.

The conclusions do not depend on the threshold: LF-NPHVI ranks first overall on all three indicators at every value, GP baselines still lead only at η=0.05\eta = 0.05, and per-condition rankings agree closely with those at τ=0.5\tau = 0.5 (median Kendall rank correlation between 0.90 and 1.00). A lower threshold mainly trades coverage for mis-selection: larger sets lower the HV gap and IGD+ but raise nIGD+ at η≥0.2\eta \ge 0.2.

We thus use τ=1/2\tau = 1/2 as a practical convention rather than an optimal threshold: at γ=1/2\gamma = 1/2 it returns exactly the designs dominated by at most one observation, and the overall rankings barely change at cutoffs of zero or two dominating observations. We have added this analysis to Appendix G.8, and a sentence in the main text for clarification, in green color.

Reviewer_mBeiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new multi-objective Bayesian optimization method based on classification for expensive problems under heteroscedastic noise. More specifically, it defines a so-called soft non-dominance indicator by subsampling of the observed objective vectors, and use it to weight a classification objective. The proposed method has been compared with several peers and the results show its competitiveness.

Critical strengths
  1. This paper works on a class of relevant optimization problems, multi-objective expensive problem under heteroscedastic noise setting.

  2. The experimental results show the competitveness of the proposed method compared to its peers.

Critical weaknesses
  1. Considering MOBO under the noise setting using classification with non-dominance is not new. In other areas (e.g., evolutionary computation), there exist a few works, such as https://ieeexplore.ieee.org/document/8281523 and https://www.sciencedirect.com/science/article/abs/pii/S0925231224004004.

  2. The Abstract could be improved. You mainly focus on the description of the proposed method, however, it would be more useful for the audience to see why you want to do so.

  3. It is unclear to me how useful the proposed soft non-dominance indicator is. In multi-objective optimization, an important issue is to distinguish between nondominated solutions, hence indicator-based fitness/acquisition function is proposed which typically convert a set of vectors (or a vector) into a scalar measure. However, it seems to me that the proposed indicator is unable to distinguish nondominated solutions (as nondominated solutions won't be dominated by any subset of the set where they are defined based on).

  4. There are some proofs, e.g., the closed form of the proposed indicator (in the limit sense) and the Pareto comliance. However, I am not sure they are importance and highly relevent. Instead, it seems to me they are a bit trivial, easy to achieve when only considering Pareto dominance.

  5. IGD+ may not be very reliable in real-world applications where problems' Pareto front is not available, see https://dl.acm.org/doi/abs/10.1145/3300148

  6. Finally, you explain why the distance-based indicators like IGD+ struggle under noise. It seems to me that non-distance indicators may face the same issue, for example the hypervolume indicator.

Questions for authors

First of all, could you please respond to the weaknesses listed above:

Secondly, could you please answer the following questions:

  1. I don't feel I get the point of using random subsets of the samples. In expectation, it seems that the indicator is just to count how many points dominating a point. Why not calculate it directly in the samples queried? Similar methods are commonly used in multi-objective optimization.

  2. I noticed that some parameter settings of the test problems considred are not very standard like ZDT, WFG and DTLZ. I am wondering any reason for such settings.

  3. What are the reference point setting in the calculation of hypervolume?

6 comments

We thank the reviewer for the careful reading and for pointing us to Pan et al. (2019) and Geng et al. (2024). We were aware of these papers and of the broader EMO literature, on which much of our performance evaluation builds: IGD+ of Ishibuchi et al. (2015) and the noisy IGD+ of Branke (2025). In fact, this literature partly motivated our work, as it typically assumes deterministic evaluations of the objectives, or numerous replications when noise is present. We study the case where dominance itself is observed through (unknown) noise and objectives are expensive-to-evaluate: each design is evaluated once, the noise may be heteroscedastic, and neither a noise model nor replications are available. With this crucial difference in mind, we answer each point below.

W1. Classification based on non-dominance. We agree that learning dominance with a classifier is established, and we now cite this line of work explicitly. However, these methods assume deterministic objectives, for which dominance labels are exact. Under noise, a single favorable observation can make a dominated design appear non-dominated, and labels may change as more data become available. Re-evaluation, the standard remedy, is not available in our setting. Even if it were, evaluations are assumed to be expensive, making data-efficiency an additional design consideration. Our contribution is to introduce a label tailored to this setting: quantifying the stability of a design’s non-dominance (Sec. 3). We extend our contribution to the evaluation of the quality of the set returned by a method under noise (Sec. 4), an issue that does not arise in the deterministic setting, and is largely underexplored in the literature.

W2. Abstract. Agreed. The revised abstract opens with the problem: dominance is observed through noise with one evaluation per design, heteroscedastic noise cannot be identified from such data, and standard protocols never charge a method for the designs it returns. The method then takes two sentences.

W3. The indicator does not distinguish non-dominated solutions. Correct: every observed non-dominated design gets ζ=1\zeta = 1. This is deliberate, and we say so in Sec. 3.1 (details in App. A). Under noise, the observed front contains designs that are there only because of a favorable draw. Ranking them against each other by those values amounts to trusting the noise. Our evaluation protocol quantifies that cost explicitly. ζ\zeta therefore spends its resolution where the evidence is more reliable: on how strongly a design is dominated. It decays with the number kk of observations that dominate a design (Lemma 1). A design dominated once keeps about half its weight (with γ=1/2\gamma = 1/2), since that dominance may itself be due to noise, while a design dominated by many observations is discarded.

Non-dominated designs are still distinguished in two ways:

  • LF-NEHVI weights each design by its hypervolume contribution, including the exclusive contribution of front points (Sec. 3.3), so front points receive different weights.

  • The labels are not the acquisition. They train a classifier over the input space, and the acquisition ranks new candidates by the classifier’s odds, an estimate of E[ζ∣x]\mathbb{E}[\zeta \mid x]. Regions where neighboring designs are consistently non-dominated score higher than an isolated design that may be a noise artifact.

W4. Relevance of the theoretical results. In our view, the value of the theory lies in what they establish for a setting that has been barely analyzed before:

  • Lemma 1 makes the indicator exact and cheap: no sampling is needed.

  • Proposition 1 (Proposition 2 in the submission) is not automatic under noise. A dominated design with more variable observations can be more likely to appear non-dominated than the design that dominates it, so dominance between true objectives need not carry over to expected utilities. Proposition 1 identifies when it does, and then the maximizers of the acquisition lie in the Pareto set. Prompted by Reviewer WMfu, we have extended this to a heteroscedastic family that includes our multiplicative noise model.

  • Proposition 4 (Proposition 3 in the initial submission) gives the exact loss decomposition that shows how much the conventional evaluation misses.

To our knowledge, these are the first such statements for noisy MOBO.

W5. IGD+ requires the Pareto front. Agreed; we now cite Li & Yao (2019). IGD+ and nIGD+ serve as benchmark diagnostics, computed against analytic or dense reference fronts (App. D). We include nIGD+ (Branke, 2025) because, to our knowledge, it is the only noisy indicator that accounts for the decision maker’s choice. Its behavior under heteroscedastic noise had not been studied, and Proposition 4 places it in our loss decomposition.

W6. Hypervolume faces the same issue. The reviewer is right, and we thank them for the correction. Our argument in Sec. 5 on distance-based indicators does not depend on distance. It rests on the scored set being estimated. Hypervolume is affected in exactly the same way. It is only Pareto-compliant on deterministic sets. Our own results show this: 36% of the HV-gap curves end above their best value (Sec. 5, Limitations). nIGD+ adds a second effect on top, since the decision maker picks by estimated rather than true objectives, which is why its curves are less monotone still (72%). We have rewritten the paragraph to cover all indicators of the returned set, HV included, with nIGD+ as the case where both effects combine.

Q1. Why random subsets rather than counting dominators? We appreciate this point. By Lemma 1, ζ\zeta is an exact function of the dominance count kk, ζ=(n−ks)/(ns)≈(1−γ)k\zeta = \binom{n-k}{s}\big/\binom{n}{s} \approx (1-\gamma)^k (Eq. 7), so it can be computed without subsampling, and our implementation of LF-NPHVI now does so (remark in Sec. 3, algorithm in the appendix). We tested both suggestions on the paper’s full design (180 conditions, 10 seeds), changing only the label of LF-NPHVI (W/T/L: per-condition wins/ties/losses on terminal HV gap at the 5% level):

  • Closed-form ζ\zeta instead of subsampling: no difference (W/T/L 5/174/1, median change in HV gap 0%). Labels are 100–160$\times$ cheaper to compute, and whole runs about 4% faster.

  • The count label 1/(1+k)1/(1+k): slightly better than our default γ=1/2\gamma = 1/2 (W/T/L 17/153/10, median −1-1%), and within about 1% of ζ\zeta at γ=0.1\gamma = 0.1.

Both belong to one family: 1/(1+k)1/(1+k) is exactly ζ\zeta averaged over a uniformly random subsample size, and labels that decay more slowly than γ=1/2\gamma = 1/2 do 1–2% better at higher noise. The differences are small, so we keep γ=1/2\gamma = 1/2. Subsampling remains the definition in Sec. 3 because it gives the label its interpretation (non-dominance under resampling) and the decay rate γ\gamma, and because it also defines the hypervolume-improvement labels of LF-NEHVI, which have no closed form we know of. Both comparisons are now in App. F; the full tables are in our response to Reviewer PJix (Q2), and a comparison with hard labels is in our response to Reviewer WMfu (Q3).

Q2. Problem settings. All problems use the standard BoTorch implementations (WFG6 from pymoo). We changed only the number of decision variables d, to study how performance depends on input dimension.

problemmd usedcommon default
DTLZ12, 315, 20m + 4
DTLZ22, 315, 20m + 9
DTLZ251414
DTLZ751424
WFG6 (k=6k = 6, l=10l = 10)416k=2(m−1)k = 2(m-1), l=20l = 20
ZDT1, ZDT222030
VehicleSafety, Penicillin, CarSideImpact3, 3, 45, 7, 7native

Q3. Reference points. These are in BoTorch’s minimization form. They are the BoTorch defaults, except for VehicleSafety (Daulton et al., 2021) and the problems without a default.

problemreference point
DTLZ1400 per objective
DTLZ21.1 per objective
DTLZ7 (m = 5)(1.1, 1.1, 1.1, 1.1, 47)
WFG6 (m = 4)(3, 5, 7, 9)
ZDT1, ZDT2(11, 11)
VehicleSafety(1698.55, 11.21, 0.29)
Penicillin (−-yield, CO$_2$, time)(25.935, 57.612, 935.5)
CarSideImpact(45.4872, 4.5114, 13.3394, 10.3942)

Both tables are now in App. D.

We hope this clarifies how our work relates to the EMO literature it builds on, and we would be grateful if the reviewer would consider revising their assessment.

We wish to thank reviewer mBei again for your constructive feedback which we believe has improved our work. As we have 24 hours left in the discussion phase, we would like to invite you to please ask for any additional clarifications that you require to make your final determination.

Reviewer_WMfuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work is on multiobjective Bayesian optimization (MOBO) when objective evaluations are corrupted by noise whose variance and distribution may depend on the design variables. Conventional MOBO algorithms generally rely on Gaussian-process (GP) surrogate models with an assumed observation-noise distribution. When this distribution is inaccurate, particularly under heteroscedastic or non-Gaussian noise, acquisition-function decisions can become unreliable. The authors propose two likelihood-free acquisition functions that avoid explicitly modeling the objectives and their noise distributions. Instead, the methods formulate candidate selection as a classification problem based on dominance relationships among previously observed objective vectors.

The central methodological contribution is a soft non-dominance indicator. Rather than classifying each observed solution as either dominated or nondominated using the entire noisy dataset, the authors repeatedly subsample the observed solutions and determine how frequently a particular solution remains nondominated.

Critical strengths
  • A principled likelihood-free acquisition framework with useful mathematical properties.
Critical weaknesses
  • The theoretical Pareto-alignment guarantees do not cover the heteroscedastic noise setting that motivates the paper. If it does, authors can clarify.
  • The experiments seem to be missing key baselines: PESMO/JESMO and qPOTS. Note that qPOTS also circumvents explicit construction of an acquisition function.
Questions for authors
  • Proposition 2 relies on a common candidate-noise distribution, whereas the motivating setting allows input-dependent noise distributions. Can the authors clarify whether any dominance-preservation guarantee holds under the multiplicative or hidden-input noise models used in the experiments? If not, could they explicitly distinguish the homoscedastic population-acquisition result from the behavior of the implemented classifier under heteroscedastic noise?
  • Could the authors compare the proposed acquisitions against a noise-aware GP acquisition function employing a heteroscedastic or robust observation model, at least on a representative subset of benchmarks? Since a variational heteroscedastic GP is already implemented for evaluating the recommended sets, a comparison using such a surrogate with qLogNEHVI would be especially informative.
  • How much improvement is obtained specifically by replacing hard non-dominance labels with the proposed subsampling-based soft labels? Could the authors compare LF-NPHVI and LF-NEHVI against otherwise identical classification-based acquisitions using the hard dominance labels of the complete observed set, while keeping the classifier architecture, training budget, and acquisition optimizer fixed?
2 comments

We thank the reviewer for the constructive review. We ran the requested baselines and ablation; the new results go in the appendix. “HV gap” is the terminal log⁡10\log_{10} hypervolume gap of the returned set (likelihood-free estimator; the oracle estimator agrees), and W/T/L counts per-condition wins/ties/losses at the 5% level over 10 seeds.

W1 / Q1. Theory under heteroscedastic noise. We thank the reviewer for highlighting this limitation which was not sufficiently discussed in the original submission. The result in Proposition 2 of the submission (now Proposition 1 in Sec. 3.5, extended as Proposition 3 in App. C.2) can, however, be extended to a class of heteroscedastic problems whose observation models are obtained by a class of invertible transformations of homoscedastic settings. In particular, this extension covers the multiplicative noise case in our experiments. We have replaced the original proposition in the revised appendix with this more general result and added two remarks explaining its application to homoscedastic and multiplicative noise. Note that the homoscedastic result also covers the contaminated noise case in our experiments, since it does not require Gaussianity. The revised text in the Sec. 3 now contains a statement of the proposition with its main assumptions. As the reviewer suggests, Sec. 3.5 now distinguishes explicitly the population acquisition, which the result covers, from its learned classifier approximation, which it does not.

The original Proposition 2 was indeed restricted to homoscedastic noise. However, it provided an important first theoretical guarantee for the proposed acquisition functions, for which no corresponding Pareto-alignment results were previously available in the literature. Establishing the result under homoscedastic noise was a natural starting point before investigating the more general heteroscedastic setting. The revised proposition now demonstrates that the guarantee extends beyond this initial setting. The hidden-input case remains an open problem that may require more specific assumptions in future work.

Empirically, at η=0.4\eta = 0.4 the likelihood-free methods rank first on HV gap under both multiplicative and hidden-input noise (Table 15, App. G.3), and they lead the heteroscedastic GP baseline of Q2 on the high-dimensional problems.

W2. Missing baselines.

  • JESMO is already among our baselines: qLB-JES is BoTorch’s lower-bound implementation of the joint entropy search of Tu et al. (2022). We now state this explicitly in Sec. 5 and App. D. We compare against JESMO because it represents the state of the art among information-theoretic acquisitions, a family that also includes PESMO (Hernández-Lobato et al., 2016), which we cite in the introduction.

  • qPOTS: we ran the authors’ package on our data (same initial designs and noise draws) on 6 problems (dd = 5–20), multiplicative and hidden-input noise, η∈{0.2,0.4}\eta \in \{0.2, 0.4\}: 24 conditions. As released it fixes the GP noise at 10−610^{-6}, so we also ran it with learned homoscedastic noise.

Mean rank on HV gap (1 = best; 6 conditions per row; high d: dd = 15–20, low d: dd = 5–7; qLogNEHVI-VHGP is the baseline of Q2):

LF-NPHVILF-NEHVIqLogNEHVIqLogNEHVI-VHGPqPOTS (default)qPOTS (learned noise)
high d, η=0.2\eta = 0.22.332.003.332.675.585.08
high d, η=0.4\eta = 0.41.502.003.833.505.334.83
low d, η=0.2\eta = 0.23.003.332.001.675.505.50
low d, η=0.4\eta = 0.42.173.333.171.835.335.17

qPOTS ranks last in every group; with learned noise it is worse than at least one likelihood-free method in 23 of the 24 conditions and never better than either. We have added it to the appendix (App. G.7).

Note as well that these three algorithms are GP-based and therefore require the specification of a likelihood model and function space prior, requirements which are not necessary for LFBO methods.

Q2. qLogNEHVI with a heteroscedastic GP. We replaced qLogNEHVI’s surrogate with one variational heteroscedastic GP (VHGP) per objective, the model we already use for evaluation, with everything else unchanged (same 24 conditions).

qLogNEHVI-VHGP vsW/T/Lbetter inHV gap change
qLogNEHVI, all 24 conditions2/22/017/24−3-3% (p=0.005p = 0.005)
LF-NPHVI, high d (dd = 15–20)1/7/43/12+6%
LF-NPHVI, low d (dd = 5–7)4/7/18/12−4-4%

The heteroscedastic surrogate helps qLogNEHVI slightly but does not close the gap to LF-NPHVI in high dimensions.

Q3. Soft vs hard labels. We expanded the ablation of App. F to 8 problems and 4 noise settings, adding multiplicative and hidden-input noise (32 conditions), and added hard labels: γ=1\gamma = 1, the non-dominance labels of the full observed set (LF-PHVI of Steinberg et al., 2025). Everything else is fixed.

soft (γ=1/2\gamma = 1/2) vs hard (γ=1\gamma = 1)LF-NPHVI, subsampled ζ\zetaLF-NPHVI, exact ζ\zetaLF-NEHVI
W/T/L, HV gap6/26/04/28/02/29/1
W/T/L, HV gap, oracle estimator7/25/08/23/14/25/3
better in, HV gap change25/32, −7-7% (p < 0.001)24/32, −5-5% (p=0.002p = 0.002)18/32, −1-1% (p=0.54p = 0.54)

Soft labels matter for LF-NPHVI: 5–7% lower HV gap, significantly better in 4–8 conditions and worse in at most one, under both homoscedastic and heteroscedastic noise. LF-NEHVI is not significantly sensitive to γ\gamma.

The exact-ζ\zeta column reflects one implementation change: Lemma 1 (Eq. 7) gives ζ\zeta in closed form, so LF-NPHVI no longer needs to subsample. Our implementation now uses Eq. (7) (noted in Sec. 3, algorithm in the appendix). Results are statistically indistinguishable from subsampling (paper’s full design: W/T/L 5/174/1, median change 0%). The soft labels are about 100$\times$ cheaper to compute, but only speed up the overall method by about 4%; see our response to Reviewer PJix (Q2).

We hope these answers address the reviewer’s concerns, and would be grateful if they would consider revising their assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their careful and constructive feedback. We have strengthened the theoretical and empirical support for the proposed methods and clarified their scope and presentation. Importantly, the two acquisition functions, the main experimental design, and the central findings remain unchanged. The main revisions are summarized below; the individual responses provide further details.

AreaRevisionsLocationReviewers
TheoryExtended Proposition 2 (Proposition 1 in Sec. 3.5 of the revised paper) to a class of heteroscedastic noise models, including the multiplicative case, and clarified the scope of the population-acquisition guarantees.Sec. 3.5; App. C.2WMfu, mBei
Additional experimentsAdded qPOTS and a heteroscedastic GP baseline; expanded ablations of soft labels, subsampling, and alternative indicators. The findings remain consistent with the original results. All LF-NPHVI results in Sec. 5 and App. G now use its exact (closed-form) labels instead of subsampled ones; the ablations of BB and γ\gamma in App. F keep subsampled labels, with exact-label results reported alongside. Exact and subsampled labels perform the same (App. F), so the reported numbers change only marginally.Sec. 5; App. F (ablations); App. G (additional results)WMfu, mBei, PJix
Methodological clarificationsClarified the purpose and limitations of soft non-dominance, its closed-form computation, the interpretation of LF-NEHVI relative to qNEHVI, and the evaluation of returned sets.Secs. 3.1, 3.3, 3.4, 4; App. A, FmBei, PJix
Presentation and evaluationRefined the abstract and related work; clarified the reported indicators, benchmark settings, and hypervolume reference points.Abstract; Secs. 1, 4, 5; App. A, DmBei, PJix

Overall, we believe these targeted revisions strengthen and clarify the original contributions without changing the proposed methodology or the paper’s main conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.