- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends Likelihood-free Bayesian Optimization (LFBO) to the framework of Multi-Objective Bayesian Optimization (MOBO). In conventional single-objective LFBO, the acquisition function is constructed using a classifier trained to solve a two-class discrimination problem (selected positive examples versus other samples), with an appropriate utility function serving as the weight (or selection criteria for positive examples). As an extension to the multi-objective setting, this study uses a soft non-dominance indicator as the utility in LFBO. This indicator measures the stability of non-dominance in the design while acknowledging the presence of noise. Furthermore, the authors propose an evaluation methodology that explicitly exposes the decision-maker's loss. Comprehensive experiments demonstrate that in low-noise scenarios, GP-based methods perform superiorly, while it is also shown that the proposed acquisition function significantly outperforms GP-based methods as the noise level and input dimensionality increases.
Critical strengths
-
Simple and Computationally Tractable Formulation for Avoiding Noise Model Estimation: The concept of constructing the LFBO utility by leveraging inter-observation dominance relationships and subsampling, rather than directly estimating heteroscedastic or non-Gaussian noise, is interesting. In particular, the closed-form derivation in terms of provides a computational advantage. While LFBO itself is an existing methodology, the utility design for noisy multi-objective optimization is highly worthy of further investigation.
-
Evaluation Design Separating Oracle Evaluation from Actual Decision-Making Quality: Traditional methods like Hypervolume (HV) and IGD+ often overlook the crucial question of whether the selected designs are truly optimal. The approach of decomposing this problem into three components—sampling, set selection, and misselection—and quantifying each is significant. Although the decomposition itself is an algebraic identity and therefore has limited theoretical novelty, the experimental finding that misselection accounts for over 40% of the total loss is valuable. This contribution can be valued independently of the proposed acquisition function.
Critical weaknesses
Unclear Mechanism of Noise Mitigation via Subsampling: It is unclear whether subsampling truly mitigates misclassification induced by noise. measures the stability of non-dominance after removing a subset of observed designs, which is not the noise uncertainty itself. Specifically, if a remarkably good objective vector is observed purely by noise, that design will lead to , causing . While this might mitigate false negatives, the mechanism for suppressing false positives resulting from extreme positive noise is not clearly defined. The ability to operate without a noise model and the robustness to noise are distinct issues. Currently, the mechanism of the proposed method appears to rely more on empirical observations than on theoretical grounding.
If my comment is due to my misunderstanding, please correct it in the rebuttal.
Questions for authors
-
Is LF-NEHVI truly a Likelihood-Free counterpart to NEHVI?: qNEHVI marginalizes over the posterior distribution of the latent objective function, whereas LF-NEHVI averages over the subsampled observed front. The uncertainties inherent in these two approaches are fundamentally different. Furthermore, since LF-NEHVI depends on the numerical value of the Hypervolume (HV), the claim that it is "invariant to the objective function's monotonic transformation and depends only on the dominance relationship" holds true for LF-NPHVI, but generally does not hold for LF-NEHVI.
-
Is the novelty and necessity of the soft non-dominance indicator sufficiently demonstrated?: Ultimately, is merely a monotonic function of . Therefore, it is necessary to compare it with LFBO methods that utilize weighting based on simple dominance count or Pareto rank. We need to determine whether the construction of subsampling itself contributes to the performance improvement.
-
Can be interpreted as a probability of belonging to the Pareto set?: represents the survival probability under subsampling, not the true probability of belonging to the Pareto set. Therefore, the explanation that selecting based on the threshold of 1/2 minimizes misclassification loss cannot be straightforwardly justified. The final decision-making quality when using different thresholds should also be confirmed.
5 comments
Thank you for the clarification. My question 3 refers specifically to the threshold of 0.5 used in Eq. (12) to construct the recommended set, not to the acquisition function or the subsampling parameter .
My main concern is the interpretation of as a probability of Pareto-set membership, to which the author answered "no". I understand represents the probability that a design remains nondominated under random subsampling of the observed data, rather than the probability that it belongs to the true Pareto set. Therefore, I though the argument that a threshold of 0.5 minimizes the expected number of misclassifications requires additional justification.
A sensitivity analysis using different thresholds (e.g., 0.25, 0.5, and 0.75) would be useful, particularly for nIGD+, to assess how the recommended-set selection affects decision quality. This is separate from the ablation of in Appendix F. I'm guessing this is not excessive requirement, but if there are reasons that the additional experiment isn't reasonable, some discussion to justify the particular value "0.5" would suffice.
Alternatively, if the results are insensitive to the threshold, it would be sufficient to report this and clarify that 0.5 is a practical convention rather than a theoretically optimal threshold. This is just for clarification, not the critial flaw in the paper.
Thank you for the clarification and suggestion. We re-scored all stored runs of the main campaign (7 methods, 16 problems, 4 noise structures, 3 noise levels, 10 seeds, no new optimization) on the three main performance metrics, with different thresholds in Eq. (12). At , depends only on the number of observations dominating design , so , and return the designs with , and , respectively.
| Threshold | nIGD+ LF-NPHVI | nIGD+ LF-NEHVI | nIGD+ best baseline | HV gap LF-NPHVI | HV gap LF-NEHVI | HV gap best baseline | IGD+ LF-NPHVI | IGD+ LF-NEHVI | IGD+ best baseline |
|---|---|---|---|---|---|---|---|---|---|
| () | 3.28 | 3.77 | 3.35 | 2.52 | 3.03 | 3.24 | 3.11 | 3.93 | 3.27 |
| (, paper) | 3.26 | 3.74 | 3.31 | 2.48 | 3.01 | 3.27 | 3.10 | 3.93 | 3.29 |
| () | 3.25 | 3.80 | 3.35 | 2.45 | 3.01 | 3.26 | 3.13 | 3.91 | 3.27 |
Mean rank among the seven methods (lower is better); the best baseline is qLogNParEGO for nIGD+ and qLogNEHVI for HV gap and IGD+.
The conclusions do not depend on the threshold: LF-NPHVI ranks first overall on all three indicators at every value, GP baselines still lead only at , and per-condition rankings agree closely with those at (median Kendall rank correlation between 0.90 and 1.00). A lower threshold mainly trades coverage for mis-selection: larger sets lower the HV gap and IGD+ but raise nIGD+ at .
We thus use as a practical convention rather than an optimal threshold: at it returns exactly the designs dominated by at most one observation, and the overall rankings barely change at cutoffs of zero or two dominating observations. We have added this analysis to Appendix G.8, and a sentence in the main text for clarification, in green color.
We thank the reviewer for the careful review. The questions on subsampling led to new experiments and a simpler implementation (Q2). “HV gap” is the terminal log10 hypervolume gap of the returned set; W/T/L counts per-condition wins/ties/losses at the 5% level over 10 seeds.
W. Subsampling and false positives. Agreed: ζ softens false negatives, not false positives. A design dominated by one lucky observation keeps ζ≈1/2 (for γ=1/2), and ζ decays geometrically in the number k of observations dominating it (Lemma 1), so a design is not immediately discarded based on a single potentially misleading observation. Conversely, a design with a favorable draw gets ζ=1.
The classifier may further mitigate such errors by learning from observations across the search space. We have clarified these limitations in Sec. 3.1 and, in more detail, in App. A, distinguishing the theoretical properties of ζ from the empirical robustness of the resulting acquisition functions. In particular, our expanded ablation shows that soft labels outperform hard labels for LF-NPHVI (see our response to Reviewer WMfu, Q3).
Q1. LF-NEHVI and NEHVI. Agreed on both points. Invariance to monotone transformations holds only for ζ and LF-NPHVI; we have restricted the claim in the abstract, conclusion and related work. LF-NEHVI shares qNEHVI’s structure (hypervolume improvement averaged over an uncertain front), but its uncertainty comes from resampling the observed front, not from a GP posterior. Sec. 3.3 now says so and no longer uses “counterpart”.
Q2. Is subsampling necessary? To compute ζ, no. As the reviewer notes, ζ is a function of k: Lemma 1 gives ζ=(sn−k)/(sn)≈(1−γ)k (Eq. 7), the B→∞ limit of subsampling. Our implementation of LF-NPHVI now uses Eq. (7) (a remark in Sec. 3, the full algorithm in the appendix). Exact labels give the same results as subsampled ones (B = 128), paired by seed:
Exact labels are 100–160$\times$ cheaper to compute, and whole runs about 4% faster. We keep subsampling as the definition in Sec. 3 because it is the general one: it also defines the hypervolume-improvement labels of LF-NEHVI, which have no closed form we know of.
On counts, we compared ζ with 1/(1+k) over the paper’s full design (16 problems × 4 noise models × 3 noise levels = 192 conditions; 180 on HV gap, which excludes DTLZ7, as it has no known maximal hypervolume), changing only the label of LF-NPHVI. The Pareto-rank label 1/r matched 1/(1+k) in the 32-condition ablation, so we did not rerun it.
Labels that decay more slowly than γ=1/2 do 1–2% better once η≥0.2. The differences among them are small, and 1/(1+k) is itself ζ averaged over a uniformly random subsample size. We keep γ=1/2 and have added both comparisons to App. F.
Q3. Is ζ a probability of Pareto membership? No. ζ is a survival frequency under subsampling, not a calibrated probability, and we have removed the misclassification claim. At γ=1/2 the 1/2 threshold of the evaluation has a direct reading instead: ζ≥1/2 exactly when at most one observation dominates the design (Eq. 13).
Could the reviewer clarify which threshold they would like varied (“the final decision-making quality when using different thresholds should also be confirmed”)? Our method uses no threshold during optimization: the acquisition maximizes the odds of a classifier trained on the soft labels. A threshold appears only in our evaluation protocol (Sec. 4), where the set returned to the decision maker is the designs with ζ≥1/2. If this is the threshold the reviewer means, we can recompute the returned sets and their decision quality (nIGD+ and HV) at other values from our stored runs. We also ablate γ in Appendix F (Tables 7 and 10).
We hope these answers address the reviewer’s concerns.