- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper describes a novel Bayesian optimization routine for simulator-scarce allocation of experimental resources by leveraging higher-order moments of simulated responses. The authors derive a closed-form expression for the exact finite-sample covariance of the sample mean and variance estimators, establish their joint asymptotic Gaussian distribution, and incorporate these results into a hierarchical multi-output GP whose posterior quantiles guide serial and parallel acquisition. Experiments on two standard benchmarks show improvement over competing methods, and modest improvement over qNEHVI in a simulated engine optimization routine.
Critical strengths
The authors' novel technique provides a promising way for practitioners to account for expensive simulation budgets when designing experiments, helping to inform more-expensive real-world experiments. Using higher-order moments is a straightforward technique that works well with a quantile-based acquisition to jointly optimize location and replication count. The authors carefully distinguish between dependence in finite-sample estimation errors and dependence between the underlying mean–variance objectives. This provides a coherent statistical foundation for jointly selecting experimental locations and replication counts. The method demonstrates competitive Pareto-front recovery across synthetic benchmarks with different noise distributions and a simulated aero-engine application. Improvements over established baselines, including qNEHVI, provide encouraging evidence of practical utility, even if some gains are modest.
Critical weaknesses
Limited evaluation of adaptive replication allocation: Although the proposed method allows for arbitrary integer replication counts above one, the authors restrict their experiments to a predefined set of {5, 10, 15} without sufficient justification. This raises questions about the method's robustness and efficiency under different replication budgets, particularly when very few or many replications are appropriate. More importantly, performance is primarily reported against BO iterations rather than cumulative simulator evaluations, making it difficult to assess whether adaptive replication genuinely improves simulation-budget efficiency. The authors would be behooved to perform sensitivity analyses across a wider range of replication counts, along with computational cost measurements,to strengthen their central claims.
Heuristic parallel batch selection: The parallel extension relies on a relatively simple distance-based repulsion criterion to diversify experimental locations. While computationally convenient, this assumes that nearby designs provide redundant information, which may not hold in domains with sharp local changes, such as molecular activity cliffs. The authors do not establish when this heuristic is preferable to covariance-aware batch selection. Furthermore, the parallel experiments use fixed replication counts, leaving the benefits of adaptive allocation in the parallel setting untested.
Questions for authors
-
Simulation-budget efficiency: Could the authors report hypervolume and IGD+ against cumulative simulator evaluations rather than BO iterations? Since adaptive HEQHVI can use more replications per iteration than fixed-r HEQHVI, this comparison seems necessary to establish whether adaptive allocation genuinely improves sample efficiency.
-
Replication count selection: What motivates restricting adaptive replication counts to {5, 10, 15}? How sensitive are the results to this choice, particularly when allowing smaller counts (e.g., 3 or 4) or a wider range of integers? It would also be informative to report the distribution of selected replication counts and the frequency of revisiting previously sampled designs.
-
Computational overhead: How does the computational cost of fitting the surrogate and optimizing HEQHVI scale with the number of allowable replication counts? Could the authors report wall-clock acquisition times relative to simulator evaluation times to clarify the practical trade-off?
-
Parallel batch selection: What justifies using distance-based MED diversification rather than a covariance-aware batch-selection strategy? Could the authors provide an ablation isolating the contribution of MED, particularly in scenarios where nearby designs exhibit substantially different responses?
11 comments
Part 2: Response to Weakness 2
W2: Parallel Batch Selection
Heuristic parallel batch selection: The parallel extension relies on a relatively simple distance-based repulsion criterion to diversify experimental locations. While computationally convenient, this assumes that nearby designs provide redundant information, which may not hold in domains with sharp local changes, such as molecular activity cliffs. The authors do not establish when this heuristic is preferable to covariance-aware batch selection. Furthermore, the parallel experiments use fixed replication counts, leaving the benefits of adaptive allocation in the parallel setting untested.
Response:
We appreciate the reviewer's thoughtful comments on distance-based diversification and covariance-aware batch selection. We agree that posterior covariance provides a more direct measure of information redundancy, particularly when the covariance model is well calibrated. Covariance-aware selection is therefore attractive when joint batch optimization is computationally manageable. MED offers a practical alternative when the single-point acquisition is already computationally demanding, as in our setting involving posterior-quantile updates and joint optimization over design locations and replication counts. It extends HEQHVI to parallel optimization through sequential batch construction, without requiring a separate joint batch acquisition.
Importantly, MED does not maximize geometric distance alone. The first batch member is selected by HEQHVI, while subsequent members balance cost-adjusted acquisition values with spatial diversification. Thus, promising designs can still be selected even when they are spatially close. This provides a useful mechanism for exploring multiple promising regions while controlling redundancy within a batch. We also recognize that geometric distance may be less informative when the response surfaces exhibit sharp local changes, making covariance-aware selection with an appropriately specified surrogate an attractive alternative.
To assess this issue, we conducted an additional experiment with continuous piecewise-linear mean and variance functions exhibiting sharp changes in slope. We compared HEQHVI–MED with qNEHVI (Daulton et al., 2021), a representative covariance-aware batch optimization method that accounts for posterior dependence among candidate designs through their joint GP predictive distribution, rather than using geometric distance for diversification. Although the Pareto-optimal designs are concentrated in , HEQHVI–MED allocated 36.00% of its evaluations to this narrow region, compared with 20.32% for qNEHVI. HEQHVI–MED also achieved higher HV ( versus ) and lower IGD$^+$ ( versus ). These results suggest that distance-based diversification can effectively identify and refine promising regions, even when the response surfaces exhibit sharp local changes.
We further examined adaptive replication in the parallel setting. Under the same per-worker budget of 250 simulator calls, adaptive HEQHVI–MED improved HV from to and reduced IGD$^+$ from to relative to fixed . These results support the practical effectiveness of combining MED-based parallel selection with adaptive replication. Following the reviewer's suggestion, we have also expanded the discussion in the revised manuscript to clarify the respective advantages and appropriate use cases of the distance-based batch selection.
Part 3: Responses to Questions 1 and 2
Q1: Simulation-Budget Efficiency
Simulation-budget efficiency: Could the authors report hypervolume and IGD+ against cumulative simulator evaluations rather than BO iterations? Since adaptive HEQHVI can use more replications per iteration than fixed-r HEQHVI, this comparison seems necessary to establish whether adaptive allocation genuinely improves sample efficiency.
Response:
We agree with the reviewer that cumulative simulator evaluations provide a more direct measure of sample efficiency than BO iterations when replication counts vary. Following this suggestion, we conducted an additional budget-controlled comparison between adaptive HEQHVI and fixed- HEQHVI on the Branin benchmark with input-dependent Gaussian noise. Both methods use the same initialization and a budget of 250 additional simulator evaluations, with each replication counted as one evaluation. We also report the corresponding HV and IGD$^+$ trajectories against cumulative simulator evaluations in Figure 2 of the revised manuscript.
As shown in Table R1 of Part 1, adaptive HEQHVI achieves a higher average HV ( versus ) and a lower average IGD$^+$ ( versus ) than fixed , corresponding to an approximately 11% reduction in IGD$^+$. These results confirm that the improvement observed in this experiment persists under equal simulation budgets, providing direct evidence of the benefit of adaptive replication in terms of sample efficiency.
Q2: Replication Count Selection
Replication count selection: What motivates restricting adaptive replication counts to {5, 10, 15}? How sensitive are the results to this choice, particularly when allowing smaller counts (e.g., 3 or 4) or a wider range of integers? It would also be informative to report the distribution of selected replication counts and the frequency of revisiting previously sampled designs.
Response:
We appreciate the reviewer's suggestion to examine replication-count selection and design revisits more closely. The original set {5,10,15} was a coarse experimental choice rather than an inherent restriction of HEQHVI. We have removed this restriction, and the revised implementation jointly optimizes the design location and any feasible integer replication count , subject to the remaining simulation budget.
Under Gaussian (Gamma) outputs, the selected replication counts are distributed as follows: in 93.76% (90.96%) of decisions, in 4.22% (6.12%), in 0.84% (1.64%), and in 1.18% (1.29%), with maximum selected counts of (). The strong preference for is consistent with the cost-adjusted acquisition criterion, which balances the expected improvement in estimation precision against the additional simulation cost. Nevertheless, the occasional selection of larger counts demonstrates that the method can increase replication effort when warranted.
Importantly, selecting a small replication count does not limit the total number of observations at a design. HEQHVI allows previously evaluated designs to be revisited, with additional replications pooled with existing observations. In the reported experiment with a budget of , 62.56% of selections under fixed and 58.48% under adaptive replication revisited previously evaluated designs. Thus, the method can refine promising designs progressively rather than committing substantial replication effort upfront. Together with the improved HV and IGD$^+$ under adaptive replication reported in Table R1, these results support the flexibility and effectiveness of the revised replication-allocation strategy.
Part 1: Response to Weakness 1
W1: Adaptive Replication Allocation
Response:
We appreciate the reviewer's important concerns regarding replication allocation and its computational efficiency. We agree that the original replication menu {5,10,15} was restrictive and that comparisons based solely on BO iterations could not establish simulation-budget efficiency. We have therefore revised the replication search and conducted additional budget-controlled experiments.
First, the revised implementation jointly optimizes the design location and any feasible integer replication count r≥2, subject to the remaining budget. The lower bound r=2 is required to compute the sample variance from a new replication block. Interestingly, the cost-adjusted HEQHVI criterion selects r=2 in 93.76% and 90.96% of decisions under Gaussian and Gamma outputs, respectively, although the largest selected counts reach 10 and 11. This behavior is consistent with the trade-off underlying our acquisition criterion: additional replications improve estimation precision but incur proportional simulation costs. Moreover, previously evaluated designs can be revisited, allowing further refinement without committing to large replication blocks upfront. Detailed selection frequencies and revisit rates are provided in our response to Question 2.
Second, we compared adaptive replication with fixed r=2 under a matched budget of B=250, excluding the common initialization. For single-point HEQHVI, the budget counts cumulative simulator calls. For the five-worker parallel HEQHVI–MED experiment, all designs in a batch share an optimized replication count rt, and B measures cumulative calls per worker; the total simulator calls are five times this amount. Thus, adaptive and fixed replication are compared under the same resources within each execution setting.
As shown in Table R1, adaptive replication improves both HV and IGD$^+$ in the serial and parallel settings. In particular, IGD$^+$ decreases by approximately 11% and 24%, respectively, with smaller but consistent improvements in HV. These results suggest that selectively allocating larger replication counts can improve Pareto-front recovery, even though the minimum count is preferred in most decisions.
Finally, expanding the replication search does not require fitting a separate GP for each candidate r; it primarily increases the acquisition-optimization workload. In our aero-engine application, selecting the next design–replication pair takes approximately 10 seconds, compared with nearly 5 minutes for a single simulator evaluation. Further computational details are provided in our response to Question 3. Together, these results provide additional evidence that adaptive replication can improve simulation-budget efficiency while maintaining manageable computational overhead.
Table R1: Adaptive versus fixed replication under a budget of B=250: total simulator calls for serial methods and per-worker calls for five-worker parallel methods.