acceptodds
Under review as a conference paper at ICLR 2027

Mean-Variance Pareto Learning with Replicate-Aware Bayesian Optimization

Abstract

Stochastic simulators often exhibit a trade-off between average performance and run-to-run variability. When the preferred balance is not known in advance, the goal is not one weighted compromise but the set of efficient designs spanning this trade-off, namely the mean–variance Pareto frontier. Learning this frontier under a limited simulation budget creates an additional difficulty: at each design, the mean and variance are estimated from the same finite collection of simulator runs. Their estimation errors therefore depend on the number of replications and can be correlated, so a sequential algorithm must decide both where to sample and how much simulation effort to allocate there. We develop a replicate-aware Bayesian optimization framework for this joint learning and allocation problem. We first derive the exact covariance of the sample mean and sample variance for any finite replication count and establish their joint asymptotic distribution. We then embed this sampling structure in a hierarchical multi-output Gaussian process that separates finite-sample estimation dependence from dependence between the underlying mean and variance surfaces. Based on this posterior, hierarchical expected quantile hypervolume improvement (HEQHVI) jointly selects a design and replication count by maximizing the expected hypervolume gain of a posterior-quantile Pareto front per simulator run. This creates an explicit trade-off between exploring new designs and reducing uncertainty at previously evaluated ones. For parallel evaluation, we extend HEQHVI using a minimum-energy design (MED) criterion that retains the best serial candidate while spreading the remaining batch members across promising but distinct regions. Synthetic benchmarks and an F119 aero-engine study illustrate how the framework learns mean–variance trade-offs while adapting replication effort.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
heteroscedastic noise, mean--variability trade-offs, multi-output Gaussian processes, Pareto optimization, replicate-aware Bayesian optimization, stochastic simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GqcVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper describes a novel Bayesian optimization routine for simulator-scarce allocation of experimental resources by leveraging higher-order moments of simulated responses. The authors derive a closed-form expression for the exact finite-sample covariance of the sample mean and variance estimators, establish their joint asymptotic Gaussian distribution, and incorporate these results into a hierarchical multi-output GP whose posterior quantiles guide serial and parallel acquisition. Experiments on two standard benchmarks show improvement over competing methods, and modest improvement over qNEHVI in a simulated engine optimization routine.

Critical strengths

The authors' novel technique provides a promising way for practitioners to account for expensive simulation budgets when designing experiments, helping to inform more-expensive real-world experiments. Using higher-order moments is a straightforward technique that works well with a quantile-based acquisition to jointly optimize location and replication count. The authors carefully distinguish between dependence in finite-sample estimation errors and dependence between the underlying mean–variance objectives. This provides a coherent statistical foundation for jointly selecting experimental locations and replication counts. The method demonstrates competitive Pareto-front recovery across synthetic benchmarks with different noise distributions and a simulated aero-engine application. Improvements over established baselines, including qNEHVI, provide encouraging evidence of practical utility, even if some gains are modest.

Critical weaknesses

Limited evaluation of adaptive replication allocation: Although the proposed method allows for arbitrary integer replication counts above one, the authors restrict their experiments to a predefined set of {5, 10, 15} without sufficient justification. This raises questions about the method's robustness and efficiency under different replication budgets, particularly when very few or many replications are appropriate. More importantly, performance is primarily reported against BO iterations rather than cumulative simulator evaluations, making it difficult to assess whether adaptive replication genuinely improves simulation-budget efficiency. The authors would be behooved to perform sensitivity analyses across a wider range of replication counts, along with computational cost measurements,to strengthen their central claims.

Heuristic parallel batch selection: The parallel extension relies on a relatively simple distance-based repulsion criterion to diversify experimental locations. While computationally convenient, this assumes that nearby designs provide redundant information, which may not hold in domains with sharp local changes, such as molecular activity cliffs. The authors do not establish when this heuristic is preferable to covariance-aware batch selection. Furthermore, the parallel experiments use fixed replication counts, leaving the benefits of adaptive allocation in the parallel setting untested.

Questions for authors
  1. Simulation-budget efficiency: Could the authors report hypervolume and IGD+ against cumulative simulator evaluations rather than BO iterations? Since adaptive HEQHVI can use more replications per iteration than fixed-r HEQHVI, this comparison seems necessary to establish whether adaptive allocation genuinely improves sample efficiency.

  2. Replication count selection: What motivates restricting adaptive replication counts to {5, 10, 15}? How sensitive are the results to this choice, particularly when allowing smaller counts (e.g., 3 or 4) or a wider range of integers? It would also be informative to report the distribution of selected replication counts and the frequency of revisiting previously sampled designs.

  3. Computational overhead: How does the computational cost of fitting the surrogate and optimizing HEQHVI scale with the number of allowable replication counts? Could the authors report wall-clock acquisition times relative to simulator evaluation times to clarify the practical trade-off?

  4. Parallel batch selection: What justifies using distance-based MED diversification rather than a covariance-aware batch-selection strategy? Could the authors provide an ablation isolating the contribution of MED, particularly in scenarios where nearby designs exhibit substantially different responses?

11 comments
AuthorsOpenReview ↗

Part 1: Response to Weakness 1

W1: Adaptive Replication Allocation

Limited evaluation of adaptive replication allocation: Although the proposed method allows for arbitrary integer replication counts above one, the authors restrict their experiments to a predefined set of {5, 10, 15} without sufficient justification. This raises questions about the method's robustness and efficiency under different replication budgets, particularly when very few or many replications are appropriate. More importantly, performance is primarily reported against BO iterations rather than cumulative simulator evaluations, making it difficult to assess whether adaptive replication genuinely improves simulation-budget efficiency. The authors would be behooved to perform sensitivity analyses across a wider range of replication counts, along with computational cost measurements,to strengthen their central claims.

Response:

We appreciate the reviewer's important concerns regarding replication allocation and its computational efficiency. We agree that the original replication menu {5,10,15} was restrictive and that comparisons based solely on BO iterations could not establish simulation-budget efficiency. We have therefore revised the replication search and conducted additional budget-controlled experiments.

First, the revised implementation jointly optimizes the design location and any feasible integer replication count r≥2r\geq 2, subject to the remaining budget. The lower bound r=2r=2 is required to compute the sample variance from a new replication block. Interestingly, the cost-adjusted HEQHVI criterion selects r=2r=2 in 93.76% and 90.96% of decisions under Gaussian and Gamma outputs, respectively, although the largest selected counts reach 1010 and 1111. This behavior is consistent with the trade-off underlying our acquisition criterion: additional replications improve estimation precision but incur proportional simulation costs. Moreover, previously evaluated designs can be revisited, allowing further refinement without committing to large replication blocks upfront. Detailed selection frequencies and revisit rates are provided in our response to Question 2.

Second, we compared adaptive replication with fixed r=2r=2 under a matched budget of B=250B=250, excluding the common initialization. For single-point HEQHVI, the budget counts cumulative simulator calls. For the five-worker parallel HEQHVI–MED experiment, all designs in a batch share an optimized replication count rtr_t, and BB measures cumulative calls per worker; the total simulator calls are five times this amount. Thus, adaptive and fixed replication are compared under the same resources within each execution setting.

As shown in Table R1, adaptive replication improves both HV and IGD$^+$ in the serial and parallel settings. In particular, IGD$^+$ decreases by approximately 11% and 24%, respectively, with smaller but consistent improvements in HV. These results suggest that selectively allocating larger replication counts can improve Pareto-front recovery, even though the minimum count is preferred in most decisions.

Finally, expanding the replication search does not require fitting a separate GP for each candidate rr; it primarily increases the acquisition-optimization workload. In our aero-engine application, selecting the next design–replication pair takes approximately 1010 seconds, compared with nearly 55 minutes for a single simulator evaluation. Further computational details are provided in our response to Question 3. Together, these results provide additional evidence that adaptive replication can improve simulation-budget efficiency while maintaining manageable computational overhead.

Table R1: Adaptive versus fixed replication under a budget of B=250B=250: total simulator calls for serial methods and per-worker calls for five-worker parallel methods.

MethodAverage HVAverage IGD$^+$
HEQHVI (adaptive-rr)0.8492260.015006
HEQHVI (r=2r=2)0.8455570.016839
HEQHVI–MED (adaptive-rr)0.8752630.003005
HEQHVI–MED (r=2r=2)0.8700280.003958
AuthorsOpenReview ↗

Part 2: Response to Weakness 2

W2: Parallel Batch Selection

Heuristic parallel batch selection: The parallel extension relies on a relatively simple distance-based repulsion criterion to diversify experimental locations. While computationally convenient, this assumes that nearby designs provide redundant information, which may not hold in domains with sharp local changes, such as molecular activity cliffs. The authors do not establish when this heuristic is preferable to covariance-aware batch selection. Furthermore, the parallel experiments use fixed replication counts, leaving the benefits of adaptive allocation in the parallel setting untested.

Response:

We appreciate the reviewer's thoughtful comments on distance-based diversification and covariance-aware batch selection. We agree that posterior covariance provides a more direct measure of information redundancy, particularly when the covariance model is well calibrated. Covariance-aware selection is therefore attractive when joint batch optimization is computationally manageable. MED offers a practical alternative when the single-point acquisition is already computationally demanding, as in our setting involving posterior-quantile updates and joint optimization over design locations and replication counts. It extends HEQHVI to parallel optimization through sequential batch construction, without requiring a separate joint batch acquisition.

Importantly, MED does not maximize geometric distance alone. The first batch member is selected by HEQHVI, while subsequent members balance cost-adjusted acquisition values with spatial diversification. Thus, promising designs can still be selected even when they are spatially close. This provides a useful mechanism for exploring multiple promising regions while controlling redundancy within a batch. We also recognize that geometric distance may be less informative when the response surfaces exhibit sharp local changes, making covariance-aware selection with an appropriately specified surrogate an attractive alternative.

To assess this issue, we conducted an additional experiment with continuous piecewise-linear mean and variance functions exhibiting sharp changes in slope. We compared HEQHVI–MED with qNEHVI (Daulton et al., 2021), a representative covariance-aware batch optimization method that accounts for posterior dependence among candidate designs through their joint GP predictive distribution, rather than using geometric distance for diversification. Although the Pareto-optimal designs are concentrated in [0.375,0.6][0.375,0.6], HEQHVI–MED allocated 36.00% of its evaluations to this narrow region, compared with 20.32% for qNEHVI. HEQHVI–MED also achieved higher HV (0.9040160.904016 versus 0.9021350.902135) and lower IGD$^+$ (0.0003130.000313 versus 0.0022070.002207). These results suggest that distance-based diversification can effectively identify and refine promising regions, even when the response surfaces exhibit sharp local changes.

We further examined adaptive replication in the parallel setting. Under the same per-worker budget of 250 simulator calls, adaptive HEQHVI–MED improved HV from 0.8700280.870028 to 0.8752630.875263 and reduced IGD$^+$ from 0.0039580.003958 to 0.0030050.003005 relative to fixed r=2r=2. These results support the practical effectiveness of combining MED-based parallel selection with adaptive replication. Following the reviewer's suggestion, we have also expanded the discussion in the revised manuscript to clarify the respective advantages and appropriate use cases of the distance-based batch selection.

AuthorsOpenReview ↗

Part 3: Responses to Questions 1 and 2

Q1: Simulation-Budget Efficiency

Simulation-budget efficiency: Could the authors report hypervolume and IGD+ against cumulative simulator evaluations rather than BO iterations? Since adaptive HEQHVI can use more replications per iteration than fixed-r HEQHVI, this comparison seems necessary to establish whether adaptive allocation genuinely improves sample efficiency.

Response:

We agree with the reviewer that cumulative simulator evaluations provide a more direct measure of sample efficiency than BO iterations when replication counts vary. Following this suggestion, we conducted an additional budget-controlled comparison between adaptive HEQHVI and fixed-rr HEQHVI on the Branin benchmark with input-dependent Gaussian noise. Both methods use the same initialization and a budget of 250 additional simulator evaluations, with each replication counted as one evaluation. We also report the corresponding HV and IGD$^+$ trajectories against cumulative simulator evaluations in Figure 2 of the revised manuscript.

As shown in Table R1 of Part 1, adaptive HEQHVI achieves a higher average HV (0.8492260.849226 versus 0.8455570.845557) and a lower average IGD$^+$ (0.0150060.015006 versus 0.0168390.016839) than fixed r=2r=2, corresponding to an approximately 11% reduction in IGD$^+$. These results confirm that the improvement observed in this experiment persists under equal simulation budgets, providing direct evidence of the benefit of adaptive replication in terms of sample efficiency.

Q2: Replication Count Selection

Replication count selection: What motivates restricting adaptive replication counts to {5, 10, 15}? How sensitive are the results to this choice, particularly when allowing smaller counts (e.g., 3 or 4) or a wider range of integers? It would also be informative to report the distribution of selected replication counts and the frequency of revisiting previously sampled designs.

Response:

We appreciate the reviewer's suggestion to examine replication-count selection and design revisits more closely. The original set {5,10,15} was a coarse experimental choice rather than an inherent restriction of HEQHVI. We have removed this restriction, and the revised implementation jointly optimizes the design location and any feasible integer replication count r≥2r\geq 2, subject to the remaining simulation budget.

Under Gaussian (Gamma) outputs, the selected replication counts are distributed as follows: r=2r=2 in 93.76% (90.96%) of decisions, r=3r=3 in 4.22% (6.12%), r=4r=4 in 0.84% (1.64%), and r≥5r\geq 5 in 1.18% (1.29%), with maximum selected counts of 1010 (1111). The strong preference for r=2r=2 is consistent with the cost-adjusted acquisition criterion, which balances the expected improvement in estimation precision against the additional simulation cost. Nevertheless, the occasional selection of larger counts demonstrates that the method can increase replication effort when warranted.

Importantly, selecting a small replication count does not limit the total number of observations at a design. HEQHVI allows previously evaluated designs to be revisited, with additional replications pooled with existing observations. In the reported experiment with a budget of 250250, 62.56% of selections under fixed r=2r=2 and 58.48% under adaptive replication revisited previously evaluated designs. Thus, the method can refine promising designs progressively rather than committing substantial replication effort upfront. Together with the improved HV and IGD$^+$ under adaptive replication reported in Table R1, these results support the flexibility and effectiveness of the revised replication-allocation strategy.

Reviewer_xHa6OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces HEQHVI, a novel method for Bayesian optimization that maximizes the expected gain in hypervolume of a Pareto front of mean and variance in a stochastic simulator. The method is replicate-aware, choosing both a design xx and the desired number of simulator runs rr on that design, jointly. It further accounts for the coupling of the estimation errors of mean and variance due to coming from the same set of simulations. The authors evaluate their method on two synthetic simulators and one simulator with practical relevance for engineering. It demonstrates state-of-the-art performance on all benchmarks.

Critical strengths
  • The method is both novel and interesting, as well as highly practically relevant.
  • The overall quality of the presentation is high.
  • The method is shown to exhibit strong performance on every benchmark.
Critical weaknesses
  • A central contribution of the paper is that the number of simulation runs r>1r > 1 is chosen jointly with the design xx. However, this necessitates that the method decide rr before seeing results from that design, and the initial guess cannot be updated as more measurements are performed. Although the simulations can be parallelized, this necessitates more total simulations than allowing the method to decide for a new xx after every simulation. This weakens the method's relevance for expensive simulators, and the trade-off is neither ablated nor discussed.
  • The evaluation is limited to a few low-dimensional design problems.
Questions for authors
  • Did the authors consider modeling the variance from simulator outcomes at different designs instead, such as with a heteroscedastic GP?
  • The method is partially motivated as providing a balance between exploration of designs xx and refinement at a single design. Reusing the same exact design rather than only similar ones can be beneficial when the simulator is highly non-smooth in the design. Could the authors add a brief discussion of the smoothness of the simulators that are considered in the paper?

Nitpicks:

  • In Figure 2, the axis labels of the top row of plots are partially hidden behind the bottom row of plots.
4 comments
AuthorsOpenReview ↗

Part 1: Responses to Weaknesses 1 and 2

W1: Adaptive Replication and Block-Based Evaluation

A central contribution of the paper is that the number of simulation runs r>1r > 1 is chosen jointly with the design xx. However, this necessitates that the method decide rr before seeing results from that design, and the initial guess cannot be updated as more measurements are performed. Although the simulations can be parallelized, this necessitates more total simulations than allowing the method to decide for a new xx after every simulation. This weakens the method's relevance for expensive simulators, and the trade-off is neither ablated nor discussed.

Response:

We appreciate the reviewer's insightful observation and agree that adapting the replication decision after each individual observation could improve sample efficiency by avoiding unnecessary evaluations. Our method adopts a complementary block-based strategy, in which the replication count is determined before execution to facilitate parallel evaluations and reduce the frequency of surrogate updates. Such an approach is also supported by existing studies: Makarova et al. (2021) use replicated observations for mean–variance estimation, Calandriello et al. (2022) investigate repeated evaluations of selected candidates, and Dai et al. (2023) study adaptive replication in parallel experimental design.

Importantly, our method remains adaptive across decision rounds: previously evaluated designs can be revisited, and subsequent replication decisions use the updated posterior. Moreover, our revised experiments allow any integer replication count r≥2r\geq 2, rather than the original set {5,10,15}. The cost-adjusted HEQHVI criterion selects r=2r=2 in 93.76% and 90.96% of decisions under Gaussian and Gamma outputs, respectively. This preference for small replication blocks is consistent with the cost-adjusted acquisition, which often favors incremental refinement over committing to larger, more costly blocks. This tendency may also be reinforced by the ability to revisit previously evaluated designs: in the Branin replication-allocation experiment, 58.48% of adaptive-rr decisions revisited existing designs, compared with 62.56% under fixed r=2r=2. Consequently, adaptive replication and exact-design revisits work together to progressively improve the estimation precision of the mean and variance by accumulating observations across successive decisions, without requiring large replication blocks upfront.

These findings suggest that within-block early stopping offers limited opportunities for additional savings in the settings examined, although it may be beneficial when larger blocks are selected. Our block-based approach therefore provides a practical balance between adaptive resource allocation, parallel execution, and the frequency of sequential feedback.

W2: Evaluation on Higher-Dimensional Problems

The evaluation is limited to a few low-dimensional design problems.

Response:

We appreciate the reviewer's suggestion and agree that evaluating higher-dimensional problems is important for assessing the applicability of our method. Accordingly, we conducted an additional 20-dimensional experiment based on a DTLZ2-type mean–variance benchmark, following the benchmarking practice in related BO studies (Makarova et al., 2021; Daulton et al., 2021, 2022). The problem features competing mean and input-dependent variance objectives, with all 20 input variables contributing to both, and uses Gaussian simulator outputs.

Table R1 reports the results under a budget of 250250. Among serial methods, adaptive HEQHVI achieves higher average HV and lower average IGD$^+$ than NEHVI. In the parallel setting, adaptive HEQHVI–MED also outperforms qNEHVI on both metrics. Moreover, adaptive replication improves upon fixed r=2r=2 in both settings. These results provide encouraging evidence that the proposed framework remains effective beyond the low-dimensional benchmarks considered in the original submission.

Table R1: Performance on the 20-dimensional mean–variance benchmark with Gaussian outputs under a budget of 250250.

MethodAverage HVAverage IGD$^+$
HEQHVI (adaptive-rr)0.2819210.146984
HEQHVI (r=2r=2)0.2784520.158304
HEQHVI–MED (adaptive-rr)0.3439240.122031
HEQHVI–MED (r=2r=2)0.3341470.135685
GPS-C0.2131080.208872
SMV0.2109950.226812
NEHVI0.2653340.167411
qNEHVI0.3267540.140987
AuthorsOpenReview ↗

Part 2: Responses to Questions 1 and 2 and Nitpick 1

Q1: Variance Modeling

Did the authors consider modeling the variance from simulator outcomes at different designs instead, such as with a heteroscedastic GP?

Response:

We appreciate the reviewer's suggestion. A heteroscedastic GP is indeed a natural alternative for learning the mean and input-dependent variance directly from simulator outputs by borrowing information across designs (Binois et al., 2018). Our approach also shares information across designs, but differs in its observation model and how replication decisions are incorporated. Specifically, we jointly model the sample mean and variance as noisy observations of two latent objective surfaces, accounting for their replication-dependent sampling covariance, including the cross-covariance induced by shared replications under non-Gaussian outputs. Together with the hierarchical bivariate GP, this structure enables tractable posterior-quantile updates for jointly selecting the design location and replication count. We emphasize that the Gaussian likelihood for these summary statistics is a moment-matched working approximation, rather than an exact raw-output likelihood.

To assess the proposed alternative directly, we conducted an additional comparison with a raw-output heteroscedastic GP on the Branin benchmark with input-dependent Gamma noise. All methods used the same initialization and an additional budget of 250 simulator calls. The heteroscedastic GP achieved (HV,IGD+)=(0.823314,0.030109)(\mathrm{HV},\mathrm{IGD}^{+})=(0.823314, 0.030109) with r=1r=1 and (0.823339,0.030243)(0.823339, 0.030243) with r=2r=2, while HEQHVI with r=2r=2 achieved (0.849626,0.015624)(0.849626, 0.015624). Thus, HEQHVI attained a modest improvement in HV and a more substantial reduction in IGD$^+$ under the same simulation budget. This direct comparison complements our existing GPS-C baseline (Wang et al., 2025), which also estimates input-dependent noise using observations from neighboring designs. Overall, these results support the effectiveness of our replication-aware framework relative to these GP-based alternatives in the tested setting.

Q2: Smoothness

Could the authors add a brief discussion of the smoothness of the simulators that are considered in the paper?

Response:

We agree with the reviewer that when the mean or variance changes sharply across the design space, revisiting an existing design can be more informative than relying on observations from nearby designs. Importantly, the proposed HEQHVI explicitly allows both new designs and previously evaluated locations to be selected. Additional replications at an existing design directly reduce its estimation uncertainty, whereas information from nearby designs is transferred through the GP surrogate and depends on its spatial modeling assumptions.

In our original experiments, the synthetic Branin and Hartmann benchmarks have smooth mean and variance surfaces. For the aero-engine simulator, however, global smoothness is not established. Our stationary Matérn-5/25/2 surrogate represents a modeling assumption rather than a guarantee about the underlying simulator, and it may smooth over sharp local changes or discontinuities. Exact-design replication helps refine local estimates but does not eliminate potential surrogate bias in such regions. Nonstationary or partition-based GP models (Snoek et al., 2014; Gramacy & Lee, 2008) offer possible alternatives for handling more complex response surfaces.

To examine this issue more directly, we conducted an additional one-dimensional experiment with continuous piecewise-linear mean and variance functions exhibiting pronounced changes in slope. The Pareto-optimal designs lie in the interval [0.375,0.6][0.375,0.6]. Across 125 single-point selections, HEQHVI evaluated only 25 distinct locations, demonstrating its ability to revisit existing designs rather than relying exclusively on spatial interpolation. As shown in Table R2, HEQHVI achieves higher HV and lower IGD$^+$ than NEHVI. In the parallel setting, HEQHVI–MED also outperforms qNEHVI, allocating 36.00% of its evaluations to the narrow Pareto region, compared with 20.32% for qNEHVI.

Overall, these results suggest that HEQHVI can effectively combine exploration with exact-design refinement, even when the response surfaces are not smooth. This distinction helps establish the scope of our modeling assumptions and the practical value of allowing exact-design revisits.

Table R2: Pareto-front recovery on the one-dimensional piecewise-linear benchmark (r=2r=2).

MethodAverage HVAverage IGD$^+$
HEQHVI0.9005600.001913
HEQHVI–MED0.9040160.000313
NEHVI0.8985910.005265
qNEHVI0.9021350.002207

Nitpick 1

In Figure 2, the axis labels of the top row of plots are partially hidden behind the bottom row of plots.

Response:

We thank the reviewer for pointing out this formatting issue. We have revised Figure 2 in the manuscript.

AuthorsOpenReview ↗

Part 3: References

References

Binois, M., Gramacy, R. B., & Ludkovski, M. (2018). Practical Heteroskedastic Gaussian Process Modeling for Large Simulation Experiments. Journal of Computational and Graphical Statistics, 27(4), 808–821.

Calandriello, D., Carratino, L., Lazaric, A., Valko, M., & Rosasco, L. (2022). Scaling Gaussian Process Optimization by Evaluating a Few Unique Candidates Multiple Times. Proceedings of the 39th International Conference on Machine Learning, PMLR 162, 2523–2541.

Dai, Z., Nguyen, Q. P., Tay, S., Urano, D., Leong, R., Low, B. K. H., & Jaillet, P. (2023). Batch Bayesian Optimization for Replicable Experimental Design. Advances in Neural Information Processing Systems, 36, 36476–36506.

Daulton, S., Balandat, M., & Bakshy, E. (2021). Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement. Advances in Neural Information Processing Systems, 34, 2187–2200.

Daulton, S., Cakmak, S., Balandat, M., Osborne, M. A., Zhou, E., & Bakshy, E. (2022). Robust Multi-objective Bayesian Optimization under Input Noise. Proceedings of the 39th International Conference on Machine Learning, PMLR 162, 4831–4866.

Gramacy, R. B., & Lee, H. K. H. (2008). Bayesian Treed Gaussian Process Models with an Application to Computer Modeling. Journal of the American Statistical Association, 103(483), 1119–1130.

Makarova, A., Usmanova, I., Bogunovic, I., & Krause, A. (2021). Risk-averse Heteroscedastic Bayesian Optimization. Advances in Neural Information Processing Systems, 34, 17235–17245.

Snoek, J., Swersky, K., Zemel, R., & Adams, R. (2014). Input Warping for Bayesian Optimization of Non-stationary Functions. Proceedings of the 31st International Conference on Machine Learning, PMLR 32, 1674–1682.

Wang, X., Hong, L. J., Jiang, Z., & Shen, H. (2025). Gaussian Process-Based Random Search for Continuous Optimization via Simulation. Operations Research, 73(1), 385–407.

Reviewer_xHa6OpenReview ↗

My comments have been addressed

Thank you for the detailed response and for the additional experiments. My review points have been addressed in full. I maintain my acceptance recommendation.

Reviewer_QDvFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a Bayesian optimization framework that learns the mean–variance Pareto frontier of stochastic simulators. It models the correlation between sample mean and variance estimation errors using a multi-output Gaussian process. The proposed HEQHVI acquisition function jointly selects the next design and the number of replications.

Critical strengths

The paper addresses a meaningful problem by considering both mean performance and variability under limited simulation budgets, and extends the proposed method to parallel optimization using minimum energy design (MED).

The proposed framework combines joint uncertainty modeling with adaptive replication allocation. Experiments on synthetic benchmarks and an aero-engine simulator show promising results.

Critical weaknesses
  1. The paper is motivated by the correlation between the estimation errors of the sample mean and variance, but does not directly demonstrate the benefit of modeling such a correlation. Comparisons with independent observation noise, independent GPs, and asymptotic covariance approximations can be included to justify the proposed modeling choices.
  2. The paper does not report the actual number of simulator runs used by adaptive-r HEQHVI or how often each replication count (5, 10, or 15) is selected. Since the results are compared over BO iterations rather than total simulation budgets, it is difficult to assess whether adaptive replication is genuinely more sample-efficient.
  3. Figure 1 looks like a standard Bayesian optimization workflow, which does not clearly show the paper's contributions. The figure also looks somewhat AI generated.
Questions for authors

Please refer to the weaknesses section.

4 comments
AuthorsOpenReview ↗

Part 1: The Response to Weakness 1.

W1: Benefit of Modeling Correlated Observation Errors

Comparisons with independent observation noise, independent GPs, and asymptotic covariance approximations can be included to justify the proposed modeling choices.

Response:

We thank the reviewer for this helpful suggestion. To assess the benefit of modeling shared-replication uncertainty, we conducted an ablation study that holds the HEQHVI acquisition criterion fixed while varying the surrogate specification. We compare the full model with three variants: (i) removing the observation-error cross-covariance while retaining the joint latent GP; (ii) fitting independent GPs to the mean and variance with replication-dependent marginal observation noise; and (iii) replacing the finite-replication covariance with its asymptotic approximation, omitting the finite-rr correction.

The experiment uses the Branin benchmark with input-dependent Gamma noise, for which the sample mean and variance have correlated estimation errors. All variants share the same initialization and use r=2r=2 replications over 125 single-point BO iterations, corresponding to 250 additional simulator calls.

As shown in Table R1, the full model achieves the highest HV and lowest IGD$^+.Removingtheobservation−errorcross−covarianceincreasesIGD. Removing the observation-error cross-covariance increases IGD^+$ from 0.0156240.015624 to 0.0164560.016456, while independent GPs yield 0.0297410.029741. Omitting the finite-replication correction also worsens both metrics. These results provide empirical support for the structured observation model in this setting.

Table R1: Average HV and IGD$^+$ for the surrogate-model comparisons on Branin with Gamma outputs.

MethodHVIGD$^+$
HEQHVI0.8496260.015624
Without observation-error cross-covariance0.8456460.016456
Two independent GPs0.8248550.029741
Asymptotic covariance approximation0.8489140.016104
AuthorsOpenReview ↗

Part 2: Responses to Weaknesses 2 and 3 and Question 1

W2: Replication Allocation and Simulation Budget

The paper does not report the actual number of simulator runs used by adaptive-r HEQHVI or how often each replication count (5, 10, or 15) is selected. Since the results are compared over BO iterations rather than total simulation budgets, it is difficult to assess whether adaptive replication is genuinely more sample-efficient.

Response:

We thank the reviewer for raising this important question about replication allocation and simulation cost.

Our original experiments reported performance over BO iterations because replicate-aware optimization naturally proceeds in sequential decision rounds: at each round, the design and its replication count are selected before the simulator outputs become available. When replications can be executed concurrently, the number of iterations measures the required feedback rounds, a relevant consideration in replicable experimental design (Dai et al., 2023). We agree, however, that iteration-based comparisons alone do not establish efficiency in terms of total simulator calls when replication counts vary.

To address this concern, we have removed the original replication menu {5,10,15} and now jointly optimize the design location and an integer replication count r≥2r\geq2, subject to the remaining simulation budget. The minimum r=2r=2 is required to compute the sample variance. The revised experiments show that r=2r=2 is selected in 93.76% of decisions under Gaussian noise and 90.96% under Gamma noise. The corresponding frequencies for r=3r=3, r=4r=4, and r≥5r\geq5 are (4.22%, 0.84%, 1.18%) and (6.12%, 1.64%, 1.29%), with maximum selected counts of 1010 and 1111, respectively.

This preference for small replication blocks is consistent with the cost-adjusted HEQHVI criterion. Additional replications improve estimation precision but typically offer diminishing marginal benefits, whereas their simulation cost increases linearly. Consequently, the expected Pareto improvement per simulator call often favors smaller rr. Importantly, choosing r=2r=2 does not limit a design to two observations: previously evaluated designs can be revisited, with new replications pooled with existing data. The method can therefore avoid committing excessive resources upfront while allocating additional replications when their expected benefit justifies the cost.

We further compared adaptive replication with fixed r=2r=2 under a budget of B=250B=250 additional simulator calls, excluding the common initialization. For serial HEQHVI, the budget counts all simulator calls. For the parallel HEQHVI-MED comparison, five workers evaluate five designs concurrently, with a common replication count rtr_t within each batch. The parallel-equivalent budget is ∑trt≤B\sum_t r_t\leq B, while the total number of simulator calls is 5∑trt5\sum_t r_t. Thus, adaptive and fixed policies are compared under matched budgets within each execution setting.

As shown in Table R2, adaptive replication achieves higher HV and lower IGD$^+$ in both settings. In particular, IGD$^+$ decreases by approximately 11% for serial HEQHVI and 24% for HEQHVI-MED, alongside modest HV improvements. These results support the benefit of adaptive replication under the specified resource budgets, even though the cost-adjusted criterion selects the minimum replication count in most decisions.

Table R2: Adaptive versus fixed replication under a budget of B=250B=250: total simulator calls for serial policies and per-worker calls for five-worker parallel policies.

MethodAverage HVAverage IGD$^+$
HEQHVI (adaptive-rr)0.8492260.015006
HEQHVI (r=2r=2)0.8455570.016839
HEQHVI-MED (adaptive-rr)0.8752630.003005
HEQHVI-MED (r=2r=2)0.8700280.003958

W3: Presentation of Figure 1

Figure 1 looks like a standard Bayesian optimization workflow, which does not clearly show the paper's contributions. The figure also looks somewhat AI generated.

Response:

We appreciate the reviewer's candid feedback. AI tools assisted with the initial visual design, although the original figure was ultimately drawn by the authors. We agree that its presentation did not sufficiently distinguish our contributions from a standard BO workflow. We have therefore completely redesigned Figure 1 to better highlight the key components of our framework: the joint estimation uncertainty induced by shared replications, its incorporation into the hierarchical multi-output GP, and the joint selection of design locations and replication counts. We have also simplified the layout and graphical elements to improve readability. We hope the revised figure conveys the main contributions more clearly.

Q1

Please refer to the weaknesses section.

Response:

Please refer to our point-by-point responses to the three weaknesses above.

AuthorsOpenReview ↗

Part 3: References

References

Dai, Z., Nguyen, Q. P., Tay, S., Urano, D., Leong, R., Low, B. K. H., & Jaillet, P. (2023). Batch Bayesian Optimization for Replicable Experimental Design. Advances in Neural Information Processing Systems, 36, 36476–36506.

AuthorsOpenReview ↗

Additional Update

In addition to our point-by-point responses to your comments, we have conducted several further experiments to strengthen the empirical evaluation and better establish the contributions of our framework. We would like to briefly highlight these additional efforts:

  1. Comparison with heteroscedastic GP.

    We added a direct comparison with a raw-output heteroscedastic GP under matched simulation budgets, considering both r=1r=1 and r=2r=2. HEQHVI achieved higher HV and lower IGD$^+$, providing further evidence for the effectiveness of our structured mean--variance surrogate.

  2. New high-dimensional benchmarks.

    We introduced two 20-dimensional benchmarks, including a DTLZ2-type problem and a more challenging Rosenbrock-coupled extension with nonlinear interactions among input variables. Across both benchmarks, HEQHVI consistently achieved higher HV and lower IGD$^+$ than NEHVI under matched simulation budgets, providing further evidence of its effectiveness in higher-dimensional optimization.

  3. Non-smooth response surfaces.

    We added a piecewise-linear benchmark with sharp changes in slope to examine exact-design revisiting and Pareto-front recovery. HEQHVI improved upon NEHVI, while HEQHVI-MED also outperformed the covariance-aware qNEHVI baseline.

  4. MED component ablation.

    We conducted a separate experiment isolating the contribution of MED diversification, keeping the surrogate, acquisition criterion, batch sizes, and simulation budgets matched. MED reduced IGD$^+$ by approximately 29.5% relative to acquisition-only batch selection.

Together with the covariance ablations, budget-matched replication comparisons, and presentation improvements made in response to your original comments, these additional results substantially strengthen the empirical support for our methodology and clarify its individual contributions.

We sincerely hope that these additional efforts and results have adequately addressed your concerns and further strengthened the manuscript. We would greatly appreciate your consideration of the revised work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for Reviewers and AC

We thank all reviewers for their valuable comments and constructive suggestions. In response to their feedback, we have revised the manuscript and conducted extensive additional experiments and analyses to further strengthen the empirical evaluation and clarify the contributions of our framework. The main updates are summarized below for the reviewers' and area chair's consideration.

  1. Surrogate-Model Comparisons and Covariance Ablations.

    We conducted ablation studies on the joint GP structure, observation-error cross-covariance, and asymptotic covariance approximation to evaluate their respective contributions. We added a direct comparison with a raw-output heteroscedastic GP under matched simulation budgets, considering both r=1r=1 and r=2r=2. HEQHVI achieved higher HV and lower IGD$^+$, providing further evidence for the effectiveness of our structured mean--variance surrogate.

  2. New High-Dimensional Benchmarks.

    We introduced two 20-dimensional benchmarks, including a DTLZ2-type problem and a more challenging Rosenbrock-coupled extension with nonlinear interactions among input variables. Across both benchmarks, HEQHVI consistently achieved higher HV and lower IGD$^+$ than NEHVI under matched simulation budgets, providing further evidence of its effectiveness in higher-dimensional optimization. HEQHVI--MED also outperformed qNEHVI in the parallel 20-dimensional experiment.

  3. Non-Smooth Response Surfaces.

    We added a piecewise-linear benchmark with sharp changes in slope to examine exact-design revisiting and Pareto-front recovery. HEQHVI improved upon NEHVI, while HEQHVI--MED also outperformed the covariance-aware qNEHVI baseline.

  4. MED Component Ablation.

    We conducted a separate experiment isolating the contribution of MED diversification, keeping the surrogate, acquisition criterion, batch sizes, and simulation budgets matched. MED reduced IGD$^+$ by approximately 29.5% relative to acquisition-only batch selection. We further evaluated MED on the standard multimodal Gramacy--Lee benchmark, achieving statistically significant improvements in both HV and IGD$^+$ over acquisition-only batch selection.

  5. Statistical Validation.

    We added pointwise 95% uncertainty intervals to Figures 2 and 3, together with paired confidence intervals and statistical tests for key ablation studies, providing stronger statistical support for the reported improvements.

  6. Adaptive Replication and Simulation-Budget Efficiency.

    We expanded replication selection from {5,10,15} to any feasible integer r≥2r\geq 2 and examined replication frequencies and exact-design revisits. Under matched simulation budgets, adaptive replication yields higher HV and lower IGD$^+$ in both serial and parallel settings, compared with fixed r=2r=2.

  7. Presentation Improvements.

    We redesigned Figure 1 to better highlight the main methodological contributions, improved the presentation of experimental results, and clarified the block-based replication mechanism.

We have addressed their specific questions and concerns in the corresponding individual responses. We hope these revisions and additional results will assist the area chair in evaluating the contributions and effectiveness of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.