- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why a flexible nonstationary multi-output GP can fit observed data well while making unreliable acquisition decisions. MOLRN-BO combines a regularized shared-spectral covariance with objective-specific RBF residuals, prequential mean correction, tempered covariance inflation, and Pareto-local qLogEHVI search with periodic global rounds. On 12 deterministic bi-objective benchmarks with 20 seeds and 110 total evaluations, it substantially improves over the original MO-LRN and obtains the best average final nHV/nIGD ranks among the evaluated methods. The strongest standard comparator, qLogNParEGO, remains statistically competitive and has better anytime ranks. Ablations distinguish improvements in off-design prediction, the combined search/bias policy, and the prediction-versus-optimization trade-off introduced by hierarchical pooling.
Critical strengths
- The diagnosis of the regression-decision mismatch is useful and supported by decision-relevant measurements. Figure 1 and Table 2a examine off-design errors, prequential uncertainty failures, and HVI optimism rather than treating training RMSE as a sufficient surrogate-selection criterion. The strictly historical candidate residuals also avoid using future evaluations in the mean correction. This is consequential practical guidance for expressive BO surrogates.
- The evaluation includes paired initial designs, 20 seeds, final and anytime metrics, adverse-tail analysis, and multiplicity-adjusted tests. The improvement over original MO-LRN is convincing, and the paper appropriately acknowledges that superiority to qLogNParEGO is not statistically established. It also reports the negative optimization result for hierarchical pooling: bias decreases while final Pareto quality slightly worsens. This makes the empirical account more informative than a collection of favorable aggregate ranks.
Critical weaknesses
- The experiments do not sufficiently isolate the incremental value of the proposed nonstationary surrogate within the complete decision pipeline. Standard stationary baselines do not receive the same local-global search and calibration machinery. Table 2a is a fixed-design predictive diagnostic without RBF residuals, while Table 2b changes localization and mode-specific bias estimation together. Consequently, the largest sequential gain could primarily reflect the acquisition-search policy rather than the shared spectral model. A stationary surrogate under the identical decision policy is an especially consequential missing control, given the close comparison with qLogNParEGO.
- The practical advantage remains uncertain because the method adds numerous fixed regularization/calibration settings and repeated covariance diagnostics, yet no controlled overhead or sensitivity comparison is provided (Sections 4.1 and 5). All optimization tests use a small deterministic, bi-objective, single-candidate budget. That scope is legitimate, but it makes the unmeasured cost and tuning dependence important when the strongest simpler baseline is already competitive. Together with the attribution gap, this supports a weak-rejection recommendation in the current version rather than an objection to the correctness of the kernel construction.
Questions for authors
- Under the identical local-global search, bias correction, and covariance-scaling pipeline, how does a stationary independent GP (and, if already available, a stationary correlated GP) compare with the shared-spectral surrogate on the existing benchmarks? This would materially change the assessment of the surrogate contribution.
- Can the search-localization and mode-specific-bias effects in Table 2b be separated, and can the effect of covariance rescaling be quantified independently? Please distinguish any existing evidence from proposed future experiments.
- What is the per-iteration fitting/calibration/acquisition overhead on a common machine, and how stable are the main conclusions to a small, prespecified perturbation of the fixed settings? Also, does the implemented model use per-mixture B_q as in the main text, or the common B specified in Appendix B.1?
2 comments
Thank you for the clarifications and additional experiments. The comparison with stationary surrogates under the same search and calibration settings addresses my main concern about the contribution of nonstationarity. The computational cost remains a limitation, but the new evidence strengthens the case for acceptance. I would raise my rating.
Response to Reviewer 3
We thank the reviewer for your suggestions.
(W1) We agree that our original ablation studies did not sufficiently disentangle the contributions of different components. Therefore, we conducted additional controlled ablation experiments. As presented in our overall notes, we introduced four additional configurations:
The comparisons A vs. B and C vs. D isolate the effect of covariance rescaling, while A vs. C and B vs. D isolate the effect of the nonstationary kernel under otherwise identical settings.
As shown in our overall notes, A outperforms C on all four metrics, while B outperforms D on three metrics and ties on the remaining one. These results provide direct evidence that the nonstationary surrogate can improve BO performance under the same decision pipeline. Covariance rescaling also improves several metrics, although its effects are not uniformly positive across all metrics.
(W2) We apologize for limiting our original evaluation to relatively low-dimensional, bi-objective problems. We have added two three-objective problems: WFG9 (20D, 3 objectives) and Penicillin (7D, 3 objectives). Our method ranks first and third, respectively, in final nHV on these two problems. Detailed results are provided in our overall notes.
Regarding computational overhead, we acknowledge that our method is substantially more expensive than simpler GP-based BO methods. A typical iteration requires approximately 10–30 minutes, with around 90% of the time spent on MLL optimization and most of the remaining time on acquisition optimization. The main difficulty comes from the complex structure of the nonstationary kernel, which makes both surrogate fitting and acquisition optimization computationally demanding.
Nevertheless, many practical applications of BO, such as hyperparameter optimization and real-world chemical experiments, involve expensive function evaluations. In these settings, the additional computational cost may be acceptable if it leads to better optimization performance or reduces the number of required experiments. We acknowledge that this trade-off depends on the application.
Regarding your questions:
(1) We have provided detailed additional controlled experiments in our overall notes. Configurations C and D use stationary correlated GPs under the same calibration and acquisition optimization procedures, allowing us to isolate the contribution of nonstationarity.
(2) We separately evaluated covariance rescaling in the new experiments, as presented in our overall notes. Regarding local–global acquisition optimization, we found that conventional gradient-based optimization struggled with the complex nonstationary kernel during GP fitting. For example, the negative MLL often decreased only from approximately 300 to above 200, resulting in poorly fitted surrogate models. This motivated our concern that conventional full-domain acquisition optimization could also struggle with the complex posterior landscape. We therefore introduced the Pareto-local/global strategy, which combines heuristic region selection and initialization with gradient-based acquisition optimization. We acknowledge that the individual contribution of this search strategy has not yet been fully separated from mode-specific bias correction.
(3) The computational cost is approximately 10–30 minutes per iteration on a CPU, with MLL optimization accounting for around 90% and acquisition optimization for most of the remaining time. GPU acceleration may reduce this cost, although we have not conducted a controlled comparison.
Regarding hyperparameter sensitivity, our original experiments showed that Q = 2 provided better off-design prediction than Q = 4. Since increasing Q increases the expressive capacity of the MO-LRN kernel, we interpret this as evidence of potential overfitting with larger spectral capacity, which motivated our choice of Q = 2. However, we acknowledge that this does not constitute a comprehensive sensitivity analysis of all regularization and calibration parameters.
Finally, we sincerely apologize for the error in Appendix B.1 (Eq. 11), which was inadvertently retained from an earlier version of our implementation. Eq. 1 and Sec. 4.3 correctly describe the implemented model: the spectral kernel parameters are shared across outputs, but the coregionalization matrices are learned separately for each spectral mixture. We will correct Eq. 11 and the corresponding description in the revised manuscript.