- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies offline data-driven multi-objective optimization, where surrogate models are trained on a fixed dataset and no further real objective evaluations are allowed. The authors argue that surrogate predictions can be overly optimistic in low-density regions and can mislead dominance judgments. They propose EBU-DR, a framework with two components. First, an Empirical-Bayes Uncertainty (EBU) adjustment uses five-fold out-of-fold predictions to estimate objective-wise shrinkage parameters, producing an uncertainty-aware objective fUA-sur that shrinks surrogate predictions toward the offline data mean when predictive signal is weak, and otherwise falls back to the uncertainty estimate. Second, a Dual-Ranking (DR) strategy concatenates the surrogate-predicted objectives and the uncertainty-aware objectives into a 2K-dimensional hybrid vector, performs non-dominated sorting in this hybrid space, and computes crowding distances using only the original K surrogate objectives. The framework is combined with GPR (RBF and Matérn), quantile regression, and BNN surrogates, and evaluated with NSGA-II on 29 benchmark problems, five offline dataset sizes, and several offline MOO baselines. The paper claims more wins than losses against surrogate-only NSGA-II, competitive HV ranks, and the best overall mean rank for GPR (Matérn)+EBU-DR, while acknowledging that improvements are concentrated in GPR configurations and that better optimization performance does not always correspond to lower surrogate prediction error.
Critical strengths
-
Offline data-driven MOO with small datasets is an important setting, and the paper correctly identifies that surrogate-only dominance can be misled by optimistic predictions in low-density regions. The illustrative example in Figure 1 makes the failure mode clear.
-
EBU-DR does not require a new surrogate architecture or a new evolutionary algorithm; it can be plugged into NSGA-II and combined with different surrogate models. This modularity is a genuine practical advantage, and the paper does evaluate four surrogate configurations rather than only GPR.
-
The paper considers 29 benchmark problems, five offline dataset sizes, multiple random permutations and optimization seeds, and a reasonably wide set of offline MOO baselines including probabilistic, deep-learning, and generative methods.
Critical weaknesses
-
EBU is essentially a shrinkage estimator derived under a Gaussian/additive-error model, and DR is a concatenation-based ranking trick. The paper does not provide a guarantee that sorting in the 2K-dimensional hybrid space corrects dominance errors, nor does it characterize when the EBU weight is consistent with the actual surrogate error. For QR and BNN surrogates, the Gaussian likelihood and additive-error assumptions behind Eq. (3)–(4) are not obviously satisfied. The fallback to s_k(x) when the covariance signal is weak is a heuristic, and the paper does not analyze its effect on the ordering of solutions. This matters because the central claim is an uncertainty-aware ranking framework, not merely an empirical heuristic.
-
The empirical evidence is narrower than the framing suggests, and the key comparison is not decisive. The strongest results are obtained with GPR surrogates, especially Matérn; QR and BNN show many ties and limited gains. The ablation compares EBU-DR only against Normal and a fixed-quantile DR variant. It does not compare against simple and standard uncertainty-aware alternatives, such as adding a variance penalty to the objective, LCB/UCB-style optimistic/pessimistic scalarization, or ensemble-variance regularization.
-
PCD and ParetoFlow are generative methods that are not subject to the same 10,000 surrogate-evaluation budget; MOBO is excluded because its results are incomplete. The paper says method-specific training and search mechanisms follow original implementations, but the comparison mixes methods with different computational budgets and different notions of “offline” evaluation.
-
Scalability and computational cost are not adequately analyzed.
Questions for authors
-
Can you add comparisons against simple uncertainty-aware selection baselines, such as adding a variance penalty to the surrogate objective, LCB/UCB-style scalarization, or ensemble-variance regularization, under the same surrogate and evaluation budget?
-
Can you provide a more formal justification for DR? Specifically, under what conditions does non-dominated sorting in the 2K-dimensional hybrid space correct a surrogate-induced dominance error, and when can it introduce a new error?
-
The EBU derivation assumes a Gaussian likelihood and additive error, but QR and BNN surrogates do not naturally satisfy these assumptions. How stable is the estimated c_k and τ_k^2 across folds and dataset sizes, and how often does the covariance-signal condition in Eq. (4) trigger the fallback branch? Can you report these statistics?
-
why is the comparison with Multiple Models on IGD+ not discussed more prominently, given that it outperforms GPR (Matérn)+EBU-DR on that metric?
Summary of the Responses
Dear AC, SAC, PC
Thank you for organizing the comments and responses. We have corrected minor errors, added experiments and analyses, and uploaded the revised manuscript. We summarize the responses here and [Response to Reviewer xxxx], starting with the Introduction section.
1. Introduction
We added a real-world example of a smart building MOP to show that offline data-driven MOO research is valuable for tackling real-world MOPs. [To 71n3]
We clarified why we chose evolutionary methods: MOEAs can explore beyond the offline data distribution (compared with generative methods), and surrogate prediction errors can affect dominance judgments during evolutionary search. [To 71n3]
2. Preliminaries and 3.Method
We have defined the general MOP and surrogate modeling, and shown the difference between offline data-driven and surrogate-assisted MOO in Section 2. We have defined surrogate predictions and uncertainty estimates in Section 3. [To 71n3]
We added detailed explanations of EBU adjustment. [To 357q]
We added an example of when EBU-DR is effective/ineffective, and the limitation that DR may remove a correct dominance relation. [To 357q]
4. Experiment
We clarified that all methods use the same offline datasets and output 100 candidate solutions, but the computational budgets may differ. [To 357q]
We corrected an error to unify the IGD+ reference settings, and included MOBO results. GPR (Mat'ern)+EBU-DR retains the best overall rank. [To Fmrc, 357q]
We added 6 discrete benchmarks, including scientific design tasks. Multiple Models with COMs ranks first, followed by QR+EBU-DR. Across all 35 benchmarks, GPR (Mat'ern)+EBU-DR retains the best overall rank. [To Fmrc]
We added tables of per-benchmark numerical results in the Appendix, following ParetoFlow and PCD. [To Fmrc]
We added an EBU-only variant experiment to examine the contribution of DR. EBU-DR achieves more wins in IGD+ and HV, especially with GPR, but no consistent advantage in MSE. [To oYJC, Fmrc]
We analyzed why EBU is often inactive for QR and BNN. In most cases, no shrinkage occurs, and EBU-DR becomes Normal. [To oYJC, 357q]
We added comparisons with uncertainty-aware baselines: fixed-quantile DR and standard-deviation penalty (fsur,k(x)+βsk(x). EBU-DR achieves more wins in IGD+ and HV but performs worse in MSE. [To 357q].
We added a Dimension Expansion Analysis, examining whether the benefits of DR come from increasing the number of objectives K -> 2K. Compared with other 3 shuffled variants, EBU-DR removes more incorrect (18.71%) than correct (4.10%) dominance relations, but it cannot guarantee that all removed relations are incorrect. Detailed investigation of the EBU-DR mechanism is provided in the Appendix. [To oYJC, 357q]
We added a Computational Cost Analysis, the additional cost of EBU-DR comes from EBU calibration (out-of-fold training and predictions) on the surrogate side and dual-ranking survival selection on the optimization side. One-time EBU calibration substantially increases training time (144%-465%), while DR adds little optimization time (3%). [To 357q]
We added experiments combining EBU-DR with other MOEAs, NSGA-III and SMS-EMOA. EBU-DR shows consistent improvements with NSGA-III relative to NSGA-II, while the results with SMS-EMOA are mixed. [To Fmrc, 357q]
5. Conclusion
We summarized the new experimental results and updated the main findings. We added future work on studying large-scale MOPs, many-objective problems, and additional real-world problems. [To oYJC, Fmrc, 71n3, 357q]
Appendix
We added EBU calibration diagnostics, including shrinkage / no-shrinkage / fallback, parameter stability across folds, and dominance changes caused by fallback. The results show that EBU calbration is surrogate-dependent and fallback may remove correct dominance relations. [To oYJC, 357q]
Appendix added the discrete benchmark settings and results, per-benchmark numerical results, dataset-size ablation results, and surrogate training costs. [Fmrc, 357q]
Thank you for your time!