- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This article proposes RESCUE, a constrained multi-objective and multi fidelity Bayesian optimization method. It learns a structural causal model from observed data or a given DAG using PC or DirectLiNGAM, and uses the intervention mean and variance obtained by do calculation as the prior mean and low rank covariance term of the multi output multi fidelity Gaussian process. The collection function adopts cost normalized causal information hypervolume knowledge gradient (VoCI HVKG), where each output reliability weight is estimated by leaving one prediction likelihood. The theory provides an arbitrary time hypervolume regret bound that does not increase with the magnitude of cross fidelity deviation when the causal prior residuals are bounded; The variant with forced target fidelity exploration further provides a sublinear convergence rate. The experiment was compared with qEHVI, MOMF, and MF HVKG on three synthesis problems and two real tasks (robot navigation, constrained XGBoost hyperparameter optimization).
Critical strengths
Introducing learnable reliability weights into the hypervolume knowledge gradient is a clean and practical design that enables the optimizer to automatically reduce weights without discarding multi fidelity structures when causal priors are lost. The improvement in sample efficiency relative to the strongest baseline HVKG on the synthetic benchmark is consistent among independent seeds, and is still partially retained in ablation after removing causal terms, indicating that modeling selection has contributed at least some of the gain.
Critical weaknesses
This paper should be rejected.
W1. Causal discovery is considered a reliable black box, but relies on PC and DirectLiNGAM under the assumption of causal sufficiency. The robot task (25 dimensional continuous parameters) only uses 1000 observation samples, and the frequency of edge occurrence fluctuates greatly during 20 runs, with a large number of edge occurrence rates below 50%. Without conducting sensitivity analysis of significance level alpha or alternative discovery algorithms, the causal prior quality supporting theory and experiments cannot be verified.
W2. The arbitrary time bound of Theorem 5.8 is represented by the posterior standard deviation of the implementation at the target fidelity. The main algorithm (used in the experiment) does not have a mandatory exploration step and cannot guarantee the shrinkage of the quantity, so it only inherits a non vanishing residual that depends on the causal prior quality. The mismatch constant xi is assumed to be bounded but has never been estimated or defined from data, making it theoretically impossible to operate.
W3. The robot navigation agent itself is relatively weak. The energy per meter test set R ² is only 0.21, and the average R ² of the four indicators is only 0.61. All methods (including baselines) are evaluated on this proxy, and systematic biases can directly affect sorting. The paper claims that residuals propagate uniformly, but does not provide quantitative verification of sorting stability under proxy noise.
W4. The absolute gain on two real problems is limited. The AUR improvement relative to HVKG is about 5% in robot navigation, but statistically insignificant in constrained XGBoost. HVKG has shared the same multi output multi fidelity kernel (only lacking causal terms), but once the acquisition function itself is complex enough, the incremental value of the causal mechanism appears limited.
W5. Intentional destruction of observation data (result - configuration for random shuffling) can still achieve comparable performance to HVKG in ablation. This weakens the core proposition that 'causal priors are the source of advantage'; A more likely explanation is that the additional flexibility or reliability of the mean function itself is at work.
W6. The computational cost is underestimated. Causal discovery, do algorithm agent training, and reliability mixing are executed every round or iteration; The wall clock only reports the collection steps. The cubic complexity of multiple output Gram matrices in high-dimensional scenarios, coupled with repeated residual likelihood calculations, makes the relatively simpler multi fidelity baseline of this method impractical.
W7. There are multiple confounding factors in the experimental design. All methods share the same initial design; The robot cost model itself is a GP fitted on the same proxy data, forming a cyclic dependency; Only investigate a single YAHPO instance; Not compared with single fidelity methods under intentional mismatch fidelity. These choices are difficult to isolate the contribution of causal components.
W8. Several values exhibit suspicious regularity (initial cost intervals between seeds are completely consistent, AUR values are close to arithmetic sequences). Although it cannot be concluded that there is fraud, it reduces the credibility of the statistical report and requires the disclosure of the original trajectories for each sub track.
Questions for authors
Can a quantitative bound or data-driven estimate of the causal residual xi be provided on each benchmark, and how the hypervolume regret changes when the residual is artificially amplified?
Why was this scheduling never used in the experimental section of the main theoretical rate dependent forced exploration (Algorithm 3)?
What is the rank correlation of the recommended Pareto set when the robot agent is replaced by a more noisy or differently trained model?
3 comments
(2/3)
W3 and Q3. We added this test to the revision (Appendix A.7, "Ranking Stability", Table 11). We keep the runs of Table 6 and recompute the regret of every recommended Pareto set on two kinds of alternative benchmark models. Five retrained models are trained on the same 300 robot runs with other random states. These are the differently trained models of Q3. Five biased models add a smooth random bias to each output of the benchmark model, sized to its test RMSE. These are the more noisy models of Q3. They treat the full test error as systematic, which is the scenario of W3.
| Benchmark model | Spearman | Kendall | AUR qEHVI | AUR MOMF | AUR HVKG | AUR RESCUE |
|---|---|---|---|---|---|---|
| Benchmark | – | – | −4025.31 | −3961.45 | −4019.72 | −4237.67 |
| Retrained | 0.975 ± 0.004 | 0.867 ± 0.183 | −3812.50 ± 106.03 | −3762.12 ± 101.22 | −3811.39 ± 109.56 | −4040.53 ± 97.12 |
| Biased | 0.791 ± 0.155 | 0.800 ± 0.183 | −3313.28 ± 30.11 | −3267.80 ± 15.70 | −3323.58 ± 42.80 | −3563.93 ± 276.87 |
Spearman is the rank correlation of the AUR of all runs with the benchmark model, and Kendall is the rank correlation of the ranking of the methods. RESCUE has the lowest AUR on every one of the ten alternative models. The only change in the ranking is between qEHVI and HVKG, which differ by 0.1% in Table 6. We revised the statement on surrogate error in Appendix A.7. It now refers to Table 11, which shows that the ranking of the methods is stable under alternative benchmark models.
W4. The gain over HVKG on robot navigation is not limited. The 5% figure is a percentage of AUR, which integrates the log HV regret, so it is not a percentage of regret. RESCUE has about 18% lower HV regret than HVKG on average over the budget, a significant and large effect (Table 6). On constrained XGBoost HPO, RESCUE also has lower AUR than HVKG, although the difference is not significant (Table 6). To isolate the causal mechanism itself, our ablation sets within RESCUE and keeps every other component fixed (Section 6.2). Removing the causal prior increases the regret of RESCUE on every problem, including both real ones (Figures 5, 9 and 13).
W5. We believe the review reads this ablation in reverse. As Section 6.2 and Appendix A.5 state, when the causal information is destroyed, RESCUE falls to the level of HVKG, and VoCI-HVKG lowers its trust in the causal prior (Figure 5). Thus the advantage of RESCUE comes from the causal prior. This ablation keeps the causal prior mean, the causal covariance term and the causal trust of RESCUE, and shuffles the outcomes in to obtain a misspecified causal prior (Table 7). Its regret is higher than that of RESCUE on every problem (Figures 5 and 9). So the gain comes from the causal relationships that RESCUE learns from , which the shuffling destroys. Falling back toward HVKG is the intended robustness, as Section 6.2 states. With low trust, RESCUE adapts accordingly and relies on , the MF-CGP without the causal prior (Eq. 2). The review's strengths describe this same mechanism.
W6. We believe the review misreads when each step runs and what the runtime figure reports. As Algorithm 1 and Appendix A.4 state, the CPM is constructed once, in Line 2, before the optimization loop begins in Line 3. This covers causal discovery and the proxy network distilled from the CPM (Appendix A.2). We now state this explicitly in the Discussion (Section 7) of the revision. The reliability mixture is part of the VoCI-HVKG acquisition (Section 4.2), which Line 6 optimizes inside the loop. Figure 8 reports "the mean wall-clock seconds per iteration" of every component and not only the acquisition, and notes that "the CPM of RESCUE is learned once per run". It lists CPM learning, GP fitting, the reliability mixture and acquisition optimization separately. The cubic complexity is not specific to RESCUE. Section 4 states the complexity of MF-CGP inference. The baselines also use a multi-output GP over all outputs (Appendix A.2).
(3/3)
W7.
- Shared initial design. Sharing the initial design is not a confound. It is what makes the comparison fair. As Section 6.1 and Appendix A.1 state, all methods start from the same initial trials in each run, so differences in results come from the methods.
- Robot cost model. The cost model does not form a cyclic dependency. Modeling an unknown evaluation cost with a GP on the log cost and weighting the acquisition by its inverse is standard practice (Snoek et al., NeurIPS 2012, Section 3.2). BoTorch provides this as its cost-aware utility, which we use unchanged (Balandat et al., NeurIPS 2020). Appendix A.3.2 states the setup. The cost of a robot run is known only after the run, so a GP on the log cost predicts it for the acquisition. After each evaluation, the budget uses its true cost in place of the prediction. We fit this GP once on evaluations of the benchmark model before optimization starts. We never refit it during optimization. No method's queries change its cost predictions, so no loop exists. RESCUE and HVKG use the same cost GP. We added this detail to Appendix A.3.2 in the revision.
- Single YAHPO instance. The number of YAHPO instances does not change how we isolate the causal contribution. Our ablation isolates it within each problem by setting and keeping every other component fixed (see W4). Constrained XGBoost HPO is one of five problems, and RESCUE has lower AUR than HVKG on all five (Table 6).
- Single-fidelity methods. Appendix A.2 already explains why a single-fidelity method reveals little here. Such a method queries only the target fidelity and is unaffected by misleading lower fidelities by construction. Each of its queries on Branin-Currin and Park costs about 121 times a query at the lowest fidelity. This cost ratio alone determines its regret over cost from the first query. For this reason, we add the fidelity to the design variables of qEHVI. Corollary 5.9 addresses robustness to mismatched fidelities directly. Its bound for RESCUE does not depend on the magnitude of the cross-fidelity bias.
W8.
- Initial cost. The initial costs differ between seeds. Table 2 reports initial cost as a range over 20 seeds: 6.5–13.1 on Branin-Currin, 11.9–20.8 on Park, 4.7–6.9 on Synthetic SCM, 1440–2152 on robot navigation, and 66–120 on constrained XGBoost HPO. Costs match only across methods within a seed, because all methods start from the same initial trials in each run, as Section 6.1 and Appendix A.1 state. This is what makes the comparison fair: every method begins from the same data and the same spent cost, so differences in results come from the methods.
- AUR values. The AUR values of Table 6 do not form arithmetic sequences, within a problem or across problems. For example, the three baseline AUR values for robot navigation are −3961.45, −4019.72 and −4025.31, which differ by 58.27 and 5.59. The only near-even values in Table 6 are the gains over MOMF on the three synthetic problems (67.31, 112.77, 158.66). Each gain is computed from the AUR of RESCUE and of MOMF on the same problem as (RESCUE − MOMF) / |MOMF|. On Park, for example, it is (22.11 − 10.39) / 10.39 ≈ 112.8%.
- Trajectories. The per-run trajectories are already public. The code linked in our Reproducibility Statement includes the run histories of every method and seed (
experiments/results/data/run_histories) and the recommended Pareto sets (experiments/results/data/fronts).
Q1. We added this estimate to Appendix A.5 in the revision (Eqs. 7 and 8, Table 8). For each run, we evaluate the noiseless outputs and the causal prior at 10,000 Sobol configuration–fidelity pairs. The estimate is the largest absolute difference between them, divided by the standard deviation of each output. This follows Assumption 5.2 with , which gives an upper bound on . To amplify the residual, we learn the CPM from 10% of and from shuffled outcomes. Both increase and the AUR on all five problems, consistent with the regret bound of Theorem 5.8.
Q2. The main theoretical result is Theorem 5.8, which holds for Algorithm 1 as used in the experiments. The sublinear rate is a secondary result for a variant (Appendix B.6), and we use Algorithm 3 only to prove it. Its forced queries incur the cost of the target fidelity independent of their expected hypervolume gain. VoCI-HVKG weighs this gain against the cost of each query (see W2). We added this to the Discussion (Section 7) in the revision.
We thank the reviewer for the detailed review, and for recognizing the causal trust as a clean and practical design. Several concerns rest on statements that the paper contradicts, in particular on the runtime (W6), the cost model (W7) and the reported values (W8). We address each point below with references to the relevant sections, tables and figures, so that each clarification can be checked directly. Table, figure and equation numbers refer to the revised version. These clarifications do not fit within the character limit of a single comment, and our response spans three comments.
(1/3)
W1.
W2.