- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MVBO, a Bayesian Optimization (BO) framework that treats each candidate kernel as a distinct "world." For each world, spectral Nyström features are constructed independently and then concatenated; the coefficients are subsequently estimated jointly via Bayesian linear regression. The resulting prior is mathematically equivalent to a Gaussian Process (GP) with a sum kernel—specifically, the sum of finite-rank kernels corresponding to each world. The authors highlight two key advantages: First, by compressing features on a per-world basis rather than compressing the aggregate sum kernel, the method preserves directions that may have small amplitudes but are nonetheless critical to the objective function. Second, because the feature dimension is fixed, the computational cost for posterior updates is , independent of the number of observations.
Critical strengths
1. The core contributions of the proposed method are honestly isolated within a controlled experimental setting. Controlled experiments along with the counterexample, clearly demonstrate that the quality of kernel approximation is distinct from the preservation of directions useful for the objective function. The authors also transparently report scenarios where their method performs poorly. This level of transparency allows for an accurate assessment of the boundaries of their contributions.
2. The theoretical and computational aspects are presented with great rigor. While the regret analysis itself follows standard procedures, the authors carefully distinguish between confidence widths tailored to different settings while also explicitly noting which assumptions may be violated in the actual implementation of the algorithm. The technical execution is highly accomplished.
Critical weaknesses
1. The claimed problem is not substantiated, and the method does not address it structurally. The only evidence that a single kernel fails on multi-regime structure is a 1D toy example, with no quantitative observation. Sum-kernel GPs, MKL, and additive GPs already combine multiple kernels. MVBO is a sum of stationary kernels with fixed global weights, so it is itself stationary and cannot represent regime transitions. The transitions in the toy example come from the linear kernel, not from the method. Its only difference from an exact sum-kernel GP is per-component truncation, and on standard benchmarks it is on par or worse (Table 5).
2. Performance gains are not reliably attributed, and the efficiency claim conflicts with the BO setting. Fig. 2 omits the most direct baselines (exact sum-kernel GP, aggregate-truncated Nyström). It also fixes baseline hyperparameters from only 10 points and reports no significance tests. The surrogate is severely overconfident (4–7% coverage, Table 8), so the gains may stem from exploration rather than better modeling. No ablation links the gains to regime identification. At , MVBO’s total overhead even exceeds that of an exact sum-kernel GP (Table 9), and all three tasks have cheap evaluations.
Questions for authors
-
Can you compare MVBO against an exact sum-kernel GP and an aggregate-truncated Nyström method with the same rank on the three applications? Given that MKL-BO uses the same kernels and fixed weights, what explains its performance difference from MVBO?
-
Is there quantitative evidence that a single-kernel or sum-kernel GP fails in these applications because of regime structure, for example prediction errors broken down by regime? Since the sum of stationary kernels is itself stationary, what mechanism, other than Nyström anchoring, allows MVBO to capture regime transitions?
-
Can you report the mean and standard deviation of final regret against a standard ARD Matérn GP baseline whose hyperparameters are refitted at each iteration, together with paired significance tests?
6 comments
Response to Reviewer Cp5N
W2. Attribution, exploration, and cheap evaluations
Thank you for asking what drives the gains and how much optimizer time matters when objective evaluations are cheap.
MvBO still reaches lower regret than aggregate Nyström on molecule and DMO when both methods choose queries using only their predicted objective values. We use the same landmarks, feature counts, and ten paired seeds (Table 24). The table gives MvBO minus aggregate Nyström in normalized regret at 200 queries.
| Acquisition | Molecule [95% CI] | Screening [95% CI] | DMO [95% CI] |
|---|---|---|---|
| UCB | −0.027 [−0.044, −0.010] | −0.007 [−0.024, +0.010] | −0.028 [−0.045, −0.011] |
| Mean only | −0.027 [−0.045, −0.009] | −0.005 [−0.024, +0.014] | −0.027 [−0.044, −0.010] |
When we train the models on the same inputs, MvBO has lower mean test RMSE than aggregate Nyström on molecule (0.142 versus 0.168) and DMO (0.121 versus 0.154, Table 23). Exact inference has the lowest RMSE. Coverage stays well below 95% for all three methods. Together, the two tests suggest that the predicted objective values help explain the gains. When we double the default UCB multiplier, the mean regret differences change sign and all paired intervals include zero (Appendix B.2.12).
Cheap evaluations and optimizer cost. At 200 queries, exact inference uses less optimizer time and reaches lower mean regret than default MvBO on all three applications (Tables 1 and 12). The applications use a force field, precomputed docking scores, and an emulator. Optimizer time excludes objective evaluations.
At 8000 Hartmann-6 queries, with the same acquisition optimizer, MvBO takes 1240 ± 62 s and exact inference takes 2680 ± 140 s. MvBO is faster on all ten seeds, with higher mean regret (0.007 versus 0.004). Exact inference is faster at both 200 and 1000 queries (Table 21).
Figure 2 and Table 1 now include both direct baselines. Our Q1 reply gives the paired comparisons, and our Q3 reply gives the independently refitted ARD baseline and paired tests.
Response to Reviewer Cp5N
Q1. Exact inference, equal rank, and MKL-BO
Thank you for requesting direct comparisons with the exact GP and aggregate Nyström, and for asking how MKL-BO differs.
We added the exact sum-kernel GP and an aggregate Nyström baseline with the same feature count as MvBO to Table 1. At 200 queries, the exact GP takes less optimizer time and has lower mean regret than default MvBO on all three tasks. MvBO reaches lower mean regret than aggregate Nyström on molecule and DMO. Both paired intervals favor MvBO.
The table reports final normalized regret as mean ± standard deviation over ten paired seeds. Negative differences favor MvBO.
| Method | Molecule (d=12) | Screening (d=50) | DMO (d=6) |
|---|---|---|---|
| MvBO (separate, r=40/40/50) | 0.075 ± 0.022 | 0.130 ± 0.030 | 0.068 ± 0.020 |
| Aggregate Nyström, 10 landmarks (rank ≤ 10) | 0.158 ± 0.038 | 0.209 ± 0.046 | 0.149 ± 0.035 |
| Aggregate Nyström, matched dimension (rank 40/40/50) | 0.103 ± 0.027 | 0.138 ± 0.033 | 0.096 ± 0.026 |
| Exact sum-kernel GP | 0.072 ± 0.021 | 0.118 ± 0.028 | 0.065 ± 0.019 |
| MKL-BO, matched kernels | 0.079 ± 0.023 | 0.127 ± 0.030 | 0.073 ± 0.021 |
| MvBO − aggregate (matched dim.) [95% CI] | −0.028 [−0.046, −0.011] | −0.008 [−0.026, +0.011] | −0.028 [−0.045, −0.010] |
| MvBO − exact [95% CI] | +0.003 [−0.009, +0.015] | +0.012 [+0.001, +0.023] | +0.003 [−0.008, +0.014] |
Default MvBO uses the ten initial observations as landmarks. To reach the same feature count, the aggregate baseline uses 40 / 40 / 50 landmarks chosen without objective values. Table 18 compares the two methods when they also use the same landmarks.
Why MKL-BO differs. MKL-BO assigns weights that sum to one to the full kernels fitted on the initial data. It keeps those weights fixed. Exact inference adds the same kernels directly. The weights can change the total prior variance and how much each kernel contributes. This can change the predictions and which points the method queries. MvBO also keeps only a subset of the features from each kernel. The MKL-BO comparison thus changes both the weights and the features. Table 18 tests the feature choice directly. If we added an overall scale of and set each weight to for kernels, MKL-BO could recover the exact sum. The reported baseline uses the weights alone.
Response to Reviewer Cp5N
W1 and Q2. Stationarity and regime evidence
Thank you for highlighting what the controlled experiments show and for asking us to clarify what MvBO can learn about changes across the domain.
MvBO uses kernels we choose in advance and has no step that learns where the objective changes behavior or assigns inputs to regions. If all components are stationary, their sum is stationary before truncation. The Nyström approximation can depend on where we place the landmarks. In Figure 4, the kernels we chose account for the oscillations and the linear slope (Section 1).
We defined the input groups before looking at objective values or fitted components. We gave the models the same training and test inputs over ten paired seeds. Table 25 reports MvBO minus aggregate Nyström in normalized root mean squared error (RMSE), with paired 95% confidence intervals.
Exact sum-kernel inference has the lowest mean prediction error in every group. On DMO, MvBO predicts better than aggregate Nyström in all three groups. The groups describe the inputs, so the test leaves open whether changes in the objective across regions explain the differences.