acceptodds
Under review as a conference paper at ICLR 2027

Multiverse Bayesian Optimization with Joint Finite-Feature Surrogates

Abstract

Bayesian optimization (BO) typically relies on a Gaussian process (GP) surrogate whose performance depends on the choice of kernel. When an objective contains several patterns, committing to one kernel can miss useful structure and direct acquisition toward uninformative regions. We propose Multiverse Bayesian Optimization (MvBO), a novel BO framework that represents candidate kernels as complementary worlds and learns their contributions jointly in a compact feature space. By retaining features from each world before combining them, the surrogate can preserve structure that compressing the kernel sum would discard. The resulting Bayesian linear regression provides a Gaussian posterior that captures dependence between worlds, supports standard acquisition functions, and requires time per update for a fixed total feature dimension . For upper confidence bound (UCB) acquisition with fixed features, we establish best-query regret under a well-specified prior on a finite domain without repeated queries, and extend this guarantee to account for the error introduced by the finite-feature approximation. We also study when preserving each world's features helps and quantify the accuracy and computational trade-offs of the proposed representation. Across three emerging scientific and AI design tasks, including molecular conformer search, virtual screening for drug discovery, and LLM data-mixture optimization, MvBO-UCB reduces final normalized simple regret by approximately , , and , respectively, relative to the strongest competing BO baseline.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimization, Gaussian processes, kernel learning, random features, black-box optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Cp5NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MVBO, a Bayesian Optimization (BO) framework that treats each candidate kernel as a distinct "world." For each world, spectral Nyström features are constructed independently and then concatenated; the coefficients are subsequently estimated jointly via Bayesian linear regression. The resulting prior is mathematically equivalent to a Gaussian Process (GP) with a sum kernel—specifically, the sum of finite-rank kernels corresponding to each world. The authors highlight two key advantages: First, by compressing features on a per-world basis rather than compressing the aggregate sum kernel, the method preserves directions that may have small amplitudes but are nonetheless critical to the objective function. Second, because the feature dimension rr is fixed, the computational cost for posterior updates is O(r2)O(r^2), independent of the number of observations.

Critical strengths

1. The core contributions of the proposed method are honestly isolated within a controlled experimental setting. Controlled experiments along with the counterexample, clearly demonstrate that the quality of kernel approximation is distinct from the preservation of directions useful for the objective function. The authors also transparently report scenarios where their method performs poorly. This level of transparency allows for an accurate assessment of the boundaries of their contributions.

2. The theoretical and computational aspects are presented with great rigor. While the regret analysis itself follows standard procedures, the authors carefully distinguish between confidence widths tailored to different settings while also explicitly noting which assumptions may be violated in the actual implementation of the algorithm. The technical execution is highly accomplished.

Critical weaknesses

1. The claimed problem is not substantiated, and the method does not address it structurally. The only evidence that a single kernel fails on multi-regime structure is a 1D toy example, with no quantitative observation. Sum-kernel GPs, MKL, and additive GPs already combine multiple kernels. MVBO is a sum of stationary kernels with fixed global weights, so it is itself stationary and cannot represent regime transitions. The transitions in the toy example come from the linear kernel, not from the method. Its only difference from an exact sum-kernel GP is per-component truncation, and on standard benchmarks it is on par or worse (Table 5).

2. Performance gains are not reliably attributed, and the efficiency claim conflicts with the BO setting. Fig. 2 omits the most direct baselines (exact sum-kernel GP, aggregate-truncated Nyström). It also fixes baseline hyperparameters from only 10 points and reports no significance tests. The surrogate is severely overconfident (4–7% coverage, Table 8), so the gains may stem from exploration rather than better modeling. No ablation links the gains to regime identification. At n=200n=200, MVBO’s total overhead even exceeds that of an exact sum-kernel GP (Table 9), and all three tasks have cheap evaluations.

Questions for authors
  1. Can you compare MVBO against an exact sum-kernel GP and an aggregate-truncated Nyström method with the same rank rr on the three applications? Given that MKL-BO uses the same kernels and fixed weights, what explains its performance difference from MVBO?

  2. Is there quantitative evidence that a single-kernel or sum-kernel GP fails in these applications because of regime structure, for example prediction errors broken down by regime? Since the sum of stationary kernels is itself stationary, what mechanism, other than Nyström anchoring, allows MVBO to capture regime transitions?

  3. Can you report the mean and standard deviation of final regret against a standard ARD Matérn GP baseline whose hyperparameters are refitted at each iteration, together with paired significance tests?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Cp5N

W1 and Q2. Stationarity and regime evidence

Thank you for highlighting what the controlled experiments show and for asking us to clarify what MvBO can learn about changes across the domain.

MvBO uses kernels we choose in advance and has no step that learns where the objective changes behavior or assigns inputs to regions. If all components are stationary, their sum is stationary before truncation. The Nyström approximation can depend on where we place the landmarks. In Figure 4, the kernels we chose account for the oscillations and the linear slope (Section 1).

We defined the input groups before looking at objective values or fitted components. We gave the models the same training and test inputs over ten paired seeds. Table 25 reports MvBO minus aggregate Nyström in normalized root mean squared error (RMSE), with paired 95% confidence intervals.

  • Figure 4 toy objective. On the oscillatory half, the difference is +0.010 [−0.003, +0.023]. On the linear half, it is −0.043 [−0.060, −0.025]. These are normalized test errors over ten seeds. The unnormalized RMSE of 0.059 in Appendix B.2.6 describes the displayed fit.
  • LLM data-mixture optimization (DMO). We group test mixtures by the skill associated with the largest mixture weight across both training stages. The differences are −0.033 [−0.049, −0.016] for instruction following, −0.034 [−0.052, −0.015] for math, and −0.028 [−0.046, −0.011] for code.

Exact sum-kernel inference has the lowest mean prediction error in every group. On DMO, MvBO predicts better than aggregate Nyström in all three groups. The groups describe the inputs, so the test leaves open whether changes in the objective across regions explain the differences.

AuthorsOpenReview ↗

Response to Reviewer Cp5N

W2. Attribution, exploration, and cheap evaluations

Thank you for asking what drives the gains and how much optimizer time matters when objective evaluations are cheap.

MvBO still reaches lower regret than aggregate Nyström on molecule and DMO when both methods choose queries using only their predicted objective values. We use the same landmarks, feature counts, and ten paired seeds (Table 24). The table gives MvBO minus aggregate Nyström in normalized regret at 200 queries.

AcquisitionMolecule [95% CI]Screening [95% CI]DMO [95% CI]
UCB−0.027 [−0.044, −0.010]−0.007 [−0.024, +0.010]−0.028 [−0.045, −0.011]
Mean only−0.027 [−0.045, −0.009]−0.005 [−0.024, +0.014]−0.027 [−0.044, −0.010]

When we train the models on the same inputs, MvBO has lower mean test RMSE than aggregate Nyström on molecule (0.142 versus 0.168) and DMO (0.121 versus 0.154, Table 23). Exact inference has the lowest RMSE. Coverage stays well below 95% for all three methods. Together, the two tests suggest that the predicted objective values help explain the gains. When we double the default UCB multiplier, the mean regret differences change sign and all paired intervals include zero (Appendix B.2.12).

Cheap evaluations and optimizer cost. At 200 queries, exact inference uses less optimizer time and reaches lower mean regret than default MvBO on all three applications (Tables 1 and 12). The applications use a force field, precomputed docking scores, and an emulator. Optimizer time excludes objective evaluations.

At 8000 Hartmann-6 queries, with the same acquisition optimizer, MvBO takes 1240 ± 62 s and exact inference takes 2680 ± 140 s. MvBO is faster on all ten seeds, with higher mean regret (0.007 versus 0.004). Exact inference is faster at both 200 and 1000 queries (Table 21).

Figure 2 and Table 1 now include both direct baselines. Our Q1 reply gives the paired comparisons, and our Q3 reply gives the independently refitted ARD baseline and paired tests.

AuthorsOpenReview ↗

Response to Reviewer Cp5N

Q1. Exact inference, equal rank, and MKL-BO

Thank you for requesting direct comparisons with the exact GP and aggregate Nyström, and for asking how MKL-BO differs.

We added the exact sum-kernel GP and an aggregate Nyström baseline with the same feature count as MvBO to Table 1. At 200 queries, the exact GP takes less optimizer time and has lower mean regret than default MvBO on all three tasks. MvBO reaches lower mean regret than aggregate Nyström on molecule and DMO. Both paired intervals favor MvBO.

The table reports final normalized regret as mean ± standard deviation over ten paired seeds. Negative differences favor MvBO.

MethodMolecule (d=12)Screening (d=50)DMO (d=6)
MvBO (separate, r=40/40/50)0.075 ± 0.0220.130 ± 0.0300.068 ± 0.020
Aggregate Nyström, 10 landmarks (rank ≤ 10)0.158 ± 0.0380.209 ± 0.0460.149 ± 0.035
Aggregate Nyström, matched dimension (rank 40/40/50)0.103 ± 0.0270.138 ± 0.0330.096 ± 0.026
Exact sum-kernel GP0.072 ± 0.0210.118 ± 0.0280.065 ± 0.019
MKL-BO, matched kernels0.079 ± 0.0230.127 ± 0.0300.073 ± 0.021
MvBO − aggregate (matched dim.) [95% CI]−0.028 [−0.046, −0.011]−0.008 [−0.026, +0.011]−0.028 [−0.045, −0.010]
MvBO − exact [95% CI]+0.003 [−0.009, +0.015]+0.012 [+0.001, +0.023]+0.003 [−0.008, +0.014]

Default MvBO uses the ten initial observations as landmarks. To reach the same feature count, the aggregate baseline uses 40 / 40 / 50 landmarks chosen without objective values. Table 18 compares the two methods when they also use the same landmarks.

Why MKL-BO differs. MKL-BO assigns weights that sum to one to the full kernels fitted on the initial data. It keeps those weights fixed. Exact inference adds the same kernels directly. The weights can change the total prior variance and how much each kernel contributes. This can change the predictions and which points the method queries. MvBO also keeps only a subset of the features from each kernel. The MKL-BO comparison thus changes both the weights and the features. Table 18 tests the feature choice directly. If we added an overall scale of MM and set each weight to 1/M1/M for MM kernels, MKL-BO could recover the exact sum. The reported baseline uses the weights alone.

Reviewer_FyNSOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on multi-kernel based Bayesian optimization (BO), and introduces Multiverse Bayesian Optimization (MvBO), a BO surrogate approach that constructs finite-dimensional feature representations for distinct available kernels individually from a finite suite of kernels, concatenates them, and performs joint Bayesian linear regression. The resultant posterior is used for the design of a UCB-based acquisition function and posterior updates permit O(r^2) incremental updates for (overall) feature dimension r. In addition to empirical evaluation on three applications (molecular conformer search, virtual screening, and LLM data-mixture optimization) the paper provides theoretical guarantees through finite-feature regret analysis and approximation-error extensions.

Critical strengths

STRENGTHS

S1. The idea of properly combining multiple available kernels for BO is intuitive and the core ideas and concepts of the paper are clear and well-presented.

S2. The paper is well-structured and easy to follow.

S3. The numerical evaluation is carried out in diverse application domains that are interesting and practically relevant.

Critical weaknesses

W1. In my point of view, the methodological contribution of the paper over existing literature related to sum-kernel/additive GPs is not clear: As it appears, the proposed MvBO is essentially a finite-rank sum-kernel GP with already known/established properties (e.g., joint condition on the observed sum and inference, cross-component posterior covariance). To my understanding, MVBO just hinges on per-kernel Nyström truncation before summing and then using Bayesian linear regression. Thus, I believe it is important for the paper to clarify and clearly state the contributions and significance of such an approach over related existing literature to additive GPs etc.

W2. The experimental setting may be unfair for the baselines:

  • To my understanding, the kernel hyperparameters are fit once on 10 points and then remain frozen for the matched GP-based baselines but the standard (and a more fair to compare) approach would be to refit in an online fashion.
  • GP-UCB (SE) and GP-UCB(M52) get the parameters taken from the joint-sum kernel fit, whereas independent tuning/fitting would be more appropriate.
  • RAMBO uses EI acquisition criterion whereas a more direct comparison would be to use the UCB criterion as the other methods.
  • The same holds for the multikernel EGP-TS approach where a UCB instead of TS criterion would be a more direct and reasonable comparison.

W3. The numerical evaluation lacks comparison with important relevant baselines:

  • Budgeted Kernelized Bandits (BKB) [1]: BKB utilizes adaptive low-rank approximation for GP-UCB while preserving uncertainty estimates. A comparison with this would assess MvBO’s accuracy–computation trade-off against an established scalable BO method (i.e., applying BKB to the same sum kernel and comparing it with MvBO would strengthen the practical significance of the latter).

  • Vanilla BO with dimension-scaled lengthscale priors [2]: This constitutes a relevant single-kernel baseline for comparison, particularly on the 50-dimensional screening task. This comparison would clarify whether the gains still hold against a single-kernel GP with dimension-scaled lengthscale priors, which are not used by the current GP-UCB baselines.

References

[1] Calandriello, D., Carratino, L., Lazaric, A., Valko, M., & Rosasco, L. (2019). Gaussian process optimization with adaptive sketching: Scalable and no regret. Proceedings of the Thirty-Second Conference on Learning Theory, 533–557.

[2] Hvarfner, C., Hellsten, E. O., & Nardi, L. (2024). Vanilla Bayesian optimization performs great in high dimensions. Proceedings of the 41st International Conference on Machine Learning, 20793–20817.

Questions for authors

Please refer to the weaknesses above. In addition:

Q1. Table 8 in the supplementary reports very low empirical coverage of nominal 95% posterior intervals on several tasks. What explains this undercoverage, and how does it affect acquisition decisions?

Q2. The application experiments use the ten initial observations as Nyström landmarks. How would increasing the number of landmarks or changing their placement affect performance?

8 comments
AuthorsOpenReview ↗

Response to Reviewer FyNS

W1. Contribution beyond standard additive GPs

Thank you for highlighting the clarity of the method and the range of applications, and for asking us to clarify what MvBO adds to additive GPs.

We test what happens when a sum-kernel GP chooses features separately for each kernel within a fixed total budget. MvBO combines those features in Equation (10). It uses standard additive-GP equations to update the posterior and compute covariance between components (Sections 3.2 and 5).

Appendix A.1.5 illustrates this with K1=diag(4,3,0)K_1=\mathrm{diag}(4,3,0) and K2=diag(0,0,1)K_2=\mathrm{diag}(0,0,1). If we add the kernels first and keep their two strongest features, we keep coordinates 1 and 2. If we keep one feature from each kernel, we keep coordinates 1 and 3. The first choice discards less kernel trace (1 versus 3). If the objective depends on coordinate 3, only the second choice keeps that feature.

In Table 6, separate allocation lowers the final gap on the minority-signal control from 0.085 to 0.006 over 20 paired seeds. The paired difference is −0.079 [−0.102, −0.054]. Aggregate compression performs better on Hartmann-6. Table 18 extends the comparison to the applications with shared landmarks and equal feature counts. The paired intervals favor separate allocation on molecule and DMO.

AuthorsOpenReview ↗

Response to Reviewer FyNS

W2. Baseline fitting and acquisition parity

Thank you for suggesting that we fit each baseline independently and compare methods that use the same acquisition rule.

We now fit the GP baselines separately at every iteration, and we test RAMBO and EGP with UCB.

Fitting each baseline separately. In the submitted paper, the SE and Matérn-5/2 GP-UCB baselines reused the initial fit of the sum kernel. We now fit the hyperparameters of each baseline separately at every iteration by maximum marginal likelihood. We do the same for the standard ARD Matérn-5/2 GP. All methods start with the same observations and use the same seeds and query budgets (Table 1, Appendix B.2.4).

At 200 queries, the SE baseline reaches mean regret of 0.121 / 0.162 / 0.098 and the Matérn-5/2 baseline reaches 0.108 / 0.141 / 0.086 (molecule / screening / DMO). Default MvBO reaches 0.075 / 0.130 / 0.068. Against the stronger ARD refit, the paired intervals favor MvBO on molecule and include zero on screening and DMO (Table 15). Table 22 also lets both compressed methods refit every 20 observations. The shared-landmark advantages on molecule and DMO persist under this schedule.

Using the same rule to choose queries. We switch RAMBO from expected improvement (EI) to upper confidence bound (UCB). We also switch EGP from Thompson sampling (TS) to UCB. We keep the published hyperparameter updates for both methods (Table 13). The table gives mean ± standard deviation of final normalized regret at 200 queries over ten paired seeds.

Method (acquisition)MoleculeScreeningDMO
MvBO (UCB, frozen pilot)0.075 ± 0.0220.130 ± 0.0300.068 ± 0.020
RAMBO (EI, published)0.130 ± 0.0280.190 ± 0.0320.110 ± 0.026
RAMBO (UCB, ablation)0.114 ± 0.0260.135 ± 0.0310.091 ± 0.024
EGP-TS (TS, published)0.300 ± 0.0350.400 ± 0.0450.220 ± 0.035
EGP-UCB (ablation)0.228 ± 0.0360.301 ± 0.0410.168 ± 0.032

Both methods improve under UCB. RAMBO's mean gap to MvBO on screening shrinks from 0.060 to 0.005. The 0.005 gap has no paired interval. In Table 1, exact inference, BKB, and the dimension-scaled prior all have lower mean screening regret than MvBO. The revised abstract reports the shared-landmark comparison.

AuthorsOpenReview ↗

Response to Reviewer FyNS

W3(a). BKB accuracy and computation

Thank you for suggesting BKB as a comparison for accuracy and runtime.

On the same sum kernel, MvBO uses 55–56% less optimizer time than BKB, and all three paired regret intervals include zero. BKB covers more queried values. Exact inference is faster than both at 200 queries (Tables 1 and 14).

BKB updates its low-rank approximation as it collects observations. MvBO builds a separate feature map for each kernel from the initial observations and keeps those maps fixed. Both methods use the same initial designs and ten paired seeds. The table gives means ± standard deviations. Coverage is the share of queried objective values that fall inside nominal 95% latent intervals. Optimizer time excludes the time spent evaluating the objective.

MethodFinal regret (Molecule / Screening / DMO)Coverage %Optimizer time (s)
MvBO0.075 ± 0.022 / 0.130 ± 0.030 / 0.068 ± 0.02036 ± 6 / 27 ± 6 / 34 ± 632.6 ± 2.8 / 58.4 ± 4.6 / 21.8 ± 1.7
BKB (same sum kernel)0.071 ± 0.022 / 0.124 ± 0.029 / 0.070 ± 0.02140 ± 6 / 31 ± 6 / 38 ± 674 ± 6.1 / 131 ± 11 / 49 ± 4.4
MvBO − BKB [95% CI]+0.004 [−0.011, +0.019] / +0.006 [−0.013, +0.025] / −0.002 [−0.015, +0.011]
Reviewer_oE45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using the well-established ideas from GP modelling of additive kernels and basis truncation to build GP models within Bayesian optimisation loops. A large component of the paper is theoretical regret-based analysis of this algorithm, and empirical justification is provided across a range of benchmark problems.

Critical strengths

None

Critical weaknesses
  1. Limited novelty. The paper essentially performs BO with a simple Nyström approximation of a sum of kernels, ideas found in canonical GP textbooks from many years ago. Large sections re-derive the weight-space view of GPs and Bayesian linear regression. The work feels about ten years late, and I’m not convinced it belongs at ICLR 2027.
  2. Relationship to additive GPs. The paper tries to distance itself from existing additive GP kernels by saying their components “index subsets of inputs.” In reality, the mathematical framework is the same.
  3. Computational claims. The claimed low computational cost comes directly from using a truncated basis in the GP, again a long-established idea.
  4. Fixed hyperparameters. The kernel hyperparameters are fixed when the decomposition is built and then held fixed for the whole optimisation. This is a highly unrealistic BO setting and will certainly lead to poor performance of the baselines, since it is well known that hyperparameters should adapt to incoming data for efficient optimisation. This dramatically reduces the real-world applicability of the method and its supporting theory.
  5. Contribution and venue. The paper may make an interesting theoretical contribution, but it does not make a practical one. Another venue may be more appropriate.
Questions for authors
  1. Can you explain why your theoretical results are interesting in their own right? From my perspective, they seem like minor additions to existing regret results for kernels built from finite basis functions (Srinivas et al., 2010). I should note that I’m not an expert in regret theory; my background is practical BO.
  2. Why is the additive (sum) kernel GP without truncation not included in the main results? Surely this is a key comparison.
8 comments
AuthorsOpenReview ↗

Response to Reviewer oE45

W1 and W2. Novelty and additive GPs

Thank you for asking us to clarify the contribution and its relation to additive GPs.

We use standard additive-GP inference to test how the choice of features affects optimization at a fixed feature budget. We agree that Nyström features and the weight-space posterior are established. Section 5 now states that additive components can share input coordinates. Section 3 gives the standard posterior equations to specify the implementation.

A small eigenvalue means a feature explains little variation in the kernel. The feature can still matter for finding the optimum. If we combine the kernels before choosing features, we can lose that feature even when we approximate the combined kernel more closely (Appendix A.1.5).

Table 6 shows both outcomes with shared landmarks and the same feature count. Separate allocation lowers the final gap from 0.085 to 0.006 on the minority-signal control, a paired difference of −0.079 [−0.102, −0.054]. Aggregate compression does better on Hartmann-6, where the paired difference is +0.020 [+0.002, +0.046]. Table 18 tests the same allocation choice on the applications. It favors MvBO on molecule and DMO, while the screening interval includes zero.

AuthorsOpenReview ↗

Response to Reviewer oE45

W3. Computational cost and longer horizons

Thank you for pointing out that a truncated basis already gives cheaper posterior updates.

At 8000 Hartmann-6 queries, MvBO runs faster than exact inference on all ten paired seeds and uses less peak memory, with higher final regret (Table 21). Exact inference is faster at 200 and 1000 queries. With a fixed feature map, the cost of a posterior update grows with the square of the feature count. To measure the effect on the full search, we ran both methods with the same acquisition optimizer.

Both methods score the same 2000 fresh Sobol candidates per iteration and take 25 Adam steps from their own best candidate. They share the initial designs, frozen sum kernel, and UCB rule. MvBO uses 40 features. Each run uses one CPU worker and one BLAS thread. Times are cumulative and exclude objective evaluations. Peak resident set size (RSS) measures process memory. Values are mean ± standard deviation.

QueriesMethodOptimizer (s)Conditioning (s)Acquisition (s)Peak RSS (MB)Normalized regret
200MvBO41.2 ± 2.10.42 ± 0.0432.4 ± 1.8186 ± 90.058 ± 0.016
200Exact33.8 ± 1.70.61 ± 0.0628.9 ± 1.5214 ± 110.041 ± 0.012
1000MvBO186 ± 91.9 ± 0.1168 ± 8248 ± 120.019 ± 0.005
1000Exact158 ± 812.4 ± 0.8131 ± 7412 ± 220.012 ± 0.004
8000MvBO1240 ± 6214.8 ± 0.81090 ± 55410 ± 180.007 ± 0.002
8000Exact2680 ± 140890 ± 481620 ± 852140 ± 1100.004 ± 0.002

At 8000 queries, mean optimizer time is 1240 s for MvBO and 2680 s for exact inference, a ratio of 2.2. The paired regret difference is +0.003 [+0.001, +0.005]. In this setup, the faster method changes between 1000 and 8000 queries. Table 3 times the posterior update by itself. Tables 19 and 20 report the earlier comparison, where the methods used different acquisition optimizers, and list the memory required to store the arrays.

AuthorsOpenReview ↗

Response to Reviewer oE45

W4. Hyperparameter adaptation and remaining limitation

Thank you for raising the concern about keeping kernel hyperparameters fixed during the search.

MvBO still has lower mean regret than aggregate Nyström on molecule and DMO when both methods refit their kernels during the search (Table 22). We tested both methods with and without refitting, using the same landmarks and feature counts. We also let the GP-UCB and ARD baselines update their own hyperparameters during the search (Table 1).

For both methods, we refit the sum kernel by exact marginal likelihood every 20 observations through observation 180. After each refit, we recompute the features at the observed points and rebuild the posterior. The table gives means ± standard deviations at 200 queries over ten paired seeds. Brackets show paired 95% confidence intervals.

RepresentationKernel scheduleMoleculeScreeningDMO
MvBOFrozen0.072 ± 0.0210.128 ± 0.0300.065 ± 0.019
MvBORefit every 200.068 ± 0.0200.122 ± 0.0290.061 ± 0.018
AggregateFrozen0.099 ± 0.0260.135 ± 0.0320.093 ± 0.025
AggregateRefit every 200.094 ± 0.0250.129 ± 0.0310.088 ± 0.024
MvBO − aggregate, both refitted [95% CI]−0.026 [−0.043, −0.009]−0.007 [−0.025, +0.011]−0.027 [−0.044, −0.010]
Refitted − frozen MvBO [95% CI]−0.004 [−0.014, +0.006]−0.006 [−0.018, +0.006]−0.004 [−0.013, +0.005]

When both methods refit, the paired intervals favor MvBO on molecule and DMO and include zero on screening. Compared with keeping the map fixed, refitting lowers MvBO's mean regret on all three tasks. All three intervals for that change include zero. Refitting adds 18.7 s on molecule, 37.2 s on screening, and 13.3 s on DMO. Default MvBO keeps the feature map fixed. The regret guarantee requires a fixed map and a wider confidence bound. Our experiment measures the cost of refitting at 200 queries. Its cost at longer horizons remains open.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their careful feedback and the area chair and program chairs for coordinating the reviews. We appreciate FyNS’s comments on the clarity of the paper and the range of applications. We also appreciate Cp5N’s comments on the controlled experiments, the care taken in stating the theoretical assumptions, and the reporting of cases where MvBO performs poorly.

We have updated the manuscript to address all the concerns raised in the reviews. The changes below summarize how we responded to the main points from FyNS, Cp5N, and oE45. Our individual replies provide the details.

  • Contribution and feature allocation (FyNS, Cp5N, oE45). We clarified that MvBO uses standard sum-kernel GP inference and keeps features from each kernel separately. Aggregate Nyström combines the kernels first and then chooses features. With the same landmarks and feature counts, MvBO reduces mean final regret by 27% on molecular conformer search and 30% on LLM data-mixture optimization. Both paired 95% confidence intervals favor MvBO. The screening interval includes zero (Table 18). Table 17 also reports the landmark count and placement comparison FyNS requested.
  • Baseline comparisons and refitting (FyNS, Cp5N, oE45). We added exact sum-kernel inference, aggregate Nyström, BKB, and independently refitted GPs to the main comparison (Figure 2 and Table 1). In response to FyNS, we also tested RAMBO and EGP with UCB (Appendix B.2.12). The advantages on conformer search and data-mixture optimization hold when both compressed models refit their kernels (Table 22).
  • Prediction, coverage, and acquisition (FyNS, Cp5N). We tested what happens when both compressed models choose queries using only the predicted objective value. The advantages on conformer search and data-mixture optimization hold in this setting (Table 24). MvBO also has lower mean prediction error than aggregate Nyström on these two tasks when both train on the same observations. To address Cp5N’s question about regimes, we report prediction errors for predefined input groups (Table 25). We limit the interpretation to prediction within those groups. Table 23 reports the prediction comparison on common inputs. Coverage remains well below the nominal 95%.
  • Runtime and practical tradeoffs (FyNS, Cp5N, oE45). At 200 queries, exact inference takes less optimizer time and reaches lower mean regret on all three applications. At 8000 Hartmann-6 queries, MvBO uses less optimizer time and peak memory under the same acquisition optimizer, with higher mean regret (Table 21). Exact inference is still faster at 1000 queries.
  • Scope of the regret guarantee (oE45). We clarify how the bound accounts for features we drop and for reuse of the initial observations. The guarantee requires a fixed feature map and a wider confidence bound (Section 4 and Appendix A.2).

We appreciate the reviewers’ suggestions and welcome any remaining questions.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear AC and Reviewers,

With the discussion phase concluding soon (Nov 18), we wanted to check in to see if you had any additional thoughts or questions regarding our rebuttal.

We appreciate the opportunity to clarify our work and hope our responses have adequately addressed your concerns. We remain available for any final discussions before the deadline.

Thank you again for your valuable feedback and time.

With thanks,

Authors of #18336

AuthorsOpenReview ↗

Summary of Revisions and Discussion Follow-Up

Dear Area Chairs and Reviewers,

Thank you for your thoughtful feedback. We have uploaded a revised manuscript and provided point-by-point responses to all reviewer concerns. The main revisions are:

  1. Clearer contribution and comparison with additive GPs. We clarify that MvBO preserves features from each component kernel before combining them. Aggregate Nyström instead combines the kernels before selecting features.

  2. Stronger baselines. We added exact sum-kernel inference, rank-matched aggregate Nyström, BKB, independently refitted GP baselines, and MKL-BO, alongside further comparisons with RAMBO and EGP.

  3. Controlled tests of what drives the gains. We matched landmarks and feature counts, tested kernel refitting and mean-only acquisition, and compared predictive error on identical observations. With matched landmarks and feature counts, MvBO reduces mean final regret relative to aggregate Nyström by 27% on molecular conformer search and 30% on LLM data-mixture optimization. The confidence interval for screening includes zero.

  4. Expanded sensitivity and statistical analyses. We examined landmark number and placement, refitted kernel hyperparameters, and a dimension-scaled lengthscale prior. We also added paired confidence intervals and signed-rank tests.

  5. Explicit computational trade-offs and limitations. At 200 queries, exact inference is faster and achieves lower mean regret. At 8,000 Hartmann-6 queries, MvBO uses less optimizer time and memory but has higher regret. The revision reports both outcomes explicitly.

  6. Clearer theoretical scope. We separate weight-learning error from finite-feature approximation error, clarify the fixed-feature and confidence-width assumptions, and discuss undercoverage and its effect on UCB exploration.

We thank Reviewer FyNS for confirming that the rebuttal addressed their concerns and for raising their score.

Before discussion closes on November 18 AoE, we would appreciate any further feedback from reviewers who have not yet responded. In particular, it would help to know whether the revisions resolve your concerns or whether any issues remain that we can address.

Thank you again for your time and careful consideration.

With thanks,

Authors of #18336

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.