- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes CAVS, a variable-selection method for high-dimensional Bayesian optimization. It fits a GP with a low-rank-plus-diagonal Mahalanobis kernel and uses the learned metric to score and select variables. The acquisition function is optimized over the selected coordinates, while the remaining coordinates are filled using historical and random candidates. The paper provides a regret analysis and compares CAVS with existing BO methods on eight real-world tasks, alongside synthetic tests of kernel fitting and variable identification under rotation.
Critical strengths
-
The low-rank-plus-diagonal kernel provides a practical way to model correlated directions with fewer parameters than a dense Mahalanobis metric. This addresses a relevant modeling issue under the small evaluation budgets typical of BO.
-
The experiments cover eight real-world tasks, including the 888-dimensional Ant problem, and CAVS achieves the best reported average rank. This provides useful evidence that the method can work across different applications.
Critical weaknesses
-
The main claim of correlation-aware variable selection is not convincingly established. The method largely follows LassoBO, and the ablations mainly show improvements in kernel prediction rather than isolating the benefit of the proposed selection score. The rotation tests also preserve the original active coordinate set and clip out-of-bound inputs, so they do not cleanly demonstrate robustness to general non-axis-aligned active subspaces. This leaves the main motivation and additional contribution insufficiently supported.
-
The optimization evaluation is not strong enough to establish a reliable advantage. The synthetic experiments focus on fitting and variable identification, without controlled end-to-end optimization tests across ambient and effective dimensions. Embedding-based comparisons are limited to ALEBO. BAxUS is not evaluated, and the more recent DSEBO is not discussed, leaving the method’s position relative to these alternatives unclear. Only five seeds are used, and several reported gains are small relative to the variation—for example, HalfCheetah regret is 500 ± 404.4 for CAVS versus 619 ± 426.5 for Vanilla BO. More evidence is needed to distinguish consistent gains from run-to-run variation.
Questions for authors
-
How much of the improvement comes from the more flexible kernel, and how much comes from the selection score? Could you compare different selection scores while keeping the same learned surrogate?
-
The rotation tests only mix the original active coordinates and then clip out-of-bound inputs. What do these tests establish about robustness to non-axis-aligned subspaces? Could clipping itself affect the results?
-
Could you provide end-to-end optimization results on higher-dimensional synthetic functions, such as at 1,000 or 10,000 dimensions? Controlling both ambient and effective dimensions would help clarify where the method works.
-
How stable are the reported gains with only five seeds? Please run more seeds and use t-tests to check whether the improvements are statistically significant, especially on tasks with large variation such as HalfCheetah and Ant.
-
Could you expand the comparison with embedding-based methods, including REMBO [1], HeSBO [2], and DSEBO [3]? Both variable selection and embedding-based search approach high-dimensional optimization through dimensionality reduction, so a broader comparison would help establish the benefit of the proposed approach.
-
What background sample schedule was used in the experiments? This affects both exploration and acquisition-optimization cost, but the practical setting is unclear.
[1] Wang et al. Bayesian Optimization in a Billion Dimensions via Random Embeddings. JAIR 2016.
[2] Nayebi et al. A Framework for Bayesian Optimization in Embedded Subspaces. ICML 2019.
[3] Qian et al. Automated Random Embedding for Practical Bayesian Optimization with Unknown Effective Dimension. IJCAI 2026.
9 comments
W2 & Q3 Optimization in higher ambient dimensions
W2. The optimization evaluation is not strong enough to establish a reliable advantage. The synthetic experiments focus on fitting and variable identification, without controlled end-to-end optimization tests across ambient and effective dimensions.
Q3. Could you provide end-to-end optimization results on higher-dimensional synthetic functions, such as at 1,000 or 10,000 dimensions? Controlling both ambient and effective dimensions would help clarify where the method works.
Thank you for this suggestion. Following your suggestion, we conducted end-to-end BO experiments on two synthetic functions at 1,000 and 10,000 ambient dimensions as case studies. CAVS ranks second to DSEBO on Rastrigin30 in 1,000D and achieves a final objective close to HeSBO on Hartmann6 in 10,000D. On Rastrigin30, CAVS reaches −151.9, second to DSEBO (−72.1). On Hartmann6, it reaches 2.61, close to HeSBO (2.62), while MCTS-VS reaches 2.85. Figure 14 (https://anonymous.4open.science/r/CAVS-re/highdim_bo.pdf) shows the optimization trajectories. LassoBO and ALEBO are omitted at 10,000D because of their computational cost, and SAASBO is omitted at both dimensions because of its high computational complexity. These two case studies provide an initial assessment of CAVS at much larger ambient dimensions, complementing our main experiments on problems with hundreds of variables. Optimizing at this scale remains challenging, with room to further improve CAVS's performance.
We have added these high-dimensional case studies in Appendix A.14 (Figure 14) and summarized the results in Section 6.4.
W2 & Q4 Stability and statistical significance
W2. Only five seeds are used, and several reported gains are small relative to the variation—for example, HalfCheetah regret is 500 ± 404.4 for CAVS versus 619 ± 426.5 for Vanilla BO. More evidence is needed to distinguish consistent gains from run-to-run variation.
Q4. How stable are the reported gains with only five seeds? Please run more seeds and use t-tests to check whether the improvements are statistically significant, especially on tasks with large variation such as HalfCheetah and Ant.
Thanks for this important question. CAVS retains the best average rank of 2.9 after expanding the evaluation. On the two tasks you highlighted, it significantly outperforms six baselines on HalfCheetah and five on Ant; it also significantly outperforms twelve baselines on MOPTA08.
Following your suggestion, we expanded the evaluation to ten seeds for every method on all eight tasks. We test per-run final recorded objective values using two-sided Welch t-tests, with Holm correction across the eight tasks for each baseline at a significance level of 0.05. Table 7 (https://anonymous.4open.science/r/CAVS-re/real_world_results.md) reports all task-wise results and significance comparisons. Figure 5 (https://anonymous.4open.science/r/CAVS-re/main_average_rank.pdf) shows CAVS's average rank throughout the evaluation budget. In the expanded comparison, CAVS achieves the best average rank of 2.9, followed by LassoBO at 3.9 and the remaining methods at 4.0–11.5. Together, the expanded seed set and statistical tests support the aggregate performance advantage and identify the task-specific significant gains.
We have updated the ten-seed results in Section 6.3 (Table 1 and Figure 5) and added the statistical comparisons in Appendix A.15 (Table 7).
W2 & Q5 Comparisons with embedding-based methods; more baselines
W2. Embedding-based comparisons are limited to ALEBO. BAxUS is not evaluated, and the more recent DSEBO is not discussed, leaving the method’s position relative to these alternatives unclear.
Q5. Could you expand the comparison with embedding-based methods, including REMBO [1], HeSBO [2], and DSEBO [3]? Both variable selection and embedding-based search approach high-dimensional optimization through dimensionality reduction, so a broader comparison would help establish the benefit of the proposed approach.
[1] Wang et al. Bayesian Optimization in a Billion Dimensions via Random Embeddings. JAIR 2016.
[2] Nayebi et al. A Framework for Bayesian Optimization in Embedded Subspaces. ICML 2019.
[3] Qian et al. Automated Random Embedding for Practical Bayesian Optimization with Unknown Effective Dimension. IJCAI 2026.
Thanks for this valuable suggestion. Following your suggestion, we have now added REMBO, HeSBO, and DSEBO, a strong recent representative of adaptive embedding methods. Across all compared methods, CAVS achieves the best average rank of 2.9, compared with 7.5 for DSEBO, 10.2 for REMBO, and 10.6 for HeSBO. This expanded comparison supports CAVS's strong aggregate performance against both embedding-based and variable-selection methods on these benchmarks. Table 1 (https://anonymous.4open.science/r/CAVS-re/real_world_results.md) provides task-wise results, and Figure 5 (https://anonymous.4open.science/r/CAVS-re/main_average_rank.pdf) shows performance across the evaluation budget.
The expanded baseline comparison is now reported in Section 6.3 (Table 1 and Figure 5).
Q6 Background sampling schedule
Q6. What background sample schedule (N_t) was used in the experiments? This affects both exploration and acquisition-optimization cost, but the practical setting is unclear.
Thanks for pointing this out. The experiments use random background vectors per iteration, generated by Latin hypercube sampling following LassoBO. Two history-based vectors constructed from the three best observations supplement these, giving seven background candidates per iteration.
We have clarified the experimental background sampling scheme in Section 4.4, Algorithm 1, and the Background sampling paragraph in Appendix A.6.
We hope our response has addressed your concerns. If we have missed anything, please let us know.
Thanks for your detailed review and constructive suggestions! Please find our responses below.
W1 & Q1 Contribution of the selection score
Thank you for this helpful suggestion. Following your suggestion, we compare our selection score Mii=wi+∣Ri,:∣22 with a diagonal-only baseline, wi, where wi=li−1 is the learned weight of variable i in the kernel’s diagonal component. This score also aligns with the coordinate-wise derivative bounds used in our subsequent regret analysis. Compared with this baseline, our selection score achieves better rank and group recall changes and better average end-to-end BO performance. With d = 10, its mean active-variable rank change is −5.399, compared with −5.058 for wi (lower is better), and its mean group recall decreases by 6.55 percentage points, compared with 7.43 percentage points for wi. Rotation comparison (https://anonymous.4open.science/r/CAVS-re/selection_score_rotation.md) reports these results.
The end-to-end BO ablation further shows that using our selection score improves average optimization performance while keeping the kernel architecture and fitting procedure fixed. Table 4 (https://anonymous.4open.science/r/CAVS-re/selection_score_results.md) reports final regret and average rank, and Figure 11 (https://anonymous.4open.science/r/CAVS-re/selection_score_rank.pdf) shows the average rank trajectory. CAVS achieves the best average BO rank of 1.2, versus 2.2 for wi and 2.5 for random selection. We evaluate all three variants on eight real-world tasks.
The end-to-end selection-score ablation is now included in Appendix A.9 (Table 4 and Figure 11), with a summary in Section 6.4.
W1 & Q2 Scope of rotation robustness
Thank you for this insightful question. The purpose of Figure 8 is to test whether variable-selection methods can continue to identify important variables when the objective depends on non-axis-aligned combinations of its active variables, with the active coordinate set held fixed. Our results show that CAVS better preserves active-variable rankings under these rotations, achieving the better mean rank change even when clipping is removed. As you pointed out, clipping can affect the results by distorting pairwise distances; CAVS's ranking advantage persists in both clipping-free experiments.
Following Reviewer BRxt's suggestion to test a clipping-free construction, we use two sampling strategies. Let s denote the number of true active variables, with normalized coordinates z and original bounds [−1,1]s. The shrunken cube restricts sampling to $
−1/s,1/s^s$ before rotation. For any orthogonal rotation Q, ∣Qz∣∞≤∣Qz∣2=∣z∣2≤1, so every rotated coordinate stays within the original bounds. The unit-ball setting samples uniformly from z∈Rs:∣z∣2≤1, whose domain is invariant under orthogonal rotation.
The shrunken-cube results (https://anonymous.4open.science/r/CAVS-re/rotation_shrunken_cube.pdf) and unit-ball results (https://anonymous.4open.science/r/CAVS-re/rotation_unit_ball.pdf) show that CAVS achieves the best mean rank change in both settings: +7.545 and −2.361, respectively (rotated minus unrotated; lower is better). Its group-recall change is also best in the shrunken cube and better than LassoBO in the unit ball. The revised Figure 8 presents the unit-ball experiment, with protocol details in Appendix A.8.