acceptodds
Under review as a conference paper at ICLR 2027

SABLE: Scalable Affine Bayesian Latent-space Exploration

Abstract

Complex objects such as engineering designs and proteins are often difficult to optimise because their design spaces exhibit high-dimensional structure and objective evaluations are expensive. Generative models provide continuous latent representations, but their latent spaces can have thousands to tens of thousands of dimensions, creating a regime where the latent dimension greatly exceeds the evaluation budget. We introduce SABLE, a Bayesian optimisation method for high-dimensional latent spaces with an isotropic distribution and limited budget. SABLE uses a Bayesian affine surrogate for prediction and acquisition, treating higher-order structure as surrogate discrepancy. After observations, the slope update has dimension at most , yielding a decomposition of each query into an in-span component optimised by expected improvement (EI) in at most dimensions and an off-span component governed by a scalar norm. We show that the off-span norm implicitly selected by affine EI can lead to underexploration, and SABLE instead prescribes it using a finite-horizon schedule, without restricting search to a learned subspace. A recompression scheme reduces information to pseudo-observations, bounding refitting cost by while keeping parameter inference, prediction, and acquisition decisions to tens of milliseconds. Across synthetic benchmarks and latent-space optimisation tasks, SABLE achieves high sample efficiency and reduces decision time by several orders of magnitude at large budgets.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian optimisation, latent space optimisation, high-dimensional optimisation, generative models, affine surrogate models, expected improvement, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_vxVYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SABLE for black-box optimization in high-dimensional latent spaces of pretrained generative models. It uses a Bayesian affine surrogate and decomposes candidate queries into components within and orthogonal to the span of previously evaluated designs. A finite-horizon schedule explicitly controls off-span exploration, while a pseudo-observation compression scheme limits model-refitting costs as observations accumulate. Experiments on controlled objectives and four scientific latent-space tasks evaluate sample efficiency, computational overhead, and the contributions of the exploration schedule and compression.

Critical strengths
  1. A concrete methodological response to two identifiable limitations. The paper addresses underexploration by standard affine EI and the growing cost of posterior refitting. The exploration schedule and compression mechanism are explicitly specified, supported by mathematical analysis, and evaluated separately. This provides useful methodological insight beyond demonstrating that a combined system works.

  2. Substantial empirical validation against relevant alternatives. The evaluation includes the closest affine-BO predecessor, several high-dimensional BO methods, and evolutionary optimizers, with 10 independent seeds. SABLE achieves the highest mean performance on three of four scientific tasks, while compression substantially reduces computational overhead. Reporting both evaluation-count and wall-clock trajectories strengthens the practical relevance of the results.

Critical weaknesses
  1. The schedule is precomputed from the evaluation horizon and a reference-resolution parameter instead of adapting to observed responses. Its justification uses an affine information-gain approximation, while the intended objectives can be nonlinear, noisy, or quantized. Although sensitivity experiments are provided, the reliability of regime selection and robustness to changing local resolvability remain insufficiently established.

  2. The stability analysis applies to conditional posteriors and to marginal EI with shared hyperparameter weights; it does not cover changes in those weights caused by approximate evidence evaluation. Furthermore, bounded posterior-refitting state does not imply bounded total memory, since exact historical readouts require additional storage growing with the observation count. These qualifications limit the scope of the theoretical and scalability claims.

Questions for authors
  1. How robust is the prescribed exploration schedule when local resolvability changes during optimization or the initial smooth/noisy regime classification is incorrect? Can the proposed regime diagnostic select an effective setting within the stated evaluation budget?
  2. How much does compression change the inferred hyperparameter weights and acquisition decisions in practice? Can these effects be separated from approximation error under fixed hyperparameters?
  3. What are the total memory and end-to-end decision-time scaling, including historical readouts and recompression, as the evaluation count grows at fixed compression budget?
6 comments

We thank the reviewer for their thoughtful and constructive assessment, and for recognising the methodological insights provided by SABLE's two complementary contributions, the substantial empirical validation against relevant alternatives, and the practical significance of its computational improvements. We revised the manuscript with additional theoretical analysis and experiments to address the reviewer's concerns.

AuthorsOpenReview ↗

Question 1: Exploration schedule, sensitivity, and regime selection (Weakness 1)

Methodological scope and schedule choice The Value of Information (VoI) schedule is motivated by an affine approximation and is not guaranteed to be optimal for general nonlinear, noisy, or quantised objectives. We have therefore revised the manuscript to emphasise that the main methodological contribution is the decoupling of off-span exploration from fitted surrogate uncertainty and not the particular VoI schedule. The latter provides one computationally inexpensive instantiation, whose definition, derivation and properties have been moved to the appendix.

Sensitivity to λref\lambda_{\mathrm{ref}} We conducted additional experiments comparing λref∈{10,105}\lambda_{\mathrm{ref}}\in\{10,10^5\}, representing the coarse- and fine-resolution regimes, on all four scientific tasks, using the same experimental protocol and ten paired seeds. The local resolvability diagnostics in Appendix D.6 identify the fine-resolution setting (λref=105\lambda_{\mathrm{ref}}=10^5) for Turbulence and both Calorimeter tasks, and the coarse-resolution setting (λref=10\lambda_{\mathrm{ref}}=10) for Peptides. At N=3,000N=3{,}000, these diagnostic-selected settings outperform the corresponding opposite regimes: 0.956 vs. 0.916 on Turbulence (10/10 paired wins), 1.166 vs. 1.023 on Peptides (7/10), 0.858 vs. 0.478 on Calorimeter DS2 (10/10), and 0.658 vs. 0.356 on DS3 (10/10). Sensitivity to the reference setting is relatively modest on Turbulence and Peptides but substantial on the calorimeter tasks, indicating that the consequences of an incorrect regime choice are objective-dependent.

Reliability of regime selection Appendix D.6 introduces a local-resolvability diagnostic based on two statistics, QQ and ZZ, requiring 40 evaluations (1.3% of the budget), and assesses its reliability across 2,000 random diagnostic sub-designs per objective. The observed (Q,Z)(Q,Z) values are (1.03,0)(1.03,0) for Turbulence, (1.01,0)(1.01,0) for DS2, (1.00,0)(1.00,0) for DS3, and (4.44,65.6)(4.44,65.6) for Peptides, consistent with the respective fine- and coarse-resolution regimes. While the diagnostic was validated separately rather than used online in the reported optimisation experiments, these results support the feasibility of selecting an appropriate initial regime within the evaluation budget. Together with the sensitivity experiments, these results show that the diagnostic's regime assignments agree with the better-performing reference setting on all four scientific tasks.

Robustness to changing local resolvability The current implementation uses a precomputed schedule and does not adapt to changes in local resolvability during optimisation. The decoupled formulation accommodates adaptive schedules without modifying the surrogate or in-span acquisition, but evaluating their effectiveness under changing resolvability, including their hyperparameter sensitivities, is left for future work.

AuthorsOpenReview ↗

Question 2: Compression and acquisition stability (Weakness 2)

We agree that the original stability analysis did not cover changes in hyperparameter weights induced by approximate marginal-likelihood evaluation. The new Appendix E.4.2 addresses this gap by establishing a total-variation bound between exact and compressed hyperparameter distributions. It also extends the marginal-EI stability analysis to separate fixed-weight compression error from errors induced by approximate evidence evaluation. Finally, we bound the resulting acquisition regret and provide a sufficient condition for preserving the selected maximiser on a common finite candidate set.

These guarantees are conditional on the realised approximation errors. The compression-budget experiments in Section 4 provide complementary empirical evidence: at N=5,000N=5{,}000, compression budgets M∈{100,300,500}M\in\{100,300,500\} yields closely matching optimisation performance, whereas aggressive compression (M=10M=10) causes substantial degradation. These experiments assess the combined approximation through optimisation performance but do not directly quantify changes in hyperparameter weights or individual acquisition decisions. Such numerical assessments remain future work.

Reviewer_v58qOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces SABLE, a Bayesian optimization method for the latent spaces of pretrained generative models, where the latent dimension D far exceeds the evaluation budget N. Queries are restricted to the sphere of radius, and the objective is modeled by a Bayesian affine (linear plus intercept) surrogate whose variance hyperparameters are marginalized on a grid. Because N observations inform at most N directions, each query splits into two parts. The part inside the span of previous queries is chosen by expected improvement (EI) in at most N coordinates. The part in a fresh orthogonal direction has its size set by a precomputed schedule that decays to zero over the budget, instead of being chosen by EI. A recompression step keeps the fitted state at about M = 300 pseudo-observations, so refitting cost stops growing with N. On synthetic spherical objectives at D = 10,000, M ≥ 100 stays within 0.05 of the uncompressed model while cutting decision time from 79.5 s to 5.5 ms. On four scientific tasks with 3,000 evaluations, SABLE attains the best mean value on three, but loses to sep-CMA-ES on Peptides, and four of the GP baselines stop at the 48-hour limit well before the full budget.

Critical strengths

Strength: The paper finds a clear failure mode of EI and tests the fix with well-designed ablations.

  1. The paper shows that standard EI explores poorly under an affine model, and that the failure goes in opposite directions depending on how the hyperparameters are fit. With MLE, EI puts most of each query into new directions. With MAP or marginalisation, it puts almost nothing there and gets stuck in the early directions. The diagnostic plots make this visible step by step: off-span norm, fitted scales, and recovery of the true linear direction (Figure 6–8).
  2. The ablation crosses three hyperparameter fits with six exploration rules, on four objectives and ten paired seeds (Table 3), and repeats the grid with noise (Table 4). It includes a natural alternative: a schedule driven by the model's own fitted variance ratio. That alternative does worse than the fixed schedule on non-affine objectives (Table 1), which supports setting exploration outside the model. On the severe objective, the schedule raises the attained value from 0.49 to 0.63.
  3. The comparison is also conservative. Baselines are given the same sphere constraint as SABLE, and the ambient CMA-ES variants confirm this constraint helps them (Fig. 4). Settings and baseline configurations are reported in full (App. H, I.7).
Critical weaknesses

Weakness 1: The benchmark design does not establish robustness beyond objectives favorable to an affine surrogate.

  1. The toy suite never removes the model’s preferred signal. All controlled tests use ZonalFunctionBench, and even its “severe” construction retains a linear component (Appendix G). There is no zero-linear-signal control or independent benchmark family. The experiments demonstrate tolerance to selected nonlinear perturbations, but do not establish robustness when useful affine structure is absent.
  2. The evaluation omits established tests of the competing methods. Neither conventional analytical benchmarks, such as shifted Ackley, nor Fan et al. 2026’s existing spherical benchmarks are included. Testing the baselines only on the selected new tasks leaves the extent of SABLE’s generalizability unresolved.
  3. The main controlled endpoints are outside the advertised small-budget regime. Table 1 and the long-horizon experiments emphasize (N=D), despite motivating the method through (N<< D). The scientific suite covers three domains, with two resolutions of the same calorimeter task, and SABLE loses to sep-CMA-ES on Peptides.
  4. The comparison mixes sample efficiency with an implementation-specific compute limit. Several GP baselines complete far fewer than 3,000 evaluations before the 48-hour cutoff. Their endpoint disadvantage cannot establish inferior sample efficiency. All runs receive four CPU cores, but no hardware-sensitivity analysis establishes how much the cutoff depends on that allocation.
  5. The exploration regime is supplied rather than selected within the evaluation budget. Scientific tasks use manually assigned ($\lambda_{\mathrm{ref}}$) regimes. The paper neither reports opposite-regime sensitivity on those tasks nor demonstrates the proposed diagnostic operating within each run’s budget. Table 4 shows that an inappropriate regime can substantially reduce performance.

Weakness 2: SABLE’s incremental additions are insufficiently isolated and justified.

  1. The core method is not new. Linear-surrogate BO on a sphere with fast acquisition optimization is already developed by Fan et al. (2026). The contribution therefore rests on the additional schedule, compression, and marginalization; not the underlying framework.
  2. The specific exploration schedule is not shown to be necessary. SABLE sets the share of each query placed in an unexplored direction (the off-span norm $t_N$) using a formula (Eq. 3) that starts at 1 and decays to 0 over the budget. It is compared only with (i) letting EI choose $t_N$ and (ii) the same formula driven by the fitted variance ratio. It is never compared with simple alternatives such as a constant t or a linear/exponential decay. This matters because Table 3 suggests many decays work: with marginalisation on the severe objective, $\lambda_{ref}$ = 10^1, 10^3 and 10^5 all beat no schedule (0.54, 0.65, 0.63 vs. 0.49). The supporting theory (Prop. D.1) only shows that the schedule decays monotonically under a toy reference model. It does not show this formula beats other decays on a real non-affine objective. The evidence supports "annealed exploration helps," not "Eq. 3 is needed."
  3. The component evidence is incomplete. The scientific comparison used to argue that compression has no sample-efficiency cost also changes MAP to marginalization. Meanwhile, controlled results show little difference between these inference treatments. A matched component ablation is needed to establish what each addition contributes.
  4. The analysis does not cover the complete implementation. Compression bounds exclude refitted hyperparameter-weight changes, and the acquisition candidate set need not contain stationary points of marginalized EI. Without measuring these errors or checking a stronger acquisition optimizer, the paper does not establish that its theoretical approximation results explain the reported gains.
Questions for authors

Major questions

  1. Will SABLE’s advantage survive independent benchmarks and weak affine signals? The paper compares against existing BO methods without evaluating their established benchmark families. Can the authors include shifted Ackley at (D=10,000), clearly distinguishing low effective dimension from full effective dimension, and explain the omission of BAxUS-style embedded Branin/Hartmann and application benchmarks? Fan et al. 2026’s SpAckley, SpCrater, SpWeber, and SpAlign already fit the spherical setting. Can the authors also test zero linear signal and (N << D). These comparisons would establish whether the gains extend beyond ZonalFunctionBench.
  2. Is SABLE more than an extension of Fan et al. 2026? The affine surrogate, spherical geometry, and fast acquisition framework already exist. What measured gain comes separately from scheduling, compression, and marginalization on scientific tasks? The current compression comparison also changes the inference treatment, while controlled experiments show little benefit from marginalization.
  3. Is Eq. 3 better than a simple decay? Eq. 3 precomputes how much of each query goes to a new direction, using a toy value-of-information model. Beating "EI chooses this amount itself" only shows that forcing some exploration helps. Can the authors compare Eq. 3 against a constant t and simple polynomial or exponential decays, tuned with a budget like the one used for λ_ref. If a tuned simple decay matches it, the contribution is generic annealing rather than the proposed formula.
  4. How sensitive is SABLE to the regime choice on real tasks? $\lambda_{ref}$ is set by hand per task (10^5 for smooth tasks, 10 for Peptides) and is never varied on the scientific benchmarks. On synthetic data, the wrong value more than halves performance (Table 4: 0.38 → 0.16). Can the authors (a) run each scientific task with the other regime, (b) report the per-task Q and Z values that Appendix D.5 summarizes, and (c) add a noisy real-world task, since all current scientific objectives are deterministic.
  5. Is SABLE's acquisition actually being maximized? SABLE does not use a numerical optimizer such as multi-start L-BFGS (as in BoTorch). It scores about 100 hand-constructed candidate directions per step (12 per hyperparameter atom plus posterior-mean directions) and picks the best. The authors note this set can miss the maximizer of marginalised EI (App. C.2). The headline ablation (schedule vs. no schedule) also uses different candidate searches for the two arms. Because acquisition lives in only d_N ≤ N dimensions, multi-start L-BFGS is cheap. Can the authors run additional ablations and report (a) the EI gap between the candidate set and multi-start L-BFGS, and (b) whether Table 1's schedule advantage holds when both arms use that optimizer?
  6. How much of the method does the theory cover? The proofs cover the affine model, the schedule under a toy model, and compression error with hyperparameters held fixed. None covers optimization of the real non-linear objective, and the hyperparameter refitting after compression is excluded. Can the authors state this scope clearly, and report the numerical size of the bounds in Prop. E.4 and Cor. E.5–E.6 on the benchmarks, compared with typical EI values.
  7. Are the GP timeouts intrinsic or partly caused by the compute allocation? Can representative baselines complete 3,000 evaluations with longer runs or suitable hardware, and does SABLE still lead at equal evaluation counts? Please profile fitting and acquisition costs and test the sensitivity to the four-core allocation. The current timeout results establish performance under that allocation, not the best achievable sample efficiency of the truncated methods.

Minor questions

  1. Can the baseline runs be reproduced exactly? Can the authors provide original repositories or adaptation sources, commit hashes, library versions, complete hyperparameters, initialization and refitting policies, hardware/thread settings, seeds, and runnable configurations. Appendix I.7 is helpful but does not replace exact implementation provenance.

  2. Acquisition choice. SABLE scores candidates with plain EI, whereas the GP baselines such as VanillaBO use LogEI (Ament et al., 2023). Since LogEI shares EI's maximizer but avoids numerical underflow, can the authors report if any and how often all candidate EI values underflow, and whether LogEI scoring changes the selected queries or the off-span allocation. For the noisy experiments (Table 4), can the authors compare the plug-in incumbent with qLogNEI (Ament et al., 2023) under the same affine posterior.

    • Ament, Sebastian, et al. "Unexpected improvements to expected improvement for bayesian optimization." Advances in neural information processing systems 36 (2023): 20577-20612.
  3. Why not test the mechanism with a nonlinear BoTorch GP? BoTorch GP baselines are already included, but the proposed scheduling mechanism is evaluated only with the affine model. Can the authors separate the benefit of scheduling from that surrogate choice? Can the authors also explain whether MSR adequately covers the relevant configuration from Xu et al.’s standard-GP paper.

  4. Which gains survive paired uncertainty analysis? Can the authors report paired performance differences and confidence intervals against TuRBO, sep-CMA-ES, and Fan et al. The percentile bands and proximity-based bolding rules are not significance tests.

  5. Does Proposition E.2 account for a changing incumbent? Is the improvement threshold identical for exact and truncated acquisitions? If it is recomputed, can the authors explain where its perturbation enters the bound and whether Equation 13 changes.

  6. Is Figure 2 based on (λref=103\lambda_{\mathrm{ref}}=10^3) or (10510^5)? Can the authors reconcile Appendix I.5 with the Section 4 default and identify the actual configuration behind each compression result.

  7. Can SABLE continue effectively after its planned horizon? Can the authors specify the behavior when (N_max) is increased during a run and whether the resulting performance differs materially from knowing the final budget initially.

  8. Do all methods receive the same number of scored evaluations? The controlled “attained value” gives model-based methods an additional recommendation evaluation. Can the authors equalize total evaluations and report whether the conclusions change.

12 comments

We thank the reviewer for their assessment and for recognising the methodological significance of the identified failure modes of standard affine EI, the systematic exploration ablations, and the fairness of our baseline comparisons. Several of the concerns raised were addressed by analyses and experiments in the original manuscript, as clarified below. To address the remaining concerns, we have improved the presentation of SABLE's principal methodological contributions and strengthened the paper with new benchmarks, objectives with negligible affine signal, additional sensitivity experiments, and extended theoretical guarantees. Together, the existing and new evidence supports SABLE's two principal contributions: decoupled off-span exploration and computationally scalable posterior refitting. We remain convinced of the quality of the paper and hope that our responses will encourage the reviewer to reconsider their assessment.

AuthorsOpenReview ↗

Question 1: Will SABLE’s advantage survive independent benchmarks and weak affine signals? (Weaknesses 1.1, 1.2 and 1.3)

Independent benchmarks

We extended the evaluation to four spherical objectives introduced by Fan et al. (2026): SpAlign, SpCrater, SpAckley, and SpWeber, at D=N∈{256,1000}D=N\in \{ 256,1000\} over ten independent runs (see Figure 5 in the updated manuscript). SABLE achieves the highest mean best-observed value in seven of the eight settings, improving on the next-best method by 0.09–0.41. The exception is SpCrater at D=256D=256, where Vanilla BO achieves 0.58 versus 0.54 for SABLE. Relative to Fan et al. (2026), SABLE's advantage increases from 0.02–0.13 at D=256D=256 to 0.21–0.42 at D=1000D=1000. These experiments provide validation beyond ZonalFunctionBench and the scientific objectives.

Weak affine signals

SABLE does not assume that the objective is affine; it uses an affine working model to guide optimisation under limited observations. The original severe benchmark already reduces the degree-one contribution to 1/71/7 of its degree-one-to-three spectral power. To address the reviewer's concern directly, we added two controlled objectives with zero degree-one spectral weight, supported on degrees 2 and 4 (zero-even) or degrees 2 and 3 (zero-orth.) in Appendix I.6. Both have a measured degree-one share below 10−410^{-4}. Appendix I.6 evaluates standard EI, a model-derived schedule, and the prescribed VoI schedule across MLE, MAP, and hyperparameter marginalisation, using D=N=1,000D=N=1{,}000 and ten paired seeds, without compression.

Prescribed exploration continues to outperform standard EI despite negligible degree-one signal. Under MAP, attained values increase from 0.45 to 0.59 on zero-even and from 0.51 to 0.65 on zero-orth.; marginalisation yields similar improvements, from 0.43 to 0.60 and from 0.52 to 0.65, respectively. The model-derived schedule does not provide comparable gains. These results show that the benefit of decoupled off-span exploration is not limited to objectives with a substantial degree-one component.

Small-budget regime (Weakness 1.3)

The N=DN=D experiments are deliberately included as additional long-horizon stress tests rather than representing SABLE's primary target regime, as stated in Section 4. All four scientific experiments are evaluated at N=3,000<DN=3{,}000<D, across latent dimensions from D=6,000D=6{,}000 to D=40,500D=40{,}500. Note that NN can be large while remaining in the regime N<DN<D; in Figure 7, we extend DS3 (D=40,500) to N=20,000, made computationally feasible by SABLE's scalability.

AuthorsOpenReview ↗

Question 2: Is SABLE more than an extension of Fan et al. 2026? (Weaknesses 2.1 and 2.3)

We acknowledge Fan et al. (2026) as the foundation for affine-surrogate BO on spherical domains and fast EI acquisition optimisation (Sections 1 and 4). SABLE builds on this foundation but addresses two distinct and substantial limitations: sample efficiency, through decoupled off-span exploration, and computational scalability, through bounded-cost posterior refitting via recompression and efficient orthogonal-frame construction. Crucially, fast acquisition optimisation does not resolve either limitation: standard affine EI can severely underexplore on non-affine objectives, while posterior refitting becomes increasingly expensive as observations accumulate.

Our controlled experiments separately assess these advances. Table 1 evaluates prescribed exploration across MLE, MAP, and hyperparameter marginalisation without compression, while Figure 2 varies only the compression budget, holding exploration and model fitting fixed. On the scientific tasks, our no-schedule ablation uses the same compressed, hyperparameter-marginalised surrogate as SABLE, differing only in exploration control. On Calorimeter DS3, prescribing off-span exploration increases the mean best-observed value from 0.39 to 0.66. Fan et al. (2026), using MAP without compression or scheduling, also attains 0.39 (against SABLE's 0.66), and requires approximately 2.5 seconds per decision (at N=3,000N = 3,000) on Calorimeter DS2, compared with 0.027 seconds for SABLE.

The computational difference becomes more pronounced at larger budgets. Without compression, posterior refitting scales as O(N2D+N3)\mathcal O(N^2D+N^3), whereas SABLE bounds the refitting cost by O(M2D+M3)\mathcal O(M^2D+M^3) for fixed compression budget MM. At N=D=10,000N=D=10{,}000, the compression experiment in Figure 2 reduces decision times from 79.5 seconds for the uncapped representation underlying Fan et al. (2026) to 0.0055 seconds with compression, a roughly 14,000-fold speedup. Our compressed no-schedule ablation also matches or exceeds Fan et al. (2026) in optimisation performance on all four scientific tasks, while requiring approximately 90 times less decision time per step (at N=3,000N = 3,000). The observed computational gap and the growing cost of uncapped refitting make that approach impractical for our N=20,000N=20{,}000, D=40,500D=40{,}500 experiment (Figure 7), where SABLE continues to improve with additional evaluations.

Finally, Table 1 shows relatively small differences between MAP and marginalisation under matched exploration rules. We therefore revised the manuscript to distinguish hyperparameter marginalisation from SABLE's two principal methodological contributions: decoupled off-span exploration and bounded-cost posterior refitting.

Reviewer_ixWiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This manuscript focuses on Bayesian optimization (BO) in high-dimensional latent spaces and proposes a Bayesian linear model for prediction and acquisition for cases where latent dimension exceeds the dataset size. This work extend recent work on high-dimensional linear BO in spherical coordinates.

The proposed method builds on assumption, or often a fact, that high-dimensional BO problem setting does not necessarily imply that geometry of data items span the full high-dimensional space. The proposed linear Bayesian model, at N observations, updates at most N-1 dimensions of the linear model, and exploration is divided into two parts: within the span of the current linear model, and outside (or complement to the span) of the current model, with the exploration outside the current linear model being controlled by a scheduled and scaled norm of the span outside the linear model for improved efficiency. For computational efficiency, this work proposes to embed, or compress, information from N observations onto M pseudo observations.

Similar to (Doumont et al 2026), using the argument that samples from a high-dimensional isotropic Gaussian concentrate on a thin spherical shell with radius \sqrt{D}, the proposed model is also implemented directly in spherical coordinates, hence also avoiding the boundary behaviour. For stability and identifiability, Bayesian linear model in the spherical coordinates is marginalised wrt prior variance of regression coefficients and noise variance using approximative Monte Carlo type marginalisation.

Evaluations include main ablations of the proposed model, including strength of compression (from N to M), affine model marginalisation vs point estimates, and scheduled exploration, across tasks that follow affine model assumption as well as varying amounts of violations from affine model. Interestingly, with the linear model overfitting the existing designs, standard EI seems stuck to the linear span defined by existing designs, whereas the proposed schedules for off-span exploration improves the results.

Proposed method is also evaluated across a set of common benchmark for scientific latent-space optimization. Overall results for the proposed method look promising; however, I would like to argue that results wrt BO step count in Figure 12 are more interesting in practise than those wrt wall clock time in Figure 4.

Critical strengths

The main strength is a very carefully constructed and derived BO method for high-dimensional latent space, where latent dimension exceeds the dimension of dataset set size.

I also found the ablations and experiments to be comprehensive and support to claims of the manuscript.

Critical weaknesses

None (except it is challenging to gain true insight into all presented technical details due to the excessive amount of supplementary material, although I think authors have made their best effort to keep the main text readable).

Questions for authors
1 comment

We thank the reviewer for their careful reading and very encouraging assessment of our work. We particularly appreciate their recognition of the careful methodological construction and derivation, the comprehensive experimental ablations, and the absence of critical weaknesses.

Presentation and supplementary material

We agree that the volume of supplementary material can make the technical details challenging to navigate. In response, we improved the presentation of the methodology in the main text and substantially shortened and streamlined Appendices B, C, D, E, F, H, and I, while preserving the essential technical results and reproducibility details.

Evaluation count versus wall-clock time

We agree that performance against evaluation count is particularly important. With the additional page available, we have included evaluation-count results in the updated Figure 6, alongside wall-clock performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.