acceptodds
Under review as a conference paper at ICLR 2027

HiBBO: HiPPO-based Space Consistency for High-dimensional Bayesian Optimisation

Abstract

Bayesian Optimisation (BO) is a powerful tool for optimising expensive black-box functions, but its effectiveness diminishes in high-dimensional spaces due to sparse data and poor surrogate model scalability. While Variational Autoencoder (VAE)-based approaches address this by learning low-dimensional latent representations, the reconstruction-based objective function often brings the functional distribution mismatch between the latent space and the original space, leading to suboptimal optimisation performance. In this paper, we first analyse the reason why reconstruction-only loss may lead to distribution mismatch and then propose HiBBO, a novel BO framework that introduces space consistency into the latent space construction in VAE using HiPPO—a method for long-term sequence modelling—to reduce the functional distribution mismatch between the latent space and the original space. Experiments on high-dimensional benchmark tasks demonstrate that HiBBO outperforms existing VAE-BO methods in convergence speed and solution quality. Our work bridges the gap between high-dimensional sequence representation learning and efficient Bayesian Optimisation, enabling broader applications in neural architecture search, materials science, and beyond.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic machine learning, HiPPO, Bayesian optimisation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oXJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the functional distribution mismatch between the VAE latent space and the original input space in high-dimensional Bayesian optimization. The authors observe that the reconstruction term in the standard VAE objective enforces pointwise similarity, but does not directly preserve the inter-point kernel relationships used by the latent GP surrogate. Therefore, they represents the original and reconstructed input sequences with HiPPO memory, and adds their discrepancy to the VAE training loss. Under a strict gap condition and a sufficiently large weight, Proposition 1 shows that the final-time regularizer can yield a smaller average kernel-statistic error than reconstruction alone. The idea is reasonable, though the connection between the theory and the training loss actually used turned out to be hard for me to pin down.

Critical strengths
  1. The motivation is solid. Reconstruction loss does not directly control the kernel geometry used by the GP surrogate, and the HiPPO penalty is simple to add to an existing VAE-BO pipeline.

  2. The evaluation covers high-dimensional synthetic functions, image optimization, discrete shape design, molecular design, and continuous control. It includes recent baselines such as INV (Chu et al., 2024), NF (Lee et al., 2025), and GPior (Ramchandran et al., 2025).

Critical weaknesses
  1. Section 3.1 defines the latent/original mismatch in terms of the mean and kernel discrepancies involving eϕe_\phi, mzm_z, kzk_z, mxm_x, and kxk_x. However, Eq. (2) compares the original input trajectory xx with the VAE-reconstructed input trajectory x‾\overline{x}, and Lemma 1 controls ∣Dk(x)−Dk(x‾)∣|D_k(x)-D_k(\overline{x})| within the input space. It is possible I am misreading the proof, but I checked twice and cannot find where eϕe_\phi enters. As written, the result only shows that for a particular input-space kernel kk, an average pairwise kernel statistic of the reconstructed trajectory is controlled by the HiPPO representation discrepancy. It does not establish that the latent/original mismatch defined in Section 3.1 is reduced or that the latent GP surrogate improves. There is also a mismatch between the theory and the training objective. Lemma 1 and Proposition 1 concerns the final-time state MtM_t, whereas Eq. (3) penalizes the concatenation CT=[M1,…,MT]C_T=[M_1,\ldots,M_T] over all prefixes. Each prefix has a different sample size and may have different constants EρE_\rho and ερ\varepsilon_\rho. The paper does not show that the assumptions of Lemma 1 hold uniformly over all prefixes, nor does it verify the strict gap condition and the requirement on λ\lambda. Tables 2-4 do not report λH\lambda_H, while Table 5 uses a HiPPO coefficient of 1, so it is unclear whether the experiments lie in the regime covered by Proposition 1.

  2. HiPPO defines a sequence representation over a time index, but the observation order in BO is not a natural temporal sequence. The initial Sobol design has an arbitrary order, and subsequent samples are appended in the order selected by the acquisition function in BO. The penalty in Eq. (3) is not permutation-invariant, whereas the target statistic DkD_k averages all pairwise kernel values and is permutation-invariant. Thus, the theory controls an order-independent quantity through an order-dependent regularizer. With low-order HiPPO memory, early samples generally enter more subsequent prefix states and can receive greater aggregate weight in Eq. (3). The exact weighting depends on the basis, discretization, and time scaling, but the paper does not discuss this or report an ordering ablation.

  3. The claimed mechanism is that HiPPO regularization reduces the latent/original mismatch, which improves the latent GP and ultimately improves BO. The experiments mainly evaluate final BO performance and do not quantify the first two links. Figure 2 and Figure 5 use only one-dimensional synthetic sequences. According to Appendix A.4, the correlation panels use a first-order polynomial kernel, while the HiPPO representation uses order 5. These examples use neither the actual surrogate kernel nor real BO data, so they provide an illustration rather than evidence for the mechanism in the reported optimization tasks.

  4. The seed protocol is uneven. Figure 3(a) does not specify the number of seeds; Figure 3(b) uses four seeds; Figures 4(a) and 4(b) use five; and Tables 2-4 each list a single seed value: 123, 1, and 2. It would help to know whether these values are representative seeds or the number of runs. The main curves also need some numerical endpoint values and a measure of uncertainty.

  5. The conclusion calls HiBBO lightweight, but I could not find timing or memory numbers. Some wall-clock and peak-memory measurements would help a lot.

Questions for authors
  1. By what rule are BO observations ordered into the HiPPO sequence? How do optimization performance and kernel statistics change under random permutation, sorting by yy, and reverse order? Has the paper considered a permutation-invariant regularizer?

  2. Can the authors report, on the real BO tasks, kernel-mismatch metrics, reconstruction error, and latent-GP calibration metrics for HiBBO and BASE?

  3. Can the authors report total runtime, per-step wall-clock time, and peak GPU memory relative to BASE and the strongest baseline, and clarify whether incremental HiPPO updates are actually used?

  4. Can the authors unify the seed protocol, provide numerical summaries, dispersion measures, and significance tests for final best values, and clarify the meaning of the seed values 123, 1, and 2 in Tables 2-4?

Reviewer_ArDeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study Bayesian optimisation (BO) in the latent space of a variational autoencoder, a problem ubiquitous when it comes to goal-oriented design of structured discrete objects like molecules and peptides. There, the quality of the learned representation often influences the performance of the surrogate, and low reconstruction error does not always imply that the latent representation preserves the relationships between inputs that are relevant for modelling the objective.

To address this issue, the authors propose HiBBO, which adds a consistency regularizer based on HiPPO representations of the original and reconstructed observation sequences. The aim is to preserve more of their geometry during VAE training and thereby improve the latent GP surrogate. After describing HIBBO, the authors provide a theoretical motivation for this construction and evaluate it on synthetic and structured optimisation problems, demonstrating the superiority of their approach.

Critical strengths
  • If one chooses to model the objective in the VAE latent space, the method is reasonable and well-motivated, and I found the main idea easy to follow.
  • The optimisation results in the experiments section are encouraging and suggest that the proposed regularizer can be useful.
Critical weaknesses
  • It seems that the methodological contribution is rather incremental. There is already a substantial line of work addressing the mismatch between the latent representation of a generative model and the BO surrogate [1,2]. Different mechanisms are employed, but the broader motivation is already established. The use of HiPPO provides another, theoretically motivated consistency penalty, but I am not yet convinced that this represents a sufficiently substantial advance.

  • The title and introduction emphasize high-dimensional BO, but the contribution and experimental setting seem more specifically concerned with latent-space BO. Apart from Ackley, the tasks involve structured objects. The competitors are also VAE-based methods. I think the positioning should reflect this more clearly, or the evaluation should include a broader comparison with methods designed for (continuous) high-dimensional BO, e.g., [3]

  • Still on LSBO, to me, the field involves many interacting choices: the generative model, retraining it, the surrogate, the regularizers, and often also a trust-region mechanism. The shared settings used here make the comparisons useful, but they do not tell us how much of the benefit will remain in a stronger or differently configured LSBO pipeline. For instance, the molecular experiment uses JT-VAE, while SELFIES-based transformer VAEs are already used in methods such as LOL-BO [4] and subsequent work like InvBO [2]. I would be interested in whether the improvements persist with such a backbone and across several molecular objectives, e.g., from the Guacamol library instead of just logP [5].

  • In the particular case of molecular drug design, currently, decoupled approaches such as COWBOYS [6] are state-of-the-art, in the sense that they outperform methods that carry BO in the latent space like InvBO and NF-BO. These decoupled approaches only employ the generative model for sampling and keep the surrogate in X\mathcal{X} space, where sometimes relevant kernels exists (e.g., tanimoto kernel on molecular fingerprints). It would be interesting to see whether HiBBO is competitive there, as it would indicate that modifying the latent space is still a relevant line of work.

[1] Advancing Bayesian Optimization via Learning Correlated Latent Space, NeurIPS 2023.

[2] Inversion-based Latent Bayesian Optimization, NeurIPS 2024.

[3] We Still Don't Understand High-Dimensional Bayesian Optimization, AISTATS 2026

[4] Local Latent Space Bayesian Optimization over Structured Inputs, NeurIPS 2022.

[5] Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 2019.

[6] Return of the Latent Space COWBOYS: Re-thinking the use of VAEs for Bayesian Optimisation of Structured Spaces, ICML 2025.

Questions for authors
  • The Ackley experiment results are reported for one seed, the MNIST ones for four seeds, the chemical design ones for 5 seeds... It would be better to have a homogeneous setting of say 5 seeds for everyone (specifically regarding the Ackley experiment)

  • Could you provide the runtime of the method against other baselines?

  • See weaknesses for other questions.

Reviewer_PH91OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of latent space optimisation. By applying the recently proposed principles of HIPPO to BO, the authors are able to regularise the latent space so that it is amenable to GP modelling. This is fresh approach over exiting latent space regularisation schemes and is empirically tested over a set of simple baseline problems.

Critical strengths
  1. The paper is clearly written and pleasant to read, and the core contributions come across well.
  2. The setup is interesting and feels like a novel take on latent space regularisation. Using HiPPO representations to encourage consistency between the original and reconstructed data is an original idea for latent space BO.
Critical weaknesses
  1. Positioning relative to prior work. This is my main criticism. Despite the paper's claim of "latent/original-space mismatch has unfortunately been ignored by the existing VAE-based BO literature", there is in fact a substantial body of existing work on regularising latent spaces for latent space BO, in particular on making latent spaces more amenable to GP modelling. The two most relevant references are LOLBO (Maus et al., 2022) and CoBO (Lee et al., 2023), which tackle essentially the same motivation as this paper. Both a discussion of these methods and an experimental comparison against them are essential for assessing the contribution.
  2. Depth of experiments. The chosen problems (MNIST, shapes, penalised LogP) are toy problems; the paper itself describes the shape task as “illustrative”. The VAE-driven BO literature has moved well beyond these benchmarks, for example the comprehensive set of test problems in González-Duque et al. (2024), or the experiments considered in LOLBO. Evaluation on more challenging, contemporary benchmarks would substantially strengthen the paper.
  3. Balance of exposition. A disproportionately large part of the paper is spent on HiPPO, an existing approach not designed for BO, rather than on the proposed BO algorithm. HiBBO itself is described only in the short Section 3.3, with the algorithm deferred to the appendix. As a result, there is little detail in the main paper about what actually happens within HiBBO steps, such as how the VAE is retrained at each number of BO steps and so its unclear how the efficient update of M helps things. Reducing the HiPPO discussion would make room to explain HiBBO more clearly.
  4. Figure 1. I struggled to understand Figure 1. It needs a much fuller explanation, both in the caption and the text. This is a missed opportunity, as a clear overview figure could do a lot to explain what is going on.
Questions for authors
  1. Why is the efficient update of the HiPPO memory (Eq. 4) important in this setting? The approach appears to require retraining the VAE (a full neural network) at each update step anyway, which would presumably dominate the computational cost.
  2. How does HiBBO relate to LOLBO and CoBO, both conceptually and empirically? Can you provide experimental comparisons against these methods?
  3. How does HiBBO perform on more challenging benchmarks, such as those in González-Duque et al. (2024) or those used in LOLBO?
  4. Could you clarify Figure 1 and expand its caption to explain each component of the diagram?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.