- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the functional distribution mismatch between the VAE latent space and the original input space in high-dimensional Bayesian optimization. The authors observe that the reconstruction term in the standard VAE objective enforces pointwise similarity, but does not directly preserve the inter-point kernel relationships used by the latent GP surrogate. Therefore, they represents the original and reconstructed input sequences with HiPPO memory, and adds their discrepancy to the VAE training loss. Under a strict gap condition and a sufficiently large weight, Proposition 1 shows that the final-time regularizer can yield a smaller average kernel-statistic error than reconstruction alone. The idea is reasonable, though the connection between the theory and the training loss actually used turned out to be hard for me to pin down.
Critical strengths
-
The motivation is solid. Reconstruction loss does not directly control the kernel geometry used by the GP surrogate, and the HiPPO penalty is simple to add to an existing VAE-BO pipeline.
-
The evaluation covers high-dimensional synthetic functions, image optimization, discrete shape design, molecular design, and continuous control. It includes recent baselines such as INV (Chu et al., 2024), NF (Lee et al., 2025), and GPior (Ramchandran et al., 2025).
Critical weaknesses
-
Section 3.1 defines the latent/original mismatch in terms of the mean and kernel discrepancies involving , , , , and . However, Eq. (2) compares the original input trajectory with the VAE-reconstructed input trajectory , and Lemma 1 controls within the input space. It is possible I am misreading the proof, but I checked twice and cannot find where enters. As written, the result only shows that for a particular input-space kernel , an average pairwise kernel statistic of the reconstructed trajectory is controlled by the HiPPO representation discrepancy. It does not establish that the latent/original mismatch defined in Section 3.1 is reduced or that the latent GP surrogate improves. There is also a mismatch between the theory and the training objective. Lemma 1 and Proposition 1 concerns the final-time state , whereas Eq. (3) penalizes the concatenation over all prefixes. Each prefix has a different sample size and may have different constants and . The paper does not show that the assumptions of Lemma 1 hold uniformly over all prefixes, nor does it verify the strict gap condition and the requirement on . Tables 2-4 do not report , while Table 5 uses a HiPPO coefficient of 1, so it is unclear whether the experiments lie in the regime covered by Proposition 1.
-
HiPPO defines a sequence representation over a time index, but the observation order in BO is not a natural temporal sequence. The initial Sobol design has an arbitrary order, and subsequent samples are appended in the order selected by the acquisition function in BO. The penalty in Eq. (3) is not permutation-invariant, whereas the target statistic averages all pairwise kernel values and is permutation-invariant. Thus, the theory controls an order-independent quantity through an order-dependent regularizer. With low-order HiPPO memory, early samples generally enter more subsequent prefix states and can receive greater aggregate weight in Eq. (3). The exact weighting depends on the basis, discretization, and time scaling, but the paper does not discuss this or report an ordering ablation.
-
The claimed mechanism is that HiPPO regularization reduces the latent/original mismatch, which improves the latent GP and ultimately improves BO. The experiments mainly evaluate final BO performance and do not quantify the first two links. Figure 2 and Figure 5 use only one-dimensional synthetic sequences. According to Appendix A.4, the correlation panels use a first-order polynomial kernel, while the HiPPO representation uses order 5. These examples use neither the actual surrogate kernel nor real BO data, so they provide an illustration rather than evidence for the mechanism in the reported optimization tasks.
-
The seed protocol is uneven. Figure 3(a) does not specify the number of seeds; Figure 3(b) uses four seeds; Figures 4(a) and 4(b) use five; and Tables 2-4 each list a single seed value: 123, 1, and 2. It would help to know whether these values are representative seeds or the number of runs. The main curves also need some numerical endpoint values and a measure of uncertainty.
-
The conclusion calls HiBBO lightweight, but I could not find timing or memory numbers. Some wall-clock and peak-memory measurements would help a lot.
Questions for authors
-
By what rule are BO observations ordered into the HiPPO sequence? How do optimization performance and kernel statistics change under random permutation, sorting by , and reverse order? Has the paper considered a permutation-invariant regularizer?
-
Can the authors report, on the real BO tasks, kernel-mismatch metrics, reconstruction error, and latent-GP calibration metrics for HiBBO and BASE?
-
Can the authors report total runtime, per-step wall-clock time, and peak GPU memory relative to BASE and the strongest baseline, and clarify whether incremental HiPPO updates are actually used?
-
Can the authors unify the seed protocol, provide numerical summaries, dispersion measures, and significance tests for final best values, and clarify the meaning of the seed values 123, 1, and 2 in Tables 2-4?