- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem of Bayesian Optimization (BO) in high-dimensional latent spaces where the objective function is computationally inexpensive, but pre-defined constraints exist. Standard Variational Autoencoders (VAEs) are trained without regard for whether inputs satisfy these constraints, leading to an intermingling of valid and invalid inputs within the latent space. The authors point out that this mixing complicates the optimization of the acquisition function and degrades sampling efficiency. The proposed method, CooL-BO, incorporates information regarding constraint satisfaction directly into the geometric structure of the latent space during the VAE pre-training phase. CooL-BO consists of the following three components:
- A linear discriminant term, , is added to the Evidence Lower Bound (ELBO) of the joint distribution .
- Class-specific Gaussian priors, , are employed for inputs that satisfy the constraints and those that do not. In each iteration, the center of each prior distribution is recalculated using the mean of the posterior distributions of the data points belonging to that respective class.
- The discriminator and the VAE are updated in an alternating fashion.
Critical strengths
1. A simple design that directly reduces the cost of constrained acquisition optimization. CooL-BO shapes the latent space so that feasibility becomes a single linear inequality . This integrates directly with TuRBO’s box trust region and removes the need for separate constraint surrogates such as those in HDSafeBO. Since it adds almost no learnable parameters, it can be plugged into existing latent BO pipelines with minimal overhead.
2. The learned representations are evaluated directly, beyond final BO performance. The paper reports latent separability (Fisher LDA ratio, linear probe) and decoded feasibility of posterior samples (Table 5), compared against conditional VAE, VampPrior, and contrastive VAE. Table 5 is the most relevant of these, since decoded feasibility matters more for BO than latent separability.
Critical weaknesses
1. The paper does not show that the addressed problem incurs a real cost. Since the constraints are known and cheap to evaluate, infeasible proposals could simply be filtered by checking after decoding, before calling the oracle. The paper does not compare against this alternative.
2. There is a logical gap between the method and sample efficiency. The linear constraint reduces optimization cost. Sample efficiency, however, depends on whether arbitrary in the half-space decodes to feasible inputs, while the discriminator only separates encoder outputs.
Questions for authors
1. Comparison with post-decoding filtering alternatives. Does the count of oracle evaluations include infeasible candidates? Could you provide a comparison of the following three approaches, all using the same TuRBO settings? (a) Standard VAE + filtering via known constraints after decoding (b) Standard VAE + constraints imposed by a linear classifier trained on the latent space (c) CooL-BO VAE + GP-based constraints This comparison would help disentangle whether the performance gains stem from the learned representations, the use of linear-constrained Bayesian optimization (BO), or the avoidance of wasted oracle evaluations on infeasible candidates.
2. Performance improvements in tasks with pre-existing linear separability. CooL-BO demonstrates performance gains even in synthetic tasks where the latent space of a standard VAE is already nearly linearly separable (as evidenced by accuracy ≥ 0.99 for Ackley20h, Keane50h, and Powell200h in Table 6). To what do you attribute these gains? If it is not due to the resolution of entanglement, could you explain what other mechanism might be at play?