acceptodds
Under review as a conference paper at ICLR 2027

BOReFT: Manifold Steering of Language Models for Black-box Optimization

Abstract

Language models are increasingly used as proposal models for black-box search, from program optimization to molecular design. Existing approaches typically improve proposals through iterative prompting or parameter updates, offering limited control over how completely and efficiently the model's search space is explored. Continuous optimization methods, such as Bayesian optimization, provide a principled way to search but require a suitable domain to operate over. To address this, we introduce BOReFT, which learns a compact, low-dimensional space of hidden-state interventions in a frozen language model, and uses this space as the search domain for Bayesian optimization with an external scoring function. Empirically, we find that the learned domain spans semantic regions and exhibits smoothness properties that support search. Theoretically, we show that semantic coverage and interpolation control the best score available in the learned space, and that decoding from this space yields a standard stochastic-bandit observation model for adaptive search. We evaluate BOReFT on the interpretable word search task "Semantle" and on three more real-world discovery tasks in de novo molecule property optimization. Compared to strong LLM baselines, BOReFT finds in Semantle a higher number of hidden targets and, on two out of three molecular objectives, achieves higher property scores. Consequently, our method provides a principled new bridge between discrete proposal spaces of LLM-based search and continuous black-box optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
black-box optimization, Bayesian optimization, large language models, representation learning, representation finetuning, latent-space search, hidden-state interventions, test-time search, scientific discovery, molecular optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sDE9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors provide an approach for applying Bayesian Optimization techniques within language models by performing a BO loop on a lower-dimensional continuous subspace of the model. The authors then apply this method with a reformulated training objective to two categories of tasks including semantle word search and molecular property optimization.

Critical strengths
  • The presentation and framing of the theoretical results is very clear, and the authors do a good job of emphasizing the important takeaways throughout the text
  • The construction of the intervention is very clean, providing an affine transformation that maps into the activation space allowing for direct intervention without any cumbersome machinery of other decoder/encoder models
  • Baselines are all fair comparisons since the models used for comparison are all additionally LoRA tuned using the same settings on the same dataset. This cleanly isolates out the effect of the method in a compelling way
Critical weaknesses
  • More seeds would give more robust statistics on the results, as the current setup only uses 5 seeds
  • The paper never reports training cost of this method, which would be useful to understand when thinking about how to apply this method
  • The metric used is inconsistent in certain cases, switching from mean to max. This can confuse comparisons of performance
Questions for authors
  • How does rank rr trade off representational gap against search gap? The box’s volume grows multiplicatively in rr, so there should be a visible optimum
  • Terminology wise, the intervention is affine in bb, the codes are a box in a linear subspace. What is the justification for “manifold”?
  • More tasks should be assessed before making broad generalizations such as "program optimization" in the abstract
3 comments
AuthorsOpenReview ↗

Author Response (1/n)

Thank you for your thoughtful review, and for highlighting the clarity of our theoretical framing, the simplicity of intervening directly in the frozen LLM's activation space, and the fairness in our evaluation setup. Below we address each of your concerns and questions.


Weaknesses

W1: More seeds would give more robust statistics on the results...

To clarify the evaluation sample sizes:

  • Semantle evaluates 5 targets per split ×\times 3 independent warm-start seeds = 15 runs on held-out targets (Table 1, Fig. 3 with ±1\pm 1 SD bands) and 15 runs on training targets (Table 4), i.e., 30 runs per method in the ablations and scaling plots (Fig. 4, Table 10). During the rebuttal we also replicated the full 30-run Semantle protocol on Qwen3.5-2B, observing the same >2×2\times held-out exact-match advantage for BOReFT over the best baseline (7/15 vs. 3/15).
  • Molecular optimization evaluates 5 seeds ×\times 3 protein objectives = 15 runs per method (Table 2; ±1\pm 1 SD trajectories in Fig. 5). We will add per-cell standard deviations alongside means in Tables 1–2 and expand molecular seed counts for the camera-ready revision.

Note that each run is a budget-500 search; therefore, running many more seeds can become expensive (in terms of both time and compute). We already observe tight standard deviation on Semantle, so don't see more seeds helping; however, given compute availability, we will aim to add more seeds for molecule optimization by the final paper version.


W2: The paper never reports training cost of this method...

Thank you for pointing out this omission. We will add a dedicated compute table to App. B. Every experiment ran on a single NVIDIA A100 GPU:

TaskOne-time Subspace TrainingSearch Cost (s / verification)Full 500-eval Seed
Semantle (Llama-3.2-1B, N=3,072N=3{,}072)8.36 A100-h4.62 s (vs. BOPRO 1.04 s, MiGrATe 2.49 s, SDPO-TTT 12.9 s)~0.53 h
Molecules (MiST-3B, N=1,024N=1{,}024)23.71 A100-h6.17 s (vs. BOPRO 5.39 s, MiGrATe 10.8 s, SDPO-TTT ~97 s)~0.86 h

Crucially, subspace training is amortized across downstream searches because a single 23.7-h ChEBI-20 run is reused across all 15 search runs spanning DRD2, GSK3$\beta$, and JNK3 (~1.58 h amortized per run, dropping further with each new target), while the trainable intervention (R,W)(R, W) itself adds only 2×r×d≈0.26M2 \times r \times d \approx 0.26\text{M} parameters.


W3: The metric used is inconsistent in certain cases, switching from mean to max. This can confuse comparisons of performance

Thank you for raising this. We consistently report both mean/max in Table 2 because molecule discovery values both average reliability (mean of per-seed discoveries) and peak candidate discovery (max across seeds, matching Cor. 4.3), whereas Semantle reports exact-match count (EM) and mean best similarity (Sim.). However, in §5.2 (ll. 88–90) our text cited both mean and max on JNK3 (0.370/0.940) while highlighting only max on GSK3$\beta$ (0.446/0.670, where BOPRO has slightly higher before-SFT mean 0.470/0.630). We will revise §5.2 to state mean ± SD and max side-by-side everywhere so comparisons are uniform throughout.

AuthorsOpenReview ↗

Author Response (2/2)

Questions

Q1. How does rank r trade off representational gap against search gap? The box’s volume grows multiplicatively in r, so there should be a visible optimum

We show this trade-off empirically in our rank sweep r∈{4,8,16,32,64,128}r \in \{4, 8, 16, 32, 64, 128\} at N=3,072N=3{,}072 (App. B.15, Fig. 8), which remains consistent with Eq. 12 and Lemma A.2(i) (diam(B)≤2σ02rκmax⁡\mathrm{diam}(\mathcal{B}) \le 2\sigma_0\sqrt{2 r \kappa_{\max}}):

Rank rrUnseen Word Recon. (n=928n=928, Greedy / Sim.)diam(B)\mathrm{diam}(\mathcal{B})Search EM (Train / Held-out / Total)Search Mean Sim.
40.009 / 0.6863.23/15 / 2/15 / 5/300.823
80.042 / 0.7153.82/15 / 1/15 / 3/300.799
160.110 / 0.7494.75/15 / 5/15 / 10/300.869
320.206 / 0.7896.70/15 / 2/15 / 2/300.794
640.699 / 0.9407.16/15 / 8/15 / 14/300.878
1280.730 / 0.94712.64/15 / 6/15 / 10/300.844

As rr grows, held-out reconstruction increases monotonically (0.009 → 0.730), shrinking the representational gap. Beyond r=64r=64, however, diam(B)\mathrm{diam}(\mathcal{B}) nearly doubles (7.1 → 12.6) and BO must search a 128128-D box under the same fixed budget Q=500Q=500, enlarging the search gap so target recovery drops from 14/30 at r=64r=64 back to 10/30 at r=128r=128 despite slightly higher representational capacity—yielding the interior optimum at r=64r=64. We will add these reconstruction and diameter columns beside Fig. 8.


Q2. Terminology wise, the intervention is affine in b, the codes are a box in a linear subspace. What is the justification for “manifold”?

In hidden-state space, h′(b)=h0+R⊤bh'(b) = h_0 + R^\top b (Eq. 2) is indeed a flat rr-dimensional affine subspace of Rd\mathbb{R}^d restricted to the box B⊂Rr\mathcal{B} \subset \mathbb{R}^r (Eq. 8). We used "intervention manifold" (following recent steering literature, e.g., Wurgaft et al., 2026) because passing h′(b)h'(b) through the remaining nonlinear transformer blocks maps B\mathcal{B} onto a curved rr-dimensional family of output distributions b↦Pbτb \mapsto P_b^\tau over sequences. That said, we agree "affine intervention subspace" / "learned code space" is more precise for B\mathcal{B} and h′(b)h'(b), and we will clarify this terminology in §1 and §3.1. Thank you for the question.


Q3. More tasks should be assessed before making broad generalizations such as "program optimization" in the abstract

Our aim with that sentence in the abstract was to illustrate how prior work uses LLMs as proposal generators broadly, whereas our evaluated claims are scoped to Semantle and de novo molecular property optimization tasks. We can revise that opening sentence to avoid being read as claiming empirical evaluation on program optimization, and note code/program search explicitly as future work in §6.


We hope our responses have helped address the concerns in your initial review. Thank you again for the helpful questions and for your consideration!

Reviewer_sDE9OpenReview ↗

Rebuttal Acknowledgement

I thank the authors for their rebuttal and the clarification against many of the points in my initial review. While I view this paper more favorably than before and feel this is a good paper, I still do not feel comfortable assigning it a 4 of maximum possible score, so I will maintain my initial score of 3.

Reviewer_NrA5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BOReFT, a black-box optimization framework that integrates representation interventions within large language models (LLMs) with Bayesian optimization (BO). Rather than directly updating model parameters or searching over discrete outputs, BOReFT first learns a low-dimensional continuous intervention space through representation interventions and a joint loss function, aiming to preserve reconstruction capability, semantic structure, and generative diversity. It then employs Bayesian optimization to iteratively search this space, guiding the language model to generate high-scoring candidates. Furthermore, the paper decomposes the optimization gap into representation and search errors and analyzes the effects of semantic coverage, interpolation error, and stochastic feedback on optimization performance. The effectiveness of BOReFT is evaluated on the Semantle word search task and three molecular property optimization tasks.

Critical strengths

S1: This work reformulates discrete sequence generation as an optimization problem over a continuous hidden-state intervention space. Compared with conventional prompt optimization, LoRA fine-tuning, and Bayesian optimization (BO) over fixed candidate embeddings, the proposed method enables BO to directly operate on continuous variables that can be decoded by the generative model.
S2: BOReFT learns the intervention space during training, whereas at inference it fixes the model and its intervention parameters and searches only over the code b. This decoupling allows the same generative space to serve different downstream scoring functions.

Critical weaknesses

W1: The core components of BOReFT are largely derived from prior work, including LoReFT’s low-rank representation interventions, VAE-style latent variable learning, KL-based self-distillation, and Gaussian process Bayesian optimization. The main contribution of this paper lies in integrating these techniques into a continuous search framework that directly generates discrete candidates.
W2: The paper actually constructs an axis-aligned hyperrectangle in the low-dimensional code space, but does not establish that the corresponding set of generative representations possesses the local geometric properties required of a manifold.
W3: The proposed regret decomposition is essentially an algebraic decomposition and does not, in itself, constitute a new optimization guarantee.

Questions for authors

Q1. Can the authors demonstrate that the performance improvements stem from the learned geometry of the intervention space, rather than merely from additional training data and teacher supervision?
Q2. Why is a 64-dimensional axis-aligned hyperrectangle an appropriate search domain for Bayesian optimization? Is there empirical evidence supporting the existence of an effective low-dimensional structure?
Q3. Can the authors demonstrate that the high-scoring molecules exhibit chemically meaningful properties and sufficient diversity, rather than merely exploiting biases in the property predictors?
Q4. When total computational cost is matched and the warm-start sets are constructed independently, does the BOReFT’s performance advantage still persist?

3 comments
AuthorsOpenReview ↗

Author Response (1/n)

Thank you for your constructive review, and for highlighting how BOReFT enables BO to act directly on continuous hidden-state coordinates inside a frozen LLM while decoupling space learning from downstream objectives. Below we address each of your concerns and questions.


Weaknesses

W1: ...largely derived from prior work...main contribution of this paper lies in integrating these techniques into a continuous search framework that directly generates discrete candidates.

While BOReFT builds on established primitives, we show that combining LoReFT with BO naively does not yield a searchable generative space; for example, reconstruction alone leaves held-out reachability undetermined (Prop. A.1). Our contribution is identifying the specific representation-learning recipe needed to turn a frozen LLM's residual stream into a compact, generalizable search domain, and verifying both theoretically (§4.2, Lemma A.2) and empirically that each component contributes positively to search performance. Removing variational training, the shared encoder, reconstruction, or on-policy forward-KL self-distillation drops Semantle target recovery from 14/30 to 3/30, 5/30, 7/30, and 9/30, respectively (§5.2, Fig. 4 right; App. B.13, Fig. 7, Table 10).


W2: ...constructs an axis-aligned hyperrectangle in the low-dimensional code space, but does not establish that the corresponding set of generative representations possesses the local geometric properties required of a manifold.

We use "intervention manifold" informally (following recent usage in works such as [1]) for the rr-dimensional affine intervention slice h′(b)=h0+R⊤b⊂Rdh'(b) = h_0 + R^\top b \subset \mathbb{R}^d (r=64≪dr = 64 \ll d; Eq. 2) parameterized over the compact box B⊂Rr\mathcal{B} \subset \mathbb{R}^r (Eq. 8), rather than claiming a curved Riemannian manifold. Regarding the local smoothness and semantic continuity needed for BO over B\mathcal{B}, we establish this both theoretically and empirically:

  • Analytical continuity: All frozen transformer layers after the intervention site are continuous, so the logits—and thus at any τ>0\tau > 0 the output probabilities Pbτ(s)P_b^\tau(s) and expected objective Fτ(b)=ES∼Pbτ[f(S)]F_\tau(b) = \mathbb{E}_{S \sim P_b^\tau}[f(S)] (Eq. 11)—vary smoothly with b∈Bb \in \mathcal{B} (§4.1 fn. 2, App. A).
  • Empirical semantic interpolation (§4.2, §5.2, App. B.10): Qualitatively, linear paths between training means in B\mathcal{B} produce coherent semantic transitions across unseen words (Fig. 2). Quantitatively, across 40 mid-path interpolations (t∈{0.4,0.5,0.6}t \in \{0.4, 0.5, 0.6\}), BOReFT's decoded embeddings track ideal semantic interpolants xαx_\alpha (Eq. 13) more closely (Δinterp=0.338\Delta_{\mathrm{interp}} = 0.338) than both an endpoint-only non-interpolating reference (0.3650.365) and all four ablated variants (0.355–0.3960.355\text{–}0.396; Fig. 4 right, Table 10), directly tracking downstream BO performance.
  • We can clarify our writing as "affine intervention subspace" in §3.1 so "manifold" is not read as a claim about differential-geometry.

[1] Wurgaft et al., 2026. Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior


W3: The proposed regret decomposition is essentially an algebraic decomposition and does not, in itself, constitute a new optimization guarantee.

We agree that Eq. 12 describes a logical identity, which is the intended role, rather than presenting a new bound. Unlike standard BO over a fixed continuous function, BOReFT has two decoupled stages, so total regret f⋆−Fτ(b)f^\star - F_\tau(b) splits into:

  • Training-time (representational) gap (f⋆−Fτ⋆f^\star - F_\tau^\star): Fixed once subspace training finishes, and measures whether high-scoring candidates exist in B\mathcal{B} at all. Prop. 4.1 (Eq. 15) is the bound on this term, controlling it via training-set semantic coverage εcov\varepsilon_{\mathrm{cov}} and decoder interpolation error εint\varepsilon_{\mathrm{int}} (Eq. 14; empirically validated in Fig. 4).
  • Test-time (search) gap (Fτ⋆−Fτ(bt)F_\tau^\star - F_\tau(b_t)): Controlled by the adaptive optimizer during search. Prop. 4.2 and Cor. 4.3 (Eq. 16) establish that noisy decodes yt=f(St)y_t = f(S_t) form an unbiased, conditionally (Δf/2)(\Delta_f/2)-sub-Gaussian bandit observation model for FτF_\tau, converting any standard cumulative-regret bound RQR_Q into a high-probability guarantee on the best sequence discovered.
AuthorsOpenReview ↗

Author Response (2/n)

Questions


Q1. ...demonstrate that the performance improvements stem from the learned geometry of the intervention space, rather than merely from additional training data and teacher supervision?

Two experiments in the paper show that data and teacher supervision without structured code geometry are insufficient.

  • Same training data, no intervention geometry ("After SFT", §5.1–§5.2): Fine-tuning all baselines via LoRA on the same representation-training targets gives them identical data exposure, yet none approaches BOReFT on held-out Semantle (≤2/15 EM vs. BOReFT's 8/15, Table 1), and SFT, in fact, degrades most baselines as proposal novelty collapses (Tables 1–2).
  • Same data and same teacher supervision, ablated geometry (§5.2, Fig. 4 right; App. B.13, Table 10): Holding the training targets and self-distillation fixed while removing the geometric components (VAE prior and shared encoder) sharply degrades both interpolation (Δinterp\Delta_{\mathrm{interp}}) and search: (a) removing variational regularization drops recovery from 14/30 (Δinterp=0.338\Delta_{\mathrm{interp}} = 0.338) to 3/30 (0.3560.356), and (b) removing the shared encoder—leaving free per-target posteriors under the exact same reconstruction + teacher-distillation loss—drops recovery to 5/30 (0.3820.382, worse than the non-interpolating endpoint reference 0.3650.365) while reducing pairwise semantic-to-code Spearman correlation from 0.430.43 to 0.310.31.

Q2. Why is a 64-dimensional axis-aligned hyperrectangle an appropriate search domain for Bayesian optimization? Is there empirical evidence supporting the existence of an effective low-dimensional structure?

Three pieces of design and empirical evidence support this choice:

  • Domain construction (App. B.8, Table 7): The axis-aligned bounding box (AABB, k=0k=0, Eq. 8) tightly encloses the learned posterior means while remaining convex and cheap to optimize over; Lemma A.2(i) bounds its diameter via the KL prior. Expanding the box even slightly by posterior spread (k=0.1,0.5,1.0k = 0.1, 0.5, 1.0) inflates volume by 105–1032×10^5\text{–}10^{32}\times, collapsing Semantle recovery from 6/15 train and 8/15 held-out down to 3/15 & 4/15 (k=0.1k=0.1) and 0/15 (k≥0.5k \ge 0.5) as repetition climbs above 0.910.91. Conversely, shrinking to a covering ellipsoid (4.1×10−4 V04.1\times 10^{-4}\,V_0) or local TuRBO-1 trust regions (6.3×10−7 V06.3\times 10^{-7}\,V_0) yields 5/15 & 5/15 and 1/15 & 1/15 on Llama-3.2-1B, making the k=0k=0 AABB the best default there (with the covering ellipsoid also serving as a strong alternative on other backbones). Moreover, our GP uses an ARD Matérn-2.52.5 kernel over a learned projection (App. B.3), letting the surrogate downweight uninformative directions inside B\mathcal{B}.
  • Empirical rank ablation (App. B.15, Fig. 10): Comparing rank r∈{4,8,16,32,64,128}r \in \{4, 8, 16, 32, 64, 128\} inside the frozen 20482048-D residual stream (r≪dr \ll d) shows that low-rank subspaces already support meaningful search even at r=4–16r = 4\text{–}16, with r=64r = 64 achieving peak target recovery (14/30) before saturating at r=128r = 128.

(continued in the next response)

AuthorsOpenReview ↗

Author Response (3/3)

Questions (contd.)

Q3. Can the authors demonstrate that the high-scoring molecules exhibit chemically meaningful properties and sufficient diversity, rather than merely exploiting biases in the property predictors?

  • We parse and sanitize every candidate with RDKit (Chem.MolFromSmiles with full valence, kekulization, aromaticity, and ring-closure checks; invalid molecules score 00, §5.1), so chemically impossible graphs are rejected. Even before any post-hoc SmiSelf repair, BOReFT's native RDKit chemical validity is 99.82% across runs (Table 6), because steering happens inside the chemistry-pretrained MiST backbone trained on curated ChEBI-20 metabolites and drug-like molecules.
  • Further, we inspected the top-found molecules and find that: (a) none of them appears in the N=1,024N=1{,}024 ($\le$90th-percentile) representation-training set (nearest-neighbor Morgan Tanimoto to training molecules is 0.19–0.280.19\text{–}0.28), and (b) all satisfy Lipinski's Rule of Five (00 violations; QED∈[0.47,0.82]\mathrm{QED} \in [0.47, 0.82]). To check whether the top generations are meaningful, and in the absence of an expert human evaluation during the rebuttal period, we use an LLM evaluation (here, Gemini 4 Carbon, pasted as generated) to check the top molecule for each property predictor:
    • DRD2: The top candidate (CN(C)CCCN(C)c1ccc2c(c1)CCCc1ccccc1-2, score 0.782, QED=0.815\mathrm{QED}=0.815, zero PAINS/Brenk alerts) is a novel molecule absent from PubChem that combines a tricyclic CNS core with a propyl-linked tertiary amine—the classic protonatable-amine + aromatic pharmacophore of tricyclic monoamine/D2 ligands.
    • GSK3$\beta$ & JNK3 (kinases): On JNK3, the top hit (0.940, QED=0.626\mathrm{QED}=0.626) is sunitinib (CID 5329102), an FDA-approved indolin-2-one multi-kinase inhibitor with documented cross-kinase JNK activity; on GSK3$\beta$, the top hit (0.670, QED=0.467\mathrm{QED}=0.467) is ibrutinib (CID 24821094), an FDA-approved aminopyrazolopyrimidine kinase inhibitor, alongside novel PubChem-absent heteroaromatic scaffolds such as a pyridyl-benzoxazole (0.500, QED=0.617\mathrm{QED}=0.617, zero PAINS/Brenk alerts).
  • Regarding diversity (App. B.6, Table 5), BOReFT maintains a low repetition rate of 0.133 (86.7% distinct molecules across the 500-query trajectory), which is more diverse than OPRO both before SFT (0.223) and after SFT (0.541), but not the lowest in the baselines we compare (e.g., SDPO-TTT is lower 0.176/0.183).

Q4. When total computational cost is matched and the warm-start sets are constructed independently, does the BOReFT’s performance advantage still persist?

  • Our main comparisons already match the black-box oracle budget (Q=500Q = 500 evaluations in Tables 1–2, Figs. 3, 5; extended only to Q=2000Q = 2000 in App. B.14, Tables 11–12, Figs. 8–9) on the exact same frozen base LLM, and our "After SFT" comparisons match training-data access as well. Per-query search wall-clock is comparable to or faster than test-time-training and tree-search baselines (e.g., 3.5×3.5\times faster than MiGrATe on Semantle and an order of magnitude faster than SDPO-TTT; App. B.5), while BOReFT's one-time, objective-free subspace training is amortized across arbitrary downstream objectives and targets (e.g., one ChEBI-20 subspace is reused across DRD2, GSK3$\beta$, and JNK3; §5.1).
  • We share the same 10 warm-start candidates per (target, seed) across all methods, following standard variance-reduction practice in BO benchmarking so that differences reflect the search algorithm rather than a lucky initial random draw (note that on molecules the 10 Sobol warm starts come from the $\le$90th-percentile training manifold and are deliberately low-scoring for every method). Drawing different warm starts per method would only inject unshared initialization variance without changing expected performance across seeds.

We hope our responses to your comments and questions have helped address the concerns in your initial review. Thank you for your consideration!

Reviewer_njhrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces BOReFT, a method which learns a low-dimensional representation for steering language models. The representation is then used with Bayesian optimization to search the space for outputs with good objective scores. The paper demonstrated that on Semantle word search and de novo molecular design tasks, BOReFT achieves better objective scores and finds more hidden targets compared to other LLM baselines.

Critical strengths

The paper showed that by learning a low dimensional representation for steering the LLM, BOReFT can combine existing methods that searches within a continuous space and find candidates that's not in the original training set. This is a promising way to make LLM proposal generation more controllable on tasks such as de novo drug design.

Critical weaknesses

I think methodologically, this paper is very similar to the cited Local latent space Bayesian optimization over structured inputs (LOLBO) paper. Both paper uses an autoregressive transformer to generate candidates, transforming a discrete optimization problem into a continuous one. The difference is the LOLBO paper trained the encoder and autoregressive decoder parts from scratch, while BOReFT uses off the shelf qwen3-0.6b embedding model as the encoder and fine-tuned Llama-3.2/Qwen2.5 models for the decoder part. I'm not fully convinced that this is a 'new' bridge between discrete proposal spaces and continuous black-box optimization. The authors should demonstrate that using fine-tuned LLMs have an advantage over the relevant latent-space BO alternatives, or narrow the contributions and relevant claims.

Questions for authors
  1. Does the scale of the chosen LLMs affect any of the results? For example would a 9B model be better on synthesizing new molecules.
  2. The base models in this paper (llama-3.2/Qwen2.5) are all ~2 years old. Given LLMs (both closed and open source ones) have become significantly better at instruction following/agentic workflows, would BOReFT still be preferable if we swap the baseline model to something more recent for example Gemma4/Qwen3.5?
  3. As a follow-up to 2, since methods like OPRO is based on prompting only, I'm curious to see how OPRO does with a frontier model, and how it compares with BOReFT in terms of practical usefulness.
3 comments
AuthorsOpenReview ↗

Author Response (1/n)

Thank you for taking the time to review our work! Reading the comments carefully, we believe the main critique might be stemming from a misunderstanding of our method. We, therefore, first summarize BOReFT as presented in §3–§4.

BOReFT Summary

  • Our main motivation with BOReFT is in being able to leverage the massive pretrained knowledge of large language models together with a principled mathematical framework, i.e. Bayesian optimization, for search and discovery, with an emphasis on being able to do this without handcrafted heuristics, such as those in many previous works that have used in-context learning.
  • Furthermore, BOReFT does not fine-tune an LLM. Given a completely frozen LLM MM, it separates representation learning from black-box search into two phases.
  • Phase 1: Objective-free manifold learning (§3.1–§3.2). At a single residual-stream site (single layer and token position; App. B.1), we introduce a shared rank-rr LoReFT intervention (r=64≪hidden dimension dr = 64 \ll \text{hidden dimension}\, d), h′(b)=h0+R⊤b,R,W∈Rr×d,b∈Rr(Eqs. 1–2),h'(b) = h_0 + R^\top b, \qquad R, W \in \mathbb{R}^{r \times d}, \quad b \in \mathbb{R}^r \quad \text{(Eqs. 1–2)}, where varying the rr-dimensional vector bb steers the frozen LLM's output distribution PbτP_b^\tau over discrete sequences. From a small set of unlabeled candidates from the search domain of interest (e.g., words or molecules) and without access to the downstream black-box objective ff, we train only (R,W)(R, W) and a small MLP gψg_\psi predicting diagonal-Gaussian code posteriors (μi,σi)=gψ(ϕ(si,di))(\mu_i, \sigma_i) = g_\psi(\phi(s_i, d_i)) (Eq. 3). The loss combines target reconstruction, a Gaussian prior keeping the codes compact, and on-policy forward-KL self-distillation (Eqs. 4–7).
  • Phase 2: Search over the fixed manifold (§3.3): After Phase I, the encoder ϕ\phi and projection gψg_\psi are discarded, and (M,R,W)(M, R, W) remain fixed. The trained posterior means define a bounded box B⊂Rr\mathcal{B} \subset \mathbb{R}^r (Eq. 8). Bayesian optimization queries coordinates bt∈Bb_t \in \mathcal{B} directly, intervenes on the frozen LLM's residual stream to decode St∼PbtτS_t \sim P_{b_t}^\tau, and updates a GP surrogate from scalar black-box evaluations yt=f(St)y_t = f(S_t) (Eqs. 9–10).
  • Theory (§4): Eq. 12 decomposes regret into a representational gap (bounded via semantic coverage εcov\varepsilon_{\mathrm{cov}} and interpolation εint\varepsilon_{\mathrm{int}} in Prop. 4.1; measured in §5.2, Fig. 4) and a search gap (governed by a sub-Gaussian bandit observation model and discovery guarantee; Prop. 4.2, Cor. 4.3).
AuthorsOpenReview ↗

Author Response (2/n)

Next, we clarify below how our method is different from the raised comparison with LOLBO (Maus et al., 2022).

Differences from LOLBO

  1. Frozen LLM + rank-r steering vs. trained VAE decoder. All weights of Llama-3.2 / Qwen2.5 are frozen in both phases (§3.2). (The "After SFT" columns in Tables 1–2 apply LoRA only to baselines for data parity; BOReFT uses no adapter, App. B.6.) Instead of training a domain sequence decoder (e.g., over SELFIES), BOReFT learns a 64-D affine slice of one hidden state inside a frozen LLM (Eq. 2).
  2. Role of ϕ\phi (unused at search time). LOLBO's sequence encoder qΦ(z∣x)q_\Phi(z\mid x) is invoked during search to encode candidates and recenter trust regions. BOReFT's ϕ\phi is never called during Phase 2 (Alg. 1); in Phase 1 it serves only as a geometric inductive bias on posteriors—explicitly ablated via encoder-free per-target posteriors (w/o shared encoder, §5.2, Fig. 4; App. B.9, Fig. 6). (Note also: on Semantle, ϕ\phi comes from frozen Llama-3.2-1B itself, whereas Qwen3-Embedding-0.6B is the black-box oracle ff; §5.1, App. B.1.)
  3. Fixed, task-agnostic domain vs. online joint VAE–GP retraining. LOLBO jointly retrains its VAE and GP online on objective-labeled data inside TuRBO trust regions. BOReFT performs zero LLM or intervention updates during search (and does not use TuRBO; App. B.8, Table 7). One objective-free manifold trained once on 1,0241{,}024 low-scoring molecules ($\le$90th percentile) is searched unchanged across DRD2, GSK3$\beta$, and JNK3 (§5.1–§5.2, Table 2).
  4. Small-data regime. Rather than training an autoencoder from scratch on 105–10610^5\text{–}10^6 molecules, BOReFT inherits the frozen LLM's chemical prior (99.82%99.82\% native SMILES validity from n=1,024n=1{,}024; Table 6) while self-distillation (Eq. 7) populates neighborhoods around anchors (ablated in Fig. 4).

Justification for "New Bridge"

  • Our claim is not, by any means, to suggest that we introduce latent-space BO, in general. However, our work, to the best of our knowledge, does provide a principled new bridge between pretrained-LLM based search and continuous BO.
  • Prior works in LLM-based search (OPRO, BOPRO, AutoDiscovery, MiGrATe, SDPO-TTT; Table 3) lack a continuous space that decodes directly into candidates, inevitably relying on either prompt heuristics or weight updates, and the most similar work to ours, InstructZero (which uses soft-prompt BO), also failing in practice (§2).
  • Prior works in classical latent-space BO (VAE-BO, LOLBO), on the other hand, require training bespoke domain autoencoders from scratch, sacrificing the ability to leverage domain knowledge from massive pretraining.
AuthorsOpenReview ↗

Author Response (3/3)

Questions

...would BOReFT still be preferable if we swap the baseline model to something more recent for example Gemma4/Qwen3.5?

Thank you for this question. To test this directly, we trained BOReFT and evaluated all search baselines on Semantle using Qwen3.5-2B under the exact protocol of §5.1 (N=3,072N=3{,}072, r=64r=64, budget Q=500Q=500, identical 5 train and 5 held-out targets ×\times 3 shared warm-start seeds). Given the stronger performance of baselines with "Before SFT" in Semantle, we use that same protocol below:

Method (Qwen3.5-2B)Train EM (↑\uparrow)Held-out EM (↑\uparrow)
Random1/150/15
AutoDiscovery3/150/15
MiGrATe5/152/15
BOPRO7/15‾\underline{7/15}3/15‾\underline{3/15}
OPRO7/15‾\underline{7/15}3/15‾\underline{3/15}
BOReFT (AABB, Eq. 8)2/153/15‾\underline{3/15}
BOReFT (Covering ellipsoid, App. B.8)9/157/15

Takeaways:

  • Upgrading from Llama-3.2-1B (Tables 1, 4) to Qwen3.5-2B improves baseline scores (e.g., OPRO held-out Sim. rises from 0.831 to 0.871, MiGrATe train EM from 0/15 to 5/15), yet every prompting, tree-search, embedding-BO, and test-time training baseline still plateaus at at most 3/15 held-out exact matches. BOReFT (Ellipsoid) recovers 7/15 held-out (0.884) and 9/15 train (0.919) targets—more than doubling the best baseline's held-out exact-match rate, matching the trend on Llama-3.2-1B (8/15 vs. 3/15, Table 1).
  • The overall advantage of searching a learned intervention manifold holds across model families, but the best-performing search-domain construction (App. B.8, Table 7) can depend on how a given backbone arranges its posterior means μi\mu_i. On Qwen3.5-2B, decoded proposals near the corners of the AABB skew toward high-frequency generic words, but restricting BO to the tighter covering ellipsoid (4.1×10−44.1\times 10^{-4} of AABB volume; App. B.8) jumps recovery from 2/15 train / 3/15 held-out (AABB) to 9/15 / 7/15. On Llama-3.2-1B, the AABB outperforms the covering ellipsoid (6/15 train, 8/15 held-out at 0.863/0.892 vs. 5/15, 5/15 at 0.873/0.852; Table 7). Note that the other hyperparameters in Qwen3.5-2B's best run (greedy T=0T=0, 1 sample, ARD Matérn-2.5, LogEI) remain unchanged from Llama-3.2-1B.

We will add this analysis in the final version of the paper; thank you!


Does the scale of the chosen LLMs affect any of the results?...on synthesizing new molecules.

Yes, performance depends on the model's pretrained knowledge and representation quality, and should improve with scale. This relates also to the representational gap (f⋆−Fτ⋆f^\star - F_\tau^\star, Eq. 12). To isolate this, we compared Qwen3.5-0.8B and Qwen3.5-2B on Semantle, holding architecture, tokenizer, training data (N=3,072N=3{,}072, r=64r=64), and the default AABB search protocol (Q=500Q=500, §5.1) fixed:

ModelUnseen Word Recon. (Greedy / Sampled)Train EM (↑\uparrow)Train Sim. (↑\uparrow)Held-out EM (↑\uparrow)Held-out Sim. (↑\uparrow)
Qwen3.5-0.8B0.532 / 0.7912/150.8270/150.765
Qwen3.5-2B0.636 / 0.8612/150.8503/150.836

Scaling from 0.8B to 2B improves unseen-word manifold reconstruction on n=928n=928 words by +10.4 points greedy recall (+7.0 sampled), and lifts held-out search from 0/15 (0.765) to 3/15 (0.836) and train similarity from 0.827 to 0.850.

On molecular optimization, our generator is MiST (Bran et al., 2025), a chemistry-pretrained Qwen2.5-3B model (§5.1, App. B.1), and a matched chemistry-pretrained checkpoint at another scale (e.g., 9B) is not available for a controlled comparison without confounding scale with domain pretraining; however, we expect scaling a chemistry-pretrained backbone to yield similar gains.


...curious to see how OPRO does with a frontier model, and how it compares with BOReFT in terms of practical usefulness.

Scaling the LLM would indeed strengthen the prior for the search task, which, as also shown in Q2 above, would benefit BOReFT as well. In particular, scaling will not alter the behavior of the OPRO search algorithm, continuing to rely on in-context optimization via prompting, which has no uncertainty over unexplored regions and has a tendency to repeat proposals (as we've shown at smaller scales in Table 5). Our main goal in this work is to demonstrate that when white-box access is available, BOReFT presents a new, principled mechanism for search and discovery that can outperform alternative techniques. In scientific discovery, furthermore, open domain models like chemistry-pretrained MiST-3B (§5.1) can often be preferable for specialized priors, privacy, reproducibility, and avoiding long prompt-history costs.


We hope our rebuttal could address concerns in your initial evaluation. Thank you for your consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.