- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors provide an approach for applying Bayesian Optimization techniques within language models by performing a BO loop on a lower-dimensional continuous subspace of the model. The authors then apply this method with a reformulated training objective to two categories of tasks including semantle word search and molecular property optimization.
Critical strengths
- The presentation and framing of the theoretical results is very clear, and the authors do a good job of emphasizing the important takeaways throughout the text
- The construction of the intervention is very clean, providing an affine transformation that maps into the activation space allowing for direct intervention without any cumbersome machinery of other decoder/encoder models
- Baselines are all fair comparisons since the models used for comparison are all additionally LoRA tuned using the same settings on the same dataset. This cleanly isolates out the effect of the method in a compelling way
Critical weaknesses
- More seeds would give more robust statistics on the results, as the current setup only uses 5 seeds
- The paper never reports training cost of this method, which would be useful to understand when thinking about how to apply this method
- The metric used is inconsistent in certain cases, switching from mean to max. This can confuse comparisons of performance
Questions for authors
- How does rank trade off representational gap against search gap? The box’s volume grows multiplicatively in , so there should be a visible optimum
- Terminology wise, the intervention is affine in , the codes are a box in a linear subspace. What is the justification for “manifold”?
- More tasks should be assessed before making broad generalizations such as "program optimization" in the abstract
3 comments
Author Response (2/2)
Questions
Q1. How does rank r trade off representational gap against search gap? The box’s volume grows multiplicatively in r, so there should be a visible optimum
We show this trade-off empirically in our rank sweep at (App. B.15, Fig. 8), which remains consistent with Eq. 12 and Lemma A.2(i) ():
| Rank | Unseen Word Recon. (, Greedy / Sim.) | Search EM (Train / Held-out / Total) | Search Mean Sim. | |
|---|---|---|---|---|
| 4 | 0.009 / 0.686 | 3.2 | 3/15 / 2/15 / 5/30 | 0.823 |
| 8 | 0.042 / 0.715 | 3.8 | 2/15 / 1/15 / 3/30 | 0.799 |
| 16 | 0.110 / 0.749 | 4.7 | 5/15 / 5/15 / 10/30 | 0.869 |
| 32 | 0.206 / 0.789 | 6.7 | 0/15 / 2/15 / 2/30 | 0.794 |
| 64 | 0.699 / 0.940 | 7.1 | 6/15 / 8/15 / 14/30 | 0.878 |
| 128 | 0.730 / 0.947 | 12.6 | 4/15 / 6/15 / 10/30 | 0.844 |
As grows, held-out reconstruction increases monotonically (0.009 → 0.730), shrinking the representational gap. Beyond , however, nearly doubles (7.1 → 12.6) and BO must search a -D box under the same fixed budget , enlarging the search gap so target recovery drops from 14/30 at back to 10/30 at despite slightly higher representational capacity—yielding the interior optimum at . We will add these reconstruction and diameter columns beside Fig. 8.
Q2. Terminology wise, the intervention is affine in b, the codes are a box in a linear subspace. What is the justification for “manifold”?
In hidden-state space, (Eq. 2) is indeed a flat -dimensional affine subspace of restricted to the box (Eq. 8). We used "intervention manifold" (following recent steering literature, e.g., Wurgaft et al., 2026) because passing through the remaining nonlinear transformer blocks maps onto a curved -dimensional family of output distributions over sequences. That said, we agree "affine intervention subspace" / "learned code space" is more precise for and , and we will clarify this terminology in §1 and §3.1. Thank you for the question.
Q3. More tasks should be assessed before making broad generalizations such as "program optimization" in the abstract
Our aim with that sentence in the abstract was to illustrate how prior work uses LLMs as proposal generators broadly, whereas our evaluated claims are scoped to Semantle and de novo molecular property optimization tasks. We can revise that opening sentence to avoid being read as claiming empirical evaluation on program optimization, and note code/program search explicitly as future work in §6.
We hope our responses have helped address the concerns in your initial review. Thank you again for the helpful questions and for your consideration!
Rebuttal Acknowledgement
I thank the authors for their rebuttal and the clarification against many of the points in my initial review. While I view this paper more favorably than before and feel this is a good paper, I still do not feel comfortable assigning it a 4 of maximum possible score, so I will maintain my initial score of 3.
Author Response (1/n)
Thank you for your thoughtful review, and for highlighting the clarity of our theoretical framing, the simplicity of intervening directly in the frozen LLM's activation space, and the fairness in our evaluation setup. Below we address each of your concerns and questions.
Weaknesses
To clarify the evaluation sample sizes:
7/15vs.3/15).Note that each run is a budget-500 search; therefore, running many more seeds can become expensive (in terms of both time and compute). We already observe tight standard deviation on Semantle, so don't see more seeds helping; however, given compute availability, we will aim to add more seeds for molecule optimization by the final paper version.
Thank you for pointing out this omission. We will add a dedicated compute table to App. B. Every experiment ran on a single NVIDIA A100 GPU:
Crucially, subspace training is amortized across downstream searches because a single 23.7-h ChEBI-20 run is reused across all 15 search runs spanning DRD2, GSK3$\beta$, and JNK3 (~1.58 h amortized per run, dropping further with each new target), while the trainable intervention (R,W) itself adds only 2×r×d≈0.26M parameters.
Thank you for raising this. We consistently report both
mean/maxin Table 2 because molecule discovery values both average reliability (meanof per-seed discoveries) and peak candidate discovery (maxacross seeds, matching Cor. 4.3), whereas Semantle reports exact-match count (EM) and mean best similarity (Sim.). However, in §5.2 (ll. 88–90) our text cited both mean and max on JNK3 (0.370/0.940) while highlighting only max on GSK3$\beta$ (0.446/0.670, where BOPRO has slightly higher before-SFT mean0.470/0.630). We will revise §5.2 to statemean ± SDandmaxside-by-side everywhere so comparisons are uniform throughout.