acceptodds
Under review as a conference paper at ICLR 2027

In-context learning for latent space Bayesian optimization

Abstract

Latent-space Bayesian optimization (LSBO) enables sample-efficient optimization over structured objects such as molecules by searching in the continuous space of a generative model. However, its surrogate is typically fitted anew for each objective from limited observations, without transferring experience across tasks. We investigate whether pretrained in-context predictors can provide such transfer in LSBO. Using prior-data fitted networks (PFNs), we introduce PILBO (_Pretrained In-context Latent-space Bayesian Optimization_), a framework combining pretrained in-context prediction with latent-space molecular generation. Its main implementation predicts from molecular representations independent of the generator and achieves the strongest aggregate performance on GuacaMol optimization objectives. We further introduce a continued-pretraining (CPT) scheme based on handcrafted synthetic molecular objectives and BO-oriented episode construction. The pretrained PFN already captures most of the benefit, with CPT yielding only modest further gains. Our analyses suggest that this limited transfer reflects a mismatch between the offline synthetic episodes used for adaptation and the adaptive prediction problems encountered during BO. Overall, pretrained in-context surrogates provide a strong alternative to repeatedly fitted LSBO surrogates.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian Optimization, In-context Learning, Latent Space Bayesian Optimization, Tabular Foundation Models, Prior-Fitted Networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YeZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve whether the per-task GP surrogate in latent-space BO can be replaced by a pre-trained in-context predictor (PFN) to enable cross-task transfer. Under limited budgets, per-task GPs suffer from early posterior and hyperparameter errors that propagate through the whole optimization trajectory. Paper proposed PILBO that has two variants, PILBO-D and PILBO-L. The authors also propose molecular-domain continued pretraining (CPT). It combines synthetic objectives built from ChEMBL base scores, Boltzmann sampling biased toward high scores, PFN-embedding-based task selection, and L2-SP regularization.

Critical strengths

1. An empirical observation suggests that a pre-trained PFN serves as a robust surrogate in decoupled LSBO. PILBO-D outperforms various LSBO baselines. Replacing the backbone with TabICLv2 results in only a marginal decline in performance, and a version using Morgan fingerprints as input also surpasses GP-COWBOYS. The ability to use an off-the-shelf tabular foundation model as a surrogate is practically significant.

2. A candid report of a negative result regarding CPT (Cross-Problem Transfer). The study transparently presents the finding that a carefully designed domain-specific CPT is largely ineffective. While tasks were selected as intended within the embedding space (Figure 4), the distribution of conditions during rollout did not align (Figure S2). This serves as a valuable cautionary note for research on PFN-for-BO: an offline synthetic prior struggles to effectively emulate the adaptive context required for Bayesian Optimization.

Critical weaknesses

1. The problem serving as the motivation has not been substantiated, nor is there any analysis demonstrating that the proposed method actually solves it. The core motivation, initial surrogate error in per-task Gaussian Processes (GPs) and its propagation, is not quantified. There is no evidence that PFNs reduce this error. The analysis section (5.3) merely addresses the failure of the ineffective CPT (Cross-Problem Transfer) approach, and even then, it only measures the overlap in the argmax regions rather than the transfer of the predictive prior. Ultimately, the claim that "PFNs improve LSBO" is supported solely by performance curves, with no further substantiation.

2. The sources of performance gains are not disentangled. PILBO-D and GP-COWBOYS differ simultaneously in their surrogates, feature representations (RDKit2D vs. Morgan), and batch acquisition strategies. Consequently, one cannot rule out the possibility that the observed gains stem from the feature representation rather than the pre-trained prior. The paper also lacks comparisons with methods in the transfer/meta-BO literature, which explicitly address cross-task transfer. Since CPT proves ineffective, the methodological contribution effectively reduces to replacing the surrogate model used in COWBOYS.

Questions for authors
  1. Can you compare per-round surrogate prediction quality (e.g., NLL, Spearman correlation, calibration on top molecules) between the PFN and a GP along the BO trajectory? This would show whether the early surrogate error cited as motivation actually exists, whether the PFN reduces it, and whether the gap narrows as context accumulates.

  2. Can you report a GP on RDKit2D descriptors with the same PI-guided pCN generation and pointwise UCB selection as PILBO-D? This would show whether the gains come from the pre-trained prior or from the representation.

2 comments
AuthorsOpenReview ↗

Individual answer - on predictive quality, attributions of the gains, and meta-BO comparisons

We thank the reviewer for their careful evaluation and insightful feedback. This prompted us to revisit several aspects of our methodology, as shown below, ultimately strengthening the manuscript.

(Weakness 1 + Question 1) Evidence for the surrogate-error motivation

We thank the reviewer for highlighting the need to substantiate our motivation beyond optimization performance curves. We have now added direct evaluations of surrogate predictive quality, fully discussed in our second general comment and reported in Section 5.2 (Figure 5; Appendix A.5).

Specifically, we compare four surrogate configurations using NLL, Spearman correlation, and ECE along saved BO trajectories, evaluating all surrogates on identical contexts and acquisition-selected molecules. PILBO-D+CPT achieves lower NLL and higher Spearman correlation than the GP variants, with particularly pronounced differences in the early rounds. We further confirm its predictive advantage on randomly sampled GuacaMol contexts. These experiments directly substantiate the predictive-quality motivation, although they do not establish a causal relationship between surrogate errors and subsequent optimization outcomes.

We have also expanded Section 5.3 and Appendices A.2–A.3 to better characterize the limitations of offline CPT, including molecular-support and task-transfer mismatches. Our new Rollout-CPT extension, discussed in the first general comment, further investigates the role of adaptive context construction.

Lastly, let us mention that the acronym CPT denotes continued pretraining and not cross-problem transfer.

(Weakness 2 + Question 2) Attributions of gains and Matched GP baseline on RDKit2D descriptors

We agree that the original comparison left some factors intertwined, despite the Morgan-fingerprint ablation for PILBO-D already included in the manuscript. As discussed in our second general comment, we have now extended this analysis with a complementary matched GP baseline. We reiterate the findings below.

We have now added GP-COWBOYS (RBF+RDKIT) using the same 212 RDKit2D descriptors, PI-guided pCN candidate-generation procedure, and pointwise UCB selection as PILBO-D-CPT. This matched comparison, together with the complementary PILBO-Morgan ablation, is now presented in the main text, Section 5.2 (Sensitivity to molecular representation) and Figure 5 (top). Shortly, PILBO-D-CPT achieves an overall score of 0.6304±0.03260.6304 \pm 0.0326 across the 8 held-out tasks, compared with 0.6166±0.03160.6166 \pm 0.0316 for GP-COWBOYS (RBF+RDKIT). The original manuscript also included a complementary experiment replacing the RDKit2D descriptors of PILBO-D+CPT with Morgan fingerprints, matching the representation used by GP-COWBOYS. This variant is now labeled PILBO-Morgan in Figure 5. This resulted in reduced performance compared to the RDKit2D descriptor representation, but still better than GP-COWBOYS. Overall, these comparisons suggest that the PFN's advantage cannot be attributed solely to its molecular representation.

Comparison with a meta-BO approach

We thank the reviewer for suggesting a comparison with meta-BO methods. We have now included HyperBO H-NLL [1]. This approach learns a GP prior from the same synthetic molecular tasks used for our CPT and conditions it on observations from each held-out objective within our LSBO framework. Appendix A.5 in the updated manuscript provides full details on the adaptation of this baseline to molecular LSBO. At 600 oracle calls, PILBO-D+CPT achieves a mean best-observed score of 0.6304 ± 0.0326, compared with 0.5771 ± 0.0279 for HyperBO, with higher mean scores on all eight tasks (Figure S7).

To caveat this comparison, it is worth noticing that even outside the PFN realm, meta-BO has not been systematically evaluated on held-out molecular LSBO objectives, at least to our knowledge. What we mean by that is that a thorough assessment of alternative meta-BO approaches would likely require a broader investigation, beyond the scope of this rebuttal. For instance, while HyperBO is a general-purpose method, its original empirical evaluation focused primarily on hyperparameter optimization, a setting substantially different from de novo molecular design. Finally, an important distinction is that PILBO builds on a tabular foundation model already pretrained on a vast and diverse corpus of synthetic tasks, even before molecular CPT, whereas our HyperBO baseline learns its prior solely from the synthetic molecular task bank. Hence, one could say that the comparison does not equalize pretraining exposure.

[1] Wang et al. Pre-trained Gaussian Processes for Bayesian Optimization. JMLR, 2024.

AuthorsOpenReview ↗

Acknowledging our rebuttal

As we have not yet heard back on our rebuttal, we would appreciate knowing whether your questions have been resolved or if any points remain. Your feedback would help us make the discussion productive and further strengthen the manuscript.

Reviewer_TEkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses a pretrained TabPFN predictor to guide molecular search through a VAE generator. Experiments on eight optimization tasks show improved aggregate performance under a fixed evaluation budget, while additional molecular pretraining provides only modest gains.

Critical strengths
  1. The benchmark results support the approach. The method achieves the strongest aggregate performance across eight objectives, suggesting that pretrained in-context predictors can be useful surrogates for molecular optimization.

  2. The ablations provide useful practical findings. The study examines input representations and continued pretraining, showing that the existing pretrained predictor accounts for most of the benefit.

Critical weaknesses
  1. Limited methodological novelty. PFNs have already been used as BO surrogates, and separating molecular prediction from latent-space generation follows COWBOYS. The main change is therefore to incorporate an existing pretrained predictor into an established framework. The proposed molecular continued pretraining adds only modest gains, leaving the methodological contribution relatively incremental.

  2. The source and robustness of the gains remain insufficiently established. The main comparison changes the predictor, molecular representation, and batch acquisition rule together. The representation ablation partially addresses this issue, but does not fully isolate the benefit of TabPFN itself. The paper also lacks direct evaluation of predictive accuracy and uncertainty calibration along the adaptive search trajectory, particularly on candidates selected for high predicted scores. These analyses would help explain the gains and assess reliability as search moves beyond the observed data.

Questions for authors
  1. Practical efficiency. The method achieves better scores under a fixed oracle budget but runs slower than GP-COWBOYS. At equal wall-clock time, or when measuring total time to reach a fixed score, does it still offer an advantage? How expensive must oracle evaluations be for the additional computation to become worthwhile?

  2. Choice of generator. What motivates using a VAE rather than a diffusion-based molecular generator? Which parts of PILBO depend specifically on the VAE formulation, and what evidence suggests that the benefits of the pretrained predictor would extend to other generative models?

2 comments
AuthorsOpenReview ↗

Individual answer 1/2 - on methological novelty, source and robustness of the gains, and practical efficiency

We thank the reviewer for their thoughtful assessment and constructive criticism, which encouraged us to critically reassess our approach.

Methodological novelty

The reviewer is correct that our approach builds on established methods such as PFNs4BO [1] and COWBOYS [2], as acknowledged in the manuscript. Nevertheless, we believe our contribution extends beyond combining these methods within the LSBO framework. Our study systematically investigates the potential of pretrained in-context surrogates for molecular LSBO, identifying the importance of predictive representations and the challenges of transferring pretrained models to this setting. In particular, our finding that carefully designed molecular CPT yields only modest gains, together with our analysis of this limitation, highlights the value of systematic empirical investigation alongside methodological innovation.

More broadly, our work contributes to a growing body of research exploring the use of tabular foundation models in biomolecular settings [3–6], with a distinct focus on sequential black-box optimization rather than molecular property prediction alone.

Lastly, we have now added an extension to our method, Rollout-CPT, which constructs pretraining episodes through sequential surrogate-guided acquisition, rather than via Boltzmann sampling, thus better matching deployment conditions. We detail this methodological extension and its evaluation in our global comment.

Source and robustness of the gains

We thank the reviewer for raising these concerns, which overlap with those of Reviewer YeZF (W1-W2, Q1-Q2). We address them jointly in our second global comment, presenting both the controlled GP comparison and the predictive accuracy and calibration analyses on BO-selected molecules and randomly sampled GuacaMol data. These results are now included in Section 5.2 and Figure 5 of the revised manuscript, with evaluation details in Appendix A.6.

Practical efficiency

As suggested by the reviewer, we compare PILBO-D-CPT and GP-COWBOYS by the time required to reach a common score. For each pair of runs on the same task and seed, we define the target retrospectively as the lower of their final scores at 600 oracle calls. Under this measure, PILBO-D-CPT is faster on average, reaching the common target in 47.89 minutes compared with 65.32 minutes for GP-COWBOYS. This analysis has now been included in Appendix A.1 (Table S5).

More broadly, the intended application is de novo molecular design, where evaluating a candidate may require biological assays or other wet-lab experiments taking days, if not weeks, orders of magnitude longer than a BO iteration. While we recognize that faster computational methods facilitate experimentation and prototyping, this difference in timescales is important when assessing practical efficiency.

[1] Müller et al. PFNs4BO: In-Context Learning for Bayesian Optimization. ICML, 2023.

[2] Moss et al. Return of the Latent Space COWBOYS: Re-thinking the Use of VAEs for Bayesian Optimisation of Structured Spaces. ICML, 2025.

[3] Lee et al. Molecular Property Prediction under Structural Shift with Tabular Foundation Models. arXiv, 2026.

[4] Guan et al. Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction? arXiv, 2026.

[5] Banaszewski and Fitzgibbon. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. arXiv, 2026.

[6] Ben Hicham et al. Tabular Foundation Models for In-Context Prediction of Molecular Properties. arXiv, 2026.

AuthorsOpenReview ↗

Individual answer 2/2 - on the choice of the molecular generation engine

On the choice of the molecular generation engine

We primarily chose a VAE because it is a well-established approach in molecular LSBO, with a long lineage of papers. Non-exhaustively, the first works employed junction-tree VAEs [1,2], followed by approaches using SELFIES-based VAEs, including LOL-BO [3], CoBO [4], and InvBO [5]. Only quite recently were alternative generative models explored, like normalizing flows for NF-BO [6], and in 2026, acquisition-guided diffusion methods [7].

Operating in a VAE framework allowed us to focus our efforts on the surrogate rather than the generative model: in our decoupled formulation PILBO-D, the PFN surrogates predict from molecular representations independently of the generator, while candidate generation and sampling must be adapted to each generative model. This setting is akin to the GP surrogate framework of COWBOYS. Our other baseline, PILBO-L, performs BO directly in latent space, so the generative model does even more work than in the decoupled approach: its latent space serves as input space for the surrogate, and therefore acquisition function maximization happens in latent space. Thus, the choice of generator plays a more central role for PILBO-L.

That being said, thanks to your suggestion, we decided to assess whether the PFN's advantage extends beyond VAEs: we replaced it with the SeqFlow normalizing-flow generator [6], adapting the acquisition function-guided generation approach of [7]. We apply the same SeqFlow generation framework to GP-COWBOYS for comparison. The results are now presented in the main text, Section 5.2 (Sensitivity to the generative model) and Figure 4. Both methods improve over their VAE-based counterparts in aggregate final score, with PILBO-D+CPT retaining its advantage (0.64030.6403 versus 0.60910.6091 at 600 oracle calls). These results suggest that the benefits of PFN-based molecular optimization extend beyond our original generator and sampling procedure.

[1] Jin et al. Junction Tree Variational Autoencoder for Molecular Graph Generation. ICML, 2018.

[2] Tripp et al. Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining. NeurIPS, 2020.

[3] Maus et al. Local Latent Space Bayesian Optimization over Structured Inputs. NeurIPS, 2022.

[4] Lee et al. Advancing Bayesian Optimization via Learning Correlated Latent Space. NeurIPS, 2023.

[5] Chu et al. Inversion-based Latent Bayesian Optimization. NeurIPS, 2024.

[6] Lee et al. Latent Bayesian Optimization via Autoregressive Normalizing Flows. ICLR, 2025.

[7] Ancilli et al. Acquisition Function-Guided Generation for Bayesian Optimization over Discrete Spaces. arXiv, 2026.

Reviewer_YVtAOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In context learning for latent space Bayesian optimization

In this work, the authors propose Prior-data fitted networks or PFNs as surrogate models for latent space Bayesian optimization for molecular generation (PILBO-D,L, PILBO-CPT-D,L). By leveraging PFNs and in-context learning, they propose an alternative to learning a separate surrogate for each optimization problem which can be limited by small budgets. They specifically investigate the effect of representations (eg. Morgan fingerprints versus RDKIT descriptors, latent versus input/output space) and continued pretraining on synthetic molecular objectives. They find PFNs to be effective surrogates for guidance on synthetic objectives and that CPT yields limited benefit. They also find that transfer between offline synthetic episodes used for adaptation and downstream tasks is ineffective.

Critical strengths

Strengths

  1. The scope of the paper is well defined. The problem is clearly stated, and the paper is well written.
  2. They investigate the effect of CPT and of direct versus latent representations.
  3. The results are compelling over the baselines Inv-BO and COWBOYS, both of which seem to be the top models for applying LSBO to molecular generation MPO tasks. PILBO-D/CPT improves performance on 6 out of 8 tasks reported.
Critical weaknesses

Weaknesses

  1. The offline mismatch section is somewhat unclear to me. This could be my lack of background in BO on synthetic objectives. In section 5.3, they report that the “discovered molecules” do not exhibit high downstream scores. Could they project some of these discovered molecules in the PFN embedded space (in Figure 4). They say they choose objectives based on embedding similarity between tasks. That would imply that the downstream tasks must lie within the PFN embedded space. Do the discovered molecules lie in densely populated regions of the embedded space or do they veer off into no-man’s land?
Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Individual response - Clarifications on section 5.3 and insights brought by Rollout CPT

We thank the reviewer for his positive assessment regarding our contribution.

We do agree that Section 5.3 may be difficult to follow at first glance, an issue that can be partially attributed to the 9-page limit. We have now revised Section 5.3 to distinguish objective resemblance on a fixed molecular panel from the molecular inputs and adaptive contexts encountered during BO.

First, we clarify that Figure 4 of the original manuscript (now Figure 6) is a diagnostic comparison of synthetic and held-out downstream objectives in the PFN embedding space. That is, each point represents an objective evaluated on a fixed panel of molecules, rather than an individual molecule. This allows us to assess how closely the synthetic objectives resemble downstream tasks according to the PFN's hidden representations. But this assessment is measured on a fixed panel of molecules, which might not be representative of the molecules encountered during downstream BO. Hence, crucially, objective resemblance on this panel need not extend to the regions actually explored during optimization. As summarized in the revised Section 5.3 and illustrated in Appendix A.2 (Figure S3), optimizing synthetic objectives that resemble downstream tasks on the fixed panel need not yield molecules with high downstream scores.

Second, we have now introduced Rollout-CPT (see our global comment), which brings pretraining context construction closer to downstream BO. Rather than sampling contexts from the fixed GuacaMol molecular pool, Rollout-CPT builds them sequentially: a surrogate and acquisition function guide the molecular generator, and each evaluated molecule is added to the context for subsequent decisions. This partly addresses the mismatch in how contexts are constructed, but does not eliminate the molecular input shift. Indeed, because synthetic and downstream objectives guide the search differently, their BO trajectories can rapidly diverge and explore different regions of molecular space.

Following up on that, we have now added experiments to further characterize this remaining input shift in Appendix A.3 of the updated manuscript. Specifically, we investigate how closely downstream BO discoveries resemble molecules encountered during pretraining. For each of the 8 held-out downstream tasks, we select the 50 highest-scoring distinct post-initialization discoveries, obtained via a PILBO-D trajectory. We then compare these molecules with the Rollout-CPT training corpus (over 200M molecular observations) and, as a reference, the GuacaMol dataset (1.27M molecules). For each of these 50 downstream reference molecules, we identify its closest neighbor in the Rollout-CPT corpus and, separately, in the GuacaMol dataset, using Tanimoto similarity between Morgan fingerprints. Interestingly, despite its much larger size, the Rollout-CPT corpus yields only modestly higher median nearest-neighbor similarity than GuacaMol on most tasks (Figure S4). This is understandable in light of the vast size of drug-like chemical space, estimated at around 106010^{60} possible molecules [1], far beyond the 200 million molecular observations in Rollout-CPT. Scoring the full Rollout-CPT corpus on the downstream objectives also shows that scores remain predominantly below the corresponding top-50 thresholds (Figure S5).

Even on MED1, where 14 of the 50 reference molecules are present in the Rollout-CPT corpus, only 875 distinct molecules in the full corpus meet or exceed the lowest score among these reference discoveries. These high-scoring molecules account for a very small fraction of the corpus, which may limit their influence during training. The empirical results are consistent with this possibility: PILBO-D+R-CPT does not improve on PILBO-D+CPT in mean final score on MED1.

[1] Reymond, J.-L. The Chemical Space Project. Accounts of Chemical Research, 2015.

AuthorsOpenReview ↗

Acknowledging our rebuttal

We thank you again for your encouraging review. We have now posted our rebuttal and revised manuscript, including several additional experiments. If you have had a chance to look at the updates, we would appreciate a brief acknowledgment or any further thoughts you may have.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global comment 1/2: experiments added to the manuscript, and introduction of Rollout-CPT baseline

We thank all reviewers for their careful assessments and constructive feedback, which have helped us substantially strengthen the manuscript. In response to the questions raised, we have expanded the experimental evaluation and clarified several aspects of our methodology. The revised manuscript now includes:

  • Rollout-based continued pretraining (R-CPT): A new adaptation strategy based on synthetic BO trajectories, evaluated against standard CPT and the original pretrained models (Section 5.2, Figure 3; Appendix C.8).

  • Controlled surrogate and representation comparisons: An additional GP-COWBOYS baseline using the same RDKit2D descriptors, candidate-generation procedure, and acquisition rule as our PFN method, complementing the Morgan fingerprint ablation (Section 5.2, Figure 5).

  • Surrogate predictive quality and calibration: Comparisons of NLL, Spearman correlation, and ECE along saved BO trajectories and on randomly sampled GuacaMol contexts, including the new matched GP baseline (Section 5.2, Figure 5; Appendix A.5).

  • Alternative molecular generation: Experiments using the SeqFlow normalizing-flow generator with acquisition-guided candidate selection, evaluating whether the PFN's advantage persists beyond the original VAE-based framework (Section 5.2, Figure 4).

  • Meta-learned GP baseline: An additional comparison with HyperBO, trained on synthetic molecular tasks (Appendix A.4).

  • Further analysis of pretraining-to-deployment transfer: Expanded diagnostics of molecular support, downstream score coverage, and the mismatch between offline task similarity and discoveries made during adaptive optimization (Section 5.3; Appendices A.2 and A.3).

  • Computational efficiency: An additional runtime comparison measuring the time required to reach a common optimization score (Appendix A.1).

We begin with a general comment introducing Rollout-CPT, a methodological extension developed independently of the reviewers' requests but relevant to our analysis of pretraining-to-deployment mismatch. A second general comment addresses concerns raised by multiple reviewers, particularly the role of molecular representation and surrogate predictive quality. We then address the remaining reviewer-specific questions individually.


Rollout-CPT

Before the rebuttal period, we extended our method with a new continued-pretraining strategy, Rollout-CPT, that partly addresses the mismatch in context construction between pretraining and deployment identified in Section 5.3 of our manuscript.

In our original molecular CPT, context and query molecules are selected through Boltzmann-weighted sampling from a fixed GuacaMol molecular pool, independently of previous observations.

In Rollout-CPT, we instead construct training episodes from synthetic BO trajectories. Starting from an initial set of molecules sampled from the GuacaMol molecular pool, a surrogate and acquisition function iteratively guide the generator toward new candidates, whose evaluations are added to the context before the next acquisition step. Consequently, contexts are constructed sequentially based on the optimization history, and newly generated molecules need not belong to the original GuacaMol pool.

To generate these synthetic BO trajectories, we use the computationally efficient LSBO method [1]. This procedure more closely reflects the adaptive context generation encountered during downstream BO.

The resulting methods, PILBO-D+R-CPT and PILBO-L+R-CPT, mildly improve aggregate performance over the original CPT.

Although the improvements remain modest and vary across tasks, these results provide further evidence of the potential of adapting PFN pretraining to the specific characteristics of LSBO. The Rollout-CPT comparison is now included in Section 5.2 and Figure 3 of the revised manuscript, with additional details and per-task results in Appendix A.3 (Figure S4), and a description in Appendix C.8. We have also revised Section~5.3 to discuss the mismatch in context construction partly addressed by Rollout-CPT and the remaining molecular input and objective-transfer gaps.

[1] Fan et al. Search at the Cost of Sampling: Nearly-Instant Latent Space Bayesian Optimization. NeurIPS, 2026.

AuthorsOpenReview ↗

Global comment 2/2: molecular representation and surrogate predictive quality

Reviewers TEkb (W2) and YeZF (W1–W2, Q1–Q2) raised overlapping concerns about the sources of our gains and surrogate predictive quality and calibration, which we address jointly here.

In BO, predictive accuracy and optimization performance are related but distinct objectives. The surrogate guides acquisition toward high-value regions rather than minimizing global prediction error. Thus, better predictive accuracy or calibration does not necessarily imply better optimization performance. This distinction has been studied explicitly in [1], which optimizes sparse GP approximations with respect to acquisition utility rather than global posterior fidelity, obtaining improved BO performance, including for molecular LSBO tasks!

Our motivation for using PFNs is their ability to transfer inductive structure learned across a large distribution of pretraining tasks, potentially providing more effective guidance under limited evaluation budgets, the typical regime in LSBO. Recent work on molecular and biomolecular property prediction further supports the potential of in-context foundation models in these settings [2–5]. However, these studies do not establish whether PFNs outperform GPs in predictive quality or BO performance for molecular optimization. Our study focuses on the latter.

Before presenting the results related to prediction quality, for a controlled comparison between PFNs and GP surrogates, we have added GP-COWBOYS (RBF+RDKit) baseline. The latter uses the same 212 RDKit2D descriptors, candidate-generation procedure, and pointwise UCB selection rule as the PFN methods. This comparison was initially requested by YeZF (Q2) with respect to BO outcomes, but it also provides a sensible way to compare prediction quality. At 600 oracle calls, PILBO-D+CPT achieves a mean best-observed score of 0.6304 ± 0.0326 across the 8 held-out tasks, compared with 0.6166 ± 0.0316 for GP-COWBOYS (RBF+RDKit). These optimization results are now presented in the main text, Section 5.2 (Sensitivity to molecular representation) and Figure 5.

Predictive accuracy and calibration. With this additional baseline being introduced, we now complement this optimization comparison. We have added a dedicated discussion in Section 5.2 and Figure 5 (middle and bottom) of the main text, with metric definitions and evaluation details in Appendix A.6. The comparison includes four surrogate configurations: PILBO-D+CPT, its Morgan-fingerprint variant (PILBO-Morgan), GP-COWBOYS, and GP-COWBOYS (RBF+RDKit). We evaluate them using negative log-likelihood (NLL), Spearman rank correlation, and expected calibration error (ECE).

We first evaluate all four surrogates along the saved downstream BO trajectories. At each round of each trajectory, all four receive the same preceding observations and predict the same next batch of molecules, selected by the method that generated the trajectory. Using identical contexts and queries provides a paired comparison, while aggregating results across trajectory sets reduces dependence on any particular acquisition policy. Spearman correlation measures agreement between predicted and observed rankings within each batch. Importantly, the queries are the candidates actually selected during adaptive search, including newly generated molecules, so this analysis evaluates predictive quality on the candidates prioritized by BO.

Figure 5 (middle) of the revised manuscript shows that PILBO-D+CPT consistently achieves lower NLL and higher Spearman correlation than the other three configurations, with particularly pronounced differences in the early rounds. It also achieves substantially lower ECE than the original GP-COWBOYS, although PILBO-Morgan and GP-COWBOYS (RBF+RDKit) generally attain lower ECE later in optimization.

Nevertheless, these results could still partly reflect “bad luck” in the particular BO trajectories evaluated. We therefore also evaluate all four surrogates on randomly sampled GuacaMol data, using identical nested contexts of 100–600 molecules and a fixed set of 1,000 query molecules within each split. Figure 5 (bottom) of the revised manuscript shows that PILBO-D+CPT achieves lower NLL and ECE and higher Spearman correlation than all three alternatives across all context sizes. These analyses provide direct evidence of improved predictive quality, in particular in early rounds.

Overall, we thank the reviewers for prompting us to do this investigation.

[1] Maus et al. Approximation-Aware Bayesian Optimization. NeurIPS, 2024.

[2] Lee et al. Molecular Property Prediction under Structural Shift with Tabular Foundation Models. arXiv, 2026.

[3] Guan et al. Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction? arXiv, 2026.

[4] Banaszewski and Fitzgibbon. Monroe: A Molecular Foundation Model for In-Context Probabilistic Inference. arXiv, 2026.

[5] Ben Hicham et al. Tabular Foundation Models for In-Context Prediction of Molecular Properties. arXiv, 2026.

AuthorsOpenReview ↗

Elaborating on the positioning of our work

Thank you for narrowing down your concern and for acknowledging that the additional experiments strengthen the evidence. We would like to clarify why we believe the contribution extends beyond showing that a combination works.

Methodological contributions

To our knowledge, molecular LSBO had not previously been systematically studied with PFN surrogates.

Beyond establishing these baselines, we designed a synthetic molecular task distribution and developed two adaptation procedures for PFN-based LSBO. Molecular CPT adapts the predictor using synthetic objectives on a fixed molecular pool. Rollout-CPT additionally constructs training contexts through sequential optimization trajectories. These constitute methodological developments beyond using an existing predictor without adaptation.

The experiments requested during rebuttal also show that our method retains its advantage with an alternative molecular generator and under matched representation and acquisition settings. Combined with the backbone ablations, these experiments provide the first comprehensive investigation of PFN integration into molecular LSBO!

As discussed in our conclusion, the next step could be moving from in-context prediction to in-context acquisition policies, as in FMTR [1] and TAMO [2]. Learning such policies over high-dimensional molecular design spaces (continuous latents or discrete) is a paper on its own. To elaborate more on that, one route is to build these policies on pretrained predictive representations, as in POLAR [3]. Precisely, our study provides empirical support for this direction in molecular LSBO, specifically via the predictive quality experiments added at the reviewer's suggestion!

Beyond methodological novelty

Crucially, we investigated why adapting the model to apparently relevant molecular tasks provided only modest additional benefits. Section 5.3 identifies a distinction between task similarity measured on a fixed molecular panel and the regions explored when those tasks are optimized. Synthetic objectives can resemble downstream tasks in the PFN’s embedding space while directing BO toward different discoveries Specifically, the embedding-selected synthetic searches produced 413,402 distinct molecules. None reached the downstream top-50 score threshold on seven of downstream tasks, and only one did so for Zaleplon (Figure S3).

Rollout-CPT partly addresses the mismatch in context construction but does not eliminate the differences in molecular coverage. Despite its much larger corpus, proximity to high-scoring downstream discoveries improves only modestly, and high downstream scores remain rare (Appendix A.3). This suggests that deployment mismatch may limit adaptation gains: neither offline task matching nor sequential pretraining ensures useful coverage of the regions explored on held-out objectives.

Following [4], which we already cite and discuss in Section 5.3, pretraining and evaluation within the same domain does not systematically improve transfer. Their experiments instead link generalization to the model’s ability to retrieve and aggregate relevant examples from the context supplied at inference. We believe this perspective is particularly relevant to BO, where successive acquisitions continually reshape the context used to select new molecules. Similarity between objectives on the original panel may therefore miss context–query relationships that matter during optimization.

Overall, our study investigates this difficulty in molecular PFN adaptation and documents what rollout-based pretraining does and does not resolve. We believe this negative finding is useful for subsequent method and dataset design.

Conclusion

We appreciate your suggestion to pursue more substantive insights or develop a more advanced combination method. To better understand how we might address this concern, we would welcome a more specific indication of which conclusions require further evidence or which methodological directions you consider particularly promising. We would be happy to investigate a focused suggestion during the remaining discussion period.

We recognize that our method builds on established components. Demonstrating their effectiveness in this setting and identifying the limits of natural adaptation strategies are also scientific contributions. We therefore respectfully believe the work should be assessed on its methodological additions AND analytical findings, rather than on component novelty alone.

[1] Nguyen et al. In-Context Batch Black-box Optimization via Flow Matching. arXiv, 2026.

[2] Zhang et al. In-Context Multi-Objective Optimization. ICLR, 2026.

[3] Huang et al. Efficient Adaptive Data Acquisition via Pretrained Belief Representations. NeurIPS, 2026.

[4] Shaheen et al. Understanding the Surprising Generalization Properties of Tabular Foundation Models. NeurIPS, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.