- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to solve whether the per-task GP surrogate in latent-space BO can be replaced by a pre-trained in-context predictor (PFN) to enable cross-task transfer. Under limited budgets, per-task GPs suffer from early posterior and hyperparameter errors that propagate through the whole optimization trajectory. Paper proposed PILBO that has two variants, PILBO-D and PILBO-L. The authors also propose molecular-domain continued pretraining (CPT). It combines synthetic objectives built from ChEMBL base scores, Boltzmann sampling biased toward high scores, PFN-embedding-based task selection, and L2-SP regularization.
Critical strengths
1. An empirical observation suggests that a pre-trained PFN serves as a robust surrogate in decoupled LSBO. PILBO-D outperforms various LSBO baselines. Replacing the backbone with TabICLv2 results in only a marginal decline in performance, and a version using Morgan fingerprints as input also surpasses GP-COWBOYS. The ability to use an off-the-shelf tabular foundation model as a surrogate is practically significant.
2. A candid report of a negative result regarding CPT (Cross-Problem Transfer). The study transparently presents the finding that a carefully designed domain-specific CPT is largely ineffective. While tasks were selected as intended within the embedding space (Figure 4), the distribution of conditions during rollout did not align (Figure S2). This serves as a valuable cautionary note for research on PFN-for-BO: an offline synthetic prior struggles to effectively emulate the adaptive context required for Bayesian Optimization.
Critical weaknesses
1. The problem serving as the motivation has not been substantiated, nor is there any analysis demonstrating that the proposed method actually solves it. The core motivation, initial surrogate error in per-task Gaussian Processes (GPs) and its propagation, is not quantified. There is no evidence that PFNs reduce this error. The analysis section (5.3) merely addresses the failure of the ineffective CPT (Cross-Problem Transfer) approach, and even then, it only measures the overlap in the argmax regions rather than the transfer of the predictive prior. Ultimately, the claim that "PFNs improve LSBO" is supported solely by performance curves, with no further substantiation.
2. The sources of performance gains are not disentangled. PILBO-D and GP-COWBOYS differ simultaneously in their surrogates, feature representations (RDKit2D vs. Morgan), and batch acquisition strategies. Consequently, one cannot rule out the possibility that the observed gains stem from the feature representation rather than the pre-trained prior. The paper also lacks comparisons with methods in the transfer/meta-BO literature, which explicitly address cross-task transfer. Since CPT proves ineffective, the methodological contribution effectively reduces to replacing the surrogate model used in COWBOYS.
Questions for authors
-
Can you compare per-round surrogate prediction quality (e.g., NLL, Spearman correlation, calibration on top molecules) between the PFN and a GP along the BO trajectory? This would show whether the early surrogate error cited as motivation actually exists, whether the PFN reduces it, and whether the gap narrows as context accumulates.
-
Can you report a GP on RDKit2D descriptors with the same PI-guided pCN generation and pointwise UCB selection as PILBO-D? This would show whether the gains come from the pre-trained prior or from the representation.
2 comments
Acknowledging our rebuttal
As we have not yet heard back on our rebuttal, we would appreciate knowing whether your questions have been resolved or if any points remain. Your feedback would help us make the discussion productive and further strengthen the manuscript.
Individual answer - on predictive quality, attributions of the gains, and meta-BO comparisons
We thank the reviewer for their careful evaluation and insightful feedback. This prompted us to revisit several aspects of our methodology, as shown below, ultimately strengthening the manuscript.
(Weakness 1 + Question 1) Evidence for the surrogate-error motivation
We thank the reviewer for highlighting the need to substantiate our motivation beyond optimization performance curves. We have now added direct evaluations of surrogate predictive quality, fully discussed in our second general comment and reported in Section 5.2 (Figure 5; Appendix A.5).
Specifically, we compare four surrogate configurations using NLL, Spearman correlation, and ECE along saved BO trajectories, evaluating all surrogates on identical contexts and acquisition-selected molecules. PILBO-D+CPT achieves lower NLL and higher Spearman correlation than the GP variants, with particularly pronounced differences in the early rounds. We further confirm its predictive advantage on randomly sampled GuacaMol contexts. These experiments directly substantiate the predictive-quality motivation, although they do not establish a causal relationship between surrogate errors and subsequent optimization outcomes.
We have also expanded Section 5.3 and Appendices A.2–A.3 to better characterize the limitations of offline CPT, including molecular-support and task-transfer mismatches. Our new Rollout-CPT extension, discussed in the first general comment, further investigates the role of adaptive context construction.
Lastly, let us mention that the acronym CPT denotes continued pretraining and not cross-problem transfer.
(Weakness 2 + Question 2) Attributions of gains and Matched GP baseline on RDKit2D descriptors
We agree that the original comparison left some factors intertwined, despite the Morgan-fingerprint ablation for PILBO-D already included in the manuscript. As discussed in our second general comment, we have now extended this analysis with a complementary matched GP baseline. We reiterate the findings below.
We have now added GP-COWBOYS (RBF+RDKIT) using the same 212 RDKit2D descriptors, PI-guided pCN candidate-generation procedure, and pointwise UCB selection as PILBO-D-CPT. This matched comparison, together with the complementary PILBO-Morgan ablation, is now presented in the main text, Section 5.2 (Sensitivity to molecular representation) and Figure 5 (top). Shortly, PILBO-D-CPT achieves an overall score of 0.6304±0.0326 across the 8 held-out tasks, compared with 0.6166±0.0316 for GP-COWBOYS (RBF+RDKIT). The original manuscript also included a complementary experiment replacing the RDKit2D descriptors of PILBO-D+CPT with Morgan fingerprints, matching the representation used by GP-COWBOYS. This variant is now labeled PILBO-Morgan in Figure 5. This resulted in reduced performance compared to the RDKit2D descriptor representation, but still better than GP-COWBOYS. Overall, these comparisons suggest that the PFN's advantage cannot be attributed solely to its molecular representation.
Comparison with a meta-BO approach
We thank the reviewer for suggesting a comparison with meta-BO methods. We have now included HyperBO H-NLL [1]. This approach learns a GP prior from the same synthetic molecular tasks used for our CPT and conditions it on observations from each held-out objective within our LSBO framework. Appendix A.5 in the updated manuscript provides full details on the adaptation of this baseline to molecular LSBO. At 600 oracle calls, PILBO-D+CPT achieves a mean best-observed score of 0.6304 ± 0.0326, compared with 0.5771 ± 0.0279 for HyperBO, with higher mean scores on all eight tasks (Figure S7).
To caveat this comparison, it is worth noticing that even outside the PFN realm, meta-BO has not been systematically evaluated on held-out molecular LSBO objectives, at least to our knowledge. What we mean by that is that a thorough assessment of alternative meta-BO approaches would likely require a broader investigation, beyond the scope of this rebuttal. For instance, while HyperBO is a general-purpose method, its original empirical evaluation focused primarily on hyperparameter optimization, a setting substantially different from de novo molecular design. Finally, an important distinction is that PILBO builds on a tabular foundation model already pretrained on a vast and diverse corpus of synthetic tasks, even before molecular CPT, whereas our HyperBO baseline learns its prior solely from the synthetic molecular task bank. Hence, one could say that the comparison does not equalize pretraining exposure.
[1] Wang et al. Pre-trained Gaussian Processes for Bayesian Optimization. JMLR, 2024.