- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Online Surrogate Repair (OSR), a system in which a scientific agent searches using inexpensive surrogate feedback, while an acquisition rule periodically selects archived candidates for oracle evaluation. The resulting labels are appended to the surrogate’s context. Experiments on synthetic tasks and the MADE materials-discovery benchmark compare acquisition rules and feedback schedules. The paper aims to show that sparse online updates can support longer searches with fewer oracle feedback queries.
Critical strengths
The distinction between global prediction error and optimization regret is useful, and the experiments illustrate that improving one need not improve the other. This is arguably the paper’s most informative observation, although it is not developed into the central contribution.
Critical weaknesses
-
The methodological novelty is severely limited. Searching with surrogate predictions, selectively evaluating candidates with the true objective, and incorporating those observations into subsequent model updates are established components of surrogate-assisted optimization. For example, Pitra et al., Doubly Trained Evolution Control for the Surrogate CMA-ES (2016), already combine optimizer-generated candidates, acquisition-based selection including EI and model updates using true evaluations. OSR’s “repair” consists of appending labelled observations to an existing model’s context, without introducing a new update mechanism, acquisition function, or adaptive schedule. Proposal-policy independence and modular separation of search from evaluation do not, by themselves, establish a substantive distinction from this literature.
-
The experiments do not adequately establish incremental value over comparable existing methods. Most comparisons involve restricted variants of the proposed pipeline, such as no repair, one-shot static acquisition, or oracle feedback after every episode. These comparisons largely confirm the expected benefits of adaptive sampling and additional inexpensive search. Archive-BO is relevant, but retains the same archive, schedule, and EI rule while changing the surrogate; it therefore primarily tests surrogate choice rather than the contribution of the OSR framework. The paper lacks convincing comparisons against established methods that likewise permit extended surrogate-guided search and sparse true evaluations under matched resource budgets. Consequently, the results provide limited evidence of an advance beyond established practice.
Questions for authors
-
What specific algorithmic contribution distinguishes OSR from existing surrogate-assisted optimization and evolution-control methods, beyond using an agent and an in-context surrogate?
-
Can the authors provide comparisons against closely related methods with matched initial information, oracle budgets, and access to inexpensive search, while reporting candidate-generation and surrogate-evaluation costs?
-
Beyond confirming that global prediction accuracy and optimization regret are different objectives, what new conclusion about their relationship is established, and how does it inform a method beyond standard acquisition rules?
6 comments
Response to Reviewer Chng 2/2
[Q2 & W2] Comparisons Against Established Methods Under Matched Budgets and Candidate/Surrogate Evaluation Costs.
We thank the reviewer for requesting established baselines and explicit evaluation budgets. At matched oracle-query allowances, LQ + TabPFN + online EI achieves the lowest mean normalized gap at both context sizes: 0.304 and 0.405, versus 0.363 and 0.537 for native lq-CMA-ES. The comparison covers all 24 BBOB functions in 5D and 10D, with three instances per task, shared initial observations (), and an 32 additional oracle evaluations. Baselines include CMA-ES, TuRBO-1, native lq-CMA-ES, and both DTS-CMA-ES variants (Table 3):
| Method | Norm. Gap () | Log Gap () | Norm. Gap () | Log Gap () |
|---|---|---|---|---|
| TuRBO-1 | 0.326(16) | 1.820(33) | 0.506(17) | 1.573(36) |
| DTS-CMA-ES (EI) | 0.348(15) | 1.774(63) | 0.556(20) | 1.413(82) |
| Native lq-CMA-ES | 0.363(16) | 1.401(26) | 0.537(14) | 1.071(29) |
| DTS-CMA-ES (variance) | 0.460(17) | 1.976(64) | 0.748(19) | 1.767(62) |
| CMA-ES | 0.568(22) | 2.304(28) | 0.856(16) | 1.959(15) |
| Fixed TabPFN + EI | 0.693(21) | 2.356(28) | 0.541(17) | 1.580(26) |
| LQ + TabPFN + online EI | 0.304(15) | 1.423(29) | 0.405(15) | 1.051(22) |
| GP + TabPFN + online EI | 0.333(18) | 1.476(28) | 0.433(18) | 1.073(24) |
| Online TabPFN + EI | 0.376(17) | 1.958(29) | 0.439(17) | 1.465(24) |
| Online TabPFN + Peak | 0.428(17) | 2.083(26) | 0.465(16) | 1.499(22) |
The paired normalized-gap improvements over native lq-CMA-ES are 0.058 [0.026, 0.091] and 0.131 [0.099, 0.164] at and (95% intervals; Appendix H). The GP combination also has positive paired intervals against DTS-CMA-ES with variance selection. Standalone Online EI reaches 0.439 at , below every standalone optimizer baseline in mean normalized gap.
The additional LLM cost is USD 0.33-0.68 per campaign at matched mean audited discovery counts. Online EI costs USD 1.35 versus 1.02 for DeepSeek and USD 2.02 versus 1.34 for Luna. Reported GPU times are 922 versus 813 and 342 versus 387 H100-seconds, respectively. We note that the additional costs are minimal compared to costly oracles such as wet-lab verification, justifying potential application of OSR in such cases (Appendix E, Table 9):
| Workflow | Operating Point | Oracle Queries () | LLM Cost ($) | GPU Time (H100-s) | Discoveries |
|---|---|---|---|---|---|
| DeepSeek-V4-Flash | Online EI (300 ep.) | 32 | 1.35 | 922 | 53.9 |
| Control, catch-up (224.5 ep.) | 224.5 | 1.02 | 813 | 53.9 | |
| Control, matched budget | 32 | 0.12 | 102 | 13.07 | |
| GPT-5.6-Luna | Online EI (300 ep.) | 32 | 2.02 | 342 | 56.9 |
| Control, catch-up (190.6 ep.) | 190.6 | 1.34 | 387 | 56.9 | |
| Control, matched budget | 32 | 0.16 | 63 | 9.63 |
Seperately, the strictly search-time Top-1 comparison gives an acceleration factor of 4.52-5.44x less oracle queries (Table 4).
[Q3 & W1] New Conclusions on the Accuracy-vs-Regret Relationship and How They Inform Our Method.
We thank the reviewer for asking how the findings inform OSR. They guide optimization-directed label selection and motivate hybrid in-context repair:
- Global fit is an insufficient repair objective: Global-UQ reduces mean NRMSE by 0.0046 but mean regret by only 0.0631; EI reduces mean NRMSE by 0.0011 and mean regret by 0.4093 (Table 1). In the agent loop, our adapted Tab-AICL baseline reaches mean regret 0.1169 versus 0.0552 for Online EI. This supports selecting context labels for their effect on optimization.
- Query dispersion alone does not determine performance: EI achieves mean regret 0.0552 at normalized step distance 0.356, between Peak (step 0.079, regret 0.0976) and Tab-AICL (step 0.915, regret 0.1169). The geometry measurements cover 620 successive acquisition pairs per method (Appendix C.2):
| Method | Incumbent Updates / World | Mean Regret () | ||
|---|---|---|---|---|
| Online Peak | 0.079 | 0.189 | 6.1 | 0.0976 |
| Online Q90-UCB | 0.189 | 0.255 | 6.0 | 0.0600 |
| Online EI | 0.356 | 0.448 | 5.6 | 0.0552 |
| Tab-AICL | 0.915 | 0.916 | 1.1 | 0.1169 |
The online comparisons also favor EI over Q90-UCB: mean regret is 0.0552 versus 0.0600 in the synthetic campaigns, and mean Top-1 is 128.3 versus 108.1 on MADE. Q90-UCB's slight lead in the pooled static study therefore does not carry over to these closed-loop comparisons. 3. Constructive extension through hybrid repair: We add classical predictions as TabPFN features and update its context through Online EI. The LQ hybrid lowers mean normalized gap by 16% at and 25% at versus native lq-CMA-ES (Table 3).
Response to rebuttal
Thank you for providing additional experiments. However, my main concern is not that the paper combines existing techniques. A meaningful integration can certainly be a valuable contribution. Rather, the paper does not sufficiently establish a non-trivial insight connecting its motivation, design choices, and empirical findings. While the additional experiments demonstrate that the proposed combination can be effective, it remains unclear what generalizable methodological understanding the reader gains beyond the established principles of surrogate-assisted optimization and acquisition-based evaluation.
I would therefore appreciate it if the authors could clarify what genuinely new insights, empirical observations, or algorithmic contributions this paper offers beyond the existing literature.
Response to Reviewer Chng 1/2
Dear Reviewer Chng,
We thank you for the rigorous evaluation, for recognizing the value of our distinction between global prediction error and optimization regret, and for pointing us to Doubly Trained Evolution Control (
DTS-CMA-ES, Pitra et al., 2016), which we now explicitly cite and benchmark against in Sections 2.2 and 4.3 (Table 3). We have substantially improved our paper, and all revised content is marked in red. We provide the following response:We thank the reviewer for asking what OSR adds to evolution control. OSR integrates retrospective archive selection and in-context repair into an agentic discovery loop whose oracle budget is independent of search length. This gives two concrete distinctions: