- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces BayesianAgent, which learns normalized expansion gains within the MLEvolve solution graph using a Gaussian-process surrogate informed by parent performance, refinement history, ancestry, and LLM potential ratings. It combines predicted gains with parent scores and uses joint Thompson sampling to select the next expansion. Evaluation covers AlphaEvolve, MLE-bench Lite, and supplementary AutoLab tasks.
Critical strengths
- Clear formulation and coherent modeling. The distinction between current parent quality, expansion gain, and expected child quality is well motivated. The resulting model is understandable and offers a concrete approach to a relevant allocation problem.
- Broad evaluation with encouraging held-out results. The study covers 15 mathematical tasks and 22 ML-engineering tasks, with broadly favorable MLE-bench scores against MLEvolve. Importantly, MLE-bench separates the validation feedback used during search from the held-out scores used for final evaluation (Tables 1–2; Section 4.1).
Critical weaknesses
- Insufficient investigation of the proposed Bayesian mechanism
W1 — Limited evidence for the acquisition design. Joint Thompson sampling is consistent with the GP posterior, but the paper provides little insight into when this particular allocation rule is preferable. The posterior-mean ablation does not distinguish generic exploration from the value of joint dependence or compare other acquisition rules (Section 4.4).
W2 — Prediction diagnostics do not validate allocation. Section 4.3 evaluates posterior means only on valid expansions selected by BayesianAgent, not posterior uncertainty calibration or decisions among alternative expansions. The virtually identical gain correlations of the full and zero-mean full-kernel GPs also leave the nonzero rating prior’s added value unclear (Table 3; Appendix C.4).
- Empirical robustness and attribution remain incomplete
W3 — Execution differences weaken the AlphaEvolve comparisons. Appendix B.2 reports changes to providers/CPU allocations and outdated resource descriptions during AlphaEvolve runs. These can affect throughput and potential ratings, so nominally matched time budgets do not establish controlled comparisons; both the main results and ablations remain difficult to attribute to allocation alone.
W4 — Run-to-run robustness is insufficiently characterized. The favorable MLE-bench results merit credit, but Tables 1–2 provide no per-task repeated-run uncertainty. The three-seed AutoLab extension has mixed task outcomes and only a small overall mean advantage over MLEvolve (0.6558 versus 0.6547), leaving the robustness and magnitude of the gains insufficiently established (Appendix F).
Questions for authors
- Regarding W1: With the same GP, candidate proposals, and budget, what evidence distinguishes joint Thompson sampling from independent marginal sampling or another acquisition rule?
- Regarding W2: What evidence supports reliable posterior uncertainty and useful cross-candidate information sharing beyond the selected-action mean-prediction analysis? Which modeling components yield a reproducible online benefit?
- Regarding W3: Which AlphaEvolve task–method runs were affected by the execution-condition changes? Do the main and ablation conclusions persist in comparisons with matched conditions?
- Regarding W4: Are repeated runs available for the main benchmarks, with paired variability in task scores and medal attainment, to distinguish reliable gains from run-to-run variation?
7 comments
Response 2/3: Decision Evidence and Component Contributions (W2)
The revised evidence connects three levels of the method: predicting gains from decision-time feedback, choosing among alternative expansions under a common history, and improving solutions over a search budget. We also provide concrete posterior updates and targeted component controls to explain how the surrogate supports these decisions.
1. Direct decision evaluation
The replay simulator starts policies from the same valid recorded drafts and reveals a recorded expansion outcome after its action is chosen. Each expansion attempt counts toward the proposal budget. Fixed-menu evaluation holds the history and available actions constant, using repeated outcomes to score the choices of different policies.
The recorded expansion histories cover 45 source graphs from all 15 AlphaEvolve tasks. The fixed menus contain 122 snapshots, 1,365 state–action pairs, and 2,947 recorded expansion outcomes, including 2,460 valid recorded expansion outcomes for alternative actions. Known failures receive zero positive incumbent improvement. The normalization reference is available to the evaluator, while policy decisions use the recorded information available at each state.
Table 1. Offline allocation comparison.
| Policy | AUC (%) ↑ | NPV (%) ↑ |
|---|---|---|
| BayesianAgent (joint TS) | 63.39 | 61.19 |
| Greedy | 61.47 | 53.22 |
| MCGS (offline adapter) | 57.85 | 34.27 |
| Random | 56.97 | 49.45 |
| Agent-rating | 58.35 | 53.90 |
AUC measures normalized incumbent improvement over 48 proposals across 14 positive-reference tasks. NPV measures choices at fixed menus across 12 positive-reference tasks. MCGS denotes the offline adaptation of MLEvolve. Section 4.4 presents the main results.
BayesianAgent attains the highest AUC and NPV among these external policies. The NPV result directly supports better expansion choices with the same recorded history and candidate set, while AUC measures how allocation translates into progress over a proposal budget.
2. How feedback informs related candidates
Appendix A.5 gives recorded examples of information sharing through posterior updates. The NOMAD example shows how an observation changes the inferred root contribution, with a larger response near the observed parent's performance. In Leaf Classification, one additional observation changes the predicted normalized gain of an unchanged candidate from another root from 0.6247 to 0.5988, with the preceding observations, candidate input, normalization, and parameters held fixed. These examples illustrate how feedback about one expansion informs other opportunities through the shared posterior.
Section 4.3 complements these examples with gain-prediction results: BayesianAgent achieves Spearman 0.2800, compared with 0.2040 for LLM rating and −0.0283 for MCGS reward, with an advantage over MCGS reward on 14 of 15 tasks. The learned model therefore provides an informative ranking signal for the allocation decisions tested above.
3. Component contributions and overall performance
Table 2. Targeted GP component controls.
| Variant | Online average rank ↓ | AUC (%) ↑ | NPV (%) ↑ |
|---|---|---|---|
| BayesianAgent (joint TS) | 4.54 | 63.39 | 61.19 |
| w/o root covariance | 4.66 | 60.61 | 61.02 |
| Zero-mean, full kernel | 5.30 | 63.95 | 57.70 |
| w/o LLM-potential terms | 4.84 | 63.23 | 57.85 |
Every component variant retains joint TS. Online ranks use the common nine-method comparison; the offline protocols are those of Table 1 above. Bold marks the best value among the variants shown. The complete controls, including Context-only and Root-only, are reported in Appendix E, Tables 11–13 and Appendix J, Table 21.
The controls identify different contributions to allocation. Removing root covariance primarily reduces replay progress. Removing the LLM prior mean, or both its mean and covariance contributions, reduces fixed-menu NPV. The zero-mean full-kernel variant retains rating covariance, while w/o LLM-potential terms removes both rating contributions and retains solution reviews.
Across the complete controls, Root-only reaches the highest replay AUC of 64.67% and the lowest NPV of 51.50%, illustrating the distinct information captured by the two evaluation criteria. Taken together, the full model combines the highest fixed-menu NPV with competitive replay AUC, supporting the combined use of root covariance, LLM potential, and joint sampling for both decision quality and search progress. It also has the best mean rank in the three-seed online ablation comparison.
The prediction analysis, recorded posterior updates, and alternative-action evaluations provide complementary evidence for the learning and allocation mechanisms. An independent posterior-coverage study is a future direction for assessing probabilistic calibration.
Response 3/3: Execution Settings and Repeated-Run Robustness
W3: Execution comparability and allocation evidence
The revised main comparisons and AlphaEvolve ablations use three seeds under the protocol in Appendix B. Both main benchmarks use GPT-6-luna through Codex with medium reasoning effort. AlphaEvolve runs receive a four-hour active-time budget, 21 CPUs across three search slots, at most 500 evolution steps, and a 1,000-second candidate execution cap. MLE-bench Lite runs receive a 12-hour allocation, one NVIDIA A100, 16 CPUs, and three search workers. These benchmark-specific resources and model access are shared across policies. AlphaEvolve CPU nodes are assigned individually on the same cluster.
Appendix F separately reports the single-seed AlphaEvolve results and describes their provider, CPU-allocation, and potential-assessment resource-description differences. The revised main results and component comparisons use the three-seed experiments.
On AlphaEvolve, the main MLEvolve baseline retains its native scheduling and memory prompts, while Greedy and Agent-rating use BayesianAgent's. Appendix E, Table 10 additionally evaluates MCGS/Top-k with BayesianAgent's scheduling and memory-prompt settings. In that three-policy comparison, BayesianAgent obtains average rank 1.68, ahead of MLEvolve at 1.94 and MCGS/Top-k at 2.38. This result strengthens the allocation comparison by evaluating both the native baseline and its parent- ion rule adapted to the BayesianAgent execution framework.
W4: Repeated-run results and the magnitude of the gains
Table 1. Three-seed main benchmark summary.
| Policy | AlphaEvolve average rank ↓ | MLE-bench Lite gold rate (%) ↑ | MLE-bench Lite medal rate (%) ↑ |
|---|---|---|---|
| BayesianAgent | 1.96 | 43.94 (6.94) | 69.70 (2.62) |
| MLEvolve | 2.33 | 39.39 (2.62) | 57.58 (2.62) |
| Greedy | 2.66 | 40.91 (4.55) | 62.12 (5.25) |
| Agent-rating | 3.06 | 40.91 (0.00) | 66.67 (2.62) |
AlphaEvolve ranks average the four-policy comparisons over 15 tasks and three seeds. MLE-bench Lite entries are mean (sample SD) across three seeds, each covering all 22 competitions. Medals are determined from official held-out scores.
BayesianAgent leads the AlphaEvolve average rank in all three seeds, with ranks of 1.73, 1.87, and 2.27. It records 22 wins, 11 ties, and 12 losses against MLEvolve across the 45 paired task–seed comparisons. The comparison with Greedy and Agent-rating supports using predicted expansion gains together with current solution quality to allocate further experiments. Appendix E, Table 9 reports each task's signed paired score difference, sample SD, and W/T/L counts.
On MLE-bench Lite, BayesianAgent obtains 46 medals versus 38 for MLEvolve and 29 golds versus 26 across 66 task–seed runs. The gains of 12.12 percentage points in mean medal rate and 4.55 points in mean gold rate demonstrate benefits at both quality thresholds. The three-seed AutoLab experiments in Appendix H provide supplementary validation in an additional application setting.
The revised manuscript's Tables 1 and 2 contain the complete task-level means and sample SDs. Together, the repeated main benchmarks, paired task-level results, and supplementary AutoLab evaluation support the empirical value of learning expansion outcomes for experimental allocation.
Response 1/3: Contributions and Acquisition Design (W1)
We thank the reviewer for recognizing the formulation of expansion gain and the held-out MLE-bench evaluation. BayesianAgent contributes a framework for learning expansion outcomes and using them to allocate experiments. Its contributions comprise three connected parts:
The acquisition rule converts this learned posterior into an expansion decision. Replacing joint TS with posterior-mean selection, Independent-TS, or GP-UCB preserves the expansion-outcome formulation, shared surrogate, and feedback-learning loop. These alternatives test how to use the model's predictions and retain the central modeling and learning contributions.
1. Why joint sampling uses the modeled relationships
Joint sampling compares candidates under a common plausible realization of their expected child quality. For two candidates,
Var(Q1−Q2)=Var(Q1)+Var(Q2)−2Cov(Q1,Q2).
The covariance determines uncertainty about their relative quality. Joint TS preserves this relationship when choosing an expansion. Independent-TS keeps the marginal means and variances and removes the sampling dependence, providing a direct comparison of how candidate correlations affect allocation.
2. Acquisition comparisons
Table 1. Acquisition rules within the BayesianAgent modeling framework.
Online ranks are from the common nine-method comparison across 15 tasks and three seeds. AUC uses 48 proposals and 14 positive-reference tasks; NPV uses fixed menus and 12 positive-reference tasks. Bold marks the best value among the acquisition rules shown.
The acquisition controls use the same proposal construction, GP fitting procedure, and resource configuration. GP-UCB selects by posterior mean child quality plus one latent posterior standard deviation. At fixed menus, the rules share the same fitted GP, recorded history, and available actions. Online and replay histories subsequently evolve according to each rule's choices. Joint TS achieves the best online rank and highest fixed-menu NPV among these acquisition controls. Its higher NPV than Independent-TS supports preserving candidate correlations when comparing expansions; its AUC is comparable to posterior-mean selection.
These experiments position joint TS as an effective use of BayesianAgent's joint posterior and demonstrate the framework's support for alternative acquisition functions. Section 4.5 and Appendices E and J provide the complete acquisition and component comparisons.