- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces BioDyad, a hierarchical agentic search framework that coordinates biomedical discovery and machine learning engineering under a fixed wall-clock budget. Within a shared Monte Carlo graph search (MCGS) backbone, its scientific hierarchy combines task-level prior guidance, node-level discovery, and biomedical memory linking candidate plans to execution outcomes. Its engineering hierarchy advances candidate programs through smoke execution, train/validation evaluation, and final full-data retraining and submission. The authors evaluate BioDyad on the 76-task BioXArena benchmark across nine biomedical domains under a two-hour per-task budget and three matched LLM backends. BioDyad achieves the highest penalized all-task score and task success rate under each backend. Component ablations, repeated runs, candidate-level diagnostics, and case studies provide additional evidence for the framework’s effectiveness.
Critical strengths
-
The framework addresses a concrete coordination problem, and the component analyses support its design. Biomedical evidence can improve predictive representations while introducing integration errors and additional execution costs. BioDyad connects biomedical discovery to program generation and execution through plan-outcome memory and staged evaluation. In Table 4, the full framework achieves a penalized average score of 0.763, compared with 0.697 for MLEvolve w/ discovery. Removing biomedical memory or staged evaluation reduces the score to 0.736 and 0.733, respectively. Table 7 further shows that the full framework produces the largest number of valid full-execution runs within the fixed budget. These results support a useful system contribution: effective biomedical resource integration depends on how discovery, memory, and execution feedback are coordinated, rather than resource access alone.
-
The broad evaluation and consistent aggregate gains across backends strengthen the empirical contribution. Under the same task interface, hardware allocation, and two-hour limit, BioDyad achieves penalized all-task scores of 0.738, 0.705, and 0.701 across the three backends. The improvements over the strongest matched alternatives are 0.023, 0.042, and 0.086. BioDyad also leads perturbation dynamics under every backend. Three independent runs on a fixed, domain-balanced 18-task subset provide preliminary support for robustness. For a paper whose main contribution is an integrated system, these results provide meaningful empirical support.
Critical weaknesses
-
The component ablations support the framework, but finer-grained attribution remains incomplete. In the primary comparison, MLEvolve and BioXArena Harness do not access the biomedical resources available to BioDyad, although Biomni accesses its own data lake and tools. The resource-matched MLEvolve w/ discovery control therefore provides an important comparison, but it covers only two domains and 16 tasks using GPT-5.4, rather than one of the three main evaluation backends. The existing ablations support the contributions of biomedical memory and staged evaluation, but do not separately establish the benefit of node-level discovery over prior guidance alone, or the additional value of biomedical plan-outcome memory over a resource-matched generic execution-feedback memory. Broader resource-matched comparisons and targeted controls would strengthen the attribution of the observed gains.
-
The budget-extension analysis characterizes candidate-pool improvement, but does not establish the corresponding benefit under the actual model-selection procedure. Section 5.5 and Appendix D explicitly report retrospective maxima of normalized hidden-test scores among completed candidates. This appropriately supports the stated conclusion that extended search can uncover better candidates. However, the deployed procedure selects a program using its pre-retraining validation score, so the analysis does not show whether the system can identify and submit those better candidates. In addition, because search selection depends on the elapsed budget fraction, the two-hour checkpoint of a 12-hour run occupies a different exploration/refinement stage from an independent two-hour run, even under the same scheduling rules. Reporting hidden-test performance for the validation-selected incumbent at each checkpoint would make the practical benefit of additional search time clearer. This limitation concerns the budget-extension analysis and does not directly undermine the two-hour main results.
Questions for authors
-
Could you extend the resource-matched MLEvolve w/ discovery comparison to at least one main evaluation backend, and report repeated-run results or paired confidence intervals for the key ablations?
-
With resources and budgets held fixed, how much does node-level discovery improve over using only the initial BIO_PLAN? How much additional benefit does biomedical plan-outcome memory provide over generic execution-feedback memory?
-
Within each task, do candidates share fixed validation examples, splits, and metric aggregation rules? If candidate plans can change the validation strategy, how are scores kept comparable for search and final program selection?
4 comments
Response to Reviewer nRRg (2/2)
W2: The budget-extension analysis characterizes candidate-pool improvement, but does not establish the corresponding benefit under the actual model-selection procedure.
Thank you for raising this insightful point.
We reported retrospective maxima of hidden-test scores in Table 6 (Table 5 in the original submission) to show that gains emerge from candidate exploration, but we agree that these candidates are not the ones the system would finally submit. In order to verify practical gains to a larger budget, we re-select candidates by validation scores and compare them in the following table:
| Domain (GPT-5.6-sol) | 2h | 4h | 6h | 8h | 10h | 12h | Gain 2 to 12h |
|---|---|---|---|---|---|---|---|
| Chemical biology, validation-selected | 0.900 | 0.923 | 0.924 | 0.924 | 0.931 | 0.929 | +0.029 |
| Chemical biology, retrospective maximum | 0.927 | 0.932 | 0.936 | 0.936 | 0.936 | 0.937 | +0.010 |
| Phenotype-disease, validation-selected | 0.580 | 0.624 | 0.613 | 0.613 | 0.620 | 0.631 | +0.051 |
| Phenotype-disease, retrospective maximum | 0.640 | 0.663 | 0.688 | 0.690 | 0.690 | 0.690 | +0.050 |
From this table:
- The most critical observation is that the gains still exist when selecting candidates by validation scores (+0.029/+0.051 on chemical biology and phenotype-disease), so the system does identify and submit much of the improvement found by extended search.
- As you rightly pointed out, the 2h checkpoint of a 12-hour run is at a different exploration stage from an independent two-hour run, which is why it falls below Table 2. Relative to Table 2, the gains are +0.003/+0.037 on chemical biology and phenotype-disease. The gain depends on headroom: chemical biology already starts high (0.926 in Table 2), while phenotype-disease has more room to improve.
We have added the validation-selected rows to Table 6 and revised Sec. 5.5 accordingly.
Q3: Within each task, do candidates share fixed validation examples, splits, and metric aggregation rules? If candidate plans can change the validation strategy, how are scores kept comparable for search and final program selection?
Thank you for this constructive question.
Within a task, candidates share the metric and its aggregation rule, but not the validation examples or split: following MLEvolve, each candidate defines its own hold-out or CV split, and a candidate plan can change it. Validation scores are therefore not strictly comparable across candidates, either during search or at final selection. We kept MLEvolve's protocol so that BioDyad and the MLEvolve baseline select programs under the same rule.
This affects which program is selected, not how results are scored: all reported numbers are hidden-test scores from the official BioXArena evaluator. Validation scores remain informative: in the 12-hour runs of Table 6, the median Spearman correlation between candidates' validation and hidden-test scores is 0.64 across the 16 tasks. A split shared across candidates would make validation scores directly comparable, and we note this in Appendix B.2.
We have clarified the validation protocol in Appendix B.2 and added the validation–test correlation to Appendix D.1.
We hope these responses address your concerns, and we would be happy to discuss any further questions.
Thank you for the detailed response and additional experiments. The new experiments strengthen the paper and sufficiently address my concerns. I appreciate the authors’ careful revisions and will maintain my rating of 3 (Weak Accept).
Thank you for your positive feedback and for recognizing our revisions. We are glad that the additional experiments have sufficiently addressed your concerns. We sincerely appreciate your thoughtful review and your support for our work.
Response to Reviewer nRRg (1/2)
Thank you for your time and valuable feedback. Our replies to the weaknesses/questions are as follows.
Thank you for this precise assessment. We agree that the resource-matched control should be tested under a main backend. We ran it on all 8 perturbation-dynamics tasks, the domain where BioDyad leads under every backend, with DeepSeek-V4-Flash Preview, one of our three main backends. MLEvolve w/ discovery uses the same cache, tools and discovery components as BioDyad. Since biomedical discovery is itself part of BioDyad's scientific hierarchy, this control isolates the contribution of the remaining components, memory and staged evaluation. Each configuration was run three times (mean ± sample SD over runs); paired 95% CIs are bootstrapped over the 8 per-task means.
From these results:
We have added these results to Sec. 5.3 and Appendix C.4 of the revised manuscript.
Thank you for this constructive question.
We apologize for the confusion caused by our description of memory. Generic execution-feedback memory is MLEvolve’s contribution rather than ours, so we retain it in all configurations and do not ablate it separately. Biomedical memory builds on this mechanism and extends it when enabled. BioDyad w/o memory (we now name it "w/o bio. memory") therefore falls back to MLEvolve’s generic memory while keeping the same resources and discovery, i.e., the resource-matched generic memory you describe. Comparing the full framework with this variant measures the benefit of the biomedical extension. We have clarified this in Sec. 4.1.2 of the revised manuscript.
Biomedical memory records are keyed by the NODE_PLAN from biomedical discovery, so the failures it retains and the repeats it avoids concern biomedical plans. This matters because integrating biomedical evidence introduces execution failures: adding discovery without biomedical memory raises invalid full runs from 248 to 358, and adding biomedical memory raises the valid share from 66.5% to 80.9% (Appendix C.3).
Following your suggestion, we ran both controls on all 8 perturbation-dynamics tasks with DeepSeek-V4-Flash Preview, one of our three main backends, with the same two-hour budget as Table 4 and three independent runs per variant (mean ± sample SD over runs).
From these results:
We have added these results to Sec. 5.3 (Tables 4 and 5) of the revised manuscript.