- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes the Evidence-Driven Discovery Loop (EDL) for automated ML development under limited time budgets. EDL organizes implementations into algorithmic directions and uses Operation-aware Evidence Control (OEC) to inspect trial histories and code before jointly selecting operations, sources, and modification plans. Experiments on MLE-Lite report higher mean Human Rank across three backbones, with additional results on NatureBench Lite. Ablations examine direction grouping, joint operation and source selection, and adaptive evidence retrieval. The authors report improvements in solution quality and search efficiency, while acknowledging additional inference costs.
Critical strengths
- EDL integrates evidence inspection with the joint selection of operations, sources, and modification plans. Allowing the controller to revise these decisions as it inspects trial histories and code is a useful approach to automated ML development.
- EDL achieves the highest mean Human Rank on MLE-Lite across all three backbones, and the evaluation accounts for planning overhead.
Critical weaknesses
The routing ablation changes more than how experiments are selected. Appendix C.6 states that Fixed route restricts Crossover to component transfer and allows new-direction Draft only when no valid source is available. Full OEC also permits pipeline fusion and new-direction actions when valid sources are available. It is therefore unclear how much of the improvement comes from evidence-driven operation and source selection and how much comes from the broader set of permitted actions.
Questions for authors
After initialization, how often did Full OEC use pipeline fusion and new-direction actions, and how did these actions contribute to the final selected solutions? This would help assess whether the restrictions in Fixed route materially affect the interpretation of the routing ablation.
4 comments
Thanks for addressing my earlier concerns. I have one follow-up question after rereading the paper.
The paper groups multiple implementations into the same direction and uses the current best valid exemplar to represent each direction. Can OEC also access the full code of other variants within the same direction? A lower-scoring variant might still contain a useful component that could be transferred to another implementation. Can OEC retrieve and use such variants, or is it limited to the current best exemplar? If the latter, could this cause it to miss useful components from other variants?
Follow-up response to reviewer Y7fX
Thank you for raising this question. OEC’s core contribution is an evidence-guided policy that jointly selects operations, sources, and modification plans. The controller interprets trial meta signals together with historical outcomes and implementation details to decide what to attempt next. This is particularly relevant to crossover: individual scores alone do not reveal whether two implementations contain complementary components or whether those components are compatible.
We acknowledge the limitation in the original implementation. Within each direction, the code-retrieval interface exposes only the current best valid exemplar; other attempts remain accessible through historical records, but their full code cannot be individually retrieved through that interface. We adopted this design because each direction groups implementations that share a broad algorithmic structure, and inspecting every variant could substantially increase context consumption and tool interactions. The current best valid exemplar provides a validated representative for comparing directions within a bounded inspection budget.
We quantified this trade-off using the historical planning snapshots. Across 292 DeepSeek and 246 Kimi planning rounds, reading every distinct valid implementation within the same directions that OEC actually inspected, instead of their exemplars alone, would require 6.41× and 6.52× as many code characters, respectively. Such exhaustive reading would exceed the existing 100,000-character per-round code-inspection budget in 65.1% and 57.7% of rounds.
The revised paper’s Section 5.4 reports our extensions to latest AI-train-Ai tasks such as nanoGPT optimizer development and AI4AI-Bench. For AI4AI-Bench, implementations and experimental evidence span multiple repository files. We therefore use Claude Code (CC) as the tool harness for both the coding agent and OEC’s planning controller. In this extension, we remove the exemplar-only access restriction: the controller can use file-reading and search tools to inspect source files and detailed evidence from archived historical nodes, including non-best variants when their source is available.
The planning traces actual use of this broader access:
- OWL/Wanda pruning: In one planning round, the controller read algorithm.py from four historical implementations, nodes 001, 003, 005, and 002, opening either the complete file or relevant sections. Their search-stage validation perplexities were 22.4225, 20.5533, 20.5299, and 118.8036, respectively, with lower values being better. Thus, inspection included an implementation substantially worse than the current best. The committed plan cited code locations in nodes 001 and 003 to identify a reconstruction target that preserves upstream pruning-induced input drift, and proposed testing a target based on the dense model’s layer outputs.
- DDPO: In another planning round, the controller fully read two different versions of pathwise_rollout.py, containing 83 and 126 lines, from nodes 004 and 005. Their search-stage aesthetic scores were 8.8534 and 7.4812, respectively, with higher values being better. The plan used evidence from both implementations to compare gradient coverage, update cost, and update count, and proposed testing whether performance under a fixed time budget was limited by update throughput.
These examples show that the CC-based extension inspects multiple historical implementations, including lower-scoring ones, and uses the retrieved evidence to identify limitations and formulate targeted experiments. We will clarify how this broader access extends the exemplar-only interface used in the original experiments while preserving OEC’s core policy.
We sincerely thank the reviewer again for their time in providing constructive feedback. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.
Best wishes,
The Authors
Thank you for your response. This addresses my concerns, and I will maintain my positive rating.
Response to reviewer Y7fX
Dear Reviewer Y7fX, Thank you for your time and effort in reviewing our paper. We address the questions and comments raised by the reviewer below.
We agree that the original Fixed-route comparison changes both operation/source selection and the permitted action set. We therefore supplement it with an audit of operation usage and final-solution provenance, a restricted-action-support condition, and a matched-source retrieval comparison.
Across 66 Full EDL/OEC task-runs (Deepseek-V4.1-Flash), we identified 1,020 post-initialization experiments, excluding Debug repairs as separate experiments. Pipeline fusion and new-direction actions accounted for 8.04% of these experiments; the remaining 91.96% were Improve or component-transfer actions. OEC selects pipeline fusion and new-direction actions relatively infrequently, using them mainly to introduce alternatives or combine existing solutions for subsequent refinement. Improve and component transfer account for most search activity, developing these starting points through incremental optimization.
¹ Outcomes include any Debug repairs associated with the originating experiment.
² Ancestry follows recorded primary, donor, fusion-source, and repair edges and includes direct selection.
The two ancestry sets overlap in two runs, giving 18/66 runs (27.27%) in their union. Concrete examples include Spooky Author Identification, where a fusion experiment improved the incumbent validation log loss from 0.2721364 to 0.2619542 and produced the final selected solution, and Denoising Dirty Documents, where a new DnCNN direction, after repairs, improved the validation score from 0.0088711 to 0.0084252 and was selected.
New-direction actions also provided starting points for subsequent development. Of the five final candidates whose ancestry included a new-direction action, four were ultimately produced by Improve. Two of these paths passed through pipeline fusion and two through component transfer before further refinement. Thus, examining only the final operation label would miss earlier exploration reused by later experiments. These records establish participation in the selected candidates’ source histories, rather than the performance loss that would result from removing an operation.
To address the weakness more directly, we added a condition that retains LLM operation/source selection while imposing the reviewer-identified action restrictions: Crossover permits component transfer only, and new-direction actions are allowed only when no valid source exists. Its post-initialization traces contain 775 Improve attempts and 238 component transfers, with no pipeline-fusion or new-direction executions. It nevertheless achieves 72.73% medal rate and 0.8565 Human Rank. This establishes that OEC can attain this performance under the restricted menu, without executing the additional operations.
Complementary controlled evidence comes from the matched-source history-retrieval experiment. Both conditions use the same deterministic source-eligibility rule, preload complete source code, prohibit additional code retrieval, and retain identical action permissions. History queries therefore cannot expand source eligibility within a planning round. Adaptive history retrieval achieves 74.24% medal rate and 0.8543 Human Rank, versus 69.70% and 0.8409 with static histories: improvements of 4.55 percentage points and 0.0134, respectively. Neither condition accepted an existing-source action outside its eligible set. This comparison supports the value of adaptive history selection under controlled source eligibility and action permissions.
These findings support distinct aspects of the design: the additional operations provide selectively used exploration and composition paths; OEC also attains strong results under restricted action support; and adaptive history retrieval has a positive effect in the matched-source comparison. We will revise the original ablation interpretation accordingly.
We sincerely thank the reviewer again for their time in providing constructive feedback. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.
Best wishes,
The Authors