acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Driven Discovery Loops for AI-Train-AI: From Trials to Better Decisions

Abstract

An emerging frontier of automated AI research is AI-train-AI, in which agents autonomously propose, implement, and evaluate machine learning solutions within a closed discovery loop. These experiments consume substantial resources and often fail or yield marginal gains. Under a limited budget, final solution quality depends on choosing when to refine or combine implementations and which candidates to build on. Each trial produces evidence beyond a scalar score, yet this evidence is often only partially used when operations and sources are selected. We introduce the Evidence-Driven Discovery Loop (EDL), which treats evidence use as part of algorithm development. It connects accumulated trial evidence to subsequent search decisions. Its outer loop captures successful and failed trials, deterministically organizes implementations by algorithmic direction, and links modification intents to evaluated outcomes. Its inner agentic policy, Operation-aware Evidence Control (OEC), starts from compact direction cards and operation-level meta signals, actively queries supporting histories and code, and constructs plans specifying operations, source roles, changes, and execution allocations. Both levels serve final solution quality within a budget that accounts for decision and execution costs. Successful and failed attempts update the evidence available to later decisions, making trial and error cumulative without changing controller parameters. Experiments on machine learning engineering tasks show strong solution quality across three model backbones, and NatureBench Lite results extend the comparison to scientific algorithm development. Selected component interventions characterize organization, action control, and evidence acquisition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI-Train-AI, Automated Machine Learning Engineering, Evidence-Driven Search, Agentic LLM Control, Search Metadata

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Y7fXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Evidence-Driven Discovery Loop (EDL) for automated ML development under limited time budgets. EDL organizes implementations into algorithmic directions and uses Operation-aware Evidence Control (OEC) to inspect trial histories and code before jointly selecting operations, sources, and modification plans. Experiments on MLE-Lite report higher mean Human Rank across three backbones, with additional results on NatureBench Lite. Ablations examine direction grouping, joint operation and source selection, and adaptive evidence retrieval. The authors report improvements in solution quality and search efficiency, while acknowledging additional inference costs.

Critical strengths
  1. EDL integrates evidence inspection with the joint selection of operations, sources, and modification plans. Allowing the controller to revise these decisions as it inspects trial histories and code is a useful approach to automated ML development.
  2. EDL achieves the highest mean Human Rank on MLE-Lite across all three backbones, and the evaluation accounts for planning overhead.
Critical weaknesses

The routing ablation changes more than how experiments are selected. Appendix C.6 states that Fixed route restricts Crossover to component transfer and allows new-direction Draft only when no valid source is available. Full OEC also permits pipeline fusion and new-direction actions when valid sources are available. It is therefore unclear how much of the improvement comes from evidence-driven operation and source selection and how much comes from the broader set of permitted actions.

Questions for authors

After initialization, how often did Full OEC use pipeline fusion and new-direction actions, and how did these actions contribute to the final selected solutions? This would help assess whether the restrictions in Fixed route materially affect the interpretation of the routing ablation.

4 comments
AuthorsOpenReview ↗

Response to reviewer Y7fX

Dear Reviewer Y7fX, Thank you for your time and effort in reviewing our paper. We address the questions and comments raised by the reviewer below.

We agree that the original Fixed-route comparison changes both operation/source selection and the permitted action set. We therefore supplement it with an audit of operation usage and final-solution provenance, a restricted-action-support condition, and a matched-source retrieval comparison.

Across 66 Full EDL/OEC task-runs (Deepseek-V4.1-Flash), we identified 1,020 post-initialization experiments, excluding Debug repairs as separate experiments. Pipeline fusion and new-direction actions accounted for 8.04% of these experiments; the remaining 91.96% were Improve or component-transfer actions. OEC selects pipeline fusion and new-direction actions relatively infrequently, using them mainly to introduce alternatives or combine existing solutions for subsequent refinement. Improve and component transfer account for most search activity, developing these starting points through incremental optimization.

OperationExecuted experimentsIncumbent validation improvements¹Directly produced the final candidate¹Present in final-candidate ancestry²
Pipeline fusion55/1,020 (5.39%)18/552/6615/66
New direction27/1,020 (2.65%)3/271/665/66

¹ Outcomes include any Debug repairs associated with the originating experiment.
² Ancestry follows recorded primary, donor, fusion-source, and repair edges and includes direct selection.

The two ancestry sets overlap in two runs, giving 18/66 runs (27.27%) in their union. Concrete examples include Spooky Author Identification, where a fusion experiment improved the incumbent validation log loss from 0.2721364 to 0.2619542 and produced the final selected solution, and Denoising Dirty Documents, where a new DnCNN direction, after repairs, improved the validation score from 0.0088711 to 0.0084252 and was selected.

New-direction actions also provided starting points for subsequent development. Of the five final candidates whose ancestry included a new-direction action, four were ultimately produced by Improve. Two of these paths passed through pipeline fusion and two through component transfer before further refinement. Thus, examining only the final operation label would miss earlier exploration reused by later experiments. These records establish participation in the selected candidates’ source histories, rather than the performance loss that would result from removing an operation.

To address the weakness more directly, we added a condition that retains LLM operation/source selection while imposing the reviewer-identified action restrictions: Crossover permits component transfer only, and new-direction actions are allowed only when no valid source exists. Its post-initialization traces contain 775 Improve attempts and 238 component transfers, with no pipeline-fusion or new-direction executions. It nevertheless achieves 72.73% medal rate and 0.8565 Human Rank. This establishes that OEC can attain this performance under the restricted menu, without executing the additional operations.

Complementary controlled evidence comes from the matched-source history-retrieval experiment. Both conditions use the same deterministic source-eligibility rule, preload complete source code, prohibit additional code retrieval, and retain identical action permissions. History queries therefore cannot expand source eligibility within a planning round. Adaptive history retrieval achieves 74.24% medal rate and 0.8543 Human Rank, versus 69.70% and 0.8409 with static histories: improvements of 4.55 percentage points and 0.0134, respectively. Neither condition accepted an existing-source action outside its eligible set. This comparison supports the value of adaptive history selection under controlled source eligibility and action permissions.

These findings support distinct aspects of the design: the additional operations provide selectively used exploration and composition paths; OEC also attains strong results under restricted action support; and adaptive history retrieval has a positive effect in the matched-source comparison. We will revise the original ablation interpretation accordingly.

We sincerely thank the reviewer again for their time in providing constructive feedback. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.

Best wishes,

The Authors

Thanks for addressing my earlier concerns. I have one follow-up question after rereading the paper.

The paper groups multiple implementations into the same direction and uses the current best valid exemplar to represent each direction. Can OEC also access the full code of other variants within the same direction? A lower-scoring variant might still contain a useful component that could be transferred to another implementation. Can OEC retrieve and use such variants, or is it limited to the current best exemplar? If the latter, could this cause it to miss useful components from other variants?

AuthorsOpenReview ↗

Follow-up response to reviewer Y7fX

Thank you for raising this question. OEC’s core contribution is an evidence-guided policy that jointly selects operations, sources, and modification plans. The controller interprets trial meta signals together with historical outcomes and implementation details to decide what to attempt next. This is particularly relevant to crossover: individual scores alone do not reveal whether two implementations contain complementary components or whether those components are compatible.

We acknowledge the limitation in the original implementation. Within each direction, the code-retrieval interface exposes only the current best valid exemplar; other attempts remain accessible through historical records, but their full code cannot be individually retrieved through that interface. We adopted this design because each direction groups implementations that share a broad algorithmic structure, and inspecting every variant could substantially increase context consumption and tool interactions. The current best valid exemplar provides a validated representative for comparing directions within a bounded inspection budget.

We quantified this trade-off using the historical planning snapshots. Across 292 DeepSeek and 246 Kimi planning rounds, reading every distinct valid implementation within the same directions that OEC actually inspected, instead of their exemplars alone, would require 6.41× and 6.52× as many code characters, respectively. Such exhaustive reading would exceed the existing 100,000-character per-round code-inspection budget in 65.1% and 57.7% of rounds.

The revised paper’s Section 5.4 reports our extensions to latest AI-train-Ai tasks such as nanoGPT optimizer development and AI4AI-Bench. For AI4AI-Bench, implementations and experimental evidence span multiple repository files. We therefore use Claude Code (CC) as the tool harness for both the coding agent and OEC’s planning controller. In this extension, we remove the exemplar-only access restriction: the controller can use file-reading and search tools to inspect source files and detailed evidence from archived historical nodes, including non-best variants when their source is available.

The planning traces actual use of this broader access:

  • OWL/Wanda pruning: In one planning round, the controller read algorithm.py from four historical implementations, nodes 001, 003, 005, and 002, opening either the complete file or relevant sections. Their search-stage validation perplexities were 22.4225, 20.5533, 20.5299, and 118.8036, respectively, with lower values being better. Thus, inspection included an implementation substantially worse than the current best. The committed plan cited code locations in nodes 001 and 003 to identify a reconstruction target that preserves upstream pruning-induced input drift, and proposed testing a target based on the dense model’s layer outputs.
  • DDPO: In another planning round, the controller fully read two different versions of pathwise_rollout.py, containing 83 and 126 lines, from nodes 004 and 005. Their search-stage aesthetic scores were 8.8534 and 7.4812, respectively, with higher values being better. The plan used evidence from both implementations to compare gradient coverage, update cost, and update count, and proposed testing whether performance under a fixed time budget was limited by update throughput.

These examples show that the CC-based extension inspects multiple historical implementations, including lower-scoring ones, and uses the retrieved evidence to identify limitations and formulate targeted experiments. We will clarify how this broader access extends the exemplar-only interface used in the original experiments while preserving OEC’s core policy.

We sincerely thank the reviewer again for their time in providing constructive feedback. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.

Best wishes,

The Authors

Thank you for your response. This addresses my concerns, and I will maintain my positive rating.

Reviewer_7HEgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Evidence-Driven Discovery Loop (EDL), a two-level framework for budgeted AI-train-AI that guides experimental decisions using accumulated trial history without updating model parameters. Its main contributions are an outer loop that distills both successful and failed trials into compact direction summaries and meta-signals, and an inner agentic policy -- Operation-aware Evidence Control (OEC) -- that adaptively retrieves historical evidence to jointly plan and allocate subsequent experimental operations. Evaluated across three LLM backbones on machine learning engineering (MLE-Lite) and scientific algorithm development (NatureBench Lite) benchmarks, EDL consistently outperforms strong baselines in both solution quality and compute efficiency, with ablation studies confirming the value of its direction grouping, joint operation selection, and adaptive retrieval mechanisms.

Critical strengths

A central contribution of this work is addressing a key bottleneck in existing AI-train-AI harnesses: decoupling source selection (typically based on scalar fitness heuristics) from the specific operation and code modification being planned. By combining compact direction summaries with an inner policy (OEC) that adaptively inspects trial history and code to jointly select operations, sources, and modification plans, the paper shifts from blind numerical node expansion to evidence-informed experiment design. This is consequential to my assessment because it substantiates this design choice with both consistent performance and efficiency gains across three LLM backbones and two benchmarks (MLE-Lite and NatureBench Lite), as well as targeted ablations that isolate the mechanism of improvement.

Critical weaknesses
  • Rather than introducing a fundamentally new learning or search algorithm, EDL/OEC operates largely as a heavily engineered, prompt-driven state machine governed by strict operational guardrails and admissibility rules (Appendices B.2-B.3). It is difficult to disentangle how much of the reported improvement stems from the core conceptual thesis (adaptive evidence inspection) versus these hand-crafted validation contracts that prevent redundant proposals -- constraints from which baseline harnesses do not benefit.

  • The empirical gains are not uniform: on NatureBench Lite (Table 2), EDL underperforms baselines like ML-Master 2.0 and MLEvolve when paired with GLM 5.2, weakening the claim of robust discovery across model backbones.

  • Figure 3(a) reports search efficiency over time using medal rates derived from best validation scores rather than held-out test performance. Because iterative search can overfit to the validation split, validation trajectories may not reflect true generalization; reporting test scores by grading the intermediate submission files generated across search time would provide a more reliable evaluation.

  • The evaluation is limited to three open-weight backbones (Kimi, GLM, and DeepSeek). It remains unclear whether EDL's structured evidence-control scaffolding provides similar gains for state-of-the-art closed-source foundation models (e.g., Gemini, GPT, and Claude) that may already exhibit stronger native long-context reasoning and planning capabilities.

Questions for authors
  • How much of EDL/OEC’s performance gain comes from adaptive evidence inspection itself versus the strict admissibility contracts and operational guardrails? Could you provide an ablation where these validation guardrails are either removed from OEC or equally applied to baseline harnesses?

  • What explains EDL’s underperformance relative to ML-Master 2.0 and MLEvolve on NatureBench Lite when using GLM 5.2 (Table 2)? Does this stem from GLM 5.2 struggling with OEC’s multi-step tool-calling protocol?

  • Since iterative search can increasingly overfit to the validation split over time, do the time-efficiency gains shown in Figure 3(a) still hold on the held-out test set? Could you report the corresponding test-set medal rate curves by grading the intermediate submission files selected at each time checkpoint?

  • Have you evaluated whether EDL provides comparable gains on frontier closed-source models (e.g., Gemini, GPT, or Claude)? Specifically, do models with stronger native long-context reasoning and planning still benefit significantly from OEC’s explicit evidence-control scaffolding, or do the returns diminish?

4 comments
AuthorsOpenReview ↗

1st Response to reviewer 7HEg

Dear Reviewer 7HEg, Thank you for your time and effort in reviewing our paper. We address the questions and comments raised by the reviewer below.

Response to W1 and Q1:

Thank you for raising the distinction between the core mechanism and operational constraints. Our starting point is to make better use of meta-signals already produced by costly experiments. Appendix A.2 shows that meta signals can improve even numerical source-selection policies. However, their usefulness depends on the intended operation, modification, and compatibility between sources, while compact statistics can omit context needed to design the next experiment. Our contribution is a decision policy that makes fuller use of trial-derived meta signals by connecting them to on-demand inspection of histories and code, and jointly selecting operations, source roles, and modification plans. Guardrails constrain admissibility, while experimental choices depend on the evolving evidence.

This strategy also transfers beyond MLE. The new Section 5.4 in our revised paper reports nanoGPT optimizer-development experiments and EDL-guided Claude Code on AI4AI-Bench, alongside NatureBench Lite. Competitive results in these settings support EDL as a reusable decision strategy for AI-train-AI, with applicability across both research tasks and coding-agent harnesses.

To examine performance attribution, we added two complementary ablations using DeepSeek-V4.1-Flash. Each condition covers 22 tasks with three search runs per task. Values below are means ± standard deviations across the three run-level task averages.

ConditionMedal rate (%) ↑Human Rank ↑
Matched-source, adaptive history retrieval74.24 ± 2.140.8543 ± 0.0180
Matched-source, static history retrieval69.70 ± 2.140.8409 ± 0.0033
Four semantic admission checks disabled75.8 ± 2.140.8428 ± 0.0054
  1. Adaptive history retrieval remains beneficial under identical validation rules. Both conditions use the same source-eligibility rule, preload complete source code, prohibit additional code retrieval, and retain identical action constraints and validation checks. They differ in whether histories are supplied statically or selected adaptively. Adaptive retrieval improves medal rate by 4.55 percentage points and Human Rank by 0.0134. Since validation rules are unchanged, this provides controlled evidence for adaptive history selection. Trace audits further confirm that neither condition accepted an existing-source action outside its eligible set.
  2. Strong performance remains attainable without four enforced semantic checks. We disable enforcement of complete source-code inspection, donor grounding, fusion-cost checks, and new-direction novelty requirements. Violations are recorded without blocking execution on these grounds; budget, structural, duplicate-intent, and execution/evaluation-isolation constraints remain enforced. Violations occur in 54/252 planning records linked to execution, confirming that the intervention is active. This condition still achieves 75.8% medal rate and 0.8428 Human Rank, with missing submissions counted as zero. These experiments provide complementary evidence: adaptive history selection adds value under identical validation rules, and the four tested semantic checks need not be mandatory for EDL to achieve strong performance.

Response to W2 and Q2:

Due to limited computational resources, our original GLM 5.2 evaluation on NatureBench Lite included only one complete search run per task. We apologize for not clearly stating this limitation and discussing the reversal. To address this concern, we have repeated the evaluation for all five frameworks, with three complete search runs per task, keeping the same evaluation settings, and per-run search budget. We report avg@3: each metric is aggregated within each repetition and then averaged across the three repetitions. The values below are means ± standard deviations across repetitions.

FrameworkRankScore ↑M-SOTA (%) ↑S-SOTA (%) ↑
EDL (ours)0.6500 ± 0.085086.67 ± 4.7150.00 ± 0.00
ML-Master 2.00.6156 ± 0.035880.00 ± 0.0033.33 ± 4.71
EvoX0.5306 ± 0.038273.33 ± 4.7130.00 ± 8.16
MLEvolve0.3789 ± 0.079156.67 ± 9.4323.33 ± 12.47
OpenMLE-Evo0.3750 ± 0.071553.33 ± 9.4326.67 ± 9.43

Under this repeated evaluation, EDL achieves the highest mean on all three metrics.

AuthorsOpenReview ↗

2nd Response to reviewer 7HEg

Response to W3 and Q3:

Thank you for pointing out this limitation. We agree that trajectories based on the best validation scores alone do not establish how solution quality evolves on held-out test data.

To address this concern, we conducted a retrospective test-set evaluation at 2, 4, 6, 8, and 12 hours for the five frameworks in Figure 3(a). At each checkpoint, we selected the incumbent implementation using only the validation information available by that time. We then re-executed the selected code in the sandbox to generate test-set results. Each selected checkpoint was independently re-executed three times. Test scores were used only for retrospective evaluation and were not fed back into search or candidate selection. The resulting test-set medal rates are reported below as mean ± standard deviation (%).

Framework2h4h6h8h12h
EDL50.00 ± 3.7165.15 ± 2.1466.67 ± 2.1469.70 ± 4.2972.73 ± 3.71
OpenMLE-Evo39.39 ± 2.1451.52 ± 4.2959.09 ± 3.7162.12 ± 2.1466.67 ± 5.67
EvoX43.94 ± 4.2953.03 ± 4.2954.55 ± 3.7156.06 ± 4.2960.61 ± 5.67
ML-Master 2.028.79 ± 2.1434.85 ± 2.1437.88 ± 4.2942.42 ± 4.2951.52 ± 5.67
MLEvolve48.48 ± 2.1453.03 ± 2.1457.58 ± 4.2959.09 ± 3.7165.15 ± 3.71

EDL has the highest mean test-set medal rate at every evaluated checkpoint. At 6 hours, it reaches 66.67%, matching the strongest baseline’s 12-hour mean (OpenMLE-Evo, 66.67%). These results support our central search-efficiency finding under held-out test evaluation.

Response to W4 and Q4:

We thank the reviewer for raising this question. We conducted an additional evaluation using GPT-6.1-sol as the backbone for EDL and all six baselines. We retained the original MLE-Lite configuration and changed only the backbone, keeping the task set, resource budgets, and evaluation protocol unchanged.

Harness / variantValid submissions ↑Medal rate (%) ↑Human Rank (%) ↑
AIRA-Evo (original)66/66 (100%)66.67 ± 2.1483.547 ± 0.250
AIRA-Evo (meta signals)66/66 (100%)69.70 ± 2.1484.685 ± 0.540
OpenMLE-Evo (meta signals)66/66 (100%)68.18 ± 3.7185.285 ± 0.383
ML-Master 2.065/66 (98.48%)71.21 ± 2.1486.207 ± 0.573
MLEvolve66/66 (100%)72.73 ± 3.7186.502 ± 0.355
EvoX65/66 (98.48%)66.67 ± 2.1483.554 ± 0.256
EDL (ours)66/66 (100%)75.76 ± 2.1487.532 ± 0.661

EDL retains gains with the closed-source backbone. It achieves the highest mean Human Rank and medal rate, exceeding the strongest baseline, MLEvolve, by 1.030 percentage points and 3.03 percentage points, respectively. The GPT-6.1-sol runs also show high submission reliability and low aggregate variability.

Russian text normalization task provides a concrete example of sustained, targeted improvement. GPT-6.1-sol decomposed the pipeline into a lookup table for frequent tokens, a contextual ranker for ambiguous tokens, and a neural model for unseen expressions. Successive experiments expanded contextual features, increased the ranker’s training data, improved decoding, and refined the ranking objective. In epoch 0, search validation accuracy increased from 99.1250% to 99.3189%. The final modification introduced pairwise candidate ranking, yielding 70 corrections and 46 regressions, a net improvement of 24 tokens over the previous ranker. The advantage also appeared in final submission results:

Russian text normalizationGPT-6.1-solDeepSeek-V4.1-FlashKimi K2.6GLM 5.2
Mean Grade across 3 epochs ↓0.02040.1820.19020.211

The central observation is the ability to identify weaknesses within a pipeline and translate them into concrete improvements to features, training objectives, and decoding strategies across successive experiments. This trajectory is consistent with EDL’s evidence-control scaffolding remaining useful with a capable backbone: accumulated experimental evidence supports continued refinement, with gains observed in both search validation and final submission quality.

We sincerely thank the reviewer again for their time in providing constructive feedback. In light of the further clarifications and additional experimental results we have provided, we respectfully ask the reviewer to raise the score of our paper. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.

Best wishes,

The Authors

AuthorsOpenReview ↗

Follow-up on our previous responses

Dear Reviewer 7HEg,

As the discussion period is coming to an end, we would appreciate your feedback on whether our responses have adequately addressed your concerns. We have provided detailed replies to each point and would be happy to clarify any remaining questions before the deadline. Many thanks for your time and consideration.

Best wishes,

The Authors

Thanks for the detailed rebuttal. My primary concerns have been satisfactorily addressed, and I have raised my score accordingly.

Reviewer_LwvLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an AI-train-AI agent workflow, and its central component is an Evidence-Driven Discovery Loop (EDL) that sits inside the search and implementation steps. EDL uses evidence from earlier experimental results to plan the next round, and each full iteration produces a portfolio of up to five experiment execution specifications. Earlier systems pass the planner a fixed context. EDL instead gives the planning agent a tool-calling loop, so the planner can inspect earlier experiments adaptively. Each portfolio action names an operation, its sources with explicit roles, a modification plan and a repeat count. Coding workers then execute the actions, and the evidence is updated for the next round. The authors evaluate on a 22-task MLE-Lite subset and a 10-task NatureBench Lite. Six harnesses are re-run on three backbones (Kimi K2.6, GLM 5.2, DeepSeek-V4.1-Flash), giving 66 task runs per configuration under a 12-hour sandbox budget and roughly one GPU per task. On MLE-Lite the paper reports submission rate, medal rate and Human Rank; on NatureBench Lite it reports RankScore, M-SOTA and S-SOTA. The claim the paper argues for is that adaptive evidence retrieval, rather than a fixed evidence bundle, is what produces the reported medal rate of 75.8 rather than 68.18.

Critical strengths
  1. The contribution is a mechanism, and its novelty is technical. An agent loop for evidence retrieval and grouping runs inside the planning agent, and three features separate it from a planner that receives a fixed context. The controller opens a planning session of up to 36 turns and issues its own queries instead of receiving a top-K prefetch. It fetches the full source code of a candidate, and reading that code is a precondition for reserving a plan, where a fixed bundle would supply a distilled note. It queries an index organised by algorithmic direction, parsed deterministically from code recipes, with per-operation statistics summarised separately. The Fixed-evidence ablation measures the cost of removing this mechanism at 7.6 medal-rate points, at 75.8 with it and 68.18 without.

  2. The evaluation protocol is stronger than is usual in this literature. Re-running every harness on three backbones removes the backbone as an explanation for the gap. It gives 66 task runs per model and harness configuration, and a standard deviation for every cell in Table 1 instead of a single number. Figure 3 plots medal rate against wall-clock time for EDL and four other harnesses. Figure 4 reports three success rates, for executable children, for parent improvements and for incumbent improvements, with failed attempts kept in the denominator.

  3. Each ablation removes one component and leaves the rest of the system intact, and Table 3 reports the effect on medal rate. The full system reaches 75.8. It reaches 69.70 without direction grouping, 69.70 with a numerical router in place of joint operation and source selection, and 68.18 with a fixed evidence bundle in place of adaptive retrieval. The three conditions map onto the three contributions the paper names, which are evidence organisation, action control and evidence acquisition. Adaptive retrieval accounts for the largest single drop, of 7.6 points. The two conditions that both report 69.70 should be confirmed as distinct values.

  4. The submission includes the implementation, so the decision procedure can be inspected and re-used. Section B.2 gives the system instruction and the initial message, Section B.3 gives the tool contracts, the character and step limits and the admissibility constraints, and Figures 9 to 11 give the response protocol. The supplementary package adds the controller and search code, the prompt builders, the per-task prompt tables as seven Parquet files of 22 rows each, vendored copies of the two baseline harnesses (AIRA-Evo and OpenEvolve), the experiment configurations and the test suite, with a REPRODUCIBILITY.md that records 492 passing tests. A reader can therefore follow what the controller sees, what it may ask for, what it must do before a plan is accepted, and which prompt produced each reported run.

Critical weaknesses
  1. The framing the paper argues for is not new, and the paper does not separate the framing from the mechanism. The problem it addresses, that search should be driven by evidence rather than by scalar ranking, is already addressed by Gome (Findings of ACL 2026; arXiv:2603.01692) and MARS (ICML 2026; arXiv:2602.02660), with earlier variants in the verbal reflections of Reflexion, the step summaries of AIDE and the experience retrieval of MLEvolve. What is new here is narrower, and is described in the first strength above. Because the abstract and the introduction argue for the framing without naming the mechanism, a reader cannot tell which of the two is being claimed.

  2. MARS (ICML 2026; arXiv:2602.02660) is the closest published system to EDL, and the experiments do not include it. MARS plans a solution, splits it into independently testable modules and refines each module by diff, and it keeps a reflective memory over differences between the current and the previous solutions. That is the same design question EDL poses. The comparison is therefore needed to support the claim that evidence organisation drives the reported gain, not merely to add a row to Table 1. The paper should either run MARS under the same runtime or state why the published MLE-bench numbers are not comparable.

  3. The paper reports how much evidence was consumed, but not what it consisted of. Section C.6 gives the volume, at 1.96 code exemplars and 3.00 histories per round, and reports that 20.2% of rounds reach outside the fixed bundle. Table 9 gives valid and improvement rates by operation. What is absent is provenance and effect. The paper does not report how often an inspected record came from a different direction than the one being edited, which is the statistic MARS reports as 63%, and it does not report how often a query changed the committed plan, although the design allows queries to change actions, sources and plans before reservation. Without those two, a reader can see that evidence was read but not that it changed any decision.

  4. In Table 2, EDL's RankScore on GLM 5.2 is 0.4652 against 0.6307 for ML-Master 2.0, while EDL leads on the other two backbones. The paper does not discuss this reversal, and it does not say whether the gap exceeds the variance the same table reports.

  5. The ablations that support the mechanism are confounded. The Fixed-evidence condition does not only disable adaptive queries. Because a source must have been read completely before a plan can be reserved, a fixed bundle also changes which sources are reachable, so the 7.6-point gap mixes the adaptivity of retrieval with source eligibility. The Fixed-route condition has the same problem in the other direction, because it also restricts the action mixture: crossover is limited to component transfer, and a new direction is allowed only when no valid source exists. Neither ablation therefore isolates the mechanism it is named after. Two further design choices are not ablated at all, namely committing a batch of five experiments per planning round rather than expanding one at a time, and the repeat count of one or two implementations per intent.

Questions for authors
  1. On tokens, adaptive retrieval costs 33.5% more than a fixed evidence bundle, at 2.477M against 1.855M per task-run (Section C.6), and on time, evidence inspection, planning and summarisation take 55.4 minutes per task-run, or 31.9% of recorded model-call time (Section C.3). Figure 3 still shows EDL reaching medal-level quality sooner than the baselines. Which mechanism explains that: fewer failed attempts before the first medal-level submission, or better targeting of each attempt?

  2. Can the related work state explicitly how EDL differs from Gome (Findings of ACL 2026; arXiv:2603.01692) and MARS (ICML 2026; arXiv:2602.02660)? Could a MARS run be completed in the response period, and if not, can the paper state why the published MLE-bench numbers are not comparable?

  3. Can the two ablations be decomposed, so that the adaptivity of retrieval is separated from source eligibility, and the change of routing is separated from the restriction of the action mixture? Ablations of the batch size and of the repeat count would help as well.

  4. Could the paper report two counts that Section C.6 does not already supply: how often an inspected record came from a different direction than the one being edited, and how often a query changed the action, sources or plan before the portfolio was reserved?

  5. Table 2 shows EDL at 0.4652 and ML-Master 2.0 at 0.6307 on GLM 5.2, while EDL leads on the other two backbones. Is the GLM gap larger than the reported variance, and if so, does the paper read it as backbone dependence or as one noisy cell?

6 comments
AuthorsOpenReview ↗

1st Response to reviewer LwvL

Dear Reviewer LwvL, Thank you for your time and effort in reviewing our paper. We address the questions and comments raised by the reviewer below.

Response to W1, W2, and Q2:

We agree that using historical evidence to guide search has substantial prior work, and that the paper should distinguish its motivation from its specific mechanisms.

  1. Our starting point is to make fuller use of the meta signals already produced by costly trials. Appendix A.2 shows that these signals can improve even numerical source-selection policies. However, their usefulness depends on the intended operation, modification, and compatibility between sources, while compact statistics can omit context essential to designing the next experiment. EDL/OEC addresses this gap through a decision policy that connects meta signals with on-demand inspection of their supporting evidence and uses both to guide experimental choices. Our contribution is this specific policy for more effective use of trial-derived information, rather than the general idea of evidence-guided search.
  2. The distinction from Gome and MARS concerns how evidence enters these decisions. Gome emphasizes diagnostic reasoning that generates update hypotheses and shares successful experience across parallel trajectories. MARS combines modular construction, diff-based refinement, and comparative reflective memory with budget-aware MCTS, using UCT for node selection. EDL/OEC focuses on actively acquiring evidence to jointly construct operations, source roles, and modification plans, including compatibility decisions in component transfer and multi-source combination.
  3. We have also extended the experimental evidence beyond MLE. While Gome and MARS report evaluations on MLE tasks, our revised paper adds nanoGPT optimizer-development and AI4AI-Bench experiments alongside NatureBench Lite. EDL/OEC performs well in both new settings. In particular, the nanoGPT results suggest that EDL’s benefits become more pronounced as evidence and meta signals accumulate during search. Detailed results and analyses are provided in the newly added Section 5.4. Competitive results in these settings support EDL as a reusable decision strategy for AI-train-AI, with applicability across both research tasks. Regarding W2’s concern about attribution, we have conducted additional experiments beyond the ablations in the paper, with detailed results and analyses provided in our responses to the subsequent comments.
  4. A controlled MARS comparison is currently limited by implementation availability. We did not find a runnable agent framework. These artifacts alone are insufficient for us to complete and validate a faithful, controlled reproduction within the response period. The published scores also cannot serve as a comparison under our runtime. Standard MARS uses one A100 with 40 GB GPU memory, 12 vCPUs, 220 GB RAM, and a 24-hour wall-clock budget. Our MLE-Lite experiments allocate approximately 12 GiB of RTX 4090 GPU memory and six vCPUs per task, with a 12-hour cumulative sandbox budget and an 18-hour combined model-plus-sandbox budget. The backbone models also differ. These differences affect implementation feasibility, execution costs, and trial counts, confounding direct score comparisons. We have revised the abstract and introduction and supplemented the related-work discussion of Gome and MARS. We will also explicitly document the MARS reproduction limitations and configuration differences. Once a runnable implementation becomes available, we will conduct a controlled comparison with matched tasks, backbone models, resource allocations, and budget accounting.

Response to W3 and Q4:

We audited all 292 DeepSeek planning rounds underlying Section C.6 and recovered their intermediate controller conversations. Provenance and adoption. Using a conservative definition—evidence originating outside every primary direction edited by the reserved portfolio, cross-direction evidence appears in 213/289 rounds with an editing target (73.7%). After within-round deduplication, this comprises 224/870 inspected histories (25.7%) and 197/573 complete code exemplars (34.4%). Of those cross-direction code exemplars, 119/197 (60.4%) are selected as sources in the reserved portfolio, connecting inspection to source adoption. Decision changes. We identify 32 substantive revisions following additional inspection, excluding wording, schema-only corrections, and forced continuation. This is 32/63 rounds with inspection between explicit drafts (50.1%; 32/292 overall). A concrete Kimi example connects revision to execution: in Leaf Classification, completing code inspection is followed by replacing MixUp/progressive-unfreezing intents with SWA and auxiliary-head experiments. The executed SWA child improves validation log loss from 0.008500 to 0.002121. Together, these traces document source adoption and substantive plan revision, with the Leaf example linking a revised plan to an executed improvement.

AuthorsOpenReview ↗

2rd Response to reviewer LwvL

Response to W5 and Q3:

Thank you for suggesting targeted ablations. Since Fixed-evidence and Fixed-route combine multiple interventions, we added five conditions examining adaptive retrieval with shared source eligibility, LLM routing with restricted action support, and batch size and repeat count. All conditions use DeepSeek-V4.1-Flash on the same 22 MLE-Lite tasks, with three runs per task and identical configured time budgets, yielding 66/66 valid final submissions each. Results are means ± standard deviations across the three run-level task averages.

Supplementary conditionMedal rate (%) ↑Human Rank ↑
Matched-source, adaptive history retrieval74.24 ± 2.140.8543 ± 0.0180
Matched-source, static history retrieval69.70 ± 2.140.8409 ± 0.0033
LLM routing with restricted action support72.73 ± 3.710.8565 ± 0.0118
Batch size 5, repeat count 174.24 ± 4.290.8482 ± 0.0092
Batch size 1, repeat count 174.24 ± 2.140.8631 ± 0.0124
  1. Adaptive history retrieval remains beneficial with source eligibility controlled. Both conditions use the same deterministic, quality-ranked source-eligibility rule, preload complete source code, and prohibit additional code retrieval. Existing-source actions must select from this preloaded set. The conditions differ in whether histories are supplied statically or selected adaptively, with identical history-query limits and other action constraints. Consequently, history queries cannot make additional sources eligible within a planning round. This shared eligibility rule applies to each independently evolving search state. Under this control, adaptive history retrieval increases medal rate from 69.70% to 74.24%, a gain of 4.55 percentage points, and Human Rank from 0.8409 to 0.8543, a gain of 0.0134. These results support the value of adaptive history selection under a shared source-eligibility rule. Trace audits also confirm that the control was respected: neither condition accepted an existing-source action outside its eligible set. Meanwhile, the adaptive controller inspected histories from directions outside that set in 19 of 313 recorded planning sessions linked to execution. Thus, the controller can draw decision-relevant evidence from a broader experimental history while respecting the prescribed source-eligibility constraints.

  2. OEC’s evidence-use strategy remains effective under restricted action support. We retain LLM operation/source selection while imposing the restrictions identified by the reviewer: Crossover permits component transfer only, and new-direction actions are allowed only when no valid source exists. The execution traces confirm that these restrictions were enforced. Post-initialization executions comprise 775 Improve attempts and 238 component transfers, with zero pipeline-fusion or new-direction executions. This condition still achieves 72.73% medal rate and 0.8565 Human Rank. Thus, OEC attains these results without pipeline fusion or post-initialization new-direction actions. A broader action set is therefore not required to achieve this level of performance, and evidence-guided planning remains competitive under restricted action support. These findings support the value of OEC’s decision strategy: combining meta signals with their underlying trial histories and code to guide the selection of operations, sources, and modification plans.

  3. Batch size: alignment with the baselines and the tradeoff between performance and planning cost. We initially chose batch size 5 to align with the evaluated AIRA-Evo and OpenMLE-Evo configurations, which generate five candidates per generation. Here, batch size denotes the maximum number of experiments committed per planning round. To examine this choice, we compare batch size 1 and batch size 5, fixing one implementation per intent and retaining five initial Draft attempts in both conditions. Both achieve 74.24% medal rate. Batch size 1 has a higher Human Rank point estimate, 0.8631 versus 0.8482, but also uses 40.1% more tokens per task-run: 3.183M versus 2.272M. Neither condition reaches the generation cap. These results show that OEC also performs well with batch size 1. Batch size 5 maintains alignment with the baseline configurations and spreads planning cost across multiple experiments, achieving the same observed medal rate with fewer tokens.

  4. Strong performance is retained with one implementation per intent.
    Both batch conditions set repeat_count=1, removing the controller’s option to implement the same intent twice. Both retain a 74.24% medal rate, showing that this performance does not require a second implementation. In the archived Full OEC trajectories we audited, second implementations account for only 10 of 1,020 controller-attributed executions (0.98%), indicating that this option is rarely used. These results show that OEC retains strong performance with one implementation per intent.

AuthorsOpenReview ↗

3rd Response to reviewer LwvL

Response to W4 and Q5:

Due to limited computational resources, our original GLM 5.2 evaluation on NatureBench Lite included only one complete search run per task. We apologize for not clearly stating this limitation and discussing the reversal. To address this concern, we have repeated the evaluation for all five frameworks, with three complete search runs per task, keeping the same evaluation settings, and per-run search budget. We report avg@3: each metric is aggregated within each repetition and then averaged across the three repetitions. The values below are means ± standard deviations across repetitions.

FrameworkRankScore ↑M-SOTA (%) ↑S-SOTA (%) ↑
EDL (ours)0.6500 ± 0.085086.67 ± 4.7150.00 ± 0.00
ML-Master 2.00.6156 ± 0.035880.00 ± 0.0033.33 ± 4.71
EvoX0.5306 ± 0.038273.33 ± 4.7130.00 ± 8.16
MLEvolve0.3789 ± 0.079156.67 ± 9.4323.33 ± 12.47
OpenMLE-Evo0.3750 ± 0.071553.33 ± 9.4326.67 ± 9.43

Under this repeated evaluation, EDL achieves the highest mean on all three metrics.

Response to Q1:

Our trajectory analysis suggests that the additional planning overhead is compensated by more productive experiments under the same wall-clock budget. Within Figure 3’s 12-hour window, including planning and execution, Kimi EDL produces 135 incumbent improvements in 539 direct Improve/Crossover attempts, compared with 46 improvements in 380 attempts for OpenMLE-Evo. Thus, EDL achieves both a higher improvement rate per attempt (25.0% versus 12.1%) and more total incumbent improvements within the same time window. The results specifically support better targeting, alongside improved execution reliability. Execution validity increases from 68.2% to 86.1%. Crucially, even conditional on successful execution, EDL advances the incumbent more frequently: 135/464 (29.1%) versus 46/259 (17.8%). The advantage therefore extends beyond avoiding execution failures: successful experiments also more often improve the best available solution.

We sincerely thank the reviewer again for their time in providing constructive feedback. In light of the further clarifications and additional experimental results we have provided, we respectfully ask the reviewer to raise the score of our paper. We are more than happy to answer any further questions from the reviewer for the rest of the discussion period.

Best wishes,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.