- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that an AI research agent working alone gets stuck on its first idea because it judges every new result against the story it has already built. To fix this, it splits the work across three agents: an executor that runs experiments, a supervisor that directs it, and an advisor (YoDA) that answers general ML questions without seeing the task details. The advisor answers from a library of 769 summarized papers and marks which parts of each answer come from those papers. On MLE-bench Low (22 Kaggle competitions, Claude Opus 5, one run), the system wins 17 medals in one hour, against 16 for each of two baselines.
Critical strengths
A simple idea, plus a useful side finding. Putting a reviewer agent outside the agent's own context is easy to adopt, and having every answer mark what comes from the papers makes the advice easy to check. The most useful part of the paper, though, is the transcript analysis. Supervisor guidance failed mostly because agents only see messages between turns, so directions often arrived after the work was done.
Critical weaknesses
- The paper doesn't show that the problem exists, and the solution is too heavy for what it gains.
- "Validation collapse" is never measured in the paper's own runs. The paper's own transcripts point to a different problem: late messages (Sec. 5.2).
- The paper library may not fit the benchmark. It is mostly about making models smaller and faster, such as quantization and pruning (Sec. 4), while Kaggle tasks need everyday skills like feature engineering and ensembling that frontier models already have. So even when the system does well, we can't tell whether the papers helped at all. The paper also doesn't say how often the advisor was consulted during MLE-bench, or what it changed.
- There are no cost numbers. One consultation costs 48k–70k tokens (App. D), but no pipeline has a per-competition cost. Matching the baselines at lower cost would count in its favor; without numbers, it looks like extra cost for no clear gain.
- The MLE-bench results don't support the claims.
- Everything is one seed, and the authors admit the 17 vs. 16 gap is within noise.
- The pipelines ran under different setups (App. A.1). The full system used an earlier setup with a 20-minute training budget. The ablations and AIDE shared a GPU and started up to five minutes early. AIBuildAI had the GPU to itself and stopped after 34 minutes on average.
- The 19 medals come from the stage with test-set feedback, yet they appear in the abstract and contributions.
Questions for authors
- How often was the advisor consulted in each competition, and did it change any outcome?
- What did each pipeline cost per competition?
- Can you rerun all pipelines under the same setup, with at least three seeds and no test-set feedback?
- How does the system compare with a fresh-context critic, the same advisor with plain retrieval, and a two-tier system with message delivery fixed?
- Can you show the problem directly? For example, count how often a solo agent keeps trying an idea that isn't working, and check whether that happens less with the advisor.