- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Agentic Reasoning for Tree Search (ARTs), where a reasoning language model inspects prior execution logs, diagnoses whether failures arise from faulty implementations or poor hypotheses, and selects which hypothesis to build on next. Across 22 MLE tasks, ARTS outperforms leading methods (i.e., AIRA, MLEvolve).
Critical strengths
- I think the idea is quite novel. It is very interesting that sometimes selecting an idea with a lower validation score may have more potential than the node with a higher validation score. This is a new observation compared to the conventional approaches, and the authors verified through comprehensive tests.
Critical weaknesses
-
While the idea is novel, the critical weakness of this paper is lack of benchmarks. Why did the author only pick 22 tasks from MLGym and MLEBench? There are more datasets in both benchmarks. The authors should provide a rationale on the selection process, otherwise, it can be seen as cherry-picked results.
-
Also, the baselines are not sufficient. I think the core search process is almost same with MARS [1], while the node selection process is the only core difference. On the other hand, the authors did not compare ARTS with MARS, even though MARS show very high score on MLEBench.
-
Performance improvement is actually quite marginal. Especially in MLEBench, while ARTS shows the highest performance overall, the improvement is not that much, or even just same as the baseline.
[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026.
Questions for authors
-
Can authors provide a results on full MLGym benchmark and full MLEBench, including hard tasks?
-
Can authors compare ARTS to MARS and other competitive baselines? I think AIRA and MLEvolve are quite weak baselines. Also they use Gemini 3 Flash and o3, while these days, there are much more stronger backbone LLMs to use.
-
Is ARTS compatible to any kind of backbone LLMs? If so, can authors provide an ablation study on this?
4 comments
Thanks for the additional experiments. These will strengthen the paper, and also addressed my major concerns. On the other hand, I would like to kindly ask some brief clarifications:
-
While the new results on the 14 hard MLE-bench tasks are highly appreciated, I also want to know why only a subset of 22 task from MLGym and MLE-bench was initially selected in the original submission. Could you provide a brief selection criteria?
-
In your new experiments, could you please confirm that whether all other baselines (AIRA, MLEvolve, and Linear) were also fully upgraded to use Gemini 4 Argon for these new runs?
I look forward to your response. Thanks!
Thank you so much for replying and raising the follow up questions!
- We chose the 22 tasks in the original submission as follows:
- MLGym: we included all tasks.
- MLE-bench: we sampled tasks uniformly at random within each difficulty split: 3 Easy, 3 Medium and 4 Hard.
After running these tasks, we found that the Easy and Medium splits are close to saturated for all methods. We kept them to show that ARTS does not degrade performance on saturated tasks. We limited the Hard split to 4 tasks because of API cost at the time. For the rebuttal, we ran all 14 Hard tasks. ARTS outperforms all baselines on this split, and we have added these results to the revised manuscript.
- Yes, All baselines (AIRA, MLEvolve and Linear) were rerun with Gemini 4 Argon under identical configurations, with the same budget, tools and evaluation protocol as ARTS.
We thank the reviewer again for the constructive discussion. If our responses resolve the remaining concerns, we would be grateful if the reviewer would consider raising their score.
Thanks for the clarification. Please include all the discussion we had in the revised paper. Since the authors well-alleviated my concerns, I am raising the score from 2 to 3.
We thank the reviewer for finding the idea novel and interesting, in particular the observation that a node with a lower validation score can hold more potential than one with a higher score.
All hard MLE-bench competitions, stronger backbone, and MARS (W1, W2, W3, Q1, Q2)
We ran all 14 hard MLE-bench competitions. Every method, including the baselines, now uses Gemini 4 Argon, a stronger and more recent backbone model, with a 24-hour budget. We added MARS, running its official code with only the backbone changed to Argon as the baseline.
Results. Test score of the best-validation solution. Bold: best. Italics: second best.
Result: ARTS achieves the best test score on 11 of 14 competitions. It beats MARS on 13 out of 14 tasks. with a median improvement is 9% over MARS.
W3: The improvement holds against the strongest baseline on every task. With the newer backbone of Gemini-4-Argon, ARTS achieves an improvement of 9% over MARS, 11% over AIRA, 13% over MLEvolve and 10% over Linear.
We thank the reviewer for suggesting this experiment since it shows that ARTS gives even stronger results as the backbone model improves, showing the effectiveness of ARTS.
Note: We would like to point that ARTS is fundamentally different from MARS which selects nodes by a UCT formula over scores, similar to AIRA. ARTS instead reasons over the code and execution logs to guide the search, instead of solely relying on the execution score.
Backbone compatibility (Q2, Q3)
Yes. ARTS works with any LLM as the scientist and the executor. The paper uses Gemini 3 Flash and o3, in three scientist/executor pairs. The new results above use Gemini 4 Argon for every method. ARTS* trains open models as the scientist: Qwen3-4B, and now Gemma 3 4B from a second model family, where it beats the frozen Gemma scientist on all 6 tasks we tested (updated Table 2). This proves that ARTS works with any backbone model.
We thank the reviewer again for these questions. We believe the new experiments resolve each of the concerns raised, and we kindly ask the reviewer to consider increasing their score. We are happy to clarify anything further.