acceptodds
Under review as a conference paper at ICLR 2027

Learning the ARTS of Search for Auto ML

Abstract

Automated machine learning (ML) research can be formulated as an iterative search over hypotheses and their code implementations. Contemporary methods navigate this space with heuristics such as MCTS that rank nodes by score alone, conflating the merit of a hypothesis with the quality of its execution. As a result, a promising hypothesis with a preliminary implementation may rank below a modest one that is well tuned. We propose Agentic Reasoning for Tree Search (ARTS), where a reasoning language model inspects prior execution logs, diagnoses whether failures arise from faulty implementations or poor hypotheses, and selects which hypothesis to build on next. As the search progresses, the model suffers reasoning degradation due to longer contexts. In response, ARTS* learns from the previously explored hypotheses using test-time RL. Across 22 tasks from MLGym and MLEBench, ARTS outperforms leading methods by a mean normalized score of 12.0%. With test-time training, a Qwen3-4B scientist matches or exceeds an o3 scientist on 6 of 12 MLGym tasks. Notably, on MetaMaze, ARTS* learns to rediscover the human-best recurrent-memory solution that heuristic methods prune away.

Primary area:
reinforcement learning
Keywords:
Auto Research, AI Agents, RL

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Rs9jOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Agentic Reasoning for Tree Search (ARTs), where a reasoning language model inspects prior execution logs, diagnoses whether failures arise from faulty implementations or poor hypotheses, and selects which hypothesis to build on next. Across 22 MLE tasks, ARTS outperforms leading methods (i.e., AIRA, MLEvolve).

Critical strengths
  1. I think the idea is quite novel. It is very interesting that sometimes selecting an idea with a lower validation score may have more potential than the node with a higher validation score. This is a new observation compared to the conventional approaches, and the authors verified through comprehensive tests.
Critical weaknesses
  1. While the idea is novel, the critical weakness of this paper is lack of benchmarks. Why did the author only pick 22 tasks from MLGym and MLEBench? There are more datasets in both benchmarks. The authors should provide a rationale on the selection process, otherwise, it can be seen as cherry-picked results.

  2. Also, the baselines are not sufficient. I think the core search process is almost same with MARS [1], while the node selection process is the only core difference. On the other hand, the authors did not compare ARTS with MARS, even though MARS show very high score on MLEBench.

  3. Performance improvement is actually quite marginal. Especially in MLEBench, while ARTS shows the highest performance overall, the improvement is not that much, or even just same as the baseline.

[1] Chen et al., MARS: Modular Agent with Reflective Search for Automated AI Research, 2026.

Questions for authors
  1. Can authors provide a results on full MLGym benchmark and full MLEBench, including hard tasks?

  2. Can authors compare ARTS to MARS and other competitive baselines? I think AIRA and MLEvolve are quite weak baselines. Also they use Gemini 3 Flash and o3, while these days, there are much more stronger backbone LLMs to use.

  3. Is ARTS compatible to any kind of backbone LLMs? If so, can authors provide an ablation study on this?

4 comments

We thank the reviewer for finding the idea novel and interesting, in particular the observation that a node with a lower validation score can hold more potential than one with a higher score.


All hard MLE-bench competitions, stronger backbone, and MARS (W1, W2, W3, Q1, Q2)

We ran all 14 hard MLE-bench competitions. Every method, including the baselines, now uses Gemini 4 Argon, a stronger and more recent backbone model, with a 24-hour budget. We added MARS, running its official code with only the backbone changed to Argon as the baseline.

Results. Test score of the best-validation solution. Bold: best. Italics: second best.

TaskMARSAIRAMLEvolveLinearARTS
Stanford COVID (↓)0.21950.21560.21600.21760.2102
BMS Molecular (↓)88.2475.3978.0171.8471.56
iWildCam (↑)0.21360.15530.12860.16900.2489
Vesuvius (↑)0.41520.30630.31260.44060.4481
SIIM COVID-19 (↑)0.24490.39430.34230.36810.4267

Result: ARTS achieves the best test score on 11 of 14 competitions. It beats MARS on 13 out of 14 tasks. with a median improvement is 9% over MARS.

W3: The improvement holds against the strongest baseline on every task. With the newer backbone of Gemini-4-Argon, ARTS achieves an improvement of 9% over MARS, 11% over AIRA, 13% over MLEvolve and 10% over Linear.

We thank the reviewer for suggesting this experiment since it shows that ARTS gives even stronger results as the backbone model improves, showing the effectiveness of ARTS.

Note: We would like to point that ARTS is fundamentally different from MARS which selects nodes by a UCT formula over scores, similar to AIRA. ARTS instead reasons over the code and execution logs to guide the search, instead of solely relying on the execution score.


Backbone compatibility (Q2, Q3)

Yes. ARTS works with any LLM as the scientist and the executor. The paper uses Gemini 3 Flash and o3, in three scientist/executor pairs. The new results above use Gemini 4 Argon for every method. ARTS* trains open models as the scientist: Qwen3-4B, and now Gemma 3 4B from a second model family, where it beats the frozen Gemma scientist on all 6 tasks we tested (updated Table 2). This proves that ARTS works with any backbone model.


We thank the reviewer again for these questions. We believe the new experiments resolve each of the concerns raised, and we kindly ask the reviewer to consider increasing their score. We are happy to clarify anything further.

Thanks for the additional experiments. These will strengthen the paper, and also addressed my major concerns. On the other hand, I would like to kindly ask some brief clarifications:

  1. While the new results on the 14 hard MLE-bench tasks are highly appreciated, I also want to know why only a subset of 22 task from MLGym and MLE-bench was initially selected in the original submission. Could you provide a brief selection criteria?

  2. In your new experiments, could you please confirm that whether all other baselines (AIRA, MLEvolve, and Linear) were also fully upgraded to use Gemini 4 Argon for these new runs?

I look forward to your response. Thanks!

Thank you so much for replying and raising the follow up questions!

  1. We chose the 22 tasks in the original submission as follows:
  • MLGym: we included all tasks.
  • MLE-bench: we sampled tasks uniformly at random within each difficulty split: 3 Easy, 3 Medium and 4 Hard.

After running these tasks, we found that the Easy and Medium splits are close to saturated for all methods. We kept them to show that ARTS does not degrade performance on saturated tasks. We limited the Hard split to 4 tasks because of API cost at the time. For the rebuttal, we ran all 14 Hard tasks. ARTS outperforms all baselines on this split, and we have added these results to the revised manuscript.

  1. Yes, All baselines (AIRA, MLEvolve and Linear) were rerun with Gemini 4 Argon under identical configurations, with the same budget, tools and evaluation protocol as ARTS.

We thank the reviewer again for the constructive discussion. If our responses resolve the remaining concerns, we would be grateful if the reviewer would consider raising their score.

Thanks for the clarification. Please include all the discussion we had in the revised paper. Since the authors well-alleviated my concerns, I am raising the score from 2 to 3.

Reviewer_DFV7OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARTS, a system that organizes machine-learning experiments as a search tree. A scientist model reviews previous code, logs, and results to decide which experiment to try next, while an executor model implements and runs it. The extension, ARTS*, trains the scientist during the search using feedback from completed experiments to improve its subsequent decisions.

Critical strengths

S1. Search decisions are grounded in code and execution logs. In the APTOS example, the scientist keeps the architecture but revises the training setup after inspecting a failed run, improving QWK from 0 to 0.92. This illustrates how the method can recover promising ideas after implementation failures.

S2. ARTS* learns experiment selection from actual outcomes. It trains both parent selection and hypothesis generation using feedback from the current task. With this training, Qwen3-4B’s mean normalized score increases from 0.72 to 1.01 across 12 MLGym tasks (Table 2).

Critical weaknesses

W1. The benefit of ARTS* under the same compute budget is unclear.

ARTS* uses 24 GPU-hours per run, while the other methods use 8 GPU-hours. Although all methods run for eight hours, their compute budgets differ. The results show that training the scientist improves performance, but do not establish whether this is a better use of resources than spending the additional compute on more experiments with a frozen scientist. A comparison under the same total compute budget is needed to support the claimed search efficiency.

W2. Final evaluation is not clearly separated from search.

Section 5 states that search is guided by validation scores and that the reported result is the best validation score reached during each run. ARTS* also uses these scores to reward and train the scientist. Using validation feedback during search is reasonable, but the paper does not clearly report a separate test evaluation of the final selected solutions. It therefore remains unclear whether the improvements hold on data that did not guide the search. Please clarify the validation/test split and report results on an independent test set.

Questions for authors

Q1. How does ARTS* compare with a frozen-scientist baseline given the same total compute budget, with the extra resources used for additional search?

Q2. Are the validation and test sets separate? Which split provides the search feedback and rewards for training the scientist?

Q3. Do the improvements hold when the final selected solutions are evaluated on an independent test set that did not guide the search or scientist training?

Q4. Are the gains from test-time training specific to Qwen3-4B? Could you evaluate ARTS* with a scientist model from another model family to check whether the improvements generalize?

3 comments

We thank the reviewer for recognizing that ARTS grounds search decisions in code and execution logs, recovering promising ideas after implementation failures, and that ARTS* learns experiment selection from actual outcomes on the current task.


1. Evaluation on a held-out test set (W2, Q2, Q3)

Setting. MLE-bench and MLGym provide only a train set and a hidden test set. In the original paper, search feedback came from the hidden test set for every method. We now separate the two. Each method builds a validation split by randomly sampling 10% of the train set and trains on the remaining 90%. All search feedback, and the ARTS* reward, come only from this validation split. We select the best-validation solution of each run and evaluate it once on the hidden test set, which never guides search or training. We reran ARTS and the baselines under this setting on all 14 hard MLE-bench competitions, with Gemini 4 Argon for every method and a 24-hour budget.

Results. Full results in Table 1 of the paper

TaskAIRAMLEvolveLinearARTS
Stanford COVID (↓)0.21560.21600.21760.2102
BMS Molecular (↓)75.3978.0171.8471.56
iWildCam (↑)0.15530.12860.16900.2489
Vesuvius (↑)0.30630.31260.44060.4481
RSNA Breast (↑)0.08060.15380.12100.1765

The improvements hold on the held-out test set. ARTS achieves the best test score on 11 of 14 competitions, with a median improvement of 11% over AIRA, 13% over MLEvolve and 10% over Linear. The full results are in Table 1 of the updated paper, and we updated Section 5 to describe this setting.


2. Matched-compute comparison (W1, Q1, Q4)

Setting. We compare ARTS using Gemini 4 Argon (24 hours) against ARTS* with Qwen3 4B scientist trained on 3 A100s for 8 hours. Both methods receive equal compute budgets. To test another model family, we also run ARTS* with Gemma 3 4B. All scores are on the held-out test set described above.

Results. Full results in Table 2 of the paper

TaskARTS, Argon (24h)Frozen Qwen3-4B (24h)ARTS*, Qwen3-4B (8h)Frozen Gemma 3 4B (24h)ARTS*, Gemma 3 4B (8h)
Stanford COVID (↓)0.21020.21910.21970.63820.2220
NFL Contact (↑)0.70240.63000.64060.55860.5637
BMS Molecular (↓)71.5673.6871.49109.6071.97
Predict Volcanic (↓)1.99M2.11M3.08M24.40M2.60M
Spaceship Titanic (↑)0.82760.83680.83790.82760.8460
Nomad 2018 (↓)0.05000.05060.05850.05610.0516

Q1: Training matches 3x more search. With a third of the search time, ARTS* with Qwen3-4B beats the 24-hour frozen scientist on 3 of 6 tasks, with a median difference of 0.1%. ARTS* also beats ARTS with Gemini 4 Argon, a much larger model, on BMS Molecular.

Q4: The gains generalize to other model family. ARTS* with Gemma 3 4B beats the 24-hour frozen Gemma scientist on all 6 tasks, with a median gain of 21%, and gives the best score of any method on Spaceship Titanic.

The full results are in Table 2 of the updated paper.


We thank the reviewer again for these questions. We believe the new experiments resolve each of the concerns raised, and we kindly ask the reviewer to consider increasing their score. We are happy to clarify anything further.

AuthorsOpenReview ↗

Kind reminder for engagement in discussion

Dear Reviewer,

We thank you again for your review. We have posted our response and revised the manuscript. We have added

  1. A 24 GPU-hour budget for every method,
  2. Held-out test results where search and rewards use only a validation split, and
  3. ARTS* with Gemma 3 4B as a second model family.

As the discussion period ends soon, we would be grateful if you could take a look and let us know if any concern remains.

AuthorsOpenReview ↗

Request for engagement

Dear reviewer,

As the discussion period nears its end, we would really appreciate you engaging in the discussion, or at least acknowledging that our rebuttal has been read.

Thanks

Reviewer_MxkVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARTS, an automated ML research framework in which a reasoning model inspects experimental histories, code, and execution logs to select promising hypotheses for further exploration. ARTS* additionally adapts the scientist through test-time reinforcement learning. Experiments across 22 tasks suggest improvements over existing search approaches.

Critical strengths
  1. The paper addresses an important failure mode in automated ML research: a promising hypothesis may receive a low score because of poor implementation. Using code and execution logs to guide subsequent search is a well-motivated approach that could improve how agents allocate expensive experimental resources.

  2. The evaluation spans 22 tasks across MLGym and MLEBench and includes comparisons with linear, tree-based, and evolutionary search. The component ablations and detailed trajectories provide useful evidence of the framework’s practical value across different task types.

Critical weaknesses
  1. As acknowledged in Section 8, the experiments match wall-clock time but not compute: ARTS* uses 24 GPU-hours per run, compared with 8 for inference-only methods. The results therefore demonstrate gains under the reported configurations, but do not establish whether task-specific training is preferable to additional search under comparable resource budgets. A cost-matched comparison would strengthen the practical efficiency claim. Separately, Appendix A.7 reports an objective-level comparison with TTT-Discover on one task; it does not establish an end-to-end advantage over that method.

  2. Evidence for the scalability of ARTS remains incomplete.* While ARTS is evaluated on 10 MLE-bench competitions selected across difficulty tiers, the main ARTS* results are limited to MLGym. Its effectiveness on complex ML engineering competitions therefore remains untested in the reported evaluation. In addition, Appendix B specifies an 8,192-token rollout sequence length and a maximum of 100 GRPO steps, but does not systematically report actual context lengths or completed updates per task. Although the method includes memory and selective inspection, the handling of growing histories within the sequence budget is insufficiently described. These gaps make it difficult to assess whether the observed gains specifically reflect improved reasoning over long histories rather than task-specific adaptation.

Questions for authors

Resource accounting: Does the eight-hour budget include all LoRA training and model-update overhead? Given the reported 24 versus 8 GPU-hours, are any compute- or cost-matched results available to determine whether ARTS* is preferable to allocating additional resources to inference-only search?

Comparison with related methods: Does Appendix A.7 evaluate the complete TTT-Discover method or only substitute its training objective? Please also explain the absence of a direct comparison with Towards Execution-Grounded Automated AI Research and clarify which claimed advantages are supported by the current comparisons.

Context management and training scale: How are the search tree, memory, and retrieved logs represented within the stated 8,192-token rollout sequence length? What is retained or truncated when this limit is exceeded? Please report actual context lengths and completed GRPO updates per task, and clarify what evidence distinguishes improved long-history reasoning from task-specific adaptation.

Evaluation coverage: Please provide the sampling procedure and difficulty assignments for the 10 selected MLE-bench competitions. Are any ARTS* results available on these competitions? If not, please clarify the intended scope of the training-related claims beyond MLGym.

3 comments

We thank the reviewer for appreciating our contribution and that we address an important failure mode, conflation of hypothesis quality with execution quality, in automated ML research. The reviewer also identified that our approach is well-motivated and the experiments with ablations provide useful evidence.


W1 W2 Q1 and Q4: Compute-matched comparison and ARTS * on MLE-bench

We ran the comparison the reviewer asked for, within the same 24 GPU-hour budget. ARTS* trains a small 4B scientist on 3 A100s for 8 hours, 24 GPU-hours in total. ARTS with Gemini 4 Argon and a frozen Qwen3-4B scientist search for 24 hours.

TaskARTS, Argon (24h)ARTS, frozen Qwen3-4B (24h)ARTS*, Qwen3-4B (8h)
BMS Molecular (↓)71.5673.6871.49
Spaceship Titanic (↑)0.82760.83680.8379
Stanford COVID (↓)0.21020.21910.2197

Table 1 and 2 of the updated paper shows the full results.

We find that A 4B scientist trained for 8 hours beats Gemini 4 Argon searching for 24 hours on BMS Molecular and Spaceship Titanic, and comes within 5% on Stanford COVID. ARTS* also matches a frozen Qwen3-4B scientist that searches three times longer on 3 of 6 tasks, with a median difference of 0.1%. Hence training the scientist is as good a use of compute as more search.


W1 and Q (related methods): Comparison with TTT-Discover

We compare TTT-Discover using Qwen3-4B model to ARTS* under the same 8-hour budget and the same GPUs.

TaskTTT-Discover (8h)ARTS* (8h)
BMS Molecular (↓)72.2871.49
Spaceship Titanic (↑)0.83560.8379
Stanford COVID (↓)0.23250.2197

Finding: Table 2 of the updated paper shows the full results. ARTS * beats TTT-Discover on 66% tasks.


Q (related methods): Comparison with Execution-Grounded Automated AI Research

Execution-Grounded search is an evolutionary search method and hence is fundamentally different from ARTS. It is more analogous to MLEvolve since it only sees a scalar score whereas ARTS involves reasoning over the execution logs to guide search.

We ran Execution-Grounded search, on all 14 hard MLE-bench competitions using Gemini 4 Argon.

TaskExecution-Grounded (24h)ARTS (24h)
BMS Molecular (↓)89.1871.56
Stanford COVID (↓)0.21880.2102
iWildCam (↑)0.14700.2489

Finding: Table 1 of the updated paper shows the full results. ARTS beats Execution-Grounded on 13 of 14 competitions and ties on the last, with a median improvement of 12% on the raw test metric.


Clarification questions

  1. Does the eight-hour budget include LoRA training and model updates?

Yes, the eight-hour budget includes all LoRA training and model updates, which run alongside the search. The model is being updates as it searches.

  1. Does Appendix A.7 evaluate the complete TTT-Discover method?

Yes and the full end-to-end comparison is above: ARTS * beats TTT-Discover on 66% tasks.

  1. How were the MLE-bench tasks selected?

In the paper, we selected 4 hard, 3 medium and 3 easy competitions at random within each difficulty tier. We have now run all 14 hard competitions. We ran the new experiments only on the hard tier because the easy and medium tiers are saturated: ARTS reaches normalized scores of 0.98 to 1.02 on these six competitions.

  1. How do the search tree, memory and logs fit within 8,192 tokens, and what is truncated?

Prompts are capped at 6,144 tokens. Nodes are compressed to one line summaries given by the scientist itself. 25 full nodes are kept in context at any given time including the root, and the top 24 nodes.

  1. What are the actual context lengths and completed GRPO updates per task?

Rollouts average 7.6k tokens, and about 20% of prompts are truncated. ARTS* completes 30 to 60 GRPO updates on MLGym and 4 to 12 on MLE-bench, since they take longer to execute. Per-task context lengths and update counts are in Appendix J of the updated paper.

  1. What distinguishes long-history reasoning from task-specific adaptation?

ARTS* is task-specific adaptation by design. It trains on the current task's tree, so the scientist learns from earlier experiments without keeping them all in context, since reasoning degrades over long contexts. This is explained in the Section 4.3 of the manuscript.


We thank the reviewer for their positive feedback and asking these questions. We ask them to consider revising their score if these concerns are resolved. If there are any concerns left, we are happy to provide more clarification.

AuthorsOpenReview ↗

Kind reminder for engagement in discussion

Dear Reviewer,

We thank you again for your review. We have posted our response and revised the manuscript. We have added

  1. A 24 GPU-hour budget for every method
  2. ARTS* on MLEBench and comparison with TTT-Discover
  3. A comparison with Execution-Grounded search, and
  4. Context-length and GRPO-update statistics.

As the discussion period ends soon, we would be grateful if you could take a look and let us know if any concern remains.

AuthorsOpenReview ↗

Request for engagement

Dear reviewer,

As the discussion period nears its end, we would really appreciate you engaging in the discussion, or at least acknowledging that our rebuttal has been read.

Thanks

Reviewer_CQRpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ARTS uses a reasoning-model scientist to inspect code and execution logs, choose a parent experiment, and propose hypotheses for a separate coding executor. Its central motivation is that a low observed score can reflect a poor implementation of a useful idea. The system combines an initial audit, explicit failure diagnosis, memory, and uniformly sampled verbalized hypothesis candidates. ARTS* additionally trains a Qwen3-4B scientist per task with LoRA and GRPO, using a rolling percentile reward. Experiments cover 22 MLGym/MLEBench tasks under eight-hour wall-clock budgets. The paper reports a 12.0% mean normalized-score improvement for the main ARTS configuration and substantial validation-score gains from test-time training.

Critical strengths
  1. The distinction between hypothesis merit and implementation quality identifies a concrete weakness in score-based research search. Inspection-driven parent selection is a useful systems contribution, and the APTOS, HMS, and Vesuvius examples explain how reading logs can lead to a specific repair rather than an uninformative change of direction (Section 6.2 and Appendix E).

  2. The evaluation includes 22 heterogeneous tasks, three-run main results, same-model configurations, and a UCB replacement inside the ARTS pipeline. Component ablations and a comparison with context compaction provide more support than an end-to-end comparison alone. The paper also discloses that ARTS with Gemini in both roles trails the strongest corresponding baseline and that ARTS* uses additional GPUs, which appropriately limits universal superiority claims.

Critical weaknesses
  1. The evaluation does not establish generalization of the selected ML solutions. Section 5 explicitly reports the best validation score obtained during search on the same validation split that guides the search and the ARTS* reward. Restoring the evaluator is a useful integrity guard, but does not address adaptive overfitting to repeatedly queried validation data. Although held-out test sets are mentioned, final independent test results are not reported. Human-best references also come from different leaderboard/test protocols, so exceeding those normalized references is not evidence of superior test performance. This is central for an AutoML contribution.

  2. The evidence that test-time training specifically remedies long-context degradation is not sufficiently isolated. ARTS* receives 24 GPU-hours versus 8 for the other methods, and the paper leaves compute-matched alternatives outside scope. The main mechanistic example is also inconsistent: Sections 6.2–6.3 say the o3 scientist tried and abandoned an LSTM, whereas Appendix H states that no LSTM/GRU node was actually executed because the sampler selected other candidates. Likewise, Section 6.1 says UCB lowers scores on all 22 tasks, but Table 5 shows UCB ahead on House Price and Kuzushiji and tied on Nomad and Jigsaw. These discrepancies weaken causal claims about the proposed components.

Questions for authors
  1. Essential validation: What are the scores of the validation-selected final artifacts on an independent test split that was inaccessible throughout search and reward computation? Please report whether the method ordering persists and separate these results from leaderboard-reference normalization.

  2. Mechanism and budget control: With equal total GPU/API expenditure, does ARTS* still outperform additional inference-only search or compaction? Can a controlled context-length or information-availability experiment distinguish relief of long-context degradation from ordinary task-specific learning?

  3. Result reconciliation: Did the o3 MetaMaze run execute a recurrent policy, or only propose one? Please reconcile the text with Appendix H and correct the all-22-task UCB claim. Please also specify the run/model provenance of the percentile-reward ablation in Table 8, whose final Fashion/Battle-of-Sexes values differ from the Qwen3-4B ARTS* results in Table 2.

3 comments

We thank the reviewer for recognizing the importance of our central claim about distinction between hypothesis merit and implementation quality. We are glad that the reviewer found our 22-task evaluation, same-model comparisons and ablations support our claims. We ran the experiments the reviewer asked for and updated the paper accordingly.


New experimental setting. As per the reviewer's suggestions, we select the best-validation solution of each run and evaluate it once on the hidden test split, which is unavailable to the agent throughout search or training. At Reviewer Rs9j's request, every method now uses Gemini 4 Argon, a newer backbone, and we run all 14 hard MLE-bench competitions. Inference-only methods run for 24 hours. All new results in the updated paper use this setting.

W1 and Q1: Test-set generalization.

MLE-bench and MLGym provide no hidden test split, so in the original paper every method received search feedback from the validation set it is evaluated on finally.

Under the new setting,the search is guided only by a validation split, and the agent never receives feedback from the hidden test set. We build the validation split by randomly sampling 10% of the train set and train on the remaining 90%.

We find that the ordering among methods still persists. ARTS achieves the best raw test score on 11 of 14 hard MLE-bench tasks, with median gains of 11% over AIRA, 13% over MLEvolve and 10% over Linear. The full results are in the updated Table 1, and Section 5 now describes this protocol.


W2 and Q2: Equal expenditure.

We would like to clarify that, the 24 versus 8 GPU-hour figure did not count the GPUs that host larger models like o3 and Gemini for the inference-only methods, and we have explained this in the paper.

We also reran ARTS for 24 hours with Gemini 4 Argon as the scientist and compared it with 8 hours of ARTS* using the trained Qwen3-4B. ARTS* has a third of the search time and a much smaller model. Even so, ARTS * beats ARTS with Argon on BMS (71.49 vs 71.56 Levenshtein distance, lower is better) and Titanic (0.838 vs 0.828 accuracy), stays within 5% on COVID, and trails by a median of 6.7% across the six tasks. For reference, AIRA and MLEvolve run with Gemini 4 Argon for 24 hours, and on the 14 hard tasks they trail ARTS with Argon by a median of 11% and 13% (updated Table 1). The full results are in the updated Table 2 of the manuscript.

W2 and Q2: Isolating the mechanism.

To ellucidate the effect of the weight update compared to extended search or compaction, we performed the following experiments:

  1. ARTS with frozen Qwen3-4B for 24 hours.
  2. ARTS with Qwen3-4B with compaction, which summarizes the history into context, for 24 hours.
  3. ARTS*, training Qwen3-4B scientist for 8 hours.

We find that, with a third of the search time, ARTS* matches the 24-hour frozen scientist (median difference 0.1%) and scores 33% higher than compaction (updated Table 2).

We then traced where the gains compared to compaction come from. We find that a) ARTS * produces 15.5% more valid experiments compared to compacted ARTS. b) ARTS * finds 2.5x more diverse solutions compared to compacted ARTS, and c) ARTS * proposes hypothesis that beat the previous hypotheses 41.5% of the times compared to 28.6% of the times for compacted ARTS.

The gain therefore comes from what ARTS* learning from the previous failures during the search, proposing more diverse hypotheses. (We have added this analysis and all the results in appendix J of the manuscript).


Clarification questions

1. W1: Human references. We use them only as a normalizer, since the score scales of each task were varying. The same normalizer apply to every method, so they do not change the ranking. We revised the claim that ARTS exceeds them in the updated manuscript.

2. W2 and Q3: o3 MetaMaze. We would like to clarify that in ARTS, the o3 scientist only proposed a recurrent policy, never executed it. On the other hand, in ARTS*, the recurrent policy was actually executed and reached 53.0. We have made this more clear in Sections 6.2 and 6.3 and Figures 10 and 11.

3. W2 and Q3: UCB claim. Thanks for catching this, we corrected Section 6.1.

4. Q3: Table 8. Since this is an ablation of the reward function, The numbers come from separate 2-hour runs with the same Qwen3-4B scientist, while Table 2 reports 8-hour runs. We have updated Appendix A.4 to states this.


We thank the reviewer for their positive feedback and asking these questions. We ask them to consider revising their score if these concerns are resolved. If there are any concerns left, we are happy to provide more clarification.

AuthorsOpenReview ↗

Kind reminder for discussion

Dear Reviewer,

Thank you again for your review. We have posted our response and revised the paper. We have added

  1. Held-out test results on all 14 MLEBench-Hard competitions
  2. 24 GPU-hour budget comparison for every method
  3. Compared ARTS* against compaction and extended search and
  4. The corrected MetaMaze and UCB claims.

As the discussion period ends soon, we would be grateful if you could take a look and let us know if any concern remains.

AuthorsOpenReview ↗

Request for engagement

Dear reviewer,

As the discussion period nears its end, we would really appreciate you engaging in the discussion, or at least acknowledging that our rebuttal has been read.

Thanks, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.