- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a learned model for acting as an environment simulator in the context of automated machine learning engineering tasks with a goal of making agents more effective with usage of actual resources. The authors SFT Qwen3.5-4B using ~500k AIDE executions on MLE-Bench lite tasks to predict exceptions, terminal output, validation metric and a bit indicated whether a submission was made given the python code, dataset summary, environment packages and wall clock time limit. The authors propose using the confidence score from the first exception token for the agent to decide on the next experiment to run either as a gate or as part of a UCB-style selection scheme. In terms of the predictive accuracy on ~1700 held-out in-distribution tasks, the model achieves 78% accuracy as compared to 41% for a zero-shot DeepSeek-v4-Flash baseline. Further, when used within an AIDE agent using the confidence gate improves the performance while reducing the real execution time by 24-46%. The evaluations are limited to the in-distribution tasks and baselines limited.
Critical strengths
- The main strength of the paper is the general design of the approach. In particular, moving beyond simply modeling the final validation performance (as is common in some classical AutoML approaches), and modeling the experiment code execution, coupled with careful design of the output space (which provides a confidence score for "free".
- Beyond the specific trained model, the dataset collected is quite substantial and could also seed future work, if released as promised.
Critical weaknesses
- I think the biggest weakness of the approach is the experimental design and in particular the lack of real OOD evaluations. For both the offline and the AIDE evaluations, the task settings and underlying agent models considered are the same as the ones used for producing the data the model is trained on. Moreover, in Appendix F the authors state "The nearest training program is the parent itself for 26% of Improve and 48% of Debug programs." So for nearly half of the
Debugtest set, the exact parent code and its ground-truth traceback were present in the training set. Even the OOD tasks reported in Section 7 appear to be picked post-hoc from a suite of 56 tasks (as mentioned in Appendix R). - Beyond the OOD evaluation, the baselines also seem lacking. A zero-shot baseline is included with some LLMs but the model is trained with ~500k traces so certainly has a compute advantage. Any sort of K-shot or other test-time compute strategy would be an important baseline to understand the benefits of the entire effort of collecting so much data and fine-tuning a model on it. The AIDE experiments don't even include this zero-shot LLM-based baseline.
(Minor) There quite a bit of related work missing in the paper for example the following:
- CWM: An Open-Weights LLM for Research on Code Generation with World Models, Copet et al., arXiv:2510.02387.
- NExT: Teaching Large Language Models to Reason about Code Execution, Ni et al., 2024.
- SWE-World: Building Software Engineering Agents in Docker-Free Environments. Sun et al., arXiv:2602.03419.
- MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. Huang et al., ICML 2024.
Questions for authors
- Given that 48% of
Debugtest scripts and 26% ofImprovetest scripts have their exact parent program and parent traceback in the training corpus, what is MLE-World's accuracy in Table 1 when the train/holdout split is partitioned such that no ancestor of any held-out test script appears in training? - Why were only 3 tasks reported out of the 56 tasks evaluated in the out-of-distribution ranking?
- How do the LLM baselines perform in the offline evaluations when provided with the parent script and its ground-truth execution log (for Improve/Debug nodes), or k-nearest neighbor executions retrieved from the 497k corpus? Similarly, how does Base AIDE perform when augmented with retrieval access over the 497k corpus?
4 comments
Dear Reviewer, thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised PDF address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.
Thanks for your response and additional results. The response was posted only 2 days earlier so I still needed some time to consider the response.
- The retrieval-augmented LLM baseline results do seem to show that fine-tuning adds real value beyond the corpus for MLE-World. Though the AIDE experiments still lack an LLM-predictor baseline and a Base AIDE with retrieval access.
- The rescoring excludes only the parent, not all ancestors. Based on what I understood, the authors' numbers imply that about 91% of held-out Improve/Debug programs have their parent in training, and the parent-excluded subset is mostly drafts and accuracy drops from 78.1% to 69.5% on it.
- My central concern about generalization beyond the training distribution remains unresolved.
Based on these concerns, I will unfortunately maintain my score leaning towards rejection.
Thank you for the follow up. We respond to the three remaining points below and provide initial results. The complete experiments will be included in the final version.
The rescoring excludes only the parent, not all ancestors
Our split is at the program level. This mirrors common practice for learned dynamics models in model based RL, where held out transitions are drawn at random from the same trajectories as the training transitions (MBPO [1]; MBRL-Lib [2]). Held out accuracy of this kind is also known not to determine downstream performance [3], so we treat the complete searches as the main evidence (table 2).
Since the reviewer asked for accuracy with no ancestor in training, we evaluated MLE-World on 1,700 programs from new searches that ran after the training corpus was fixed. No program, parent, or earlier ancestor has an exact or near duplicate in training.
| Predictor, programs (%) | Terminal output | Exception | Metric |
|---|---|---|---|
| MLE-World | 77.9 ± 1.0 | 86.4 ± 0.8 | 78.1 ± 1.9 |
| DeepSeek with 4 retrieved executions (N=4) | 53.5 ± 1.1 | 66.7 ± 1.1 | 37.4 ± 2.2 |
| MLE-World, new search trees | 71.4 ± 1.4 | 80.6 ± 1.1 | 67.7 ± 2.9 |
We acknowledge that accuracy drops on new trajectory. Even so, MLE-World on new search trees still outperforms the much larger prompted model, DeepSeek with 4 retrieved executions. We expect this gap to narrow with more training data, a larger model, and reasoning before prediction, which the Conclusion lists as future work.
LLM predictor baseline and Base AIDE with retrieval in the AIDE experiments
LLM predictor. We did not run BoN with a prompted LLM predictor. DeepSeek's inference is slow, needing much more GPU time than MLE-World, which would hinder a time limited search. The faster Coder model predicts poorly even with 8 retrieved executions (28.7% terminal output and 39.5% metric accuracy)
Retrieval. We tested retrieval in the BoN setting on purpose, so that it can be compared directly with BoN guided by MLE-World under the same candidates and budget:
| BoN feedback source | Mean percentile | Below MLE-World |
|---|---|---|
| MLE-World | 50.3 | |
| Retrieval 1NN | 45.0 | 5.3 ± 2.6 |
| Retrieval 5NN | 44.1 | 6.2 ± 5.3 |
| Base AIDE (one candidate) | 45.4 |
Search with retrieved outcomes stays at Base AIDE's level. Using the corpus to fine-tune the agent itself, rather than adding retrieved executions to its prompt, is left for future work.
Generalization beyond the training distribution
Our setting, in which a world model is trained on experience from the same tasks it is later used on, is the standard setting in model based RL and in robot learning. PETS [4], MuZero [5], Dreamer [6], and MBPO [1] each learn a dynamics model from interaction with an environment and then plan or train a policy in that same environment, and DayDreamer [7] does the same on physical robots. These works ask whether a learned model improves decisions on the task that produced its data. We ask the same question for ML engineering: "given execution logs accumulated on a set of ML tasks, can a learned model of execution make later searches on those tasks better and cheaper?" Generalization to unseen tasks is therefore not the goal of the paper, but since the reviewer requested it, we provide initial results for three kinds of shift.
| Shift | Initial result |
|---|---|
| New programs | The search gains hold on new trees (table 2) |
| New agent model: Gemma-4-E4B, which wrote no training program (App. U) | Every guided policy ends above Base AIDE |
| Unseen tasks, prediction only | Ahead of a prompted model on most tasks, but still below the training tasks |
We see that MLE-World carries over to new programs, with an initial signal for a new agent and for unseen tasks. We have not run complete searches on unseen tasks. We will add them to the final version and state this scope in the abstract.
References
[1] Janner et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.
[2] Pineda et al. MBRL-Lib: A Modular Library for Model-based Reinforcement Learning. arXiv:2104.10159, 2021.
[3] Lambert et al. Objective Mismatch in Model-based Reinforcement Learning. L4DC 2020.
[4] Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. NeurIPS 2018.
[5] Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature 2020.
[6] Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020.
[7] Wu et al. DayDreamer: World Models for Physical Robot Learning. CoRL 2022.
We thank the reviewer for the careful and constructive review. In summary, we (1) report Table 1 on held out programs whose parent is not in training, (2) explain how the unseen tasks were selected, and (3) point to retrieval baselines from the same corpus for both prediction and complete searches.
Q1 and W1. Parent programs in training. The quoted statistic counts parent and child links across the split. It does not mean held out programs are duplicated: no exact or near duplicate of a held out program is in training (Appendix F). A child also differs from its parent in outcome: copying the parent's record matches the terminal output for only 22.3% of programs (Table 1). We rescored Table 1 on the held out programs whose parent is not in training:
Terminal output and exception accuracy are unchanged. Metric accuracy is lower, so part of it on Improve and Debug programs benefits from a parent seen in training. We will add this table.
On agent models: the training records come from five generator models. The search agent, Qwen3.5-9B, produced 11% of them, and no reported run is in training. We agree that the evaluation is within task and state this as a limitation (Appendix A).
Q2. Why three of 56 unseen tasks. We agree that the reported unseen task results do not establish generalization of complete agent searches. Although generalization to unseen tasks was outside the primary scope of this project, we provide these exploratory results to inform future research and show an initial signal of generalization ability. We initially evaluated small subsets from 56 unseen tasks to identify where execution prediction might transfer, then selected three promising tasks for further evaluation on separate confirmation samples. This exploratory analysis identifies possible transfer conditions rather than representative performance across unseen tasks.
MLE-World shows some generalization when an unseen task shares structure with training tasks: the two image classification tasks retrieve mostly image classification programs and reach 68.0% and 53.0%. Stanford (RNA sequences in JSON) has no related training task; its nearest programs come from an unrelated tabular task, and all 100 programs are predicted to fail, although 35 succeeded.
Q3 and W2. Baselines with access to the corpus. Prediction. The revised Table 1 gives the prompted models k nearest executions retrieved from the same 497,056 records (Appendix H). Whenever the parent is in training and ranks among the nearest programs, the model receives the parent script with its real execution log.
Retrieval helps the prompted models, but MLE-World remains well ahead, at 13 to 1,061 times less GPU inference time than the prompted models without retrieval.
Search. We ran BoN with outcomes retrieved from the same corpus in place of MLE-World predictions, with four candidates per step for 12 h:
Search with retrieved outcomes stays at Base AIDE's level (Figure 5, Appendix Q.1). The reason is that a copied record is a poor prediction for a new program. On the held out programs, copying the record of the most similar training program from the same task matches the exception class for 41.1% and the terminal output for 28.0%, vs 86.4% and 77.9% for MLE-World (Table 1, Nearest program). BoN selects among four candidates by their predicted outcome and validation score, so with retrieved outcomes it mislabels most candidates and the extra candidates add little.
Related work. Thank you for the pointers. The revised Section 2 cites SWE-World, CWM, NExT, and MLAgentBench.