acceptodds
Under review as a conference paper at ICLR 2027

MLE-WORLD: PREDICTING MACHINE LEARNING EXPERIMENT OUTCOMES WITH A LEARNED WORLD MODEL

Abstract

AI research agents can propose, execute, and improve solutions to machine learning tasks, but they face a critical bottleneck: execution cost. An agent must wait for each candidate's execution to finish before it receives feedback, which makes it hard to scale at test time by evaluating more candidates. We introduce MLE-World, a learned world model for machine learning engineering that predicts a candidate solution's terminal output (including error traces, training logs, and validation scores) from its code and a description of the task's data, without executing the program. We train MLE-World by supervised fine-tuning Qwen3.5-4B on agent trajectories that pair solution code with observed execution outputs. Evaluating a candidate then requires a single inference call to a 4B model, so an agent can screen several candidates per step and execute only one. Because these predictions are not always accurate, MLE-World also reports a confidence score for each predicted outcome, which we utilize in two ways. First, we propose a confidence gate to only execute solutions with low confidence while relying on the world model to simulate executions with high confidence. Second, because greedily selecting the candidate with the best validation score does not encourage exploration, we propose a selection rule inspired by upper confidence bounds (UCB) to add an exploration bonus based on confidence to predicted validation scores. On 17 known MLE-bench Lite tasks, all five gated policies raise mean leaderboard percentile from 36.9 to 46.449.2, and the gate cuts real execution by 2446%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
world models, LLM agents, machine learning engineering (MLE), test time scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_khsJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a learned model for acting as an environment simulator in the context of automated machine learning engineering tasks with a goal of making agents more effective with usage of actual resources. The authors SFT Qwen3.5-4B using ~500k AIDE executions on MLE-Bench lite tasks to predict exceptions, terminal output, validation metric and a bit indicated whether a submission was made given the python code, dataset summary, environment packages and wall clock time limit. The authors propose using the confidence score from the first exception token for the agent to decide on the next experiment to run either as a gate or as part of a UCB-style selection scheme. In terms of the predictive accuracy on ~1700 held-out in-distribution tasks, the model achieves 78% accuracy as compared to 41% for a zero-shot DeepSeek-v4-Flash baseline. Further, when used within an AIDE agent using the confidence gate improves the performance while reducing the real execution time by 24-46%. The evaluations are limited to the in-distribution tasks and baselines limited.

Critical strengths
  • The main strength of the paper is the general design of the approach. In particular, moving beyond simply modeling the final validation performance (as is common in some classical AutoML approaches), and modeling the experiment code execution, coupled with careful design of the output space (which provides a confidence score for "free".
  • Beyond the specific trained model, the dataset collected is quite substantial and could also seed future work, if released as promised.
Critical weaknesses
  • I think the biggest weakness of the approach is the experimental design and in particular the lack of real OOD evaluations. For both the offline and the AIDE evaluations, the task settings and underlying agent models considered are the same as the ones used for producing the data the model is trained on. Moreover, in Appendix F the authors state "The nearest training program is the parent itself for 26% of Improve and 48% of Debug programs." So for nearly half of the Debug test set, the exact parent code and its ground-truth traceback were present in the training set. Even the OOD tasks reported in Section 7 appear to be picked post-hoc from a suite of 56 tasks (as mentioned in Appendix R).
  • Beyond the OOD evaluation, the baselines also seem lacking. A zero-shot baseline is included with some LLMs but the model is trained with ~500k traces so certainly has a compute advantage. Any sort of K-shot or other test-time compute strategy would be an important baseline to understand the benefits of the entire effort of collecting so much data and fine-tuning a model on it. The AIDE experiments don't even include this zero-shot LLM-based baseline.

(Minor) There quite a bit of related work missing in the paper for example the following:

  • CWM: An Open-Weights LLM for Research on Code Generation with World Models, Copet et al., arXiv:2510.02387.
  • NExT: Teaching Large Language Models to Reason about Code Execution, Ni et al., 2024.
  • SWE-World: Building Software Engineering Agents in Docker-Free Environments. Sun et al., arXiv:2602.03419.
  • MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. Huang et al., ICML 2024.
Questions for authors
  • Given that 48% of Debug test scripts and 26% of Improve test scripts have their exact parent program and parent traceback in the training corpus, what is MLE-World's accuracy in Table 1 when the train/holdout split is partitioned such that no ancestor of any held-out test script appears in training?
  • Why were only 3 tasks reported out of the 56 tasks evaluated in the out-of-distribution ranking?
  • How do the LLM baselines perform in the offline evaluations when provided with the parent script and its ground-truth execution log (for Improve/Debug nodes), or k-nearest neighbor executions retrieved from the 497k corpus? Similarly, how does Base AIDE perform when augmented with retrieval access over the 497k corpus?
4 comments

We thank the reviewer for the careful and constructive review. In summary, we (1) report Table 1 on held out programs whose parent is not in training, (2) explain how the unseen tasks were selected, and (3) point to retrieval baselines from the same corpus for both prediction and complete searches.

Q1 and W1. Parent programs in training. The quoted statistic counts parent and child links across the split. It does not mean held out programs are duplicated: no exact or near duplicate of a held out program is in training (Appendix F). A child also differs from its parent in outcome: copying the parent's record matches the terminal output for only 22.3% of programs (Table 1). We rescored Table 1 on the held out programs whose parent is not in training:

Held out programs (%)NTerminal outputExceptionMetric
All (Table 1)1,70077.9 ± 1.086.4 ± 0.878.1 ± 1.9
Parent not in training66977.5 ± 1.686.4 ± 1.369.5 ± 3.8
Draft programs (no ancestor)56178.3 ± 1.786.5 ± 1.474.0 ± 3.9

Terminal output and exception accuracy are unchanged. Metric accuracy is lower, so part of it on Improve and Debug programs benefits from a parent seen in training. We will add this table.

On agent models: the training records come from five generator models. The search agent, Qwen3.5-9B, produced 11% of them, and no reported run is in training. We agree that the evaluation is within task and state this as a limitation (Appendix A).

Q2. Why three of 56 unseen tasks. We agree that the reported unseen task results do not establish generalization of complete agent searches. Although generalization to unseen tasks was outside the primary scope of this project, we provide these exploratory results to inform future research and show an initial signal of generalization ability. We initially evaluated small subsets from 56 unseen tasks to identify where execution prediction might transfer, then selected three promising tasks for further evaluation on separate confirmation samples. This exploratory analysis identifies possible transfer conditions rather than representative performance across unseen tasks.

Unseen taskRelated training tasksMLE-WorldDeepSeek
RANZCR (successful runs)Plant Pathology, MLSP Birds68.0 ± 4.818.6 ± 2.3
APTOSPlant Pathology, Dog Breed53.0 ± 5.649.3 ± 5.0
3D Object DetectionDenoising Documents, MLSP Birds48.7 ± 7.735.7 ± 8.2
Stanford COVID Vaccinenone2.0 ± 0.519.0 ± 1.7
Pooled46.9 ± 4.128.7 ± 2.5

MLE-World shows some generalization when an unseen task shares structure with training tasks: the two image classification tasks retrieve mostly image classification programs and reach 68.0% and 53.0%. Stanford (RNA sequences in JSON) has no related training task; its nearest programs come from an unrelated tabular task, and all 100 programs are predicted to fail, although 35 succeeded.

Q3 and W2. Baselines with access to the corpus. Prediction. The revised Table 1 gives the prompted models k nearest executions retrieved from the same 497,056 records (Appendix H). Whenever the parent is in training and ranks among the nearest programs, the model receives the parent script with its real execution log.

Predictor (%)Terminal outputExceptionMetric
DeepSeek, zero shot41.1 ± 1.157.2 ± 1.218.6 ± 1.7
DeepSeek + 4 retrieved53.5 ± 1.166.7 ± 1.137.4 ± 2.2
Coder + 8 retrieved28.7 ± 1.141.4 ± 1.239.5 ± 2.2
Copy the parent's record22.3 ± 1.034.2 ± 1.248.8 ± 2.3
MLE-World77.9 ± 1.086.4 ± 0.878.1 ± 1.9

Retrieval helps the prompted models, but MLE-World remains well ahead, at 13 to 1,061 times less GPU inference time than the prompted models without retrieval.

Search. We ran BoN with outcomes retrieved from the same corpus in place of MLE-World predictions, with four candidates per step for 12 h:

BoN feedback sourceMean percentileBelow MLE-World
MLE-World50.3
Retrieval 1NN45.05.3 ± 2.6
Retrieval 5NN44.16.2 ± 5.3
Base AIDE (one candidate)45.4

Search with retrieved outcomes stays at Base AIDE's level (Figure 5, Appendix Q.1). The reason is that a copied record is a poor prediction for a new program. On the held out programs, copying the record of the most similar training program from the same task matches the exception class for 41.1% and the terminal output for 28.0%, vs 86.4% and 77.9% for MLE-World (Table 1, Nearest program). BoN selects among four candidates by their predicted outcome and validation score, so with retrieved outcomes it mislabels most candidates and the extra candidates add little.

Related work. Thank you for the pointers. The revised Section 2 cites SWE-World, CWM, NExT, and MLAgentBench.

Dear Reviewer, thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised PDF address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Thanks for your response and additional results. The response was posted only 2 days earlier so I still needed some time to consider the response.

  • The retrieval-augmented LLM baseline results do seem to show that fine-tuning adds real value beyond the corpus for MLE-World. Though the AIDE experiments still lack an LLM-predictor baseline and a Base AIDE with retrieval access.
  • The rescoring excludes only the parent, not all ancestors. Based on what I understood, the authors' numbers imply that about 91% of held-out Improve/Debug programs have their parent in training, and the parent-excluded subset is mostly drafts and accuracy drops from 78.1% to 69.5% on it.
  • My central concern about generalization beyond the training distribution remains unresolved.

Based on these concerns, I will unfortunately maintain my score leaning towards rejection.

Thank you for the follow up. We respond to the three remaining points below and provide initial results. The complete experiments will be included in the final version.

The rescoring excludes only the parent, not all ancestors

Our split is at the program level. This mirrors common practice for learned dynamics models in model based RL, where held out transitions are drawn at random from the same trajectories as the training transitions (MBPO [1]; MBRL-Lib [2]). Held out accuracy of this kind is also known not to determine downstream performance [3], so we treat the complete searches as the main evidence (table 2).

Since the reviewer asked for accuracy with no ancestor in training, we evaluated MLE-World on 1,700 programs from new searches that ran after the training corpus was fixed. No program, parent, or earlier ancestor has an exact or near duplicate in training.

Predictor, programs (%)Terminal outputExceptionMetric
MLE-World77.9 ± 1.086.4 ± 0.878.1 ± 1.9
DeepSeek with 4 retrieved executions (N=4)53.5 ± 1.166.7 ± 1.137.4 ± 2.2
MLE-World, new search trees71.4 ± 1.480.6 ± 1.167.7 ± 2.9

We acknowledge that accuracy drops on new trajectory. Even so, MLE-World on new search trees still outperforms the much larger prompted model, DeepSeek with 4 retrieved executions. We expect this gap to narrow with more training data, a larger model, and reasoning before prediction, which the Conclusion lists as future work.

LLM predictor baseline and Base AIDE with retrieval in the AIDE experiments

LLM predictor. We did not run BoN with a prompted LLM predictor. DeepSeek's inference is slow, needing much more GPU time than MLE-World, which would hinder a time limited search. The faster Coder model predicts poorly even with 8 retrieved executions (28.7% terminal output and 39.5% metric accuracy)

Retrieval. We tested retrieval in the BoN setting on purpose, so that it can be compared directly with BoN guided by MLE-World under the same candidates and budget:

BoN feedback sourceMean percentileBelow MLE-World
MLE-World50.3
Retrieval 1NN45.05.3 ± 2.6
Retrieval 5NN44.16.2 ± 5.3
Base AIDE (one candidate)45.4

Search with retrieved outcomes stays at Base AIDE's level. Using the corpus to fine-tune the agent itself, rather than adding retrieved executions to its prompt, is left for future work.

Generalization beyond the training distribution

Our setting, in which a world model is trained on experience from the same tasks it is later used on, is the standard setting in model based RL and in robot learning. PETS [4], MuZero [5], Dreamer [6], and MBPO [1] each learn a dynamics model from interaction with an environment and then plan or train a policy in that same environment, and DayDreamer [7] does the same on physical robots. These works ask whether a learned model improves decisions on the task that produced its data. We ask the same question for ML engineering: "given execution logs accumulated on a set of ML tasks, can a learned model of execution make later searches on those tasks better and cheaper?" Generalization to unseen tasks is therefore not the goal of the paper, but since the reviewer requested it, we provide initial results for three kinds of shift.

ShiftInitial result
New programsThe search gains hold on new trees (table 2)
New agent model: Gemma-4-E4B, which wrote no training program (App. U)Every guided policy ends above Base AIDE
Unseen tasks, prediction onlyAhead of a prompted model on most tasks, but still below the training tasks

We see that MLE-World carries over to new programs, with an initial signal for a new agent and for unseen tasks. We have not run complete searches on unseen tasks. We will add them to the final version and state this scope in the abstract.

References

[1] Janner et al. When to Trust Your Model: Model-Based Policy Optimization. NeurIPS 2019.

[2] Pineda et al. MBRL-Lib: A Modular Library for Model-based Reinforcement Learning. arXiv:2104.10159, 2021.

[3] Lambert et al. Objective Mismatch in Model-based Reinforcement Learning. L4DC 2020.

[4] Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. NeurIPS 2018.

[5] Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature 2020.

[6] Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020.

[7] Wu et al. DayDreamer: World Models for Physical Robot Learning. CoRL 2022.

Reviewer_BC1yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MLE-World, a Qwen3.5-4B model fine-tuned on approximately 497K code–execution pairs to predict exception classes, terminal output, validation scores, and submission status. Within AIDE, these predictions support candidate screening and several search policies. An execution gate substitutes predicted feedback for selected high-confidence executions, while a UCB-inspired rule adds an exploration bonus based on the probability of a clean run. On 17 MLE-bench Lite tasks seen during fine-tuning, the method improves mean leaderboard percentile and reduces real execution time. The paper also reports prediction results on three tasks excluded from fine-tuning.

Critical strengths
  1. The execution bottleneck is practically relevant. Predicting both failures and successful execution records is potentially more useful than returning only a scalar preference. The experiments provide evidence that the model helps avoid unsuccessful runs and improves the rate of valid submissions.

  2. The empirical analysis includes useful controls. Beyond evaluating several search policies over three seeds per task, the paper includes corpus baselines, a matched greedy control for the exploration bonus, a random-execution gate, and fixed-state diagnostics. These analyses help identify where the improvements come from. The paper also acknowledges its in-distribution evaluation and the limitations of its uncertainty proxy.

Critical weaknesses
  1. The methodological advance over existing approaches appears limited. Supervised execution prediction, candidate screening, confidence-based simulation, and uncertainty bonuses are established ideas. The dedicated training corpus and integration into MLE search are useful engineering contributions, but the distinction from closely related systems such as DSWorld and ForeAgent is not sufficiently supported by direct comparisons. In particular, it remains unclear whether generating terminal logs is necessary, rather than using a simpler learned failure/score predictor with comparable task knowledge.

  2. Generality and practical cost-effectiveness are not established. The main experiments use the same 17 tasks that supplied the training data, whereas Base AIDE has no equivalent access to the accumulated execution experience. The three unseen-task results are prediction-only and were selected following an earlier ranking over 56 tasks, so they are not strong evidence of broad transfer. Collecting approximately 500K training examples is also a substantial prerequisite. Although SFT cost is reported, data-collection cost and the number of future searches needed to amortize it are not analyzed. The demonstrated use case is therefore closer to repeated search on known tasks than general-purpose ML research assistance.

  3. The evidence for reliable quality prediction and exploration is less convincing than the evidence for failure screening. Appendix P shows that screening improves test percentile by 5.51 points, while ranking the screened pool by predicted metric adds only 0.68 ± 0.79 points. Metric accuracy within 5% relative error does not establish that small differences between competing candidates are ranked correctly. Likewise, confidence in the exception class is not directly uncertainty about model quality. The controlled bonus experiment is encouraging, but does not fully explain whether the benefit comes from broader exploration, better failure avoidance, or subsequent differences in the search trajectory.

Questions for authors
  1. Can you provide a prospectively selected task-held-out evaluation of complete agent searches, rather than prediction accuracy alone? What were the results across the original 56 unseen tasks, and how were the three reported tasks selected?

  2. How does the method compare with Base AIDE given retrieval access to the same execution corpus, or a similarly trained predictor that outputs only failure status and validation score? An end-to-end ablation would clarify the value of generating terminal logs beyond the target ablation in Appendix D.

  3. Does exception-class confidence still predict validation-score error when conditioning on programs that actually run successfully? Please also report this relationship on candidates generated during search, since the balanced offline holdout has a different distribution.

  4. Can you report within-pool pairwise ranking accuracy or selection regret for validation scores? How should the limited additional benefit of metric ranking in Appendix P be reconciled with the motivation for outcome-guided optimization?

  5. What is the cost of collecting the training corpus, and when do the savings outweigh collection and SFT costs? Including generator, reviewer, and world-model inference costs would also make the practical efficiency comparison more complete.

  6. Predicted timeouts are never executed, even when confidence is low. How often does this rule incorrectly discard a successful candidate, and what is the effect of verifying low-confidence timeout predictions? Please clarify the priority of this rule relative to mandatory verification of a new predicted best.

4 comments

We thank the reviewer for the detailed and constructive feedback. We (1) report the 56 task screening process for the unseen tasks, (2) compare MLE-World with a predictor trained without terminal output and with retrieval inside complete searches, (3) report confidence diagnostics from actual searches, and (4) discuss cost and report a timeout audit.

Q1. Unseen tasks. We agree that the unseen results are prediction only and do not establish generalization of complete searches. Generalization to unseen tasks was outside our primary scope; these exploratory results give an initial signal. We screened small subsets from 56 unseen tasks, then evaluated three promising tasks and one unrelated task on separate confirmation samples.

Unseen task (terminal output, %)Related training tasksMLE-WorldDeepSeek
RANZCR (successful runs)Plant Pathology, MLSP Birds68.0 ± 4.818.6 ± 2.3
APTOSPlant Pathology, Dog Breed53.0 ± 5.649.3 ± 5.0
3D Object DetectionDenoising Documents, MLSP Birds48.7 ± 7.735.7 ± 8.2
Stanford COVID Vaccinenone2.0 ± 0.519.0 ± 1.7
Pooled46.9 ± 4.128.7 ± 2.5

MLE-World shows some generalization when an unseen task shares structure with training tasks: the two image classification tasks retrieve mostly image classification programs and reach 68.0% and 53.0%. Stanford has no related training task; its nearest programs come from an unrelated tabular task, and all 100 programs are predicted to fail, although 35 succeeded.

Q2 and W1. Terminal logs, retrieval, and related systems.

Predictor (%), training tasks / unseen tasksTerminal outputExceptionMetric
MLE-World77.9 / 46.986.4 / 54.678.1 / 20.7
Trained without terminal outputn/a83.9 / 60.378.9 / 4.4
DeepSeek + retrieval53.5 / 28.366.7 / 49.237.4 / 10.8

Training tasks: Tables 1 and 8; unseen tasks: Table 25. Without terminal output, metric accuracy matches MLE-World on training tasks but collapses on unseen tasks (4.4% vs 20.7%), so learning to predict logs helps scores transfer.

BoN feedback source, 12 hMean percentileBelow MLE-World
MLE-World50.3
Retrieval 1NN45.05.3 ± 2.6
Retrieval 5NN44.16.2 ± 5.3
Base AIDE (one candidate)45.4

Predicted logs matter for policies that read the text: with them, Sequential, Fusion, and Pairwise reach comparable performance in less search time (Figure 6); e.g., Sequential reaches percentile 40 at 2.8 h versus 5.2 h. Retrieval from the same corpus leaves BoN at Base AIDE's level. Unlike DSWorld and ForeAgent, MLE-World predicts the full execution record, and we test it against a predictor without logs and against retrieval.

Q3 and W3. Confidence, metric error, and the bonus. Metric accuracy within 5% on successful runs, by confidence quintile:

Quintile (%)12345
Held out, 484 runs (Figure 9a)44.670.584.295.2100.0
255 ungated searches, 5,863 runs52.860.962.968.483.8

Confidence thus identifies more reliable score predictions offline and during search. On the bonus we agree: u=1−pcleanu = 1 - p_{\mathrm{clean}} is outcome uncertainty, used as a proxy for uncertainty about quality, and we will clarify this in the paper.

Q4. Ranking within a pool.

Selection among four (Table 22)PercentileGainValid (%)
Uniform choice10.926.1
Screen16.45.5 ± 1.241.9
Screen and ranking17.16.2 ± 1.642.5
Execute all four19.88.8 ± 2.647.1

We agree that ranking adds a modest benefit after screening; together they recover 70% of the gain from executing all four. Pairwise ranking accuracy within a pool is 71.5% (Appendix R.1). Predicted records also help agents screen failures and revise code before real feedback.

Q5 and W2. Cost. Our intended use case reuses execution logs generated while researchers develop and evaluate ML methods. We will release our corpus and checkpoint so others need not repeat collection and SFT.

We focus on the execution bottleneck: experiments need isolated resources, whereas model inference can be batched on shared servers. Confidence gating reduces real execution time by 30–40% relative to ungated search (Table 3).

Q6. Timeouts. Predicted timeouts are excluded before both low confidence verification and verification of a new predicted best. Among 1,150 Base AIDE Improve candidates, 36 were predicted timeouts: 24 timed out, 9 failed otherwise, and 3 (8.3%) ran successfully. None of the 85 real improvements was lost. Two of the three had confidence below 0.5, so verifying low confidence timeout predictions is a reasonable refinement we will note.

Thank you for the additional experiments. The comparisons against the predictor without terminal logs and against retrieval, along with the timeout audit, are useful clarifications.

My main concern remains, however: the end-to-end gains are still demonstrated on the 17 tasks used for training, and the authors acknowledge that the unseen-task results are exploratory and selected from a larger pool rather than a held-out evaluation of complete searches. The modest gain from metric ranking after screening also leaves the value of outcome prediction beyond failure screening unclear. I will therefore maintain my score.

Thank you for reading the new experiments and for stating the remaining concerns precisely. Your feedback has helped us improve the quality of the paper, and we are always happy to clarify further. We have updated the paper to address both concerns; please refer to the revised PDF for the results below.

1. Complete searches outside the training distribution

We agree on the facts: no complete search in the paper runs on an unseen task, and Appendix T is prediction only. For now, the updated paper adds complete searches with a new agent on the seen tasks: Gemma-4-E4B, which wrote none of the training programs (new Appendix U, Table 27, Figure 14). We will add agent runs on unseen tasks in the final version.

Gemma-4-E4B agent, 8 hMean percentilevs Base AIDEValid (%)
Base AIDE47.274.5
Sequential48.9+1.7±4.596.1
Pairwise48.0+0.8±5.694.1
Fusion53.8+6.6±3.388.2
BoN52.9+5.7±4.796.1
UCB54.0+6.8±4.492.2
BoN + gate55.7+8.5±4.594.1

Takeaway: with an agent from a different model family that is not included in the training corpus, All eight guided arms end above Base AIDE, by +0.8 to +8.5 points, the same size as in Table 2 (+1.7 to +6.7).

The setting we claim is a world model of known tasks, as in model based RL: tasks with accumulated logs, such as a benchmark on which new agents are developed. Appendix U is that use.

2. Value beyond failure screening

The gain from ranking by predicted metric looks small (+0.7 raw score after screening, Table 22), but screening and ranking together almost reach real execution performance. They score 17.1, against 19.8 for executing all four candidates and keeping the best by real validation, and recover 70% of that gain (+6.2 of +8.8) without running any candidate. Little room is left for ranking to add more. This is also an analysis of one decision at fixed states. The more important evidence is the complete agent run.

In complete searches, predicted terminal outputs help with the screen held fixed (Figure 6). Both arms of Figure 6 keep the exception, metric, and submission fields; only the predicted terminal outputs are removed. With predicted terminal output, the agent finds a good solution sooner: all three policies reach percentile 40 between 1.1 and 2.4 hours earlier, for example Sequential at 2.8 h instead of 5.2 h. After 3 h of search they are 5.6 to 11.8 points ahead (Sequential: 41.7 vs 29.9). At fixed states, Fusion from predicted terminal outputs reaches 20.6 against 14.0 from code alone (+6.6±2.9), the same as with real terminal outputs (20.4, Table 22).

Screening itself is outcome prediction, and it targets the largest loss: Base AIDE spends 6.5 of its 9.9 execution hours per trial on runs that crash or time out (Figure 8). Simpler predictors do not deliver it. BoN with retrieved outcomes stays at Base AIDE's level (45.0 vs 50.3, Figure 5), and a judge that reads code alone reaches 13.0 against 16.4 for the screen (Table 22).

Dear Reviewer, thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised PDF address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Reviewer_fD7HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns a world model of the machine learning engineering executor: a compact language model that predicts a candidate program's terminal output, exception class and validation score from its code and the task's data overview, without running it. The predictions drive candidate selection inside AIDE through four adapted search policies plus a confidence-based selection rule, and the model's confidence decides when a prediction is trusted and when a real run is requested. On MLE-bench Lite tasks, predictions agree with observed terminal output far more often than prompted baselines at a fraction of the GPU time, and gated search raises mean leaderboard percentile while executing substantially less.

Critical strengths

1.The problem is well posed and the decomposition is clean: predicting the record rather than a scalar score is what makes the feedback useful for repair, and predicting the exception class first gives both a debuggable field and a free confidence signal from one token's distribution. 2.The controls are more thorough than this genre usually offers — corpus baselines that show the gain comes from reading the program rather than recalling the task, a random-gating control at matched execution counts, and per-stage ablations that isolate the uncertainty bonus from the rest of the policy. 3.The limitations section is unusually direct about the in-distribution setting, the missing timeout prediction, and the fact that a missed timeout still costs a full run, which makes the reported numbers easy to interpret.

Critical weaknesses

1.The world model is trained and evaluated on the same benchmark tasks, so what it demonstrates is a within-task simulator rather than a model that transfers; more importantly, the value proposition is circular, since deploying it requires the accumulated logs it is supposed to save. 2.The task set is filtered to the tasks where the agents make progress, which conditions both the baseline rate and the reported gains on the regime where AIDE already works, and breaks comparability with published numbers on the benchmark. 3.The confidence signal is validated as a ranking device only in distribution, while gate thresholds are tuned per policy on a development run over the same tasks; and the exploration bonus scores uncertainty about whether a program will run cleanly, not uncertainty about the metric it is ranking, so it optimizes a related but different quantity.

Questions for authors

1.On a genuinely new task with no logged executions, how is the corpus bootstrapped, and how quickly does gate quality degrade as the number of cold-start runs shrinks? 2.How much of the percentile gain survives a control that gives the base agent the same candidate-generation budget — sampling several programs per step and executing one chosen without the world model?

2 comments

We thank the reviewer for the careful and constructive review. We (1) clarify the intended use and scope, (2) add an initial transfer signal on unseen tasks, and (3) report controls for confidence, thresholds, and candidate budget.

W1 and Q1. Scope, circularity, transfer, and new tasks.

We agree on scope: MLE-World models the tasks it was trained on, and transfer was outside our primary scope (Appendix A). Our intended use case reuses execution logs generated while researchers develop and evaluate ML methods. These logs contain the code, terminal outputs, and validation scores needed for training. We will release our corpus and checkpoint so others can reuse this investment without repeating collection and SFT.

Initial signal of transfer. Four unseen tasks (Appendix T; terminal output accuracy, %, ± SE).

Unseen taskRelated training tasksMLE-WorldDeepSeek
RANZCR (successful runs)Plant Pathology, MLSP Birds68.0 ± 4.818.6 ± 2.3
APTOSPlant Pathology, Dog Breed53.0 ± 5.649.3 ± 5.0
3D Object DetectionDenoising Documents, MLSP Birds48.7 ± 7.735.7 ± 8.2
Stanford COVID Vaccinenone2.0 ± 0.519.0 ± 1.7
Pooled46.9 ± 4.128.7 ± 2.5

MLE-World shows some generalization when an unseen task shares structure with training tasks: the two image classification tasks retrieve mostly image classification programs and reach 68.0% and 53.0%. Stanford (RNA sequences in JSON) has no related training task; its nearest programs come from an unrelated tabular task, and all 100 programs are predicted to fail, although 35 succeeded.

New task. The corpus is the execution log of ordinary agent runs. We have not measured how accuracy grows with logged runs on a new task and leave it to future work. Any new best candidate is always executed for real (Section 5.2), so a weak world model cannot publish an unverified result.

W2. Task filter.

We keep 17 of 22 Lite tasks based on corpus coverage. On the four excluded image tasks only 2.3 to 11.6% of logged executions finish cleanly with a submission (38.6% on kept tasks); the fifth has no executions (Appendix B). This is not a regime where AIDE already works: Base AIDE averages 45.4 percentile at 12 h, below the leaderboard median (Table 2). The filter applies to every policy, and gains are paired on the same tasks, seeds, and budget. We do not compare against published MLE-bench numbers; every row is rerun in our harness.

W3. Confidence, thresholds, and the bonus.

Confidence. Beyond the held out set (Figure 9), we checked 1,150 Improve programs from the 51 Base AIDE trials, written during new searches and absent from training. Outcome predictions are correct 43% of the time in the lowest pmax⁡p_{\max} quintile and 99% in the highest (AUROC per task 0.79 ± 0.02).

Thresholds. Each pair is chosen among three pairs fixed in advance from held out operating points, on one 3 h development seed that no reported run uses (Appendix O). The outcome is not sensitive to the choice: in Table 19, 12 of 15 gated cells exceed the ungated column, and the single pair (0.50, 0.56) does so for all five policies. Our claim is execution savings at maintained quality: a random gate with no threshold reaches a similar percentile (47.4 vs 47.2), while the confidence gate uses 19% less execution time (Table 20).

Bonus. We agree. u=1−pcleanu = 1 - p_{\mathrm{clean}} is outcome uncertainty, which Section 5.1 calls "an empirical exploration proxy, not a formal confidence bound". It tracks metric error (Figure 9a), but it is not the metric's uncertainty. We will say so and state the proxy in the Introduction.

Q2. Same candidate budget without the world model.

One step (Table 22). On 153 recorded states, each with four executed and graded candidates:

Selection among the same fourMean percentileValid (%)
Uniform choice10.926.1
MLE-World screen and ranking17.1 (+6.2 ± 1.6)42.5
Execute all four, keep best19.8 (+8.8 ± 2.6)47.1

Uniform choice is the expected score of one sample, so four candidates add nothing without selection. MLE-World recovers about 70% of the gain from executing all four; a code-only LLM judge reaches 13.0.

Full search (Figure 5, Appendix Q.1). BoN with four candidates per step, selecting with outcomes retrieved from the same corpus instead of predicted ones, reaches 45.0 (1NN) and 44.1 (5NN) at 12 h, vs 45.4 for Base AIDE and 50.3 for BoN with MLE-World (5.3 ± 2.6 and 6.2 ± 5.3 points higher). The larger budget alone leaves the agent at Base AIDE's level, because a copied record is a poor prediction for a new program: on held out programs, the most similar training program's record matches the exception class for 41.1% and the terminal output for 28.0%, vs 86.4% and 77.9% for MLE-World (Table 1, Nearest program), so BoN mislabels most candidates. The arms ran in separate cohorts, so this is descriptive.

Dear Reviewer, thank you again for your review. As the discussion period ends today, we wanted to check whether our response and the revised PDF address your concerns, and we would be glad to clarify anything before the deadline. If they do, we would appreciate it if you would consider updating your score.

Comments on the paper

1 comment

We thank the Area Chair and all reviewers for their time and constructive feedback. The reviews raise three questions: whether retrieval from the training corpus would do as well as a world model, whether predicting terminal logs is needed, and how far the results hold beyond the training tasks and agent.

Scope, and results beyond the training setting (all reviewers). MLE-World is trained on experience from the same tasks it is later used on. This is the standard setting for learned world models in model based RL and robot learning: PETS (Chua et al., 2018), MuZero (Schrittwieser et al., 2020), Dreamer (Hafner et al., 2020), and DayDreamer (Wu et al., 2022) each learn a model of an environment and act in that same environment. These works ask whether a learned model improves decisions on the task that produced its data. We ask the same question for ML engineering: "given execution logs accumulated on a set of ML tasks, can a learned model of execution make later searches on those tasks better and cheaper?" Generalization to unseen tasks is therefore not the goal of the paper, but since the reviewers requested it, we provide initial results for two kinds of shift. We ran complete searches with a Gemma-4-E4B agent, whose model family wrote no training program, and extended prediction to unseen tasks. Every guided policy has a higher observed mean than Base AIDE with this agent, and more valid submissions (App. U). On unseen tasks, MLE-World predicts well when they resemble training tasks and fails on an unrelated one (App. T). Guidance thus holds for a new agent, while transfer to new tasks depends on similarity.

New experiments. Since submission we ran 18 further search arms of 51 twelve hour trials each, trained one more world model, and evaluated retrieval variants of both prompted models. Main results now use a 12 h budget instead of 8 h, which reduces the influence of early, noisy publications. Every guided policy still has a higher observed mean than Base AIDE (Table 2).

1. The gains come from the learned model, not from access to its training data ([R1 fD7H] Q2, [R2 BC1y] W1/Q2, [R3 khsJ] W2/Q3). In the prediction evaluation, we gave the prompted models retrieved training executions. The strongest reaches 53.5% terminal output accuracy, and copying the most similar training program 28.0%, against 77.9% for MLE-World (Table 1). In search, BoN that selects with retrieved outcomes instead of predictions stays at Base AIDE's level (Fig. 5).

2. Predicted terminal logs help the agent during search ([R2 BC1y] W1/Q2). We ran searches that hide the predicted logs from MLE-World. With the logs, all three policies that read them find better solutions sooner; Sequential reaches percentile 40 in about half the time (Fig. 6).

3. Held out programs are new programs ([R3 khsJ] W1/Q1). Many held out programs have their parent in training, but none has an exact or near duplicate there (App. F), and this is common setting on model based RL (random from the same trajectories as the training transitions). To test the effect, we rescored MLE-World on programs from new search trees. There is some drops, but MLE-World still outperforms the much larger prompted model. We expect the gap to narrow with more training data, a larger model, and reasoning before prediction, which the Conclusion lists as future work.

4. Confidence, ranking, and the bonus ([R1 fD7H] W3, [R2 BC1y] W3/Q3/Q4). In actual searches, score predictions remain more accurate at higher confidence. Ranking by predicted score adds little after screening, but the two together recover about 70% of the gain from executing all four candidates (Table 22). The main evidence is the complete search, where guidance gives every policy a higher observed mean than Base AIDE (Table 2). Regard to bonus on UCB, we agree that the bonus uses outcome uncertainty as a proxy for uncertainty about quality (Sec. 5.1).

5. Cost and task selection ([R1 fD7H] W1/W2, [R2 BC1y] W2/Q5/Q6). The intended use reuses logs that accumulate while agents are developed on a benchmark, and we will release the corpus and checkpoint. At 12 h the gate saves 30 to 40% of real execution time (Table 3). The 17 tasks are kept by corpus coverage and are not easy: Base AIDE stays below the leaderboard median (App. B). An audit of skipped timeouts found no real improvement lost.

What remains open. Complete searches on unseen tasks are not yet run. The paper limits its claims to known tasks with accumulated logs (App. A), and we will add these searches to the final version.

References Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. NeurIPS 2018. Schrittwieser et al. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature 2020. Hafner et al. Dream to Control: Learning Behaviors by Latent Imagination. ICLR 2020. Wu et al. DayDreamer: World Models for Physical Robot Learning. CoRL 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.