- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents OpenMLE, a full-stack framework for improving LLMs' capabilities on MLE tasks. It has three parts:
- OpenMLE-Gym: a collection of 5,758 executable, quality-checked MLE tasks.
- OpenMLE-Train: a post-training recipe that teaches a model four program-editing operators (draft, improve, debug, crossover) through execution-verified SFT followed by RL.
- OpenMLE-Evo: a redesigned AIRA-Evo-style search harness that uses structured experience records to guide parent selection and memory.
On MLE-Bench Lite, both post-training and the new harness improve results. The enhanced system improves a model with 3B active parameters to a level comparable with frontier proprietary agents under the same compute budget, and preliminary results also suggest generalizability to other modalities and scientific tasks.
Critical strengths
-
The release is a substantial contribution to the open research community. The task corpus is much larger than existing executable MLE resources, and releasing tasks, sandbox, training trajectories, code, and weights together is rare in this area. This gives the community a valuable foundation for studying how to train models for iterative ML engineering rather than only evaluating them.
-
The experiments are carefully designed and also broad. For example, the authors separate the effect of the model from the effect of the harness by holding one fixed while varying the other, replicate the post-training gain on a second backbone, and evaluate many models under a common protocol. The analysis showing that the new harness achieves its gains with far fewer tokens is also a useful empirical finding.
Critical weaknesses
-
The evaluation is a bit noisy to support several claims made by the authors. MLE-Bench Lite has only 22 tasks, and with three runs the standard deviation of Medal Average is about 7 to 9 points (according to Table 2). Several reported gains are within that range, including each component of the harness ablation and the comparison with frontier proprietary agents, whose numbers come from single runs. Another concern is that the selected program is run three times on the test split and the best result is kept, which might optimistically inflate reported scores, and it is unclear whether the external baselines received the same treatment.
-
The paper offers limited insights into which design choices matter. Most components adapt existing techniques, so the value lies largely in the overall engineering and the release. However, probably due to budget constraint the training recipe is not ablated: for instance, there is no comparison of the reward processing, progress bonus, or operator mix against simpler alternatives. Also, about 75% of the SFT data is for the draft operator, so it is unclear how much the model actually learns to improve, debug, and recombine existing solutions.
Questions for authors
- OpenMLE-Evo-Max adds experience priors, lessons distilled from public Kaggle competition materials and given to the search before it starts. What exactly are these materials? Beyond exact name matching, did you check for dataset-level or near-duplicate overlap between MLE-Bench and either the OpenMLE-Gym training tasks or the sources of these "experience priors"?
6 comments
Response to Reviewer qSqM (2/3)
(Response to Reviewer qSqM, part 2 of 3.)
W2. Insights beyond the release
We agree that most components build on prior techniques: MLE-Smith-style task construction (Qiang et al., 2025), GSPO (Zheng et al., 2025) with an entropic advantage (Yuksekgonul et al., 2026), and an AIRA-Evo population loop (Toledo et al., 2025). What the stack adds is the ability to change one part while holding the others fixed, and the paper reports four findings from such comparisons: (1) at a fixed training budget, adding executable training tasks keeps improving the trained agent (Figure 6, both backbones); (2) SFT and RL each add to the result on two backbones (Table 1); (3) the harness gain comes from spending fewer tokens per node, not from evaluating more nodes: OpenMLE-Evo evaluates 12.4% fewer nodes with 42% fewer tokens, and the share of Improve calls that set a new best roughly doubles (Appendix E.3); and (4) training and search gains are complementary: OpenMLE-35B benefits from the improved harness, and the harness benefits the four other models we paired it with (Figure 7). The analyses below on the training signal and on individual operators add evidence at the component level.
W2. Ablating the training recipe
As you note, a full 200-step RL run costs about 2.5 × 10⁴ sandbox GPU-hours, so the submission ablates the recipe only at the stage level (SFT vs. SFT+RL, Table 1). We add component-level evidence from an offline analysis of the logged RL rollout groups of the released OpenMLE-35B; each group consists of 16 candidates that share the same task, operator, parent context, and prompt.
Reward processing. Compared with static leaderboard bounds, the task-adaptive bounds of Appendix C.4 increase the mean within-group reward range from 0.682 to 0.742 (+8.9%), so they separate candidates within a group more clearly.
| Operator (candidates that beat their parent) | Count | Mean reward, λ = 0 | Mean reward, λ = 0.5 |
|---|---|---|---|
| Improve | 1,153 | 0.554 | 0.712 |
| Debug | 1,409 | 0.415 | 0.622 |
| Crossover | 1,016 | 0.598 | 0.696 |
Progress bonus. Holding the candidates, groups, parents, and base reward fixed and changing only the bonus coefficient, λ = 0.5 raises the mean reward of candidates that beat their parent for all three revision operators (table above), so verified improvements receive more credit.
Advantage. The mean of the largest within-group advantage is 6.04 with the entropic advantage, compared with 1.90 with the GRPO z-score (Shao et al., 2024), so the best candidate in each group receives a much larger update.
These numbers measure the learning signal each design produces, not its effect on final performance. Full-scale ablations remain future work, and the revised PDF will state this in Appendix C.4.
W2. Does the model learn to improve, debug, and recombine?
74% of the SFT examples are Draft (19,436 of 26,259), mainly because the parallel path collects complete solutions. Revision is trained mostly in the RL stage: RL samples Improve, Debug, and Crossover for half of all rollout groups (0.17, 0.17, and 0.16; Table 7), and RL contributes 12.1 of the 21.2 points on Qwen3.6-35B-A3B (48.48% → 60.61%; Table 1). To check this directly, we tracked the validation reward of each operator throughout RL on the held-out Gym validation set of Figure 15 (mean of the first and of the last three validation points):
| Operator | Validation reward, start of RL | Validation reward, end of RL |
|---|---|---|
| Draft | 0.198 | 0.289 |
| Improve | 0.138 | 0.356 |
| Debug | 0.202 | 0.360 |
| Crossover | 0.130 | 0.297 |
All three revision operators improve, and by more than Draft. The search logs show the same trend (see our response to Reviewer P8LV): from the base model to SFT to OpenMLE-35B, the share of Improve calls that beat their parent rises from 17.7% to 30.4% to 32.4%, the share of Crossover calls that beat the stronger parent from 11.7% to 20.7% to 28.8%, and the share of Debug chains repaired within ten calls from 91.8% to 94.2% to 95.2%. The revised PDF will add the per-operator validation curves to Appendix C.4.
References
- Qiang et al. (2025). MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline. arXiv:2510.07307.
- Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
- Toledo et al. (2025). AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench. arXiv:2507.02554.
- Yuksekgonul et al. (2026). Learning to Discover at Test Time. arXiv:2601.16175.
- Zheng et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071.
Response to Reviewer qSqM (3/3)
(Response to Reviewer qSqM, part 3 of 3.)
Q. What the experience priors are
The experience priors are text guidance given to OpenMLE-Evo-Max before search starts, in three parts, all included in the supplementary code (skills/): (1) twelve category playbooks, one per modality × task type (e.g., image classification, tabular regression, text sequence-to-sequence), listing task routing, validation schemes, model families, and common failure modes; (2) one task skill per target task, generated offline from the task's public description, the matching playbooks, and the execution constraints, without web access and without any data, code, or solution from the target competition; and (3) one shared note on common runtime and submission-format errors. The playbooks were distilled from public Kaggle notebooks and write-ups of competitions outside MLE-Bench; the next answer describes how MLE-Bench material was excluded.
The priors are used only in the Evo-Max rows. Every post-training and harness result in Sections 5.1 and 5.2 is obtained without them. In Table 2, adding the priors to OpenMLE-Evo raises Medal Average by 1.5 points but lowers Human Rank (0.7647 → 0.7283), while adding asynchronous parallel search on top raises Medal Average by a further 9.1 points. The revised PDF will describe the priors in full in Appendix E.1.
Q. Overlap beyond exact name matching
Yes, at the dataset level, for both the training tasks and the priors. For OpenMLE-Gym, beyond exact name matching, we also filtered tasks by Kaggle competition ID, which removes different datasets derived from the same competition.
When building the experience library, we excluded at the source all material of every MLE-Bench competition: data, discussions, notebooks, solutions, leaderboards, and write-ups. For the remaining competitions we used only notebooks and write-ups, and we further removed competitions or documents that share a source with, or explicitly refer to, an MLE-Bench competition, based on task names and descriptions. For example, for dogs-vs-cats-redux-kernels-edition we excluded not only the competition with that name but also dog-vs-cat-classification and dogs-vs-cats, which use the same dataset. No web search was allowed during library construction or during the experiments, so the information sources are fixed and auditable.
The revised PDF will describe both procedures in Appendices B.1 and E.1.
We hope these responses address your concerns, and we would be glad to clarify anything further.
Response to Reviewer qSqM (1/3)
We thank the reviewer for the detailed review and the concrete points on statistics and contamination. We agree with both main concerns. Below we (i) clarify the statistical support using Human Rank, (ii) quantify the effect of repeated test execution (with the mean of three executions instead of the best, OpenMLE-35B with OpenMLE-Evo scores 56.06% rather than 60.61%), (iii) add component-level evidence on the training recipe and on each operator, and (iv) describe the experience priors and our decontamination procedure in detail.
W1. Noise in the evaluation
We agree that Medal Average over the 22 MLE-Bench Lite tasks (Chan et al., 2024) is noisy: a medal is a binary outcome per task, so the metric is sparse and varies considerably across runs. For this reason we also report Human Rank, the fraction of human leaderboard participants that a submission surpasses, which is continuous and has much lower variance (Table 13). The post-training gains are far larger than this noise on both backbones: Human Rank rises from 0.5828 ± 0.0278 to 0.7647 ± 0.0376 on Qwen3.6-35B-A3B and from 0.5573 ± 0.0074 to 0.7055 ± 0.0505 on Qwen3-30B-A3B.
We agree that the Medal Average differences between ablation rows are within run-to-run noise. In Human Rank, the three search mechanisms each improve the result (Table 2): adding on-demand memory to the selection-only configuration raises Human Rank from 0.6986 ± 0.0370 to 0.7647 ± 0.0376; adding guided selection to the memory-only configuration raises it from 0.7168 ± 0.0263 to 0.7647; and adding asynchronous search raises it from 0.7283 ± 0.0103 to 0.8126 ± 0.0388. Each increase is larger than the run-to-run standard deviation of both configurations and agrees in direction with Medal Average; with three runs each, we treat these as consistent evidence, not individually significant results. The exception is the experience priors, which raise Medal Average slightly (60.61% → 62.12%) but lower Human Rank (0.7647 → 0.7283); we will state this explicitly in the revised text.
W1. Comparison with frontier proprietary agents
We describe OpenMLE-35B with OpenMLE-Evo-Max as on par with GPT-5.5 + Codex and do not claim that it is better. Human Rank supports this reading: our system reaches 0.8126 ± 0.0388 over three runs, and GPT-5.5 + Codex reaches 0.7833, about 0.76 standard deviations below our mean (Table 12). The frontier references are single runs because evaluating proprietary models costs considerably more, as noted in Appendix E.2; within our budget we ran each of them once.
W1. Repeated test execution and treatment of external baselines
You are right that keeping the best of three executions inflates the reported scores somewhat: model training inside the program is stochastic, so the same program scores slightly differently across executions. We recomputed OpenMLE-35B with OpenMLE-Evo using the mean over the three executions of all 66 task-runs:
The difference (4.5 points of Medal Average and 0.023 of Human Rank) does not change our conclusions: it is much smaller than the post-training gain measured under a single protocol (21.2 points and 0.182 with best of three). In 6 of the 66 task-runs (9.1%) the three executions differ in Human Rank by at least 0.10, and in 6 a medal is gained or lost. The revised PDF will add the mean-over-executions statistics for our models to the appendix.
Why three executions. Search and final selection never use test data: the program is selected by validation score alone, and that single program is then executed on the test package. We executed it three times because, in our shared evaluation environment, hardware and platform faults can make a correct program fail, which would heavily underestimate its Human Rank and medal. Two examples from our logs:
The Codex and Claude Code references were executed once. We manually inspected every failed reference submission and found none caused by hardware faults, so their scores were not affected by such faults.
References