- Rating
- 3
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents M1, an MLE agent trained through general and domain-specific SFT followed by Windowed Interaction RL (WIRL). It introduces a task construction pipeline and an interaction harness, and evaluates the trained model on MLE and related optimization tasks.
Critical strengths
-
The performance gains over the base model are substantial.
-
Evaluation under both the proposed harness and Claude Code provides useful evidence of transfer.
-
Comparisons with RL baselines and ablations of trend rewards and policy correction help assess the training method.
Critical weaknesses
-
The connection between window-level optimization and long-horizon decision making is insufficiently established. WIRL assigns a shared reward based on the current execution outcome to decisions within each window. This enables training from intermediate feedback, but it is unclear how it credits exploratory actions whose benefits emerge only in later windows.
-
The methodological novelty is not sufficiently established. While the empirical gains are promising, the paper reads more like a technical report describing an effective training pipeline than a research paper establishing a distinct methodological advance. The combination of a custom harness, staged SFT, and RL involves substantial data collection and training effort, but the paper does not convincingly establish what new, generalizable insight the proposed recipe provides. In particular, it remains unclear how much of the improvement depends on the distinctive methodological choices rather than the overall investment in task-specific data and post-training. My concern is therefore not simply the absence of additional ablations, but the limited justification of the method’s novelty beyond a successful integration of existing ideas.
-
The need for the custom harness is unclear. The paper does not clearly explain what essential capability the M1 harness provides beyond a general-purpose coding harness such as Claude Code, potentially augmented with context management. Could comparable or better results be obtained by collecting trajectories and performing the same post-training directly within that environment? Evaluating the trained M1 model in Claude Code demonstrates transfer, but does not answer this training-time question. Without a clearer account of which limitations of existing harnesses necessitate the proposed design, it is difficult to assess the custom harness as a research contribution.
Questions for authors
See weaknesses. The concerns above are substantive and could change my recommendation.
4 comments
Official Comment to Reviewer SQ4j (Part 2)
[W2] Methodological novelty: how much of the improvement comes from method design versus data and training effort, and what generalizable insights the recipe provides.
We thank the reviewer for this question. We have made the following additions in the revised manuscript:
- Section 5.3: We add an MLE SFT comparison without outcome-guided trajectory filtering and supervision truncation, with the mean and SEM over three evaluations reported in Appendix D.1.
- Section 3.2: We clarify how experiment-level outcomes recorded by the Harness determine the supervision boundaries in SFT and the execution windows in WIRL.
- Appendix D.1: We add an SFT comparison in which MLE teacher trajectories collected using M1 Harness are replaced with trajectories collected using Claude Code (Table 12).
- Introduction and Conclusion: We clarify the unifying principle underlying the three method components.
(1) M1 addresses the challenging problem of autonomous optimization in real-world long-horizon agentic MLE.
Real-world MLE tasks involve data analysis, code modification, model training, and evaluation, with costly execution and delayed feedback, and the agent must maintain experimental states and adjust optimization directions over thousands of interaction steps. Existing studies advance this direction with different trade-offs in task scale, training granularity, and optimization organization. ML-Agent [1] collects expert trajectories limited to 15 steps and 30 minutes and mainly uses single-step RL on intermediate decisions. SandMLE [2] uses miniature synthetic environments to reduce execution costs and enable trajectory-level RL. Frontis-MA1 [3] trains predefined program evolution operators and organizes long-horizon search through an external evolutionary framework.
M1 instead trains a model to organize and advance the complete, open-ended MLE experimentation process on real historical competition tasks, rather than optimizing local actions or predefined operators. Individual teacher trajectories reach about 3,000 steps over 12–24 hours, so the model learns long-horizon optimization from real execution feedback.
(2) M1's methodological contribution lies in the coordinated design of training mechanisms for this challenge.
① Model-directed Interaction Harness. We design a dedicated Interaction Harness that gives the model control over experimental state management, solution branching, and optimization directions. This enables the model to autonomously organize open-ended, long-horizon experiments rather than relying on predefined workflows.
② Outcome-guided Hierarchical SFT. Through Hierarchical Agentic SFT, we use actual experimental outcomes to select valuable interaction trajectories and determine supervision boundaries, rather than treating complete trajectories as uniformly positive demonstrations.
③ Feedback-aligned Policy Optimization (WIRL). We organize the multi-turn decisions associated with an experimental outcome into a single optimization unit and combine trend signals, window-level length normalization, and asynchronous policy correction, allowing the model to continuously learn from intermediate experimental feedback.
These designs address environment organization, behavioral supervision, and policy optimization respectively, and together constitute a post-training approach for real-world MLE.
(3) Additional experiments further validate the effectiveness of key methodological designs.
① Effect of the Interaction Harness. We replace MLE teacher trajectories collected using M1 Harness with trajectories collected using Claude Code; the scientific teacher trajectories were already collected using Claude Code, and all other configurations are identical. Each resulting model is evaluated three times under both harnesses (Any Medal, mean ± SEM):
| MLE Trajectory Source | M1 Harness (%) | Claude Code (%) |
|---|---|---|
| Claude Code | 53.0 ± 4.0 | 56.1 ± 3.0 |
| M1 Harness (Ours) | 62.1 ± 1.5 | 60.6 ± 4.0 |
The model trained with M1 Harness trajectories outperforms the one trained with Claude Code trajectories under both evaluation harnesses, so the Harness design contributes to performance rather than merely providing an execution environment. Its performance under Claude Code (60.6%) is also close to that under M1 Harness (62.1%), indicating that the learned capabilities transfer to a general-purpose coding harness.
General-purpose coding harnesses do not distinguish experiments from other operations or record each experiment's outcome, whereas M1 Harness records them explicitly, and these records define the supervision boundaries in SFT and the execution windows in WIRL. The three designs are therefore tightly coupled.
Official Comment to Reviewer SQ4j (Part 3)
② Effect of Outcome-guided Supervision. We use the same raw teacher trajectories, initial checkpoint, and training configurations, removing only outcome-guided trajectory filtering and supervision boundary selection:
| Supervision Strategy | Any Medal (%) |
|---|---|
| Outcome-guided filtering and truncation (Ours) | 62.1 ± 1.5 |
| Without filtering and truncation | 56.1 ± 3.0 |
Removing outcome-guided filtering and truncation reduces performance by 6.0 points. Organizing long-horizon supervision by experimental outcomes therefore preserves effective exploration and failure recovery, and avoids treating subsequent degraded behavior as positive demonstrations.
③ Effect of WIRL. As shown in Table 4, under the same MLE-SFT initialization, trajectory-level GRPO and PPO-style reach 56.1% and 60.6% Any Medal, whereas WIRL reaches 72.7%. Removing the local trend reward or policy correction reduces it to 66.7% and 65.2%, respectively.
M1 also improves in other environments and tasks, including Claude Code, AIME scaffold optimization, and representative MLS-Bench-Lite algorithm optimization tasks, so the learned optimization capabilities are not limited to the training harness.
(4) M1 provides three generalizable insights for training long-horizon agents.
First, autonomous experiment organization can itself become a post-training objective in real-world MLE tasks. Models should learn not only to generate code or execute local operations, but also to maintain experimental states, explore alternative solutions, analyze execution feedback, and autonomously determine subsequent optimization directions throughout the complete experimentation process. This suggests that the ability to organize long-horizon tasks can be transferred from externally designed workflows into learnable strategies within the model itself.
Second, long-horizon trajectory supervision should not treat complete interactions as uniformly positive demonstrations. Instead, experimental outcomes should guide the selection of effective exploration, failure-recovery behaviors, and supervision boundaries. This principle is also relevant to scientific experimentation, algorithm optimization, and other tasks where verifiable feedback becomes available only after multiple rounds of interaction.
Third, the granularity of RL optimization should align with environmental feedback events when outcomes depend on multiple decisions, rather than being restricted to individual actions or complete trajectories. Rewards should reflect both the absolute quality of solutions and local optimization progress. Under asynchronous execution, window-level length normalization and policy-lag correction are also needed to control the effects of interaction units with different lengths and samples collected from outdated policies. These considerations provide a useful approach to policy optimization for other long-horizon agent tasks with costly execution feedback.
(5) The role of data and training effort.
We acknowledge that domain-specific task construction and staged training provide an important foundation (Table 3). Beyond it, the controlled experiments above show that the selection of supervision content, the intermediate feedback structure of the interaction environment, and the granularity of policy optimization each add further gains.
We again thank the reviewer for the constructive feedback.
References
[1] Liu, Zexi, et al. "ML-Agent: Reinforcing LLM agents for autonomous machine learning engineering." arXiv preprint arXiv:2505.23723 (2025).
[2] Zhou, Yuhang, et al. "Synthetic sandbox for training machine learning engineering agents." arXiv preprint arXiv:2604.04872 (2026).
[3] Yang, Junlin, et al. "Frontis-MA1: Training an AI4AI model towards recursive self-improvement in machine learning engineering." arXiv preprint arXiv:2607.28568 (2026).
Official Comment to Reviewer SQ4j (Part 4)
[W3] Necessity of the custom interaction harness.
We thank the reviewer for this question. We have made the following additions in the revised manuscript:
- Section 3.2: We further clarify how experiment-level outcomes recorded by the Harness determine the supervision boundaries in SFT and the execution windows in WIRL.
- Appendix D.1: We add an SFT comparison replacing M1 Harness MLE teacher trajectories with Claude Code trajectories (Table 12).
M1 Harness is designed for M1 and provides a unified experiment structure for trajectory collection, post-training, and evaluation; both outcome-guided SFT and WIRL build on it.
(1) Limitations of general-purpose harnesses and the key capabilities of M1 Harness. In general-purpose coding harnesses, an experiment is one of many shell commands: the harness does not distinguish experiments from other operations or record each experiment's outcome. Compaction and summarization ease the length limit, but experimental states remain text in the conversation, and a summarized history may lose the code and execution results of previous solutions.
M1 Harness instead turns candidate executions and submission updates into explicit events through execute_code and update_answer. Candidate code, execution logs, and metadata are stored in solution nodes forming an experiment tree by parent–child revision, so the model can retrieve any historical solution to compare alternatives, backtrack, or change direction, while a separate submission tool keeps exploration from overwriting the selected submission (Appendix B).
This difference also determines how effectively a model learns from the trajectories. Both harnesses compact history near the context limit, but M1 Harness keeps candidate code and execution logs in solution nodes independently of compaction. For a small model such as M1, the supervision signal therefore shifts from inferring over a long history which operations were effective to learning complete analysis–modification–execution–outcome units, making it easier to learn what to do at each optimization stage and which changes lead to improvement.
(2) Controlled SFT comparison using trajectories collected with Claude Code. To answer the training-stage question directly, we replace the MLE teacher trajectories collected with M1 Harness by Claude Code trajectories during MLE SFT, keep all other data and configurations unchanged, and evaluate each SFT model three times under both harnesses (Any Medal, mean ± SEM):
| MLE Trajectory Source | M1 Harness (%) | Claude Code (%) |
|---|---|---|
| Claude Code | 53.0 ± 4.0 | 56.1 ± 3.0 |
| M1 Harness (Ours) | 62.1 ± 1.5 | 60.6 ± 4.0 |
The model trained with M1 Harness trajectories performs better under both harnesses. Even when both collection and evaluation use Claude Code, its 56.1% remains below the 60.6% of the model trained with M1 Harness trajectories.
(3) Experiment-level outcomes provide execution windows for WIRL. WIRL treats the multi-turn decisions between two consecutive execute_code outcomes as an execution window and uses that outcome as a shared reward for the decisions within it; a completed window can be used immediately for asynchronous updates. With the same MLE-SFT initialization and training tasks, WIRL reaches 72.7% Any Medal against 56.1% for GRPO and 60.6% for PPO-style optimization, both trained on terminal rewards over complete trajectories (Table 4). Building these windows requires explicit records of experiment events and outcomes, which M1 Harness provides and general-purpose harnesses do not record.
(4) Complementary use of general-purpose harnesses. M1 also uses data from general-purpose environments: the general agent initialization stage includes Claude Code-style trajectories, and the scientific teacher trajectories are collected with Claude Code (Appendix C.1). They provide general programming, tool-use, and scientific problem-solving experience, complementing the MLE trajectories from the M1 Harness.
(5) Additional comparison on algorithm optimization tasks. We evaluate the same M1 model with both harnesses on 12 representative MLS-Bench-Lite tasks, one per domain:
| Evaluation Harness | Average Verifier Reward (Representative MLS-Bench-Lite Tasks) |
|---|---|
| Claude Code | 0.351 |
| M1 Harness (Ours) | 0.425 |
M1 Harness addresses these limitations, brings measurable gains under the same SFT configuration, and gives outcome-guided SFT and WIRL the experiment-level structure they require.
Official Comment to Reviewer SQ4j (Part 1)
[W1] Connection between window-level optimization and long-horizon decision making, and credit assignment for exploratory actions.
We thank the reviewer for raising this question. We have made the following additions to the revised manuscript:
WIRL decomposes long-horizon optimization into decision units bounded by experimental feedback and uses reward design and interaction history to align optimization across windows with the objective of the entire trajectory.
(1) Window boundaries align with the feedback structure of long-horizon MLE. In long-horizon MLE, the environment provides verifiable feedback only when an experiment finishes, and each experimental outcome depends on multiple preceding decisions involving analysis, modifications, and configuration. WIRL groups all decisions between two consecutive experimental outcomes into an execution window and assigns the outcome of the corresponding experiment as a shared reward. Exploratory decisions such as data analysis, feature exploration, and hypothesis testing typically occur before the next experiment is launched. They therefore belong to the same window as the experiment they support and receive credit from its outcome.
(2) Window rewards are aligned with the final objective. Window rewards are primarily determined by the absolute quality of the solution, using the same task-specific medal thresholds as the final evaluation. The reward tiers are −1.0, −0.6, −0.4, 1.0, and 2.0, while the local trend term Tm contributes only ±0.2 to reflect progress relative to the previous valid submission. Since the final submission is selected from the candidate solutions produced during experimentation, improving solution quality within individual windows is consistent with improving the final outcome. The trend term has a small magnitude: within the same quality tier, a temporary score decrease receives a reward only 0.2 lower than an unchanged score. Furthermore, executing an experiment does not change the current submission candidate, which is updated only when the model selects a node through
update_answer. Historical solutions are preserved in the experiment tree, so exploration does not overwrite the current submission.(3) Information from earlier experiments is not lost when a window ends. Window boundaries determine which decisions receive a reward, but do not interrupt the interaction context. The trajectory continues across windows, and information from previous experiments, including feature performance, rejected hypotheses, and analysis findings, is carried into subsequent windows. Therefore, when the value of an earlier exploratory action becomes apparent in a later experiment, the model can use the information retained from that exploration. The improved outcome in the later window is achieved using this information, and the corresponding reward is assigned to the subsequent decisions that use it.
(4) Empirical comparison. In Table 4, trajectory-level GRPO and PPO-style optimization assign the final trajectory outcome as a shared reward to all decisions along the trajectory. With the same MLE-SFT initialization and training tasks, and the shared optimization settings listed in Table 10, GRPO and PPO-style achieve Any Medal rates of 56.1% and 60.6%, respectively, both below the MLE-SFT starting point of 62.1%, whereas WIRL reaches 72.7%. Removing the trend term reduces WIRL's performance to 66.7%. These results indicate that, in long-horizon MLE, execution-window optimization based on completed experiments improves long-horizon decision-making more effectively than assigning a shared reward to the entire trajectory.
Explicit multi-window return estimation, such as discounted returns across windows or long-horizon value functions, could further assign credit to exploratory decisions whose benefits emerge over multiple experiments. We identify this as a direction for future work in Section 7.
We again thank the reviewer for the question, which has helped us clarify the connection between window-level optimization and long-horizon decision-making.