- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MLToolBench, a benchmark and tool-augmented environment for training machine learning engineering agents to use diagnostic tools effectively. The authors construct a library of 67 diagnostic tools and propose SPICE, a turn-level reward mechanism that estimates action utility by comparing action likelihoods with and without privileged solution context. Experiments across two model sizes demonstrate substantial improvements in task success rates over SFT and outcome-based RL baselines.
Critical strengths
-
The paper addresses a meaningful problem in MLE agent training. The paper identifies a practical gap between providing diagnostic tools and teaching agents to use them effectively. The proposed diagnostic interface covers diverse ML development scenarios and is motivated by recurring agent failures.
-
The paper provides substantial empirical improvements with useful ablations. The results demonstrate strong gains across two model sizes, with additional evaluations on held-out tasks. The comparisons with outcome-only RL, TIPS, and shuffled SPICE rewards help isolate the contribution of turn-level feedback.
Critical weaknesses
- The SPICE reward may not accurately reflect the actual utility of diagnostic actions.
SPICE measures how much a verified solution context increases the likelihood of an action, but this does not necessarily reflect the action's causal contribution to task success. An action strongly associated with the final solution may not be the most useful diagnostic step during exploration. Can the authors validate whether SPICE scores correlate with actual action utility, for example through counterfactual interventions?
- The contribution of the tool library is difficult to disentangle from task-specific engineering.
The 67 diagnostic tools are constructed based on observed failure modes, potentially encoding substantial task-specific knowledge. It is therefore unclear whether the performance gains primarily come from learning generalizable diagnostic strategies or from the carefully designed tool library. How well does the approach generalize to unseen diagnostic tools or failure modes not covered by the library?
- The computational overhead of SPICE is not sufficiently characterized.
SPICE requires additional likelihood evaluations conditioned on verified solution summaries, introducing computational overhead beyond standard outcome-based RL. The reported improvements in training efficiency may therefore not translate into lower overall training costs. Could the authors provide a compute-matched comparison with outcome-only RL, accounting for reward computation and solution-bank construction?
Questions for authors
- The training and evaluation tasks are relatively limited, and the OOD evaluation remains within the same five ML domains. Could the authors evaluate on more diverse and realistic MLE tasks, such as MLE-Bench, to demonstrate generalization beyond the synthetic task distribution?
2.The reported evaluation results are based on multiple rollouts from the same trained checkpoint rather than independent training runs. Could the authors report results across multiple training seeds, particularly for OOD tasks, to establish the statistical robustness of the improvements?
4 comments
Response to W2 - SPICE benefits from diagnostic tools absent during training.
We tested dependence on the original diagnostic library by replacing its endpoints while keeping the tasks fixed. SPICE still benefited from the replacement tools. We separately evaluated transfer to public research codebases.
(1) SPICE benefited from tools it had never seen during SFT or RL. We replaced the MLToolBench diagnostics with four from Evidently (v0.4.33). They report data drift, missing values, classification quality by feature, and probability calibration. We tested the frozen 35B SFT, outcome-only RL, TIPS, and SPICE policies on five existing tabular tasks spanning House Prices, Spaceship Titanic, two Titanic instances (one healthy), and Breast Cancer Wisconsin. We retained their data splits, starting code, metrics, and success criteria.
Each policy had either A, base actions only, B, base actions plus MLToolBench diagnostics, or C, base actions plus applicable Evidently diagnostics. We disabled the original diagnostic endpoints in C. Workspaces, data permissions, dependencies, decoding, seeds, and budgets were matched. We supplied no new training, demonstrations, task-specific hints, or solution summaries. With 10 rollouts per task, these evaluations comprised 600 runs, 50 per cell (see the revised manuscript, Appendix G.7, Table 22).
| 35B method | A: Base actions Success (%) | B: +MLToolBench Success (%) | C: +Evidently Success (%) | C–A gain (pp) |
|---|---|---|---|---|
| SFT | 40.0 | 46.0 | 42.0 | +2.0 |
| Outcome-only RL | 44.0 | 50.0 | 46.0 | +2.0 |
| TIPS | 46.0 | 56.0 | 52.0 | +6.0 |
| SPICE | 48.0 | 70.0 | 66.0 | +18.0 |
Each A/B/C cell reports the percentage of successful rollouts. B and C add diagnostics to the same base actions. C–A is the difference in percentage points (pp). On these five tasks, adding Evidently raised success by 18.0 points for SPICE, 6.0 for TIPS, and 2.0 each for SFT and outcome-only RL (see the revised manuscript, Appendix G.7, Table 22). Separately, on the full 25 ID and 10 OOD tasks, the original Qwen3.5-35B-A3B checkpoint reached 34.4%/29.0% ID/OOD success under A, 35.6%/31.0% under B, and 36.0%/31.0% under C (see the revised manuscript, Table 1 and Appendix G.7).
(2) The gain also appeared on existing research codebases. We evaluated the same checkpoints on MLS-Bench-Lite. Our response to Reviewer 7s87 (Q1) gives the results and explains which sources overlap with our training suite (see the revised manuscript, Section 3.3 and Appendix G.2, Table 15). This tests transfer to public research tasks. The evaluation did not assign a seen/unseen label to individual failure mechanisms.
Response to W3 - SPICE retains higher success under the same measured elapsed-time budget.
To account for reward computation and solution-bank preparation, we compared performance under the same measured elapsed-time budget on the same hardware. Including initial-summary generation and online scoring, SPICE reached 65.2% ID and 46.0% OOD success, versus 48.0% and 44.0% for outcome-only RL. Both methods accounted for shared SFT preparation consistently (see the revised manuscript, Appendix G.8, Table 24).
(1) SPICE retained higher success under the same elapsed-time budget. We used the cumulative time of 100 outcome-only RL updates as the budget, charging initial-summary generation and all online scoring and bank updates to SPICE. SPICE completed 92 RL updates. We chose the checkpoint by measured cumulative time and evaluated both methods on the same 25 ID and 10 OOD tasks, with 10 rollouts per task and matched evaluation settings.
| 35B method | RL updates | ID success (%) ↑ | OOD success (%) ↑ |
|---|---|---|---|
| Outcome-only RL | 100 | 48.0 | 44.0 |
| SPICE | 92 | 65.2 | 46.0 |
| SPICE (full training) | 100 | 69.2 | 48.0 |
At the time limit, SPICE exceeded outcome-only RL by 17.2 percentage points on ID and 2.0 on OOD.
(2) Scoring and online bank updates added 9.4% to a fixed SPICE workload. All three 35B methods used the same SFT initialization, training configuration, and eight H200 GPUs with BF16 (see the revised manuscript, Appendix A.3, Table 3). The timings below average all 100 updates of each full run, rather than the prefix used to select the 92-update checkpoint. They include rollouts, environment execution, rewards, online bank updates where applicable, and policy optimization. We excluded startup, evaluation, and checkpoint saving. Each run retained the response lengths and tool calls its policy produced (see the revised manuscript, Appendix G.8, Table 23).
| 35B method | ID / OOD success (%) | Training time (s/step) ↓ | Auxiliary overhead (%) ↓ |
|---|---|---|---|
| Outcome-only RL | 48.0 / 44.0 | 142.4 | 0.0 |
| TIPS | 55.2 / 45.0 | 165.6 | 12.8 |
| SPICE | 69.2 / 48.0 | 163.2 | 9.4 |
To isolate auxiliary overhead, we timed each method with and without its auxiliary modules on its own fixed workload. Both timings used the same rollout batch, model and optimizer state, reference bank, and execution settings.
We restored the starting state between measurements and timed the complete workload so overlapping operations counted once. Unlike this paired profiling, the actual s/step comparison also reflects differences in the rollouts generated by each policy.
(3) Initial bank preparation took 3.2 seconds per trajectory on average. We reused verified trajectories from the shared SFT stage. The extra work was one Qwen3.5-35B-A3B inference per candidate to summarize its decisions (see the revised manuscript, Appendix G.8). During RL we sampled up to eight references per task per update. Subsequent likelihood scoring, verification, and summarization are included in step times (see the revised manuscript, Appendix D.2).
Response to Q1 - SPICE improves performance on the external MLS-Bench-Lite benchmark.
We evaluated transfer beyond our synthetic task suite with the 30-task MLS-Bench-Lite evaluation. SPICE scored higher than all four controls on this public benchmark, where agents improve algorithms or components in existing research codebases. Its 12 research areas include robotics, reinforcement learning, and optimization, which lie outside our original five domains. Other areas overlap. Some training templates also draw on MLS-Bench task sources. We evaluated the frozen checkpoints on the benchmark's own scaffolds (MLS-Bench).
We tested the original Qwen3.5-35B-A3B and four trained checkpoints (SFT, SFT + outcome-only RL, SFT + TIPS, and SFT + SPICE). All five used the same Harbor adapter, diagnostics, decoding settings, task-specific hardware, and five hours per task. We followed the benchmark's editable scopes, seed policies, verifier, and score aggregation. The evaluation used no additional fine-tuning, injected faults, or privileged solution summaries.
| 35B method | MLS-Bench-Lite score [95% CI] ↑ |
|---|---|
| Original Qwen (no SFT or RL) | 12.8 |
| SFT | 16.2 [14.6, 18.1] |
| SFT + outcome-only RL | 18.1 [14.9, 20.4] |
| SFT + TIPS | 19.2 [18.1, 21.2] |
| SFT + SPICE | 24.6 [21.3, 26.0] |
On the official 0–100 scale, SFT + SPICE reached 24.6, 11.8 points above the original checkpoint and 8.4 above SFT. With a shared SFT initialization, SPICE scored 5.4 points above TIPS and 6.5 above outcome-only RL. Brackets give 95% confidence intervals for the trained checkpoints (see the revised manuscript, Section 3.3 and Appendix G.2, Table 15).
Response to Q2 - SPICE's advantage persists across three independent RL training seeds.
To check whether the OOD gain depends on one RL run, we trained Qwen3.5-35B-A3B SPICE and outcome-only RL with three independent RL seeds per method, starting from the same SFT checkpoint. The OOD advantage persisted across these runs. We retained the original run (seed 42) and added seeds 1337 and 2026.
Every run began from the same SFT checkpoint and used the same 80 training tasks, 100 RL updates, rollout budgets, tools, and optimizer settings. The seeds varied task order and rollout sampling. SPICE runs shared the initial bank and update rule, then updated the bank from their own histories. We evaluated the 100-update checkpoint from every run on the same 25 ID and 10 OOD tasks, with 10 rollouts per task, common evaluation seeds, and matched budgets (see the revised manuscript, Section 3.5 and Appendix G.9, Table 25).
| 35B method | Split | Seed 42 (%) | Seed 1337 (%) | Seed 2026 (%) | Mean ± SD (%) |
|---|---|---|---|---|---|
| Outcome-only RL | ID | 48.0 | 46.8 | 48.8 | 47.9 ± 1.0 |
| Outcome-only RL | OOD | 44.0 | 42.0 | 43.0 | 43.0 ± 1.0 |
| SPICE | ID | 69.2 | 68.0 | 68.8 | 68.7 ± 0.6 |
| SPICE | OOD | 48.0 | 47.0 | 49.0 | 48.0 ± 1.0 |
SPICE's mean advantage over the three RL seeds was 20.8 points on ID and 5.0 points on OOD. Each SPICE run exceeded all three outcome-only runs on both splits. On OOD, success ranged from 47–49% for SPICE and 42–44% for outcome-only RL. SD is the sample standard deviation of success rates across the three RL training seeds.
Thank you for your comments on diagnostic utility, generalization, computational cost, and robustness across training runs. We hope the following clarification and experiments can help to address your concerns.
Response to W1 - Masking high-scoring diagnostic information reduces task success more.
High-scoring SPICE calls were more useful according to independent labels, and masking their information reduced task success more. Under the SPICE policy, masking high-scoring returns reduced success by 26.1 percentage points, versus 5.0 for low-scoring returns; the labeling evaluation also showed that SPICE ranked useful calls more accurately than TIPS (see the revised manuscript, Appendices G.1 and G.6, Tables 14 and 21).
(1) Masking information from high-scoring calls reduces final success more. We grouped read-only calls from each policy by SPICE score before inspecting continuation outcomes. We excluded the evaluated trajectory from its reference bank. Each pair of continuations began from the same saved history and workspace, with the same seed, decoding settings, and remaining budget. Masking withheld measurements, conclusions, and suggestions while preserving the structure and execution status of the return. Later tools operated normally.
Success drop subtracts masked-return success from original-return success in percentage points, before rounding. Brackets give 95% task-bootstrap confidence intervals. Both continuations incurred the call cost, so the contrast measures the value of the returned information (see the revised manuscript, Appendix G.6, Table 21; Appendix B.3, Table 6 for the original masking and paraphrase controls).
(2) SPICE ranks calls judged useful more reliably than TIPS. Both methods scored the same calls on 10 held-out ID and 10 held-out OOD tasks, using the same frozen 35B SFT checkpoint and five verified references per task. We evaluated 200 trajectories, randomly sampling three successful and two failed runs per task from each source: fresh SFT rollouts and historical Gemini 3.8 rollouts with guidance. Reference and evaluation trajectories were disjoint.
Gemini 3.8 Flash and Claude Opus 5 (via Claude Code) independently labeled calls as useful, redundant/irrelevant, or uncertain, based on the task, preceding history, and tool return. Judges were blinded to scores, references, future steps, and final outcomes. GPT-5 adjudicated disagreements; human annotators separately audited 20 trajectories under blinded conditions (see the revised manuscript, Appendix G.1).
AUROC (area under the ROC curve) is the probability that a randomly drawn useful call scores above a randomly drawn redundant or irrelevant call, counting a tie as half (0.5 is chance; 1.0 is perfect ranking). The high–low usefulness gap is the percentage of useful calls in the top score quartile minus that in the bottom quartile, in percentage points. Both metrics exclude uncertain labels (see the revised manuscript, Section 3.4 and Appendix G.1, Metrics, and Table 14 for aggregation and confidence-interval procedures).
SPICE supplements the terminal task-success reward with local feedback, so task completion remains part of the training objective (see the revised manuscript, Section 2.3).