acceptodds
Under review as a conference paper at ICLR 2027

MLToolBench: Learning Tool-Augmented Agents for Machine Learning Development

Abstract

Machine learning development agents must connect local evidence to code changes and verify that apparent improvements are valid. General code execution exposes evidence, but does not determine what to inspect or how observations should guide subsequent actions. We introduce toolbench, a diagnostic interface layer over MLAgentBench, and train policies to use it through guided demonstrations, supervised fine-tuning, and reinforcement learning. During RL, outcome-based GRPO optimizes task success, while SPICE uses verified solutions as training-only context to reweight supervision on sampled tool-call tokens. This signal measures solution-conditioned likelihood support rather than causal credit. We train on 80 tasks and evaluate on 25 held-out in-domain and 10 out-of-domain tasks. Diagnostic access alone is insufficient: adding toolbench reduces unguided Gemini 3.6 Flash from 66.8% to 57.2% in-domain, whereas tools with guidance reach 74.0%. With toolbench, the complete SFT-plus-RL pipeline improves Qwen3-8B from 24.0%/13.0% to 52.8%/39.0% ID/OOD success, and Qwen3.5-35B-A3B from 34.8%/31.0% to 64.4%/49.0%. These results show that diagnostic interfaces become useful when paired with training that connects observations to interventions.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
coding agent, tool use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7s87OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MLToolBench, a benchmark and tool-augmented environment for training machine learning engineering agents to use diagnostic tools effectively. The authors construct a library of 67 diagnostic tools and propose SPICE, a turn-level reward mechanism that estimates action utility by comparing action likelihoods with and without privileged solution context. Experiments across two model sizes demonstrate substantial improvements in task success rates over SFT and outcome-based RL baselines.

Critical strengths
  1. The paper addresses a meaningful problem in MLE agent training. The paper identifies a practical gap between providing diagnostic tools and teaching agents to use them effectively. The proposed diagnostic interface covers diverse ML development scenarios and is motivated by recurring agent failures.

  2. The paper provides substantial empirical improvements with useful ablations. The results demonstrate strong gains across two model sizes, with additional evaluations on held-out tasks. The comparisons with outcome-only RL, TIPS, and shuffled SPICE rewards help isolate the contribution of turn-level feedback.

Critical weaknesses
  1. The SPICE reward may not accurately reflect the actual utility of diagnostic actions.

SPICE measures how much a verified solution context increases the likelihood of an action, but this does not necessarily reflect the action's causal contribution to task success. An action strongly associated with the final solution may not be the most useful diagnostic step during exploration. Can the authors validate whether SPICE scores correlate with actual action utility, for example through counterfactual interventions?

  1. The contribution of the tool library is difficult to disentangle from task-specific engineering.

The 67 diagnostic tools are constructed based on observed failure modes, potentially encoding substantial task-specific knowledge. It is therefore unclear whether the performance gains primarily come from learning generalizable diagnostic strategies or from the carefully designed tool library. How well does the approach generalize to unseen diagnostic tools or failure modes not covered by the library?

  1. The computational overhead of SPICE is not sufficiently characterized.

SPICE requires additional likelihood evaluations conditioned on verified solution summaries, introducing computational overhead beyond standard outcome-based RL. The reported improvements in training efficiency may therefore not translate into lower overall training costs. Could the authors provide a compute-matched comparison with outcome-only RL, accounting for reward computation and solution-bank construction?

Questions for authors
  1. The training and evaluation tasks are relatively limited, and the OOD evaluation remains within the same five ML domains. Could the authors evaluate on more diverse and realistic MLE tasks, such as MLE-Bench, to demonstrate generalization beyond the synthetic task distribution?

2.The reported evaluation results are based on multiple rollouts from the same trained checkpoint rather than independent training runs. Could the authors report results across multiple training seeds, particularly for OOD tasks, to establish the statistical robustness of the improvements?

4 comments

Thank you for your comments on diagnostic utility, generalization, computational cost, and robustness across training runs. We hope the following clarification and experiments can help to address your concerns.


Response to W1 - Masking high-scoring diagnostic information reduces task success more.

High-scoring SPICE calls were more useful according to independent labels, and masking their information reduced task success more. Under the SPICE policy, masking high-scoring returns reduced success by 26.1 percentage points, versus 5.0 for low-scoring returns; the labeling evaluation also showed that SPICE ranked useful calls more accurately than TIPS (see the revised manuscript, Appendices G.1 and G.6, Tables 14 and 21).

(1) Masking information from high-scoring calls reduces final success more. We grouped read-only calls from each policy by SPICE score before inspecting continuation outcomes. We excluded the evaluated trajectory from its reference bank. Each pair of continuations began from the same saved history and workspace, with the same seed, decoding settings, and remaining budget. Masking withheld measurements, conclusions, and suggestions while preserving the structure and execution status of the return. Later tools operated normally.

35B policySPICE score groupOriginal-return success (%)Masked-return success (%)Success drop pp [95% CI]
Outcome-only RLHigh51.947.2+4.6 [+1.0, +8.8]
Outcome-only RLLow46.347.2-0.9 [-3.1, 0.0]
SPICEHigh76.550.4+26.1 [+19.7, +33.1]
SPICELow63.058.0+5.0 [+1.8, +8.4]

Success drop subtracts masked-return success from original-return success in percentage points, before rounding. Brackets give 95% task-bootstrap confidence intervals. Both continuations incurred the call cost, so the contrast measures the value of the returned information (see the revised manuscript, Appendix G.6, Table 21; Appendix B.3, Table 6 for the original masking and paraphrase controls).

(2) SPICE ranks calls judged useful more reliably than TIPS. Both methods scored the same calls on 10 held-out ID and 10 held-out OOD tasks, using the same frozen 35B SFT checkpoint and five verified references per task. We evaluated 200 trajectories, randomly sampling three successful and two failed runs per task from each source: fresh SFT rollouts and historical Gemini 3.8 rollouts with guidance. Reference and evaluation trajectories were disjoint.

Gemini 3.8 Flash and Claude Opus 5 (via Claude Code) independently labeled calls as useful, redundant/irrelevant, or uncertain, based on the task, preceding history, and tool return. Judges were blinded to scores, references, future steps, and final outcomes. GPT-5 adjudicated disagreements; human annotators separately audited 20 trajectories under blinded conditions (see the revised manuscript, Appendix G.1).

SplitMethodAUROC [95% CI] ↑High–low usefulness gap pp [95% CI] ↑
IDTIPS0.512 [0.389, 0.625]+9.0 [-10.7, +28.2]
IDSPICE0.779 [0.747, 0.822]+34.8 [+22.6, +49.5]
OODTIPS0.496 [0.450, 0.588]-1.6 [-2.8, +2.1]
OODSPICE0.713 [0.689, 0.747]+16.7 [+7.2, +31.6]

AUROC (area under the ROC curve) is the probability that a randomly drawn useful call scores above a randomly drawn redundant or irrelevant call, counting a tie as half (0.5 is chance; 1.0 is perfect ranking). The high–low usefulness gap is the percentage of useful calls in the top score quartile minus that in the bottom quartile, in percentage points. Both metrics exclude uncertain labels (see the revised manuscript, Section 3.4 and Appendix G.1, Metrics, and Table 14 for aggregation and confidence-interval procedures).

SPICE supplements the terminal task-success reward with local feedback, so task completion remains part of the training objective (see the revised manuscript, Section 2.3).

Response to W2 - SPICE benefits from diagnostic tools absent during training.

We tested dependence on the original diagnostic library by replacing its endpoints while keeping the tasks fixed. SPICE still benefited from the replacement tools. We separately evaluated transfer to public research codebases.

(1) SPICE benefited from tools it had never seen during SFT or RL. We replaced the MLToolBench diagnostics with four from Evidently (v0.4.33). They report data drift, missing values, classification quality by feature, and probability calibration. We tested the frozen 35B SFT, outcome-only RL, TIPS, and SPICE policies on five existing tabular tasks spanning House Prices, Spaceship Titanic, two Titanic instances (one healthy), and Breast Cancer Wisconsin. We retained their data splits, starting code, metrics, and success criteria.

Each policy had either A, base actions only, B, base actions plus MLToolBench diagnostics, or C, base actions plus applicable Evidently diagnostics. We disabled the original diagnostic endpoints in C. Workspaces, data permissions, dependencies, decoding, seeds, and budgets were matched. We supplied no new training, demonstrations, task-specific hints, or solution summaries. With 10 rollouts per task, these evaluations comprised 600 runs, 50 per cell (see the revised manuscript, Appendix G.7, Table 22).

35B methodA: Base actions Success (%)B: +MLToolBench Success (%)C: +Evidently Success (%)C–A gain (pp)
SFT40.046.042.0+2.0
Outcome-only RL44.050.046.0+2.0
TIPS46.056.052.0+6.0
SPICE48.070.066.0+18.0

Each A/B/C cell reports the percentage of successful rollouts. B and C add diagnostics to the same base actions. C–A is the difference in percentage points (pp). On these five tasks, adding Evidently raised success by 18.0 points for SPICE, 6.0 for TIPS, and 2.0 each for SFT and outcome-only RL (see the revised manuscript, Appendix G.7, Table 22). Separately, on the full 25 ID and 10 OOD tasks, the original Qwen3.5-35B-A3B checkpoint reached 34.4%/29.0% ID/OOD success under A, 35.6%/31.0% under B, and 36.0%/31.0% under C (see the revised manuscript, Table 1 and Appendix G.7).

(2) The gain also appeared on existing research codebases. We evaluated the same checkpoints on MLS-Bench-Lite. Our response to Reviewer 7s87 (Q1) gives the results and explains which sources overlap with our training suite (see the revised manuscript, Section 3.3 and Appendix G.2, Table 15). This tests transfer to public research tasks. The evaluation did not assign a seen/unseen label to individual failure mechanisms.

Response to W3 - SPICE retains higher success under the same measured elapsed-time budget.

To account for reward computation and solution-bank preparation, we compared performance under the same measured elapsed-time budget on the same hardware. Including initial-summary generation and online scoring, SPICE reached 65.2% ID and 46.0% OOD success, versus 48.0% and 44.0% for outcome-only RL. Both methods accounted for shared SFT preparation consistently (see the revised manuscript, Appendix G.8, Table 24).

(1) SPICE retained higher success under the same elapsed-time budget. We used the cumulative time of 100 outcome-only RL updates as the budget, charging initial-summary generation and all online scoring and bank updates to SPICE. SPICE completed 92 RL updates. We chose the checkpoint by measured cumulative time and evaluated both methods on the same 25 ID and 10 OOD tasks, with 10 rollouts per task and matched evaluation settings.

35B methodRL updatesID success (%) ↑OOD success (%) ↑
Outcome-only RL10048.044.0
SPICE9265.246.0
SPICE (full training)10069.248.0

At the time limit, SPICE exceeded outcome-only RL by 17.2 percentage points on ID and 2.0 on OOD.

(2) Scoring and online bank updates added 9.4% to a fixed SPICE workload. All three 35B methods used the same SFT initialization, training configuration, and eight H200 GPUs with BF16 (see the revised manuscript, Appendix A.3, Table 3). The timings below average all 100 updates of each full run, rather than the prefix used to select the 92-update checkpoint. They include rollouts, environment execution, rewards, online bank updates where applicable, and policy optimization. We excluded startup, evaluation, and checkpoint saving. Each run retained the response lengths and tool calls its policy produced (see the revised manuscript, Appendix G.8, Table 23).

35B methodID / OOD success (%)Training time (s/step) ↓Auxiliary overhead (%) ↓
Outcome-only RL48.0 / 44.0142.40.0
TIPS55.2 / 45.0165.612.8
SPICE69.2 / 48.0163.29.4

To isolate auxiliary overhead, we timed each method with and without its auxiliary modules on its own fixed workload. Both timings used the same rollout batch, model and optimizer state, reference bank, and execution settings.

Overhead=(TwithTwithout−1)×100%.\mathrm{Overhead}=\left(\frac{T_{\mathrm{with}}}{T_{\mathrm{without}}}-1\right)\times100\%.

We restored the starting state between measurements and timed the complete workload so overlapping operations counted once. Unlike this paired profiling, the actual s/step comparison also reflects differences in the rollouts generated by each policy.

(3) Initial bank preparation took 3.2 seconds per trajectory on average. We reused verified trajectories from the shared SFT stage. The extra work was one Qwen3.5-35B-A3B inference per candidate to summarize its decisions (see the revised manuscript, Appendix G.8). During RL we sampled up to eight references per task per update. Subsequent likelihood scoring, verification, and summarization are included in step times (see the revised manuscript, Appendix D.2).

Response to Q1 - SPICE improves performance on the external MLS-Bench-Lite benchmark.

We evaluated transfer beyond our synthetic task suite with the 30-task MLS-Bench-Lite evaluation. SPICE scored higher than all four controls on this public benchmark, where agents improve algorithms or components in existing research codebases. Its 12 research areas include robotics, reinforcement learning, and optimization, which lie outside our original five domains. Other areas overlap. Some training templates also draw on MLS-Bench task sources. We evaluated the frozen checkpoints on the benchmark's own scaffolds (MLS-Bench).

We tested the original Qwen3.5-35B-A3B and four trained checkpoints (SFT, SFT + outcome-only RL, SFT + TIPS, and SFT + SPICE). All five used the same Harbor adapter, diagnostics, decoding settings, task-specific hardware, and five hours per task. We followed the benchmark's editable scopes, seed policies, verifier, and score aggregation. The evaluation used no additional fine-tuning, injected faults, or privileged solution summaries.

35B methodMLS-Bench-Lite score [95% CI] ↑
Original Qwen (no SFT or RL)12.8
SFT16.2 [14.6, 18.1]
SFT + outcome-only RL18.1 [14.9, 20.4]
SFT + TIPS19.2 [18.1, 21.2]
SFT + SPICE24.6 [21.3, 26.0]

On the official 0–100 scale, SFT + SPICE reached 24.6, 11.8 points above the original checkpoint and 8.4 above SFT. With a shared SFT initialization, SPICE scored 5.4 points above TIPS and 6.5 above outcome-only RL. Brackets give 95% confidence intervals for the trained checkpoints (see the revised manuscript, Section 3.3 and Appendix G.2, Table 15).


Response to Q2 - SPICE's advantage persists across three independent RL training seeds.

To check whether the OOD gain depends on one RL run, we trained Qwen3.5-35B-A3B SPICE and outcome-only RL with three independent RL seeds per method, starting from the same SFT checkpoint. The OOD advantage persisted across these runs. We retained the original run (seed 42) and added seeds 1337 and 2026.

Every run began from the same SFT checkpoint and used the same 80 training tasks, 100 RL updates, rollout budgets, tools, and optimizer settings. The seeds varied task order and rollout sampling. SPICE runs shared the initial bank and update rule, then updated the bank from their own histories. We evaluated the 100-update checkpoint from every run on the same 25 ID and 10 OOD tasks, with 10 rollouts per task, common evaluation seeds, and matched budgets (see the revised manuscript, Section 3.5 and Appendix G.9, Table 25).

35B methodSplitSeed 42 (%)Seed 1337 (%)Seed 2026 (%)Mean ± SD (%)
Outcome-only RLID48.046.848.847.9 ± 1.0
Outcome-only RLOOD44.042.043.043.0 ± 1.0
SPICEID69.268.068.868.7 ± 0.6
SPICEOOD48.047.049.048.0 ± 1.0

SPICE's mean advantage over the three RL seeds was 20.8 points on ID and 5.0 points on OOD. Each SPICE run exceeded all three outcome-only runs on both splits. On OOD, success ranged from 47–49% for SPICE and 42–44% for outcome-only RL. SD is the sample standard deviation of success rates across the three RL training seeds.

Reviewer_UFXKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to train machine learning agents to gather and use diagnostic evidence. The authors introduce MLToolBench, which adds 67 diagnostic tools and a task suite to the MLAgentBench framework. Their training pipeline starts with SFT on teacher demonstrations collected using solution-informed guidance, followed by reinforcement learning. The main method, SPICE, assigns intermediate rewards based on how a verified solution summary changes the likelihood of a sampled tool call.

The authors train Qwen3-8B and Qwen3.5-35B-A3B on 80 tasks and evaluate them on 25 in-domain and 10 out-of-domain tasks from their suite. The full pipeline improves task success, and matched RL comparisons favor SPICE over outcome-only rewards and TIPS. The gains are much larger in-domain than out-of-domain.

Critical strengths

The matched RL comparisons provide direct evidence that SPICE helps within the evaluated setting. The authors hold the SFT initialization, tool interface, and rollout budget fixed. For the 35B model, SPICE reaches 69.2% in-domain success, compared with 48.0% for outcome-only RL and 55.2% for TIPS. The observation-masking and score-shuffling experiments also test whether diagnostic information and the assignment of scores to actions matter. These controls support the method more convincingly than a comparison between the original model and the full training pipeline alone.

Critical weaknesses

The evaluation is limited to the authors’ own task suite. Many tasks contain deliberately configured preprocessing, data, or modeling problems, while the diagnostic tools were designed around recurring agent failures. The OOD split holds out data sources or prediction targets, but stays within the same five domains and can include familiar problem types. It is therefore unclear how well the gains would transfer to ML development tasks outside this setup. Results on an external agent benchmark or naturally occurring ML workflows would materially strengthen the paper’s practical contribution.

The paper does not sufficiently establish the advantage of the full training pipeline over simpler alternatives. A missing baseline is a closed-source model given a reusable diagnostic guide, built only from training tasks, together with MLToolBench. The guided Gemini experiment uses solution-informed hints, so it does not answer this question. Meanwhile, adding MLToolBench reduces the performance of unadapted closed-source models, without enough analysis to explain why. All RL experiments also start from an SFT checkpoint, leaving the need for the guided-demonstration cold start untested. These gaps do not invalidate the matched SPICE comparisons, but they make it difficult to judge how much of the proposed training pipeline is necessary.

Questions for authors

1.Clarification based on existing runs: What types of failures become more common when Gemini and Claude receive MLToolBench? Do the trajectories suggest unnecessary calls, budget exhaustion, poor tool selection, or incorrect use of diagnostic outputs? This would help explain when the tool library helps and when it interferes.

2.Suggested additional comparison: Could you evaluate Gemini or Claude with MLToolBench and a fixed, reusable diagnostic guide derived only from training tasks, without access to test-task reference solutions? This comparison would directly affect my assessment of the benefit of parameter training over providing general instructions for using the tools.

3.Clarification, with a suggested experiment if results are unavailable: Have you tried outcome-only RL or SPICE directly from the original Qwen checkpoints, without SFT? If so, please report the results. Otherwise, this comparison would help determine whether the guided-demonstration cold start is necessary. My question concerns the role of SFT in the pipeline, rather than the validity of the RL comparisons that already share the same initialization.

2 comments

Thank you for your suggestions on external evaluation and simpler alternatives to the training pipeline. We hope the following clarification and experiments can help to address your concerns.


Response to W1 - SPICE's gains extend to MLS-Bench-Lite.

We tested transfer beyond our configured suite on the 30-task MLS-Bench-Lite benchmark. SPICE scored higher than SFT, outcome-only RL, and TIPS. The benchmark uses existing research codebases across 12 areas, including robotics, reinforcement learning, and optimization outside our five domains. Some of our synthetic training templates draw on MLS-Bench task sources.

We evaluated the original Qwen3.5-35B-A3B and four trained checkpoints (SFT, SFT + outcome-only RL, SFT + TIPS, and SFT + SPICE). All five used the same Harbor adapter, diagnostic tools, decoding settings, task-specific hardware, and five hours per task. We used the official scaffolds, editable scopes, verifier, and score aggregation, with no additional fine-tuning, injected faults, or privileged solution summaries (MLS-Bench and Harbor protocol).

35B methodMLS-Bench-Lite score [95% CI] ↑
Original Qwen (no SFT or RL)12.8
SFT16.2 [14.6, 18.1]
SFT + outcome-only RL18.1 [14.9, 20.4]
SFT + TIPS19.2 [18.1, 21.2]
SFT + SPICE24.6 [21.3, 26.0]

On the official 0–100 scale, SFT + SPICE reached 24.6, up 11.8 points from the original checkpoint and 8.4 points from SFT. From the same SFT checkpoint, TIPS reached 19.2 and outcome-only RL reached 18.1. Brackets give 95% confidence intervals for the trained checkpoints (see Section 3.3 and Appendix G.2, Table 15).


Response to W2 and Q2 - Training the same 35B model outperforms the tested reusable guides.

We evaluated Gemini and Claude with MLToolBench and reusable guides built only from training tasks, without parameter updates or test-task solutions. Both models benefited. On the original Qwen 35B, the best guide reached 39.2% ID and 32.0% OOD success, while training the same backbone with SFT and SPICE reached 69.2% and 48.0%.

(1) Gemini's test performance is sensitive to the choice of guide. We built three versions from Gemini's Original-condition trajectories on the 80 ID training tasks. G1 described recurring diagnostic needs and their frequency. G2 added the tools and evidence to check. G3 added how to interpret findings, when to act on them, and when to stop checking. We built each version independently three times and froze each guide before testing. None used test-task solutions, task-specific hints, or revisions based on test results. We evaluated Gemini 3.6 Flash without parameter updates on 25 held-out ID and 10 OOD tasks, with 10 rollouts per task. The MLToolBench conditions shared tool descriptions, decoding settings, and budgets. Guide rows report mean success and sample standard deviation (SD) across the three independently constructed guides.

Gemini 3.6 Flash conditionID success (%)OOD success (%)
Original (without MLToolBench)65.252.0
MLToolBench61.246.0
MLToolBench + G163.7 ± 1.346.7 ± 2.1
MLToolBench + G267.6 ± 3.954.7 ± 2.5
MLToolBench + G368.4 ± 1.744.3 ± 2.1

The best-performing guide changed across splits: G3 gave the highest mean ID success, whereas G2 exceeded G3 on OOD by 10.3 percentage points before rounding. Thus, stronger ID performance from a guide did not carry over to OOD tasks (see the revised manuscript, Appendix G.3.2, Table 17). In a separate single-guide evaluation, Claude Sonnet 5 improved from 62.4% to 68.8% on ID and from 43.0% to 47.0% on OOD with MLToolBench (see the revised manuscript, Appendix G.3.1, Table 16, and Appendix H for the guide).

(2) Training yielded larger gains than prompting the same 35B backbone. We tested each ID-derived guide once on the original Qwen3.5-35B-A3B checkpoint, using MLToolBench and the same evaluation protocol.

35B conditionID success (%)OOD success (%)
Original checkpoint, no added guide35.631.0
Original checkpoint + G136.431.0
Original checkpoint + G239.232.0
Original checkpoint + G338.832.0
SFT + SPICE69.248.0

Training improved success over the strongest tested guide by 30.0 points on ID and 16.0 on OOD (see the revised manuscript, Appendix G.3.3, Table 18). The tested guides recovered only a small part of the improvement from training this backbone. This comparison measures the combined gain from SFT and RL; the comparisons between RL objectives start from the same SFT checkpoint.

Response to W2 and Q1 - Gemini misuses diagnostic flags, while Claude repeats checks without new evidence.

To understand why diagnostic access reduced success, we compared failures with and without MLToolBench. Output misuse became more common in Gemini, and unnecessary or repeated calls became more common in Claude. We reviewed 1,000 ID trajectories, 250 per model and tool condition across 25 tasks. We assigned each failed run one primary failure category and used all 250 runs as the denominator. We considered both standard tools and MLToolBench diagnostics and kept the other/uncertain category for cases with insufficient evidence (see the revised manuscript, Appendix G.4, Table 19).

Primary failure mode, n (%)Gemini OriginalGemini +MLToolBenchClaude OriginalClaude +MLToolBench
Unnecessary or repeated calls19 (7.6)25 (10.0)10 (4.0)33 (13.2)
Budget exhaustion19 (7.6)14 (5.6)23 (9.2)13 (5.2)
Inappropriate tool selection or invocation28 (11.2)16 (6.4)25 (10.0)19 (7.6)
Misinterpretation or misuse of outputs11 (4.4)31 (12.4)20 (8.0)21 (8.4)
Other / uncertain10 (4.0)11 (4.4)10 (4.0)8 (3.2)
All unsuccessful runs87 (34.8)97 (38.8)88 (35.2)94 (37.6)

A run could have several contributing factors. The table counts it once under its primary failure.

(1) Gemini acted on an outlier flag without establishing a data defect. Output misuse rose from 4.4% to 12.4%. In a Titanic run, the label-balance tool advised no resampling. The outlier tool flagged 23.7% of rows because the zero-inflated feature Parch had IQR bounds of [0,0]. Those bounds alone gave no reason to discard the flagged rows. Gemini removed training rows, added class weights, and changed the prediction threshold. Validation accuracy fell from 79.72% to 73.43% (see the revised manuscript, Appendix I.1).

(2) Claude delayed execution by checking an unchanged file. Unnecessary or repeated calls rose from 10/250 (4.0%) to 33/250 (13.2%). In a Spaceship Titanic run, Claude edited train.py at step 20. At step 21 it repeated a missing-value check on train.csv, expecting the preprocessing edit to change the report on the original file. After further checks and edits, Claude started execution at turn 29 with 135 seconds left. The run timed out before it produced a submission (see the revised manuscript, Appendix I.2).

Diagnostic access can hurt performance when agents act on flags without checking their relevance or repeat checks without obtaining new evidence. The failure counts show how frequently these patterns occur, and the cases illustrate how they disrupt subsequent decisions and task completion (see the revised manuscript, Appendix G.4, Table 19, and Appendix I).


Response to W2 and Q3 - RL works without SFT, but SFT improves success under the same RL-update budget.

RL improves the original checkpoints even without SFT, while guided-demonstration SFT further improves performance under the same RL-update budget. We tested both outcome-only RL and SPICE on both backbones. For 35B SPICE, SFT raises ID/OOD success from 58.8%/43.0% to 69.2%/48.0%, gains of 10.4/5.0 percentage points after the same 100 RL updates (see the revised manuscript, Appendix G.5, Table 20).

Within each backbone, we fixed the 80 training tasks, 100 RL updates, rollout budgets, optimizer settings, policy prompts, and tool interfaces. We evaluated all checkpoints with MLToolBench on 25 ID and 10 OOD tasks, with 10 rollouts per task (see the revised manuscript, Section 3.5 and Appendix G.5, Table 20).

Training conditionID success (%) ↑OOD success (%) ↑
Qwen3-8B; all conditions use MLToolBench at evaluation
No SFT, no RL24.820.0
Full SFT only38.432.0
Full SFT + outcome-only RL39.233.0
Full SFT + SPICE52.436.0
Direct outcome-only RL (no SFT)29.624.0
Direct SPICE (no SFT)43.631.0
Qwen3.5-35B-A3B; all conditions use MLToolBench at evaluation
No SFT, no RL35.631.0
Full SFT only44.441.0
Full SFT + outcome-only RL48.044.0
Full SFT + SPICE69.248.0
Direct outcome-only RL (no SFT)41.236.0
Direct SPICE (no SFT)58.843.0

For outcome-only RL, SFT increased ID/OOD success by 9.6/9.0 points for 8B and 6.8/8.0 points for 35B. For SPICE, the corresponding gains were 8.8/5.0 and 10.4/5.0 points.

The comparison holds the RL budget fixed. SFT adds a separate training stage. Direct SPICE used the same initial verified-solution bank, update rule, policy snapshots for scoring, and reward calibration, so it retained access to privileged references during RL.

Reviewer_8iz2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces MLToolBench, a diagnostic tool suite and training framework designed to improve the ability of large language model (LLM) agents to perform machine learning engineering (MLE) tasks. The authors argue that simply providing agents with diagnostic tools is insufficient, as agents must also learn when to invoke these tools and how to incorporate the resulting evidence into subsequent decisions. To address this challenge, the paper develops a library of 67 diagnostic tools covering data inspection, code verification, and experiment diagnosis across five ML domains. The proposed training pipeline combines guided supervised fine-tuning (SFT) with reinforcement learning (RL). In particular, the authors introduce Solution-Privileged Information Credit Estimation (SPICE), a turn-level reward mechanism that measures how privileged solution context changes the likelihood of sampled tool actions, providing intermediate feedback beyond terminal task rewards.

Critical strengths
  1. A comprehensive diagnostic tool suite. MLToolBench provides 67 executable diagnostic tools across five ML domains, including computer vision, NLP, tabular learning, time series, and graph learning. The tools cover different stages of ML development, such as data inspection, code verification, and experiment diagnosis. This provides a useful environment for studying tool-augmented ML agents.
  2. Promising empirical results and relatively comprehensive ablations. The proposed framework achieves substantial improvements across both evaluated Qwen backbones. The authors also compare SPICE against outcome-only RL and TIPS under matched training configurations. Additional experiments involving reward shuffling, diagnostic-observation masking, and tool-use statistics provide useful evidence about the contributions of different components.
Critical weaknesses
  1. Limited methodological novelty and insufficient justification of SPICE. The conceptual distinction from existing likelihood-based credit assignment and reward-shaping methods, particularly TIPS and related hindsight-based approaches, could be articulated more clearly.
  2. Insufficient evidence that SPICE learns genuinely effective diagnostic decisions. While the reported improvements are encouraging, the experiments do not conclusively establish that SPICE accurately assigns credit to useful diagnostic actions. The reward-shuffling experiment provides partial evidence, but it does not directly measure whether actions receiving higher SPICE scores are more beneficial to downstream decisions.
  3. Potential confounding factors in the training pipeline. The proposed framework combines multiple components, including solution-informed guidance, teacher-generated demonstrations, supervised fine-tuning, and reinforcement learning. Although the paper includes several ablations, the contribution of guidance quality is not fully separated from that of tool-use demonstrations.
Questions for authors

Could the authors further clarify the key methodological differences between SPICE and existing likelihood-based credit assignment methods, particularly TIPS?

2 comments

Thank you for your comments on SPICE's methodological contribution, diagnostic credit, and the role of guidance in training. We hope the following clarification and experiments can help to address your concerns.


Response to W1 and Q1 - SPICE uses long solution traces to assign credit to individual tool actions.

We study how smaller open-weight models learn to inspect ML environments and act on their findings through tools built from failure analysis, guided demonstrations, and RL. Within this approach, SPICE turns verified task-solution knowledge into signed, turn-level feedback for tool actions, alongside the final outcome reward.

(1) SPICE vs. TIPS: long solution traces for local action credit. SPICE uses a long solution trace to score a local tool action, so details of the decision process inform credit at the current turn. TIPS uses the final answer as its prediction target in its QA experiments (Section 3.1). In our length-normalized ML adaptation, a multi-step solution summary replaces this answer label. A diagnostic action may affect only a small part of the summary, so averaging likelihood changes across the entire reference can dilute its local signal. SPICE instead conditions on the summary and averages the score only over the current action's tokens.

Let hh be the pre-action history, aa the sampled tool call, oo its return, cc a verified solution summary, and qq a frozen checkpoint of the updated policy model. For one reference, the scores before reward calibration are

=\frac{1}{|c|} \left[\log q(c\mid h,a,o)-\log q(c\mid h)\right],$$ $$s_{\mathrm{SPICE}} =\frac{1}{|a|} \left[\log q(a\mid h,c)-\log q(a\mid h)\right].$$ The first is our length-normalized ML adaptation of TIPS. It asks whether the interaction makes the reference more likely. SPICE holds the history and sampled action fixed and asks how solution knowledge changes support for that action. Neither SPICE branch sees the current return or future steps. **SPICE's scoring formulation extends beyond diagnostic calls to code edits and execution** (see the revised manuscript, Section 2.3 and Appendix D.2). **(2) SPICE vs. hindsight-based methods: signed local rewards alongside terminal success.** An ML rollout can fail because of a later edit or execution error even after a useful diagnostic check. SPICE is designed to retain feedback on such intermediate actions. [Self-Distilled RLVR](https://arxiv.org/html/2604.03128v1#S4.SS1) uses positive weights to rescale outcome-derived token advantages, while [HCAPO](https://arxiv.org/html/2603.08754v1#S4.SS1) reweights future returns using outcome-conditioned ratios. SPICE instead converts the reference-induced likelihood change into an additive, signed reward at each tool turn. **A reference-supported diagnostic action can receive positive auxiliary reward even when the rollout fails**, while terminal success remains part of the combined training return. [SEED](https://arxiv.org/html/2607.14777v1#S3.SS3) also turns privileged guidance into a learning signal, but extracts that guidance from completed on-policy trajectories for token-level distillation. SPICE uses verified demonstrations or successful solutions from earlier RL iterations of the same task. These references score the sampled action in its pre-action history, without using the scored rollout's completion. Thus, existing task knowledge supplies local rewards for new attempts, rather than each attempt supplying its own hindsight context (see the revised manuscript, Section 2.3 and Appendix D.2). **(3) Empirical evidence.** **SPICE ranks calls labeled useful more reliably than TIPS:** AUROC is 0.779 vs. 0.512 on ID and 0.713 vs. 0.496 on OOD (see the revised manuscript, Appendix G.1, Table 14). **High-scoring SPICE calls provide information with greater downstream value:** under the SPICE policy, masking their diagnostic information reduces final success by 26.1 pp, versus 5.0 pp for low-scoring calls (see the revised manuscript, Appendix G.6, Table 21). Thank you for asking us to clarify SPICE's motivation and methodological contribution. We have made the rationale for using solution knowledge to score local tool actions explicit in the methodology, with a fuller comparison to TIPS in the appendix (see the revised manuscript, Section 2.3 and Appendix D.2). -----

Response to W2 - SPICE scores track diagnostic usefulness and downstream information value.

We added two direct tests of what a high SPICE score means: whether the call is useful, and whether its information affects final task success. SPICE ranks calls labeled useful more reliably than TIPS on both ID and OOD tasks. Under the SPICE policy, masking information from high-scoring calls reduces final success by 26.1 percentage points, compared with 5.0 for low-scoring calls.

(1) SPICE ranks calls judged useful more reliably than TIPS. Both methods scored the same calls on 10 held-out ID and 10 held-out OOD tasks with the same frozen 35B SFT checkpoint and five verified reference summaries per task. For each task, we sampled fresh SFT rollouts and historical rollouts from Gemini 3.8 with guidance. We randomly drew three successful and two failed trajectories from each source. These 200 evaluation trajectories were separate from the reference trajectories (see the revised manuscript, Appendix G.1).

SplitMethodAUROC [95% CI] ↑High–low usefulness gap pp [95% CI] ↑
IDTIPS0.512 [0.389, 0.625]+9.0 [-10.7, +28.2]
IDSPICE0.779 [0.747, 0.822]+34.8 [+22.6, +49.5]
OODTIPS0.496 [0.450, 0.588]-1.6 [-2.8, +2.1]
OODSPICE0.713 [0.689, 0.747]+16.7 [+7.2, +31.6]

AUROC (area under the ROC curve) is the probability that a randomly drawn useful call scores above a randomly drawn redundant or irrelevant call, counting a tie as half. An AUROC of 0.5 is chance and 1.0 is perfect ranking. The high–low usefulness gap is the percentage of useful calls in the top score quartile minus the percentage in the bottom quartile, in percentage points (pp). We excluded uncertain labels, computed the metrics within each task and trajectory source, and weighted sources and tasks equally. Brackets give 95% task-bootstrap confidence intervals (see the revised manuscript, Appendix G.1, Metrics, and Table 14). SPICE's high–low gap was positive on both splits.

Gemini 3.8 Flash and Claude Opus 5 (via Claude Code) independently labeled each call useful, redundant/irrelevant, or uncertain and cited evidence from the task, preceding history, and tool return. We withheld automatic scores, reference summaries, model identities, future steps, and final outcomes. GPT-5 adjudicated disagreements. In a separate blinded audit of 20 of the 200 trajectories, human labels agreed with Gemini on 92.5% of calls and with Claude on 94.0%. For the disputed calls that GPT-5 adjudicated, agreement with the human labels was 97.5%.

(2) Masking information from high-scoring calls reduces final success more. We grouped calls from the SPICE policy by score before inspecting outcomes. From the same saved history and workspace, we continued each run twice with the same seed, decoding settings, and remaining budget. One continuation received the actual return. The other saw the same structure and execution status with the diagnostic information masked. Both incurred the cost of the call.

With 119 calls per group, masking reduced success by 26.1 pp [19.7, 33.1] in the high-score group and 5.0 pp [1.8, 8.4] in the low-score group, relative to each group's unmasked success rate (see the revised manuscript, Appendix G.6, Table 21).


Response to W3 - Guidance improves SFT beyond tool-enabled demonstrations.

To separate collection-time guidance from tool-enabled demonstrations, we added the missing condition: the same teacher with MLToolBench, but without solution-informed guidance. Adding guidance under this comparison improved the SFT student's success by 3.2 percentage points on ID and 6.0 on OOD. We trained a Qwen3.5-35B-A3B student under each collection condition and evaluated it before RL. All students used MLToolBench on 25 ID and 10 OOD tasks, with 10 rollouts per task (see the revised manuscript, Section 3.5 and Appendix G.10, Table 26).

GroupDemonstration collectionID success (%)OOD success (%)
ANo MLToolBench, no guidance39.232.0
B (new)MLToolBench, no guidance41.235.0
CMLToolBench + solution-informed guidance44.441.0

Giving the teacher MLToolBench (A to B) improved student success by 2.0 points on ID and 3.0 on OOD. Adding guidance with the same tools (B to C) yielded a further 3.2 points on ID and 6.0 on OOD. We removed guidance from the student's inputs, so it reached the student through the demonstrations. All comparisons between RL objectives used the same SFT checkpoint. We report the separate initialization experiment in our response to Reviewer UFXK (W2 and Q3) (see the revised manuscript, Appendix G.5, Table 20).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.