acceptodds
Under review as a conference paper at ICLR 2027

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

Abstract

We introduce ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. Its construction pipeline provides a systematic methodology for building tool-use environments: it converts stepwise solutions from the MATH dataset into reusable tools with descriptions and typed schemas, validates their behavior and usability, and applies human repair and authoring to unresolved cases. The human-authored ToolMATHHard split complements the automatic pipeline by recovering unresolved cases, extending evaluation beyond successful automatic trajectories and mitigating validation-related coverage bias. Gold tools and graded distractors model connected operations, overlapping functionality, and missing capabilities. Beyond final accuracy, ToolMATH emphasizes three evaluation axes: (1) Adaptability measures how much Gold-only success is retained when gold tools are replaced entirely by distractors; (2) Robustness measures success retention when distractors are added alongside gold tools; and (3) Tool Connectivity measures answer accuracy as a function of the maximum depth of verified tool-call dependencies. Behavior-conditioned metrics and trace-level failure analyses distinguish reliable tool use, tool avoidance, adaptive substitution, and failures under unreliable catalogs. Together, this construction methodology and these diagnostics provide a controlled testbed for studying how models adapt to changing tool availability, remain robust to distractors, and maintain correctness across connected tool-use trajectories.

Primary area:
datasets and benchmarks
Keywords:
Tool augmentation, Planning, Multi-tool

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QZCYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ToolMATH, a math-grounded benchmark for evaluating long-horizon tool use under different tool-catalog conditions. It converts MATH solutions into reusable tools and evaluates models under Gold-only, Gold-present, and Distractors-only settings. The paper further studies distractor similarity, tool-use behavior, dependency depth, and failure patterns, and introduces TOOLMATH-HARD as a more challenging human-authored split.

Critical strengths
  1. The Gold-only, Gold-present, and Distractors-only settings provide a clear way to study model behavior under different levels of tool availability and distraction.

  2. The benchmark construction is extensive, combining automatic tool generation and validation with equivalence screening, human repair, and the TOOLMATH-HARD split.

  3. The paper goes beyond final accuracy by analyzing tool-call behavior, dependency depth, and failure types, providing a more detailed view of how models use or avoid tools.

Critical weaknesses
  1. Many models can solve the tasks without calling tools, which makes it unclear how well the benchmark isolates long-horizon tool-use ability from general problem-solving ability. Including tasks where tool use is necessary would strengthen the evaluation.

  2. Although the paper provides behavior-conditioned analyses, the proposed metrics do not fully isolate tool-use capabilities. Adaptability may reflect no-tool reasoning rather than alternative tool use, while Robustness may remain high simply because a model avoids tools, as illustrated by GPT-5 achieving 99.2% accuracy under Gold-present with only a 5.1% tool-call rate. Tool Connectivity also depends on model-selected trajectories, making dependency depth difficult to compare across different tool-use strategies.

  3. The paper has several writing and presentation issues, including fragmented discussions and inconsistencies in reference metadata and formatting, which affect the overall clarity and presentation quality.

Questions for authors
  1. Can the authors evaluate on a subset where tool use is necessary to obtain the correct answer?

  2. For Adaptability and Robustness, can the authors clarify how much of the reported performance actually depends on successful tool use?

  3. Since Tool Connectivity uses model-generated dependency depth, how should it be compared across models with different tool-use strategies? Would controlling the required dependency depth provide a clearer evaluation?

4 comments

Thank you for your comments. We would first like to address errors in the previous version of the paper.

1. Corrections in the uploaded revision: We have changed the typo of Claude's Gold-present accuracy, and the figure-axis label for tool connectivy is max dependency depth, not the number of tool calls.

Furthermore, we present a new analysis to identify which individual tool calls actually participated to output the answer. These results are not yet incorporated into the manuscript, but we would value your perspective on this approach.

DAG construction

The original method linked earlier outputs to later inputs by rule-based equality, without an LLM judge. We now use gpt-6-luna to judge information use from the full solution trace.

  • Nodes: Each call occurrence is (tool name, actual input, observed output). Uses of the same tool are separate nodes. The final answer is an additional node.
  • Edges: For nn calls, examine n(n+1)2\frac{n(n+1)}{2} forward pairs. Connect an earlier call to a later call or the answer when its output directly supplies information used to form the target input or answer, including transformations. Equality or order alone is insufficient. Forward edges ensure a DAG.
  • Depth: Count tool-call nodes on the longest path ending at the answer, excluding the answer node. Other branches remain visible. Missing answers are isolated NULL nodes.

This methodology enables to show whether each tool is logically used to output the final answer, while also providing their depth. Human audit resulted that 675/692 (97.54%) pairwise edge decisions correct and 92/100 (92%) DAGs correct overall.

Examples of different tool trajectories in a same problem

Due to space limitations, please refer to our response to the above reviewers regarding the same point.

Paired behavior and correctness transitions

For the same 100 problems per model, we compare Gold-only with L1 distractor-only (k=5). G1 = no calls; G2 = answer-connected; G3 = calls without confirmed connection. Missing answers count as incorrect and belong to G1 or G3.

All 3x3 behavior counts follow: Each cell lists destinations G1,G2,G3, and columns give the starting group.

ModelFrom G1From G2From G3
GPT-4o-mini8,0,917,3,3114,1,17
Llama3-8B1,0,40,9,310,9,46
Qwen2.5-7B3,0,316,2,396,3,28

All 2x2 correctness counts follow: C = correct, W = incorrect, and columns give start/end outcomes.

ModelC/CC/WW/CW/W
GPT-4o-mini36221032
Llama3-8B223471
Qwen2.5-7B39151729

For GPT-4o-mini and Qwen2.5-7B, most initially answer-connected runs move to G1 or G3 and retains overall accuracy, while for Llama3-8B, they move to G3 and accuracy significantly decreases in distractor-only settings. The correctness table separately tracks retained successes, losses, and recoveries. Together they describe output-use changes alongside answer outcomes.

Comparability across models

For 2,0002,000 common answered problems under Gold + L3 distractors, we measure accuracy and mean depth separately by original MATH difficulty, thereby holding the problem set of each group fixed across models .

This table shows accuracy, shown in the format of percent (correct/total).

Model12345
GPT-4o-mini90.7 (244/269)82.7 (416/503)73.6 (362/492)61.7 (238/386)47.7 (167/350)
Llama3-8B62.5 (168/269)50.3 (253/503)40.9 (201/492)28.8 (111/386)14.9 (52/350)
Qwen2.5-7B87.4 (235/269)79.7 (401/503)67.7 (333/492)56.0 (216/386)40.6 (142/350)

This table shows depth, shown in the format of mean depth / mean depth excluding questions with 0 depth (the number of problems with non-zero depth). Zero depth means no answer-connected path.

Model12345
GPT-4o-mini1.01/1.34 (203)1.08/1.36 (399)1.12/1.49 (369)1.08/1.60 (260)0.94/1.60 (205)
Llama3-8B0.93/1.33 (189)1.00/1.41 (356)1.12/1.53 (359)1.09/1.58 (267)0.99/1.63 (212)
Qwen2.5-7B0.96/1.19 (217)1.02/1.30 (394)0.97/1.33 (359)0.88/1.39 (246)0.79/1.36 (204)

Accuracy declines with difficulty. At higher difficulties, nonzero depth generally rises while the fraction of answer-connected runs falls. The mean depth decreases across Levels 3, 4, and 5 for every model. This is since fewer connected trajectories pull down the mean even though the remaining paths tend to be deeper. These separate measurements on a fixed problem set reveal what a single depth curve would not.

We would welcome your perspective on what these DAGs reveal about how tool outputs enter solutions.

Thank you for the authors' efforts. However, the response has raised even more serious concerns. The substantial changes to both the definition of Tool Connectivity and the criteria for identifying dependency edges suggest more fundamental issues with the paper's methodology. These changes make it difficult to establish the validity of the original experimental results and conclusions. I will maintain my original rating and recommendation of rejection.

Thank you for raising the concern regarding methodological consistency, and we would like to further clarify the differences.

The new LLM-based DAG analysis is not a replacement for the original method. It is also not used to redefine the original experimental results reported in the paper.

In the original analysis, a deterministic rule, whether the earlier tool’s output matched a later tool’s input, was used to connect dependency edges between tool calls. The new analysis is a separate follow-up direction in which gpt6-luna judges whether an earlier output was logically used in a later input or in the final answer, which not only includes same values, but also includes cases involving transformed values. In other words, the original method provides an explicit, rule-based approximation of input-output reuse, whereas the new method attempts to broaden this criteria and capture semantic relationships as well.

Furthermore, the new analysis was introduced to investigate a question that the original experiments could not fully answer: whether a tool call actually contributes information to the final answer, rather than just being connected to another call or appearing in a successful trajectory. Its preliminary results suggest that this distinction can reveal behavioral differences hidden by aggregate accuracy and simple tool call counts.

We apologize that our previous response may have made these approaches appear to be changes to the existing experimental protocol in the paper. Our intention was not to retroactively alter the basis of our submitted results-but to share a potential direction for a subsequent study and obtain feedback on its value. Thanks again for the helpful feedback.

Thank you for the further clarification. I agree that the motivation of this paper is interesting and potentially valuable. However, in my view, the current version of the work still needs further improvement, particularly regarding the concerns I raised earlier.

I sincerely appreciate the authors' continued responses and efforts to clarify these issues. I hope the authors can further consider my previous comments, strengthen the methodology and experimental evaluation, and also further polish the writing and presentation of the paper. I believe these improvements would make the motivation and contributions of the work much clearer.

Thank you again for your time and effort.

Reviewer_4NtnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ToolMATH, a benchmark for diagnosing tool-use behavior under controlled changes in tool availability and distractor similarity. It converts MATH solution steps into executable tools with hidden implementations, producing a main split of 7,699 questions and 12,369 tools. A complementary HARD split contains 329 questions and 362 human-authored tools for cases unresolved by automatic construction and validation. The evaluation considers Gold-only, Gold-present, and Distractors-only catalogs, measuring answer accuracy, invocation behavior, success retention under catalog changes, and accuracy conditioned on observed tool-dependency depth.

Critical strengths
  1. The evaluation targets relevant challenges in tool use. Overlapping functionality, distracting alternatives, and unavailable intended tools are meaningful challenges beyond basic function calling. Controlling catalog composition, distractor similarity, and distractor count provides a useful experimental framework for studying these factors.
  2. The benchmark includes substantial construction and validation effort. The pipeline combines tool-level checks, question-level execution, human repair, and complementary human authoring. Mathematical tasks also provide objectively checkable answers.
  3. The behavioral analysis reveals differences hidden by aggregate accuracy. Models with similar answer accuracy exhibit substantially different invocation rates. The distinction between call count and verified output-to-input dependency depth is also useful. The paper explicitly acknowledges several limitations in interpreting these results, including self-selected invocation groups and the absence of causal guarantees from observed tool calls.
Critical weaknesses
  1. The benchmark does not sufficiently establish task dependence on tools, weakening its validity as an evaluation of long-horizon tool-use capability. Strong models achieve near-ceiling main-split accuracy while making few tool calls: GPT-5 obtains 99.2% accuracy with a 5.1% invocation rate under Gold-present in Table 1. Even on HARD, the same-question controls in Table 5 show no consistent benefit from providing tools. Claude achieves 93.92% without tools versus 89.97% with Gold-only tools; GPT-5 achieves 93.92% versus 90.88%; and Gemini achieves 93.31% versus 92.40%. Consequently, high success retention after removing or adding tools can largely reflect internal mathematical reasoning or tool avoidance. This leaves a gap between the benchmark’s demonstrated value as a behavioral diagnostic and its intended assessment of tool adaptation, robustness, and long-horizon execution. A task subset with verified dependence on tool-provided information or computation would strengthen the latter interpretation.
  2. The gold/distractor distinction is tied to a reference solution rather than task-level necessity. Section 3 defines gold tools as those intended for the reference computation and permits alternative solutions. Non-gold tools may therefore be useful components of a valid alternative solution, while gold tools may be unnecessary for answering the question. Removing gold tools does not establish that a required capability is missing, and success under Distractors-only does not by itself demonstrate adaptive tool substitution. The invocation-conditioned decomposition helps describe behavior, but even successful trajectories containing calls do not establish that those calls contributed to the answer. The evaluation needs a clearer separation of internal reasoning, useful alternative tool composition, and incidental tool invocation. A detailed human analysis of a subset of trajectories is needed to examine whether and how the tool calls contribute to successful solutions.
  3. These issues also raise concerns about the comparability of the experimental results and the interpretation of the analyses.
    • Tool-Acc and NoTool-Acc are not directly comparable measures of capability across models. These metrics condition on each model’s own invocation decisions, producing different question subsets and difficulty distributions. In Table 4, Claude’s HARD NoTool-Acc of 100% covers only 10 questions, whereas Gemini’s and GPT-5’s no-call groups contain 211 and 154 questions, respectively. Their conditional accuracies therefore cannot establish relative unaided reasoning or tool-use ability. Within a model, the difference between Tool-Acc and NoTool-Acc also does not estimate the benefit of tools. The paper acknowledges this selection issue, but the resulting metrics primarily support descriptive behavioral comparisons; capability comparisons require matched questions and controlled tool-access conditions.
    • Section 5.4 does not sufficiently explain the mechanisms behind shorter observed dependency chains. Depth bins contain model-selected trajectories, and different models can place different questions into the same bin. Within a model, lower accuracy at greater depth also mixes task difficulty with execution behavior. More importantly, GPT-5’s shorter observed curve could arise from selecting more efficient tools, completing intermediate computations internally, or bypassing tools altogether. These mechanisms have different implications for tool-use capability. Interpreting this pattern as a conservative policy that avoids long chains requires comparisons of trajectories for the same questions under the same tool catalogs, showing tool selection, internal computation, total call count, dependency depth, and the use of returned outputs.
  4. The HARD set selection process leaves its sources of difficulty and its coverage insufficiently characterized. HARD recovers cases unresolved by automatic construction and validation, whose failures can reflect unsuitable extracted tools or limitations of the validator models. Its construction also introduces separately human-authored tools. Thus, differences from the main split combine question selection, validator-dependent selection, and tool construction. Longer reference solutions and lower answer accuracy support greater empirical difficulty, but do not isolate difficulty in tool composition. The paper should report the size of the unresolved pool, how the final 329 questions were selected, the exclusion criteria, and a breakdown of the original failure causes.
Questions for authors
  1. Can you identify and evaluate a subset where successful completion demonstrably depends on tool outputs? How do the Adaptability, Robustness, and dependency-depth results change on that subset?
  2. Among successful Distractors-only trajectories, how often does success involve useful alternative tool composition, incidental calls, or entirely internal reasoning? Can a trace audit or tool-output intervention substantiate this distinction?
  3. Can you provide matched trajectory case studies explaining why stronger models exhibit shorter dependency chains? In particular, please consider conducting a human analysis of a sufficiently large subset to distinguish more efficient tool selection, internal completion of intermediate steps, and complete tool avoidance, and report how frequently each behavior occurs.
  4. How many questions remained unresolved after automatic validation and human repair, and how were the 329 HARD questions selected from that pool? What proportion originally failed because of tool construction problems versus unsuccessful use of valid tools?
  5. Table 4 reports Claude’s main accuracy as 97.8% on Gold-present configuration, whereas Table 1 reports 97.4%.
1 comment

Thank you for your comments. We would first like to address errors in the previous version of the paper.

1. Corrections in the uploaded revision: We have changed the typo of Claude's Gold-present accuracy, and the figure-axis label for tool connectivy is max dependency depth, not the number of tool calls.

2. ToolMATH-HARD selection: Of 5,801 unresolved questions, 400 were selected without a predefined selection criterion. Excluding 71 questions that are poorly suited to tool representation left 329.

Furthermore, we present a new analysis to identify which individual tool calls actually participated to output the answer. These results are not yet incorporated into the manuscript, but we would value your perspective on this approach.

DAG construction

The original method linked earlier outputs to later inputs by rule-based equality, without an LLM judge. We now use gpt-6-luna to judge information use from the full solution trace.

  • Nodes: Each call occurrence is (tool name, actual input, observed output). Uses of the same tool are separate nodes. The final answer is an additional node.
  • Edges: For nn calls, examine n(n+1)2\frac{n(n+1)}{2} forward pairs. Connect an earlier call to a later call or the answer when its output directly supplies information used to form the target input or answer, including transformations. Equality or order alone is insufficient. Forward edges ensure a DAG.
  • Depth: Count tool-call nodes on the longest path ending at the answer, excluding the answer node. Other branches remain visible. Missing answers are isolated NULL nodes.

This methodology enables to show whether each tool is logically used to output the final answer, while also providing their depth. Human audit resulted that 675/692 (97.54%) pairwise edge decisions correct and 92/100 (92%) DAGs correct overall.

Behavior among successful distractor-only runs

On the same 100 problems per model, we classify correct distractor-only runs (k=5) into the three groups below. Counts in each row sum to its correct total.

ModelLevelNo callsAnswer-connectedCalls, unconfirmedCorrect/total
GPT-4o-miniL12431946/100
GPT-4o-miniL320111849/100
Llama3-8BL10156/100
Llama3-8BL30538/100
Qwen2.5-7BL11813756/100
Qwen2.5-7BL313114064/100

For GPT-4o-mini and Qwen, most correct runs have no confirmed answer-connected calls. On the other hand, Llama's L3 successes more often contain connected calls while achieving far fewer successes overall, implying unrelated tool use in general.

Examples of different tool trajectories in a same problem

For C(5,1), Llama3-8B uses factorial_cl(5)=120, factorial_cl(1)=1, and factorial_cl(4)=24 under Gold-only, then computes 120/(1*24)=5. Under L3 distractor-only, combinatorial_choose(5,1)=5 supplies the answer. Although both DAGs have depth 1, three independent factorial outputs versus a direct combination result. This reveals a different division of computation between tools and the model's reasoning even when the depth itself is unchanged.

Comparability across models

For 2,0002,000 common answered problems under Gold + L3 distractors, we measure accuracy and mean depth separately by original MATH difficulty, thereby holding the problem set of each group fixed across models .

This table shows accuracy, shown in the format of percent (correct/total).

Model12345
GPT-4o-mini90.7 (244/269)82.7 (416/503)73.6 (362/492)61.7 (238/386)47.7 (167/350)
Llama3-8B62.5 (168/269)50.3 (253/503)40.9 (201/492)28.8 (111/386)14.9 (52/350)
Qwen2.5-7B87.4 (235/269)79.7 (401/503)67.7 (333/492)56.0 (216/386)40.6 (142/350)

This table shows depth, shown in the format of mean depth / mean depth excluding questions with 0 depth (the number of problems with non-zero depth). Zero depth means no answer-connected path.

Model12345
GPT-4o-mini1.01/1.34 (203)1.08/1.36 (399)1.12/1.49 (369)1.08/1.60 (260)0.94/1.60 (205)
Llama3-8B0.93/1.33 (189)1.00/1.41 (356)1.12/1.53 (359)1.09/1.58 (267)0.99/1.63 (212)
Qwen2.5-7B0.96/1.19 (217)1.02/1.30 (394)0.97/1.33 (359)0.88/1.39 (246)0.79/1.36 (204)

Accuracy declines with difficulty. At higher difficulties, nonzero depth generally rises while the fraction of answer-connected runs falls. The mean depth decreases across Levels 3, 4, and 5 for every model. This is since fewer connected trajectories pull down the mean even though the remaining paths tend to be deeper. These separate measurements on a fixed problem set reveal what a single depth curve would not.

We would welcome your perspective on what these DAGs reveal about how tool outputs affect the model's output on answers.

Reviewer_ES7wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ToolMATH is a mathematics-grounded benchmark intended to diagnose long-horizon tool use under controlled changes to the available tool catalog. It derives executable Python tools from MATH solutions, exposes their descriptions and typed input schemas while hiding implementations, and combines automated validation with human repair. The main split contains 7,699 questions and 12,369 tools; a complementary HARD split contains 329 questions and 362 human-authored tools. Evaluation varies gold-tool availability, distractor similarity, and catalog size. The paper reports answer accuracy, invocation-conditioned metrics, success retention under catalog changes, dependency-depth diagnostics, and trace-level failure analyses across six models, with additional controller comparisons.

Critical strengths
  1. The benchmark supports controlled interventions on the tool catalog for the same underlying problem. Gold-only, Gold-present, and Distractors-only conditions, together with nested distractor budgets and executable tools, offer a useful environment for studying sensitivity to tool availability and overlap.

  2. The paper provides relevant controls and acknowledges important interpretation limits. In particular, it distinguishes invocation-conditioned accuracy from causal tool benefit, and Table 5 compares tool-assisted and unaided performance on the same HARD questions. The human-authored HARD split also broadens coverage beyond successful automatic construction trajectories. These choices make the resource more useful than a benchmark reporting only aggregate task accuracy.

Critical weaknesses
  1. Unclear distinctive contribution. Although the paper defines several evaluation axes, it does not sufficiently establish what important limitation of existing benchmarks they collectively address or what substantive new understanding the resulting evaluations provide.

  2. Insufficient differentiation from related benchmarks. ToolHop [1] already evaluates multi-hop tool use with executable tools and verifiable answers, but is not discussed. Missing-tool evaluation is also covered by the cited FAIL-TaLMs work [2]. The added measurement value of combining mathematics with catalog interventions remains insufficiently demonstrated.

  3. Mathematical reasoning and tool-use capability remain difficult to separate. GPT-5 achieves 99.2% main-split accuracy while calling tools on only 3.9%/5.1% of Gold-only/Gold-present questions (Table 1). The HARD controls likewise show no consistent frontier-model benefit from tool access (Table 5). Such results can reveal avoidance or disruption, but do not establish that the diagnostics reliably distinguish tool-composition ability from unaided mathematical reasoning.

  4. Dependency verification is underspecified. Section 4.4.1 does not sufficiently explain how output-to-input dependency edges are verified, particularly for transformed outputs or coincidentally matching values. Without an operational procedure and validation, the reliability of the central dependency-depth metric is difficult to assess.

  5. Figure 2 has an ambiguous statistical basis. Its caption specifies maximum dependency depth, whereas the axis says "Tool Calls" and the title includes "With Dedup & Exclude Distractors." Per-depth sample counts and uncertainty are also missing. These issues make it difficult to assess what the curves measure and how reliable the estimates are, especially at greater depths.

[1] Junjie Ye et al. ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use. ACL 2025. [2] Eduardo Treviño et al. Benchmarking Failures in Tool-Augmented Language Models. NAACL 2025.

Questions for authors
  1. Contribution clarification: What is the single most important capability or failure mode that ToolMATH measures more reliably than existing multi-hop and missing-tool benchmarks? Please identify a concrete result demonstrating this added diagnostic value, including the distinction from ToolHop [1].

  2. Evidence clarification: Beyond the same-question controls in Table 5, what evidence supports interpreting the proposed diagnostics as measures of tool-composition capability? Which conclusions remain valid after accounting for differences in mathematical reasoning ability and self-selected tool use?

  3. Metric clarification: How are output-to-input dependency edges verified in practice, including transformed outputs and coincidentally matching values? Does Figure 2 actually use maximum verified dependency depth or a filtered count of tool calls? Please reconcile the plot labels with Section 4.4.1 and provide per-depth sample counts and validation of the dependency extraction.

[1] Junjie Ye et al. ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use. ACL 2025.

3 comments

Thank you for your comments. We would first like to address errors in the previous version of the paper.

1. Corrections in the uploaded revision: We have changed the typo of Claude's Gold-present accuracy, and the figure-axis label for tool connectivy is max dependency depth, not the number of tool calls.

Furthermore, we present a new analysis to identify which individual tool calls actually participated to output the answer. These results are not yet incorporated into the manuscript, but we would value your perspective on this approach.

DAG construction

The original method linked earlier outputs to later inputs by rule-based equality, without an LLM judge. We now use gpt-6-luna to judge information use from the full solution trace.

  • Nodes: Each call occurrence is (tool name, actual input, observed output). Uses of the same tool are separate nodes. The final answer is an additional node.
  • Edges: For nn calls, examine n(n+1)2\frac{n(n+1)}{2} forward pairs. Connect an earlier call to a later call or the answer when its output directly supplies information used to form the target input or answer, including transformations. Equality or order alone is insufficient. Forward edges ensure a DAG.
  • Depth: Count tool-call nodes on the longest path ending at the answer, excluding the answer node. Other branches remain visible. Missing answers are isolated NULL nodes.

This methodology enables to show whether each tool is logically used to output the final answer, while also providing their depth. Human audit resulted that 675/692 (97.54%) pairwise edge decisions correct and 92/100 (92%) DAGs correct overall.

Answer-connected calls across models

For 2,000 shared problems under Gold + L3 distractors:

ModelAnswer-connected calls / all calls
GPT-4o-mini2,519/5,565 (45.3%)
Llama3-8B2,377/7,033 (33.8%)
Qwen2.5-7B2,149/4,658 (46.1%)

The DAG identifies which outputs are used, a distinction that total call counts and final accuracy do not provide.

Examples of different tool trajectories in a same problem

For C(5,1), Llama3-8B uses factorial_cl(5)=120, factorial_cl(1)=1, and factorial_cl(4)=24 under Gold-only, then computes 120/(1*24)=5. Under L3 distractor-only, combinatorial_choose(5,1)=5 supplies the answer. Although both DAGs have depth 1, three independent factorial outputs versus a direct combination result. This reveals a different division of computation between tools and the model's reasoning even when the depth itself is unchanged.

Comparability across models

For 2,0002,000 common answered problems under Gold + L3 distractors, we measure accuracy and mean depth separately by original MATH difficulty, thereby holding the problem set of each group fixed across models .

This table shows accuracy, shown in the format of percent (correct/total).

Model12345
GPT-4o-mini90.7 (244/269)82.7 (416/503)73.6 (362/492)61.7 (238/386)47.7 (167/350)
Llama3-8B62.5 (168/269)50.3 (253/503)40.9 (201/492)28.8 (111/386)14.9 (52/350)
Qwen2.5-7B87.4 (235/269)79.7 (401/503)67.7 (333/492)56.0 (216/386)40.6 (142/350)

This table shows depth, shown in the format of mean depth / mean depth excluding questions with 0 depth (the number of problems with non-zero depth). Zero depth means no answer-connected path.

Model12345
GPT-4o-mini1.01/1.34 (203)1.08/1.36 (399)1.12/1.49 (369)1.08/1.60 (260)0.94/1.60 (205)
Llama3-8B0.93/1.33 (189)1.00/1.41 (356)1.12/1.53 (359)1.09/1.58 (267)0.99/1.63 (212)
Qwen2.5-7B0.96/1.19 (217)1.02/1.30 (394)0.97/1.33 (359)0.88/1.39 (246)0.79/1.36 (204)

Accuracy declines with difficulty. At higher difficulties, nonzero depth generally rises while the fraction of answer-connected runs falls. The mean depth decreases across Levels 3, 4, and 5 for every model. This is since fewer connected trajectories pull down the mean even though the remaining paths tend to be deeper. These separate measurements on a fixed problem set reveal what a single depth curve would not.

Relation to existing benchmarks

ToolHop studies executable multi-hop use; FAIL-TaLMs studies unavailable tools. Our proposed diagnostic can examine which tools actually leads to an answer and how catalog changes alter that computation.

We would welcome your perspective on what these DAGs reveal about how tool outputs enter solutions.

Thank you for the clarification and additional analysis. The answer-connected DAG definition and human audit alleviate my concerns about dependency measurement. I also recognize the distinction in focus: the new analysis traces how tool outputs enter the final answer, beyond reporting task success or total call counts.

However, my main concern remains the benchmark’s distinctive evaluation value. The response does not yet sufficiently establish what specific tool-use capability these diagnostics reliably assess or how the new results substantiate ToolMATH’s positioning as a long-horizon tool-use benchmark. It is also unclear which aspects of ToolMATH’s construction enable insights that could not be obtained by applying the same DAG analysis to existing benchmarks. The additional results provide finer descriptions of tool-use behavior, but their substantive implications for capability evaluation remain insufficiently demonstrated.

Therefore, while the response addresses some measurement concerns, it does not fully resolve my primary concern about the contribution. I maintain my original rating.

Thank you for clarifying the remaining concerns.

We believe that the key distinction for our work is to observe the change in tool-use behavior and to evaluate how models compose and reuse tool outputs under controlled changes to the tool environment.

ToolMATH provides reusable gold tools derived from mathematical solutions, together with distractors extracted from other questions whose availability and similarity with the gold tools can be systematically controlled. This allows to examine whether the logic leading to its answer changes, which may be filled by the model calling tools or fallback with logical reasoning, and also whether the tools called were actually used for the model's final answer.

For example, in our preliminary analysis, Qwen2.5-7B maintained nearly the same accuracy between Gold-only and Level-1 Distractors-only. However, the questions where the model had answer-connected tool calls substantially decreased (from 57 to 5), while questions involving tool calls with no connection to the final answer increased (from 37 to 70). Llama3-8B showed a different pattern: its accuracy dropped from 25% in Gold-only to 6% in Level-1 Distractors-only. Also, while 99 of the 100 questions still contained tool calls, 81 of them had no tools having logical connection to the final answer. These results suggest distinct behavioral patterns. Namely, Qwen2.5-7B can maintain its overall accuracy despite a substantial shift from answer-connected tool use, yet still tries to call tools that are not related to the final answer. On the other hand, Llama3-8B shows a more substantial decline in the task while continuing to invoke tools, though most of the tool calls were not logically used to output the final answer. Thus, the two models show different behavioral tendencies regarding tool invocation, which accuracy and simple tool call rate alone cannot distinguish.

We are currently seeing these preliminary results as an important direction for strengthening the benchmark's positioning, yet we don't mean that the current analysis in the paper has already resolved the concerns. However, we would appreciate your perspective on whether this clarification addresses them, and what point would be considered convincing. Thank you again for the helpful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.