- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces ToolMATH, a math-grounded benchmark for evaluating long-horizon tool use under different tool-catalog conditions. It converts MATH solutions into reusable tools and evaluates models under Gold-only, Gold-present, and Distractors-only settings. The paper further studies distractor similarity, tool-use behavior, dependency depth, and failure patterns, and introduces TOOLMATH-HARD as a more challenging human-authored split.
Critical strengths
-
The Gold-only, Gold-present, and Distractors-only settings provide a clear way to study model behavior under different levels of tool availability and distraction.
-
The benchmark construction is extensive, combining automatic tool generation and validation with equivalence screening, human repair, and the TOOLMATH-HARD split.
-
The paper goes beyond final accuracy by analyzing tool-call behavior, dependency depth, and failure types, providing a more detailed view of how models use or avoid tools.
Critical weaknesses
-
Many models can solve the tasks without calling tools, which makes it unclear how well the benchmark isolates long-horizon tool-use ability from general problem-solving ability. Including tasks where tool use is necessary would strengthen the evaluation.
-
Although the paper provides behavior-conditioned analyses, the proposed metrics do not fully isolate tool-use capabilities. Adaptability may reflect no-tool reasoning rather than alternative tool use, while Robustness may remain high simply because a model avoids tools, as illustrated by GPT-5 achieving 99.2% accuracy under Gold-present with only a 5.1% tool-call rate. Tool Connectivity also depends on model-selected trajectories, making dependency depth difficult to compare across different tool-use strategies.
-
The paper has several writing and presentation issues, including fragmented discussions and inconsistencies in reference metadata and formatting, which affect the overall clarity and presentation quality.
Questions for authors
-
Can the authors evaluate on a subset where tool use is necessary to obtain the correct answer?
-
For Adaptability and Robustness, can the authors clarify how much of the reported performance actually depends on successful tool use?
-
Since Tool Connectivity uses model-generated dependency depth, how should it be compared across models with different tool-use strategies? Would controlling the required dependency depth provide a clearer evaluation?
4 comments
Thank you for the authors' efforts. However, the response has raised even more serious concerns. The substantial changes to both the definition of Tool Connectivity and the criteria for identifying dependency edges suggest more fundamental issues with the paper's methodology. These changes make it difficult to establish the validity of the original experimental results and conclusions. I will maintain my original rating and recommendation of rejection.
Thank you for raising the concern regarding methodological consistency, and we would like to further clarify the differences.
The new LLM-based DAG analysis is not a replacement for the original method. It is also not used to redefine the original experimental results reported in the paper.
In the original analysis, a deterministic rule, whether the earlier tool’s output matched a later tool’s input, was used to connect dependency edges between tool calls. The new analysis is a separate follow-up direction in which gpt6-luna judges whether an earlier output was logically used in a later input or in the final answer, which not only includes same values, but also includes cases involving transformed values. In other words, the original method provides an explicit, rule-based approximation of input-output reuse, whereas the new method attempts to broaden this criteria and capture semantic relationships as well.
Furthermore, the new analysis was introduced to investigate a question that the original experiments could not fully answer: whether a tool call actually contributes information to the final answer, rather than just being connected to another call or appearing in a successful trajectory. Its preliminary results suggest that this distinction can reveal behavioral differences hidden by aggregate accuracy and simple tool call counts.
We apologize that our previous response may have made these approaches appear to be changes to the existing experimental protocol in the paper. Our intention was not to retroactively alter the basis of our submitted results-but to share a potential direction for a subsequent study and obtain feedback on its value. Thanks again for the helpful feedback.
Thank you for the further clarification. I agree that the motivation of this paper is interesting and potentially valuable. However, in my view, the current version of the work still needs further improvement, particularly regarding the concerns I raised earlier.
I sincerely appreciate the authors' continued responses and efforts to clarify these issues. I hope the authors can further consider my previous comments, strengthen the methodology and experimental evaluation, and also further polish the writing and presentation of the paper. I believe these improvements would make the motivation and contributions of the work much clearer.
Thank you again for your time and effort.
Thank you for your comments. We would first like to address errors in the previous version of the paper.
1. Corrections in the uploaded revision: We have changed the typo of Claude's Gold-present accuracy, and the figure-axis label for tool connectivy is max dependency depth, not the number of tool calls.
Furthermore, we present a new analysis to identify which individual tool calls actually participated to output the answer. These results are not yet incorporated into the manuscript, but we would value your perspective on this approach.
DAG construction
The original method linked earlier outputs to later inputs by rule-based equality, without an LLM judge. We now use gpt-6-luna to judge information use from the full solution trace.
This methodology enables to show whether each tool is logically used to output the final answer, while also providing their depth. Human audit resulted that 675/692 (97.54%) pairwise edge decisions correct and 92/100 (92%) DAGs correct overall.
Examples of different tool trajectories in a same problem
Due to space limitations, please refer to our response to the above reviewers regarding the same point.
Paired behavior and correctness transitions
For the same 100 problems per model, we compare Gold-only with L1 distractor-only (k=5). G1 = no calls; G2 = answer-connected; G3 = calls without confirmed connection. Missing answers count as incorrect and belong to G1 or G3.
All 3x3 behavior counts follow: Each cell lists destinations G1,G2,G3, and columns give the starting group.
All 2x2 correctness counts follow: C = correct, W = incorrect, and columns give start/end outcomes.
For GPT-4o-mini and Qwen2.5-7B, most initially answer-connected runs move to G1 or G3 and retains overall accuracy, while for Llama3-8B, they move to G3 and accuracy significantly decreases in distractor-only settings. The correctness table separately tracks retained successes, losses, and recoveries. Together they describe output-use changes alongside answer outcomes.
Comparability across models
For 2,000 common answered problems under Gold + L3 distractors, we measure accuracy and mean depth separately by original MATH difficulty, thereby holding the problem set of each group fixed across models .
This table shows accuracy, shown in the format of percent (correct/total).
This table shows depth, shown in the format of mean depth / mean depth excluding questions with 0 depth (the number of problems with non-zero depth). Zero depth means no answer-connected path.
Accuracy declines with difficulty. At higher difficulties, nonzero depth generally rises while the fraction of answer-connected runs falls. The mean depth decreases across Levels 3, 4, and 5 for every model. This is since fewer connected trajectories pull down the mean even though the remaining paths tend to be deeper. These separate measurements on a fixed problem set reveal what a single depth curve would not.
We would welcome your perspective on what these DAGs reveal about how tool outputs enter solutions.