- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents ToolGraph, a tool-use agent harness that compiles dependencies between tools into a persistent, weighted graph using parameter matching and counterfactual probing. The graph provides action recommendations to frozen LLM planners and is refined using execution outcomes. The authors investigate three capabilities of the framework, including continual graph updates, test-time scaling through graph-guided action scoring, and transfer across disjoint tool catalogs. Experiments on ToolComp, τ²-Bench, and Toolathlon show improvements in task completion, with the LLM-based graph reader outperforming other graph-reading strategies.
Critical strengths
-
The paper introduces a unified framework that supports continual learning, test-time scaling, and cross-catalog transfer through a shared tool-dependency graph without modifying the underlying LLM planner.
-
The proposed counterfactual probing mechanism evaluates tool dependencies by comparing the information gained from chaining tools against using individual tools, providing evidence beyond simple parameter matching.
-
The paper provides extensive experiments across three benchmarks and three LLM backbones, including comparisons of four graph-reading strategies, inference cost analysis, and evaluations of continual graph updates and cross-catalog transfer.
Critical weaknesses
I appreciate the authors' efforts. However, I have several concerns that I believe should be addressed to strengthen the paper.
-
My main concern is the lack of a systematic study showing that incorrect tool selection or tool arguments are the primary causes of task failures. I believe that selecting an incorrect read tool or providing incorrect arguments to a read tool may not necessarily lead to task failure. In contrast, incorrect write operations and policy violations can have more significant consequences, particularly in environments such as τ-Bench, as discussed in [1]. Although the authors demonstrate that constructing ToolGraph improves task completion, these results do not establish that tool dependencies are the primary source of failures. I encourage the authors to conduct a comprehensive failure analysis to identify the dominant error types and clarify which failures ToolGraph actually mitigates.
-
The authors propose a method for constructing tool-dependency graphs. However, several previous studies [2–5] have also explored dependency-based graph construction, primarily for generating tasks and interactive environments. Although these works use dependency graphs for different downstream purposes, comparing their graph construction approaches with ToolGraph would help clarify the novelty of the proposed method and whether its construction strategy provides additional benefits.
-
Using a portion of the benchmark tasks for training raises concerns about potential data contamination and the interpretation of generalization results. Although τ²-Bench provides an official train/test split, the original paper does not establish this split as its evaluation protocol. The training tasks were originally introduced for exploring agent training rather than defining a standard evaluation split for τ²-Bench. I encourage the authors to use external training data [6-7] for their graph updates and evaluate on the complete τ²-Bench evaluation set. This would provide stronger evidence that the improvements generalize beyond the benchmark-specific training tasks.
-
Numerous training-based and training-free methods have been proposed to improve LLM agents' tool-use performance, but the paper lacks sufficient comparisons with these approaches. In particular, I suggest including EvoTool [8] and CAST [9] as additional baselines on τ²-Bench, where applicable. These comparisons would help clarify how ToolGraph performs relative to alternative tool-use improvement techniques and whether its graph-based guidance offers complementary benefits.
References:
[1] IRMA: How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on τ-Bench, https://arxiv.org/abs/2508.20931
[2] APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay, https://arxiv.org/abs/2504.03601
[3] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis, https://arxiv.org/abs/2601.05808
[4] VULCAN: Where Agents Learn by Living in Simulated Tool Environments, https://openreview.net/forum?id=Ht4HB4pIQq
[5] ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training, https://arxiv.org/abs/2602.06820
[6] Simia: https://huggingface.co/datasets/Simia-Agent/Simia-Tau-SFT-90k-Hermes
[7] AReaL: https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data
[8] EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection, https://arxiv.org/abs/2603.04900
[9] CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents, https://arxiv.org/abs/2608.30147
Questions for authors
Please read the weaknesses.
2 comments
My main concerns have been addressed. I am willing to increase my score.
W1: failure analysis. We analysed the 94 failures of our method (TG Q_llm) on τ², pooled over the three planners.
What the table shows:
What ToolGraph mitigates.
The revised paper adds this analysis as an appendix and uses it to explain the graph controls.
W2: APIGen-MT, EnvScaler, VULCAN, ScaleEnv. All four are now cited. They build tool-dependency graphs to synthesize tasks or environments for training, so the graph feeds a data generator. Ours is the artifact a frozen planner reads and updates at inference.
W3: splits and external data.
W4: EvoTool and CAST. Both are now cited. They train or evolve the planner, while ToolGraph changes only what a frozen planner reads, so they are complementary rather than competing. Combining them is a natural follow-up.