- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies dependency-aware tool planning for LLM-based agents. Existing methods often rely on external tool graphs for retrieval or planning, while autoregressive tool generation may suffer from error accumulation when an incorrect tool is selected early in the sequence.
To address these issues, the authors propose GRAFT, which represents each tool as a dedicated token and uses graph-token edge reconstruction to learn directed tool dependencies within the LLM. The method further introduces on-policy tool context distillation, where a student generates tool sequences from user queries, while a frozen teacher receives additional information, including decomposed subtasks and ground-truth tool sequences, to provide step-level supervision.
The authors evaluate GRAFT on four tool-planning datasets using three LLM backbones. Results show consistent improvements in sequence accuracy and dependency-related metrics over existing baselines. An additional evaluation on τ²-bench Retail reports higher task completion rates than PLaG and ToolGen.
Critical strengths
-
S1: Method Design.
The idea of learning directed tool dependencies within the LLM is interesting and well motivated. The combination of graph-token edge reconstruction and on-policy distillation is reasonable, and the paper is clearly presented. Compared with methods that rely on external graph information, GRAFT can generate tool plans directly from user queries without requiring additional graph context at inference time. The ablation and link prediction experiments also provide evidence that graph supervision contributes to planning performance.
-
S2: Experimental Results and Efficiency.
The experiments are generally well designed and cover four tool-planning datasets and three LLM backbones. GRAFT consistently outperforms ToolGen in sequence accuracy, while reducing inference-time token consumption. The authors also provide ablation studies and additional analyses of learned tool dependencies and graph robustness. The evaluation on τ²-bench Retail is a useful addition, as it shows improvements in task completion beyond offline sequence matching.
Critical weaknesses
-
W1: Contribution of On-Policy Distillation.
While the ablation results show that distillation improves performance, the contribution of the on-policy strategy remains unclear. In Eq. (9), the teacher has access to both the subtasks and the ground-truth tool sequence, providing strong supervision. The current experiments do not sufficiently distinguish the benefit of on-policy training from that of privileged teacher supervision. More direct comparisons with alternative training strategies would strengthen the claimed contribution.
-
W2: Scope of Dependency-Aware Planning.
Appendix B states that the evaluation retains samples with chain-structured dependencies, which simplifies the original graph planning problem. Real workflows may involve multiple prerequisites, branching, or parallel tool calls. Although the τ²-bench results are encouraging, the evaluation is limited to one domain with 15 tools. It remains unclear how well GRAFT generalizes to more complex dependency structures.
Questions for authors
-
Q1: On-Policy Distillation.
The teacher has access to the ground-truth tool sequence during distillation. Could the authors compare OPD with off-policy distillation and budget-matched SFT? A comparison with gold-label supervision on student-generated prefixes, with appropriate handling of off-path states, would also help clarify the specific benefit of the proposed distillation strategy.
-
Q2: General Tool Dependencies.
Appendix B indicates that the evaluation mainly uses chain-structured tool sequences. How would GRAFT handle workflows involving multiple prerequisites, branching, or parallel tool calls? It would be helpful to clarify whether the current planning formulation can represent and enforce such dependencies.
-
Q3: Error Recovery.
Although the interactive evaluation includes replanning after execution failures, it does not isolate the model's ability to recover from incorrect tool prefixes. Could the authors evaluate GRAFT on deliberately corrupted prefixes and compare it with the SFT-only model? This would provide more direct evidence of the robustness gained from OPD.
-
Q4: Interactive Evaluation.
The interactive evaluation in Section 4.7 only compares against PLaG and ToolGen. Since other graph-based methods, such as GTool and GNN4Plan, are already included in the offline experiments, why are they not evaluated in the interactive setting? Including these baselines under the same executor setup would make the end-to-end results more convincing.
-
Q5: Code Availability and Reproducibility.
I was unable to access the anonymous code repository linked in the paper. Could the authors check the link and make the implementation available? While Section 4.1 describes the shared experimental setup, some details of how ToolGen was adapted to the current planning task remain unclear, particularly its training and decoding settings.
3 comments
Thank you for your sincere response and additional experiments. This rebuttal has addressed most of my concerns. Although generalization to branching and parallel dependencies remains insufficiently evaluated, I am satisfied overall and I will raise my rating.
We sincerely thank you for your constructive feedback and for recognizing our efforts by raising the score.
Thank you for your thoughtful review and useful suggestions.
W1 / Q1. Contribution of on-policy distillation.
We compare GRAFT with the SFT-only baseline and an off-policy distillation variant. The off-policy variant uses the same frozen teacher and privileged context as GRAFT, but replaces student-generated prefixes with gold prefixes. We also evaluate OPD without the gold tool sequence in the teacher prompt. Results on Llama-3.2-3B are:
Off-policy distillation outperforms SFT-only, showing the benefit of privileged teacher supervision. With identical teacher context, OPD further improves EM and ACPL. This supports learning from student-generated prefixes. OPD without gold tools also outperforms SFT-only, showing that the gains do not rely solely on the reference tool sequence.
W2 / Q2. General tool dependencies.
Our offline evaluation focuses on chain-structured tool sequences, as described in Appendix B. However, our interactive evaluation on τ²-bench Retail includes conditional workflows, where tool selection depends on order status, user confirmation, and execution feedback. GRAFT handles these scenarios through stepwise planning based on execution outcomes. The results are show in Table 3.
For branching dependencies, different branches can be serialized into valid tool sequences. However, a branching workflow may have multiple valid execution orders. Therefore, evaluating such workflows requires metrics that account for multiple valid execution paths. Our experiments across multiple benchmarks have demonstrated GRAFT's ability to model tool dependencies. We leave a systematic evaluation of branching dependencies to future work.
Q3. Recovery from incorrect tool prefixes.
We add a corrupted-prefix experiment. After generating the first tool token, we insert a random incorrect tool token and continue decoding. For each test sample, GRAFT and SFT-only receive the same injected token. We measure the exact match of the generated continuation against the ground-truth suffix (Suffix EM).
Under corrupted prefixes, GRAFT exceeds SFT-only by more than 4 points on both datasets in Suffix EM. This provides direct evidence of greater robustness to erroneous prefixes. Original EM and Suffix EM use different evaluation targets, so we report them separately.
Q4. Additional interactive baselines.
We now add GNN4Plan and GTool under the same interactive evaluation setup on τ²-bench Retail:
GRAFT achieves the best task completion rate and valid graph edge rate among the evaluated methods.
Q5. Code availability and reproducibility.
The anonymous repository link expired in early November. Access has now been restored.
For a fair comparison, we adapt ToolGen's tool-token generation approach to our tool-sequence planning task. Each tool is represented by a dedicated token, and the model is trained to generate an ordered tool sequence directly from the user query. This follows ToolGen's retrieval-oriented formulation but extends the output from individual tools to complete tool sequences. We do not use its end-to-end agent-tuning stage, which additionally models tool arguments and execution feedback. We will clarify this in the paper.