- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces Lightweight Tool Probing (LTP), a framework designed to improve LLM agents' multi-turn tool-use capabilities by exposing them to actual tool responses (gathered in offline stage) rather than relying solely on tool documentation. The proposed method allows an agent to explore tools in an isolated sandbox before task execution, store observed responses in a reusable evidence bank, and retrieve relevant examples during subsequent tasks. The authors also introduce execution diagnostics to measure response, argument-constraint, and reference-binding coverage, aiming to better understand tool-use failures. Experiments across multiple LLMs and benchmarks, show improvements in task success with relatively low additional token costs. Furthermore, the authors demonstrate that trajectories generated using LTP can improve supervised fine-tuning performance across four student models, even without access to the evidence bank at inference time.
Critical strengths
-
Practical and efficient approach to improving tool use: The paper addresses a relevant challenge in multi-turn tool use and proposes a relatively simple approach that reuses observed tool responses without requiring parameter updates or repeated exploration during task execution. The results show meaningful improvements over plain tool calling with relatively low additional token costs, highlighting the practical potential of the method.
-
Promising experimental results across inference and fine-tuning: The authors evaluate LTP across multiple models and benchmarks, demonstrating improvements in task success beyond the primary experimental setting. Furthermore, the finding that LTP-generated trajectories improve supervised fine-tuning performance across four student models, even without retrieving evidence at inference time, is an interesting result that extends the potential usefulness of the approach beyond inference-time prompting.
Critical weaknesses
-
Limited novelty and insufficient differentiation from existing approaches: The proposed method appears closely related to prior work on reusable tool knowledge, including DRAFT, Play2Prompt, and ToolAtlas, as well as retrieval-based few-shot prompting. Although the authors discuss some of these approaches, the paper does not sufficiently establish what distinguishes LTP from them. In particular, the lack of controlled comparisons between raw tool responses and alternative representations, such as detailed tool descriptions or input-output demonstrations, makes it difficult to assess the novelty and necessity of the proposed method.
-
Insufficient ablations and experimental evidence to support the broader claims: The paper lacks systematic ablations of important components, including the probing budget, evidence quality, retrieval strategy, and context-size limits, making it unclear which factors drive the reported improvements. Additionally, while multiple models and benchmarks are considered, the main baseline comparisons are concentrated on one model and benchmark, and several results rely on single runs without statistical uncertainty estimates. These limitations are particularly consequential given the relatively small performance differences between LTP and the strongest baselines, weakening the evidence for the method's general effectiveness and superiority.
Questions for authors
-
Novelty and differentiation from existing approaches: What are the key methodological differences between LTP and existing approaches such as DRAFT, Play2Prompt, ToolAtlas, and retrieval-based few-shot prompting? In particular, is there evidence that retaining raw tool responses provides advantages over detailed tool descriptions, reusable tool-use instructions, or input-output demonstrations under comparable conditions?
-
Ablation studies and evidence quality: How sensitive is LTP's performance to its main design choices, including the probing budget, retrieval strategy, number of retrieved records, and context-size limit? Have the authors analyzed the quality, diversity, and coverage of the collected evidence and their relationship to downstream performance?
-
Scalability and preparation costs: How does LTP's performance and computational cost change with the number and complexity of available tools? Given the initial offline probing cost, how many subsequent tasks are needed for LTP to become more cost-effective than alternative approaches, and how sensitive is this trade-off to toolset size?
-
Statistical significance and generalizability: Are the reported improvements consistent across multiple inference runs or random seeds? In particular, is the 1.37 percentage-point improvement over the strongest baselines on BFCL statistically significant? To what extent do the authors expect the findings to generalize beyond the model-benchmark combinations evaluated in the paper?
-
Fairness of the SFT comparison: How do the authors distinguish improvements arising from higher-quality LTP-generated trajectories from those attributable to differences in trajectory length, tool-call structure, or the additional loss-masked evidence included during training? Have they considered comparisons with matched training-token budgets or LTP trajectories with the retrieved evidence removed?
-
Validity of the proposed explanation and diagnostics: What evidence supports the conclusion that the improvements specifically result from better understanding of tool responses rather than additional demonstrations or prompting effects? Given that the execution diagnostics are closely related to task-success requirements, how do the authors establish that these metrics capture meaningful improvements in tool-use behavior beyond their expected correlation with task success?
Thank you for the detailed responses and additional experiments. The new ablations, repeated evaluations, and SFT controls resolve most of my concerns. I still have reservations about the novelty relative to existing approaches, the advantage of raw responses over faithful summaries, and the limited scalability analysis and baseline comparisons beyond BFCL. Nevertheless, I appreciate the substantial effort to address the review, and the additional evidence strengthens the paper. In light of these improvements, I am raising my score.