acceptodds
Under review as a conference paper at ICLR 2027

Lightweight Tool Probing: Showing LLM Agents What Their Tools Return

Abstract

Tool documentation tells an agent how to call a tool, but only calling it shows what it returns. Multi-turn function calling runs on exactly this missing knowledge. Arguments must match the formats a tool expects, and identifiers returned by one call must feed the next. Existing remedies either keep the agent reading, rewriting the documentation into better text about the tool, or keep it paying, rediscovering tool behavior inside every task; and task success alone cannot show whether such guidance helps. We introduce Lightweight Tool Probing (LTP): before any task, a model probes a toolset once in an isolated sandbox and keeps the raw responses it observed in a frozen evidence bank; during a task, it retrieves at most three short response excerpts per step, while identifiers still come from the live environment. LTP needs no gold solutions, no task feedback, and no parameter updates. To locate where the evidence helps, we also introduce execution diagnostics that score how much of a task’s requirements an execution covers. On BFCL, LTP lifts DeepSeek-V4-Flash from 41.75% to 49.75% at 1.09 times the tokens of plain tool calling, outperforming all five baselines, the strongest of which need at least twice the tokens. The gain travels to GLM-5.3 and Qwen3.5-397B, and to ACEBench, AppWorld, and WorkBench. It is interpretable. The tasks LTP rescues gain argument-constraint coverage. Even its trajectories are better training data. They beat plain trajectories for all four student models, by up to 10.75 points. **Showing an agent what its tools actually return is a cheap complement to telling it how to call them.**

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, tool use, task-agnostic tool probing, training data generation, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zebzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces Lightweight Tool Probing (LTP), a framework designed to improve LLM agents' multi-turn tool-use capabilities by exposing them to actual tool responses (gathered in offline stage) rather than relying solely on tool documentation. The proposed method allows an agent to explore tools in an isolated sandbox before task execution, store observed responses in a reusable evidence bank, and retrieve relevant examples during subsequent tasks. The authors also introduce execution diagnostics to measure response, argument-constraint, and reference-binding coverage, aiming to better understand tool-use failures. Experiments across multiple LLMs and benchmarks, show improvements in task success with relatively low additional token costs. Furthermore, the authors demonstrate that trajectories generated using LTP can improve supervised fine-tuning performance across four student models, even without access to the evidence bank at inference time.

Critical strengths
  • Practical and efficient approach to improving tool use: The paper addresses a relevant challenge in multi-turn tool use and proposes a relatively simple approach that reuses observed tool responses without requiring parameter updates or repeated exploration during task execution. The results show meaningful improvements over plain tool calling with relatively low additional token costs, highlighting the practical potential of the method.

  • Promising experimental results across inference and fine-tuning: The authors evaluate LTP across multiple models and benchmarks, demonstrating improvements in task success beyond the primary experimental setting. Furthermore, the finding that LTP-generated trajectories improve supervised fine-tuning performance across four student models, even without retrieving evidence at inference time, is an interesting result that extends the potential usefulness of the approach beyond inference-time prompting.

Critical weaknesses
  • Limited novelty and insufficient differentiation from existing approaches: The proposed method appears closely related to prior work on reusable tool knowledge, including DRAFT, Play2Prompt, and ToolAtlas, as well as retrieval-based few-shot prompting. Although the authors discuss some of these approaches, the paper does not sufficiently establish what distinguishes LTP from them. In particular, the lack of controlled comparisons between raw tool responses and alternative representations, such as detailed tool descriptions or input-output demonstrations, makes it difficult to assess the novelty and necessity of the proposed method.

  • Insufficient ablations and experimental evidence to support the broader claims: The paper lacks systematic ablations of important components, including the probing budget, evidence quality, retrieval strategy, and context-size limits, making it unclear which factors drive the reported improvements. Additionally, while multiple models and benchmarks are considered, the main baseline comparisons are concentrated on one model and benchmark, and several results rely on single runs without statistical uncertainty estimates. These limitations are particularly consequential given the relatively small performance differences between LTP and the strongest baselines, weakening the evidence for the method's general effectiveness and superiority.

Questions for authors
  • Novelty and differentiation from existing approaches: What are the key methodological differences between LTP and existing approaches such as DRAFT, Play2Prompt, ToolAtlas, and retrieval-based few-shot prompting? In particular, is there evidence that retaining raw tool responses provides advantages over detailed tool descriptions, reusable tool-use instructions, or input-output demonstrations under comparable conditions?

  • Ablation studies and evidence quality: How sensitive is LTP's performance to its main design choices, including the probing budget, retrieval strategy, number of retrieved records, and context-size limit? Have the authors analyzed the quality, diversity, and coverage of the collected evidence and their relationship to downstream performance?

  • Scalability and preparation costs: How does LTP's performance and computational cost change with the number and complexity of available tools? Given the initial offline probing cost, how many subsequent tasks are needed for LTP to become more cost-effective than alternative approaches, and how sensitive is this trade-off to toolset size?

  • Statistical significance and generalizability: Are the reported improvements consistent across multiple inference runs or random seeds? In particular, is the 1.37 percentage-point improvement over the strongest baselines on BFCL statistically significant? To what extent do the authors expect the findings to generalize beyond the model-benchmark combinations evaluated in the paper?

  • Fairness of the SFT comparison: How do the authors distinguish improvements arising from higher-quality LTP-generated trajectories from those attributable to differences in trajectory length, tool-call structure, or the additional loss-masked evidence included during training? Have they considered comparisons with matched training-token budgets or LTP trajectories with the retrieved evidence removed?

  • Validity of the proposed explanation and diagnostics: What evidence supports the conclusion that the improvements specifically result from better understanding of tool responses rather than additional demonstrations or prompting effects? Given that the execution diagnostics are closely related to task-success requirements, how do the authors establish that these metrics capture meaningful improvements in tool-use behavior beyond their expected correlation with task success?

Reviewer_FoWsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates a practical problem in multi-turn tool use: LLM agents may know how to call an API from its documentation but still fail to use the API correctly because they do not understand the actual responses returned by previous calls.

The paper proposes Lightweight Tool Probing (LTP). Before solving tasks, an agent explores tools in an isolated sandbox and stores their raw responses in a reusable evidence bank. During task execution, it retrieves relevant response examples to guide tool calls, while obtaining state-dependent information from the live environment. The paper also introduces three diagnostic metrics: response coverage, argument-constraint coverage, and binding coverage.

Validation. The authors evaluate LTP on BFCL, ACEBench Agent, AppWorld Test, and WorkBench, across several LLMs. On BFCL, DeepSeek-V4-Flash improves from 41.75% to 49.75% success, with approximately 9% additional token cost.

The results are promising and suggest that reusable tool-response evidence can improve tool-use reliability without model updates. However, the paper does not yet establish clearly how much its gains come specifically from retaining raw responses, how it differs from closely related tool-exploration and memory methods, or how robust the gains are across repeated runs and changing environments.

Critical strengths

Practical problem and promising results. LTP addresses a real challenge in multi-turn tool use by allowing agents to reuse tool-response evidence. On BFCL, it improves DeepSeek-V4-Flash success from 41.75% to 49.75% with only about 9% additional token cost.

Broad initial evaluation and useful diagnostics. The paper evaluates multiple models and benchmarks, and proposes response, argument-constraint, and binding coverage metrics to help analyze execution failures. The SFT results also suggest that LTP-generated trajectories may provide useful training data.

Critical weaknesses
  1. The novelty of LTP is not yet sufficiently established. The paper acknowledges closely related work, particularly Play2Prompt and ToolAtlas, but does not clearly explain why its approach represents a substantial advance over them. Play2Prompt also interacts with tools to learn their behavior and reuses the resulting information to improve tool instructions, while ToolAtlas builds reusable tool-side memory through probing. Thus, the general idea of exploring tools and reusing the acquired knowledge is not new. LTP's use of raw tool responses instead of rewritten instructions or structured memory is a reasonable design choice, but the paper has not sufficiently demonstrated that this representation is the key source of its claimed gains. I encourage the authors to sharpen the novelty claim and provide a direct comparison with these closest methods, ideally using the same tools, tasks, and exploration and inference token budgets. An ablation comparing raw responses with summarized observations and tool-use examples would also help establish whether LTP's specific design offers a meaningful advantage over existing approaches.

  2. The experiments show promising results, especially the 8-point improvement on BFCL with only about 9% more tokens. The authors also test multiple models and three additional environments, which is a strength. However, several issues limit the strength of the contribution claims.

First, the experiments do not isolate the main source of the gains. LTP combines sandbox probing, raw-response storage, and budgeted retrieval, but there is no sufficiently controlled comparison showing that raw responses outperform summaries or structured tool-use examples under matched token budgets. Second, the main results rely on limited repeated-run evidence, making it difficult to assess the stability of the reported improvements, especially for models with smaller gains. Third, although results on ACEBench, AppWorld, and WorkBench are encouraging, the overall evaluation remains concentrated on BFCL and a limited set of models and environments. The generalization claim should therefore be narrowed or supported with broader tests.

The execution diagnostics provide useful evidence about trajectory quality, but the observed association between coverage and success does not establish that retrieved evidence causes the improvement. The SFT results are also promising, although they use only 127 task–teacher pairs that both methods successfully solve, limiting conclusions about the broader value of LTP-generated data. I encourage the authors to add matched-budget ablations, repeated runs with uncertainty estimates, and stronger tests of causal mechanism and generalization. These experiments would clarify which parts of LTP are essential and how robust its benefits are.

Questions for authors

Novelty: How does LTP substantially differ from the closest methods, particularly Play2Prompt and ToolAtlas? Can the authors clarify the specific technical contribution beyond probing tools and reusing observations?

Ablations: Can the authors compare raw responses against summarized observations and structured tool-use examples under the same probing and inference token budgets? This is important for establishing whether raw-response retention is essential.

Robustness: Can the authors provide repeated runs and confidence intervals or paired significance tests for the main results, particularly the smaller improvements on other models?

Generalization and cost: How does LTP perform when tool behavior or environment state changes, and how many tasks must reuse a bank before its upfront probing cost becomes worthwhile?

Diagnostics and SFT: Can the authors provide stronger evidence that retrieved evidence causes the observed coverage improvements, and test whether LTP-generated trajectories improve SFT beyond the 127 tasks that both methods already solve?

Reviewer_7HuXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Lightweight Tool Probing (LTP), a training-free framework designed to improve the tool-use capabilities of LLM agents by exposing them to actual tool responses before executing downstream tasks. The authors argue that conventional tool documentation primarily describes how tools should be invoked but often fails to provide sufficient information about their outputs, argument constraints, and dependencies between successive tool calls. To address this limitation, LTP allows an agent to explore a toolset in an isolated sandbox and store the observed responses in a reusable evidence bank. During task execution, relevant response excerpts are retrieved to guide tool selection and argument construction without requiring additional model training or task-specific feedback.

Critical strengths
  1. LTP provides a straightforward approach to improving tool-use performance without requiring additional model training, gold solutions, or task-specific feedback. The reusable evidence bank is an appealing design choice, as it allows agents to leverage previously observed tool behavior across different tasks.
  2. The authors evaluate LTP across several benchmarks and LLM backbones, comparing it against multiple existing approaches. The reported improvements are promising, particularly given the relatively low additional token consumption compared with the evaluated baselines.
  3. The proposed execution diagnostics go beyond final task success by examining response evidence, argument constraints, and reference binding. The analysis provides useful insights into how tool-response evidence may influence agent performance.
Critical weaknesses
  1. Limited methodological novelty. The core idea of probing tools, storing their responses, and retrieving relevant observations during subsequent tasks is conceptually close to existing approaches involving tool exploration, retrieval-augmented generation, and agent memory.
  2. Limited analysis of the probing and retrieval strategies. The effectiveness of LTP depends heavily on which tools are explored, which responses are retained, and how relevant evidence is retrieved during task execution. However, the paper provides limited evidence about the relative importance of these decisions.
  3. Insufficient evidence about the effectiveness of the proposed diagnostics. The execution diagnostics measure whether tool interactions satisfy annotated requirements, but their relationship to actual evidence utilization remains somewhat indirect. Although the authors report correlations between coverage metrics and task success, these results do not establish that the diagnostics accurately capture the underlying reasons for agent failures.
Questions for authors
  1. Novelty compared with existing tool exploration and memory methods.
  2. Impact of probing and retrieval strategies.

Comments on the paper

1 comment

Thank you for the detailed responses and additional experiments. The new ablations, repeated evaluations, and SFT controls resolve most of my concerns. I still have reservations about the novelty relative to existing approaches, the advantage of raw responses over faithful summaries, and the limited scalability analysis and baseline comparisons beyond BFCL. Nevertheless, I appreciate the substantial effort to address the review, and the additional evidence strengthens the paper. In light of these improvements, I am raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.