- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tries to address whether schema based tool invocations can be replaced by natural language interfaces for tool calling, using Tool Primitives, which wraps every tool with an LLM interface that internally handles the schema resolution and execution. This effectively moves the burden of input preparation from the general purpose agent to a tool-specific LLM interface that can be specialized for each tool. Additionally, the authors argue in favor of a system where the tools are retrieved at inference time instead of provided in the form of a descriptive large catalog. To that end, they introduce ToolFace (a tool repo with over 25k functions with schemas, implementation) and wrap every tool in the repo as a Tool primitive.
Further, they introduce HEART - a harness engineering framework via agent native reusable tool primitives. This is effectively a meta harness that takes a query, decomposes it into a plan, identifies missing information and asks the users for the same, grounds every step of the plan with actual values from the query or from prior turns, and creates a natural language request. This is then validated by the primitive, and HEART then validates each result against task completion, validity of the arguments and execution. It also uses feedback from failures to retry (within bounded limits). HEART ensures that only the tools that are needed are retrieved, and ensures that the tool primitives are provided with a crisp natural language input to actually do the work. The system uses qwen3 8b for the LLM (decomposition, value grounding, validation, tool primitives). This system is tested against 5 tool use benchmarks (ToolBench, NESTFUL, \tau^2 bench, ACEBench, BFCLv4). Tool retrieval is performed via semantic search on the descriptors.
The main results show that HEART does better than all commercial baselines as well as over finetuned baselines (albeit the models used here are older than the current set of models) on ToolBench. They also show convincing improvement over all models tested on NESTFUL, better performance on \tau^2 bench (with lower API cost - higher token count though) and also outperforms other compared models on BFCL (web search, memory) and ACEBench.
The paper includes ablations that help attribute the success rates to different components - and show that the toolface/tool primitives contribute the most (at the cost of an additional LLM call per tool use). Removing the validation step causes the next big drop, and the decompositon/resolution each cause a much more moderate drop, showing that every piece of the system is contributing to the increased success rate.
Additionally, the revision budget is an interesting piece of information - maximum performance is seen at 5 revisions, though the difference between 3 and 5 is minimal.
Finally they also test their system against 50 end to end real world tasks across 5 domains, and shows that their method achieves a much higher task completion rate over commercial large models (gpt, claude, gemini). An extra ablation to provide the baselines with the tools via function calling only raised the completion to at most 32% - proving that the tool availabilty is not the only contributing factor. The authors also do some analysis into prompt injection attacks, and show that they achieve 0% attack success rate.
Critical strengths
- The system is well defined, divides up the work logically and isolates the tool use problems from the task solving. This allows for better division of labor, and potentially less context junk in the main agent from logs from tool use.
- Good variety of benchmarks tested, plus additional real world tasks.
Critical weaknesses
-
Analysis on the component attribution is useful, but could be more detailed - removing the helpers also changes how the tools are provided to the agent. Attributing all of the difference to just the abstraction may not be prudent, and it would be good to separate the effect of tool search, tool access from the tool primitives themselves. For example, keep tool search, tool access constant - change ONLY whether the requests use plain language + tool primitive OR structured inputs. That would be a cleaner way to test if abstraction is the gain-causer. If this was the experiment that was run for the ablation already, it was not very clear what was being held constant.
-
The 50 task results are hard to verify, as we don't know what the success criteria are. Are they fixed? Were they vetted by human reviewers? Were there independent judges? Since this is such a strong claim, the validation criteria needs to be better explained.
-
The prompt injection prevention claims are hard to reconcile - while the attack surface doesn't exist in the LLM's input prompt, the attack surface still exists on the prompt for the tool primitives. The testing effectively tests - can an attacker make the agent choose the wrong tool; but it does not test - can an attacker make the agent use the wrong inputs? Saying zero attack success gives us a false sense of security - but not all attack modes were considered. This is more about scope of the testing - which is acknowledged to be limited in the appendix, but it would help to soften the claim even in the main paper.
Questions for authors
- On page 16, algorithm 1, I'm not entirely sure if I'm reading this right. The break on line 15 would break out of the inner for loop on validation failure, and we would still return r_k. Effectively, we almost always return r_k after the first iteration of the while loop. That seems.. wrong? When do we ever go into a second revision?
6 comments
W1. Analysis on the component attribution is useful, but could be more detailed - removing the helpers also changes how the tools are provided to the agent. Attributing all of the difference to just the abstraction may not be prudent, and it would be good to separate the effect of tool search, tool access from the tool primitives themselves. For example, keep tool search, tool access constant - change ONLY whether the requests use plain language + tool primitive OR structured inputs. That would be a cleaner way to test if abstraction is the gain-causer. If this was the experiment that was run for the ablation already, it was not very clear what was being held constant.
R1. Thank you for suggesting this comparison. In Table 6, we removed ToolFace and Tool Primitives together while keeping decomposition, value resolution, and validation in place. The drop from 75.1 to 16.1 on ToolBench therefore measures the combined effect of changing tool access and invocation. Following your suggestion, we ran an additional experiment on ToolBench and NESTFUL to examine the contribution of the invocation interface more directly.
We compared two ways of calling a retrieved tool. In both settings, value resolution could read the tool’s raw API schema. In the first (HEART w/ raw API schemas), it sent a natural-language request to the Tool Primitive, which constructed the arguments and called the tool. In the second (HEART w/ schema-based invocation), value resolution constructed the arguments and called the tool directly. Since direct invocation uses one fewer LLM call per step, we also tested a budget-matched variant that used the extra call to refine the arguments before execution.
All three variants used the same ToolFace repository, semantic retriever, backbone, decomposition and validation prompts, and revision budget (B = 3). Each replayed HEART’s first-pass plan with the same five candidate tools at each step. Revised steps could query the retriever again. Across all steps, 94.6% of candidate lists matched HEART’s exactly. The table reports the mean of three runs per setting.
| Variant | ToolBench Avg. Pass | NESTFUL Full Acc. |
|---|---|---|
| HEART | 75.1 | 0.44 |
| HEART w/ raw API schemas | 75.0 | 0.43 |
| HEART w/ schema-based invocation | 33.9 | 0.13 |
| HEART w/ schema-based invocation + self-refinement (budget-matched) | 44.6 | 0.17 |
| w/o ToolFace & Tool Primitives (Table 6) | 16.1 | 0.06 |
Giving value resolution access to raw schemas had little effect while it continued to call tools through the primitives. With the schemas visible in both settings, switching from natural-language requests to direct schema-based calls reduced ToolBench Avg. Pass from 75.0 to 33.9 and NESTFUL Full Accuracy from 0.43 to 0.13. Self-refinement recovered part of that loss, reaching 44.6 and 0.17 at HEART’s call budget. These results support a contribution from the invocation interface even when the systems use the same retriever and start with identical tool candidates. We report the new comparison in Appendix F.10, Table 19, and clarify in Sec. 4.3 which components the original Table 6 ablation removed.
W2. The 50 task results are hard to verify, as we don't know what the success criteria are. Are they fixed? Were they vetted by human reviewers? Were there independent judges? Since this is such a strong claim, the validation criteria needs to be better explained.
R2. Thank you for asking how we determined task success. Yes, the criteria are fixed, we reviewed them before grading, and two annotators judged every run independently. The original submission described the success criterion only briefly, in Appendix F.9, and noted in the Table 8 caption that human annotators judged the runs. Each task has written success criteria, available in the anonymous repository, and we checked before grading that they were reasonable and reflected what the user requested. The same criteria applied to every system. For a hotel booking, for example, the run had to leave a confirmed reservation in the final account state with the requested location, dates, number of guests, room type, and guest details, and finding a suitable hotel without booking it counted as a failure. For lookup tasks, the criteria specified the information the system had to retrieve. Each annotator labeled a run as successful only if every condition held. Table 8 covers 200 runs from HEART and the three frontier models, and Table 18 adds 150 runs from the same three models with access to ToolFace. The annotators agreed on 328 of the 350 runs (93.7%) and resolved the remaining 22 by discussion.
As an additional check, Claude-4.6-Sonnet and GPT-5.4 each evaluated all 350 trajectories using the task criteria, the trajectory, and the final account state.
| Judge | Agreement | Cohen's κ |
|---|---|---|
| Annotator 1 vs. Annotator 2 | 328/350 (93.7%) | 0.86 |
| Claude-4.6-Sonnet | 331/350 (94.6%) | 0.88 |
| GPT-5.4 | 327/350 (93.4%) | 0.85 |
Both LLM judges gave HEART the same completion rate as the human evaluation, 84% (42/50). Because both judges were also evaluated systems, we checked their agreement on their own trajectories. Claude-4.6-Sonnet matched the human labels on 95 of its 100 trajectories, and GPT-5.4 matched on 93 of 100. The tasks and tool lists are available in the anonymous repository. The revised manuscript describes the grading protocol in Appendix H.1 and reports judge agreement in Appendix H.2, Table 25.
W3. The prompt injection prevention claims are hard to reconcile - while the attack surface doesn't exist in the LLM's input prompt, the attack surface still exists on the prompt for the tool primitives. The testing effectively tests - can an attacker make the agent choose the wrong tool; but it does not test - can an attacker make the agent use the wrong inputs? Saying zero attack success gives us a false sense of security - but not all attack modes were considered. This is more about scope of the testing - which is acknowledged to be limited in the appendix, but it would help to soften the claim even in the main paper.
R3. Thank you for pointing out this. We have revised the main text and the Table 9 caption to state explicitly that the observed 0.0% attack success rate applies to tool selection under the ToolHijacker evaluation. That experiment measures whether a malicious tool document can cause HEART to select the wrong tool. It does not measure whether untrusted content can affect the arguments passed to a correctly selected tool. In particular, an earlier tool’s output can reach both value resolution and the receiving primitive, so the concern you raise also applies to HEART.
To examine this case, we evaluated HEART and the three frontier models on 629 AgentDojo security cases involving instructions embedded in tool outputs. The frontier models used native function calling. We also evaluated a version of HEART that records the source of each argument and rejects sensitive values supported only by tool outputs.
| Method | Utility, no attack (%) | Utility under attack (%) | ASR (%) |
|---|---|---|---|
| GPT-5.4 | 77.3 | 62.8 | 18.6 |
| Claude-4.6-Sonnet | 80.4 | 73.4 | 8.9 |
| Gemini-3.1-Pro | 72.2 | 57.9 | 21.5 |
| HEART | 76.3 | 70.6 | 6.4 |
| HEART + provenance check | 74.2 | 71.4 | 1.9 |
HEART’s ASR was 6.4% on these cases. Adding the provenance check reduced ASR to 1.9% and raised utility under attack from 70.6% to 71.4%, while utility without an attack fell from 76.3% to 74.2%. We now describe the ToolHijacker threat model alongside the original result in Sec. 4.4, qualify the corresponding claims in the Introduction, Sec. 4.5, and Conclusion, and report the AgentDojo results separately in Appendix F.13, Table 24.
Thank you very much for the careful and constructive review. We appreciate your recognition that the system is well defined, divides the work in a logical way and keeps tool-use problems apart from task solving, which can keep the caller's context cleaner, as well as the variety of benchmarks and the added real-world tasks. We are especially grateful for your close reading of Algorithm 1 and for the cleaner control you suggested for the component-contribution ablation, which we have now run. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of your comments below.
W0. The main results show that HEART does better than all commercial baselines as well as over finetuned baselines (albeit the models used here are older than the current set of models) on ToolBench.
R0. Thank you for pointing this out. We selected Vicuna, Alpaca, and ToolLLaMA for our ToolBench evaluation because they were used as baselines in the original ToolBench paper (Qin et al., 2023). Our submission also included more recent models fine-tuned for tool calling, such as xLAM-2 and ToolACE-2, in the NESTFUL and BFCLv4 experiments. Their ToolBench results, however, were missing from the original comparison.
To address this gap, we evaluated xLAM-2-8B-FC-R, xLAM-2-70B-FC-R, and ToolACE-2-Llama-3.1-8B on ToolBench. We also expanded the comparison to three open-weight models with native function calling, namely gpt-oss-120b, GLM-4.5-Air, and Kimi-K2-Instruct. We evaluated the added baselines under the same protocols as the other baselines in Tables 1, 2, and 5: DFSDT on all six ToolBench subsets, one-shot and three-shot ICL on NESTFUL, and both the Web Search and Memory categories of BFCLv4. For ToolACE-2-Llama-3.1-8B, GLM-4.5-Air, and Kimi-K2-Instruct, the BFCLv4 results are taken from the public BFCL V4 leaderboard, as for the other open-weight models in Table 5. The table summarizes the main metric of each benchmark.
On ToolBench, xLAM-2-70B-FC-R performs best among the three added fine-tuned models at 64.7 Avg. Pass, 10.4 points below HEART’s 75.1. The gap is smaller when we include the models with native function calling: Kimi-K2-Instruct reaches 70.6, within 4.5 points of HEART. Kimi-K2-Instruct is also the closest added baseline on NESTFUL (0.36 vs. 0.44), while gpt-oss-120b comes closest on BFCLv4 Web Search (77.50 vs. 86.00). We have incorporated these results into Tables 1, 2, and 5 and updated the baseline descriptions in Sec. 4.1.