acceptodds
Under review as a conference paper at ICLR 2027

Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives

Abstract

Large language models (LLMs) augmented with external tools have demonstrated remarkable capability in solving complex real-world tasks. Existing approaches remain brittle in multi-step and multi-turn settings, where the model must map one tool's output into the next tool's input schema, and degrade further as the tool catalogue grows. We introduce Tool Primitives, which replace schema-based invocation with a natural language interface: each tool is wrapped with an LLM that resolves its own schema and executes the underlying function, so callers describe what they want instead of emitting schema-compliant arguments, and dependent calls pass results to one another in natural language. Building on them, we host ToolFace, a repository of 25,519 functions from which only the tools retrieved for the current query enter context, bounding what the model reasons over by the retrieval budget rather than by the size of the repository. A primitive, however, answers only the request it is handed. We therefore propose HEART, a Harness Engineering framework via Agent-native, Reusable Tool Primitives, which decomposes a query into an invocation plan, grounds each step's arguments from the query and from earlier results, and validates every returned result, feeding failures back as targeted revisions. Experiments on five benchmarks demonstrate that HEART outperforms SFT-based models by 10% on average and surpasses GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro by 6% on average while reducing API cost by up to 85%. On 50 real-world tasks, HEART achieves 84% task completion, the average of three frontier commercial models (22%).

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Harness Engineering, LLM Tool Use, Tool Primitives

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XSnrOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to address whether schema based tool invocations can be replaced by natural language interfaces for tool calling, using Tool Primitives, which wraps every tool with an LLM interface that internally handles the schema resolution and execution. This effectively moves the burden of input preparation from the general purpose agent to a tool-specific LLM interface that can be specialized for each tool. Additionally, the authors argue in favor of a system where the tools are retrieved at inference time instead of provided in the form of a descriptive large catalog. To that end, they introduce ToolFace (a tool repo with over 25k functions with schemas, implementation) and wrap every tool in the repo as a Tool primitive.

Further, they introduce HEART - a harness engineering framework via agent native reusable tool primitives. This is effectively a meta harness that takes a query, decomposes it into a plan, identifies missing information and asks the users for the same, grounds every step of the plan with actual values from the query or from prior turns, and creates a natural language request. This is then validated by the primitive, and HEART then validates each result against task completion, validity of the arguments and execution. It also uses feedback from failures to retry (within bounded limits). HEART ensures that only the tools that are needed are retrieved, and ensures that the tool primitives are provided with a crisp natural language input to actually do the work. The system uses qwen3 8b for the LLM (decomposition, value grounding, validation, tool primitives). This system is tested against 5 tool use benchmarks (ToolBench, NESTFUL, \tau^2 bench, ACEBench, BFCLv4). Tool retrieval is performed via semantic search on the descriptors.

The main results show that HEART does better than all commercial baselines as well as over finetuned baselines (albeit the models used here are older than the current set of models) on ToolBench. They also show convincing improvement over all models tested on NESTFUL, better performance on \tau^2 bench (with lower API cost - higher token count though) and also outperforms other compared models on BFCL (web search, memory) and ACEBench.

The paper includes ablations that help attribute the success rates to different components - and show that the toolface/tool primitives contribute the most (at the cost of an additional LLM call per tool use). Removing the validation step causes the next big drop, and the decompositon/resolution each cause a much more moderate drop, showing that every piece of the system is contributing to the increased success rate.

Additionally, the revision budget is an interesting piece of information - maximum performance is seen at 5 revisions, though the difference between 3 and 5 is minimal.

Finally they also test their system against 50 end to end real world tasks across 5 domains, and shows that their method achieves a much higher task completion rate over commercial large models (gpt, claude, gemini). An extra ablation to provide the baselines with the tools via function calling only raised the completion to at most 32% - proving that the tool availabilty is not the only contributing factor. The authors also do some analysis into prompt injection attacks, and show that they achieve 0% attack success rate.

Critical strengths
  • The system is well defined, divides up the work logically and isolates the tool use problems from the task solving. This allows for better division of labor, and potentially less context junk in the main agent from logs from tool use.
  • Good variety of benchmarks tested, plus additional real world tasks.
Critical weaknesses
  • Analysis on the component attribution is useful, but could be more detailed - removing the helpers also changes how the tools are provided to the agent. Attributing all of the difference to just the abstraction may not be prudent, and it would be good to separate the effect of tool search, tool access from the tool primitives themselves. For example, keep tool search, tool access constant - change ONLY whether the requests use plain language + tool primitive OR structured inputs. That would be a cleaner way to test if abstraction is the gain-causer. If this was the experiment that was run for the ablation already, it was not very clear what was being held constant.

  • The 50 task results are hard to verify, as we don't know what the success criteria are. Are they fixed? Were they vetted by human reviewers? Were there independent judges? Since this is such a strong claim, the validation criteria needs to be better explained.

  • The prompt injection prevention claims are hard to reconcile - while the attack surface doesn't exist in the LLM's input prompt, the attack surface still exists on the prompt for the tool primitives. The testing effectively tests - can an attacker make the agent choose the wrong tool; but it does not test - can an attacker make the agent use the wrong inputs? Saying zero attack success gives us a false sense of security - but not all attack modes were considered. This is more about scope of the testing - which is acknowledged to be limited in the appendix, but it would help to soften the claim even in the main paper.

Questions for authors
  • On page 16, algorithm 1, I'm not entirely sure if I'm reading this right. The break on line 15 would break out of the inner for loop on validation failure, and we would still return r_k. Effectively, we almost always return r_k after the first iteration of the while loop. That seems.. wrong? When do we ever go into a second revision?
6 comments

Thank you very much for the careful and constructive review. We appreciate your recognition that the system is well defined, divides the work in a logical way and keeps tool-use problems apart from task solving, which can keep the caller's context cleaner, as well as the variety of benchmarks and the added real-world tasks. We are especially grateful for your close reading of Algorithm 1 and for the cleaner control you suggested for the component-contribution ablation, which we have now run. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of your comments below.


W0. The main results show that HEART does better than all commercial baselines as well as over finetuned baselines (albeit the models used here are older than the current set of models) on ToolBench.
R0. Thank you for pointing this out. We selected Vicuna, Alpaca, and ToolLLaMA for our ToolBench evaluation because they were used as baselines in the original ToolBench paper (Qin et al., 2023). Our submission also included more recent models fine-tuned for tool calling, such as xLAM-2 and ToolACE-2, in the NESTFUL and BFCLv4 experiments. Their ToolBench results, however, were missing from the original comparison.

To address this gap, we evaluated xLAM-2-8B-FC-R, xLAM-2-70B-FC-R, and ToolACE-2-Llama-3.1-8B on ToolBench. We also expanded the comparison to three open-weight models with native function calling, namely gpt-oss-120b, GLM-4.5-Air, and Kimi-K2-Instruct. We evaluated the added baselines under the same protocols as the other baselines in Tables 1, 2, and 5: DFSDT on all six ToolBench subsets, one-shot and three-shot ICL on NESTFUL, and both the Web Search and Memory categories of BFCLv4. For ToolACE-2-Llama-3.1-8B, GLM-4.5-Air, and Kimi-K2-Instruct, the BFCLv4 results are taken from the public BFCL V4 leaderboard, as for the other open-weight models in Table 5. The table summarizes the main metric of each benchmark.

ModelToolBench Avg. PassNESTFUL one-shot Full Acc.BFCLv4 Web Search Overall
xLAM-2-8B-FC-R58.40.006.50
xLAM-2-70B-FC-R64.70.0415.00
ToolACE-2-Llama-3.1-8B55.90.008.50
gpt-oss-120b67.90.3377.50
GLM-4.5-Air65.30.2973.50
Kimi-K2-Instruct70.60.3666.50
HEART75.10.4486.00

On ToolBench, xLAM-2-70B-FC-R performs best among the three added fine-tuned models at 64.7 Avg. Pass, 10.4 points below HEART’s 75.1. The gap is smaller when we include the models with native function calling: Kimi-K2-Instruct reaches 70.6, within 4.5 points of HEART. Kimi-K2-Instruct is also the closest added baseline on NESTFUL (0.36 vs. 0.44), while gpt-oss-120b comes closest on BFCLv4 Web Search (77.50 vs. 86.00). We have incorporated these results into Tables 1, 2, and 5 and updated the baseline descriptions in Sec. 4.1.

W1. Analysis on the component attribution is useful, but could be more detailed - removing the helpers also changes how the tools are provided to the agent. Attributing all of the difference to just the abstraction may not be prudent, and it would be good to separate the effect of tool search, tool access from the tool primitives themselves. For example, keep tool search, tool access constant - change ONLY whether the requests use plain language + tool primitive OR structured inputs. That would be a cleaner way to test if abstraction is the gain-causer. If this was the experiment that was run for the ablation already, it was not very clear what was being held constant.
R1. Thank you for suggesting this comparison. In Table 6, we removed ToolFace and Tool Primitives together while keeping decomposition, value resolution, and validation in place. The drop from 75.1 to 16.1 on ToolBench therefore measures the combined effect of changing tool access and invocation. Following your suggestion, we ran an additional experiment on ToolBench and NESTFUL to examine the contribution of the invocation interface more directly.

We compared two ways of calling a retrieved tool. In both settings, value resolution could read the tool’s raw API schema. In the first (HEART w/ raw API schemas), it sent a natural-language request to the Tool Primitive, which constructed the arguments and called the tool. In the second (HEART w/ schema-based invocation), value resolution constructed the arguments and called the tool directly. Since direct invocation uses one fewer LLM call per step, we also tested a budget-matched variant that used the extra call to refine the arguments before execution.

All three variants used the same ToolFace repository, semantic retriever, backbone, decomposition and validation prompts, and revision budget (B = 3). Each replayed HEART’s first-pass plan with the same five candidate tools at each step. Revised steps could query the retriever again. Across all steps, 94.6% of candidate lists matched HEART’s exactly. The table reports the mean of three runs per setting.

VariantToolBench Avg. PassNESTFUL Full Acc.
HEART75.10.44
HEART w/ raw API schemas75.00.43
HEART w/ schema-based invocation33.90.13
HEART w/ schema-based invocation + self-refinement (budget-matched)44.60.17
w/o ToolFace & Tool Primitives (Table 6)16.10.06

Giving value resolution access to raw schemas had little effect while it continued to call tools through the primitives. With the schemas visible in both settings, switching from natural-language requests to direct schema-based calls reduced ToolBench Avg. Pass from 75.0 to 33.9 and NESTFUL Full Accuracy from 0.43 to 0.13. Self-refinement recovered part of that loss, reaching 44.6 and 0.17 at HEART’s call budget. These results support a contribution from the invocation interface even when the systems use the same retriever and start with identical tool candidates. We report the new comparison in Appendix F.10, Table 19, and clarify in Sec. 4.3 which components the original Table 6 ablation removed.

W2. The 50 task results are hard to verify, as we don't know what the success criteria are. Are they fixed? Were they vetted by human reviewers? Were there independent judges? Since this is such a strong claim, the validation criteria needs to be better explained.
R2. Thank you for asking how we determined task success. Yes, the criteria are fixed, we reviewed them before grading, and two annotators judged every run independently. The original submission described the success criterion only briefly, in Appendix F.9, and noted in the Table 8 caption that human annotators judged the runs. Each task has written success criteria, available in the anonymous repository, and we checked before grading that they were reasonable and reflected what the user requested. The same criteria applied to every system. For a hotel booking, for example, the run had to leave a confirmed reservation in the final account state with the requested location, dates, number of guests, room type, and guest details, and finding a suitable hotel without booking it counted as a failure. For lookup tasks, the criteria specified the information the system had to retrieve. Each annotator labeled a run as successful only if every condition held. Table 8 covers 200 runs from HEART and the three frontier models, and Table 18 adds 150 runs from the same three models with access to ToolFace. The annotators agreed on 328 of the 350 runs (93.7%) and resolved the remaining 22 by discussion.

As an additional check, Claude-4.6-Sonnet and GPT-5.4 each evaluated all 350 trajectories using the task criteria, the trajectory, and the final account state.

JudgeAgreementCohen's κ
Annotator 1 vs. Annotator 2328/350 (93.7%)0.86
Claude-4.6-Sonnet331/350 (94.6%)0.88
GPT-5.4327/350 (93.4%)0.85

Both LLM judges gave HEART the same completion rate as the human evaluation, 84% (42/50). Because both judges were also evaluated systems, we checked their agreement on their own trajectories. Claude-4.6-Sonnet matched the human labels on 95 of its 100 trajectories, and GPT-5.4 matched on 93 of 100. The tasks and tool lists are available in the anonymous repository. The revised manuscript describes the grading protocol in Appendix H.1 and reports judge agreement in Appendix H.2, Table 25.

W3. The prompt injection prevention claims are hard to reconcile - while the attack surface doesn't exist in the LLM's input prompt, the attack surface still exists on the prompt for the tool primitives. The testing effectively tests - can an attacker make the agent choose the wrong tool; but it does not test - can an attacker make the agent use the wrong inputs? Saying zero attack success gives us a false sense of security - but not all attack modes were considered. This is more about scope of the testing - which is acknowledged to be limited in the appendix, but it would help to soften the claim even in the main paper.
R3. Thank you for pointing out this. We have revised the main text and the Table 9 caption to state explicitly that the observed 0.0% attack success rate applies to tool selection under the ToolHijacker evaluation. That experiment measures whether a malicious tool document can cause HEART to select the wrong tool. It does not measure whether untrusted content can affect the arguments passed to a correctly selected tool. In particular, an earlier tool’s output can reach both value resolution and the receiving primitive, so the concern you raise also applies to HEART.

To examine this case, we evaluated HEART and the three frontier models on 629 AgentDojo security cases involving instructions embedded in tool outputs. The frontier models used native function calling. We also evaluated a version of HEART that records the source of each argument and rejects sensitive values supported only by tool outputs.

MethodUtility, no attack (%)Utility under attack (%)ASR (%)
GPT-5.477.362.818.6
Claude-4.6-Sonnet80.473.48.9
Gemini-3.1-Pro72.257.921.5
HEART76.370.66.4
HEART + provenance check74.271.41.9

HEART’s ASR was 6.4% on these cases. Adding the provenance check reduced ASR to 1.9% and raised utility under attack from 70.6% to 71.4%, while utility without an attack fell from 76.3% to 74.2%. We now describe the ToolHijacker threat model alongside the original result in Sec. 4.4, qualify the corresponding claims in the Introduction, Sec. 4.5, and Conclusion, and report the AgentDojo results separately in Appendix F.13, Table 24.

Reviewer_vUzzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a general harness that contains three components: 1) a harness framework (named as HEART); 2) a tool set (named as ToolFace); 3) a llm that maps instructions into structured API calls (named as Tool Primitives).

When user has a query, HEART first plans or decomposes the query into several steps to finish the task. Each step is described in natural language. Then, for each step, HEART will retrieves the tool set in ToolFace to see what's most related tool connecting with current step. Once matched, the tool and current step will both provide to Tool Primitives, letting which to construct the tool parameters, execute the tool, get results and provide to the HEART to do next step execution. HEART plays as a role to decide next steps, connect all components, how to handle the failure function calls, etc.

The authors evaluate the whole framework using Qwen3-8B for both HEART and Tool Primitives components on five tool-use benchmarks. The results are positive. An 8B model as backbone even show better performance than frontier models such GPT-5.4, even though it increases latency significantly.

Critical strengths
  1. The paper evaluates HEART across five public benchmarks. On several settings, the improvements are large.
  2. Ablation studies are solid. For example, authors show that HEART or Tool Primitives substituted by other stronger LLM such as GPT-5.4 can further push the performance.
  3. The proposed method requires no model training and could be practically useful for direct applying to a system hoping to have better tool calling performance.
Critical weaknesses
  1. Each component in this paper is not technically novel. HEART and Tool Primitives are essentially two llms with pre-defined instructions, while ToolFace is a collection of functions covering many use cases.
  2. The reported performance on the 50 real-world tasks is impressive, but the benchmark is relatively small and curated by the authors.
  3. The latency of the proposed method is indeed much higher than the compared baselines, which might restricts it to be a practical option for a real system.
Questions for authors

Q1: Would it be possible to replace Tool Primitive interface with conventional schema-based function calling?

Q2: Can you provide more details about the evaluation protocol for the 50 real-world tasks? Like how they are constructed, what's the human evaluation pipeline, etc.

9 comments

We appreciate your thoughtful review and are encouraged by your recognition of our work: the evaluation across five public benchmarks with the gains you noted in several settings, the ablation studies, including the substitution of stronger LLMs such as GPT-5.4 (Table 12), and a training-free method that could be useful in practice. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We provide detailed responses to the remaining concerns below.


W1. Each component in this paper is not technically novel. HEART and Tool Primitives are essentially two llms with pre-defined instructions, while ToolFace is a collection of functions covering many use cases.
R1. We would like to clarify that the novelty of HEART lies in how tools are exposed to the model. HEART is a harness engineering framework for composing and executing Tool Primitives.

Prior work has used prompting, as in ReAct and HuggingGPT, or fine-tuning on curated tool-use trajectories, as in Hammer, ToolACE, and the xLAM series. With conventional schema-based invocation, the caller must read each tool’s schema and generate valid arguments. This creates two difficulties. First, when calls depend on one another, the caller must identify the relevant value in one tool’s output and fit it to the next tool’s input schema. For example, GPT-4o achieved 28% full-sequence accuracy on NESTFUL. Second, a larger catalogue puts more tool descriptions in context. Prior work reported accuracy drops of 7%–85% as these descriptions grew from 8K to 120K tokens.

We address these two limitations by changing how the caller interacts with tools. A Tool Primitive wraps an executable function, accepts a natural-language request, and constructs the arguments required by its API. When one call depends on another, HEART identifies the relevant value in the earlier result and passes it to the next primitive with a description of what to do. The receiving primitive handles the conversion to its own input schema. To support a large catalogue, ToolFace exposes 25,519 executable tools through this common interface, and only the tools retrieved for a query enter the caller’s context. HEART coordinates the task as a whole. It plans the calls, tracks where each step’s inputs come from, and checks whether each result satisfies the step’s goal. All LLM-based roles share the same backbone and require no additional training.

The differences are also stated in Sec. 2: prior methods couple how tools are stored, represented and called, and HEART separates the three.

W2. The reported performance on the 50 real-world tasks is impressive, but the benchmark is relatively small and curated by the authors.

R2. Thank you for raising this concern. The primary empirical goal of our paper is to study how well HEART performs on established tool-use benchmarks. We evaluate HEART on five public benchmarks under their official protocols. We include the 50 author-written real-world tasks as a preliminary study of how HEART performs in end-to-end user scenarios.

For the 50-task study, we supplement the completion rates in Tables 8 and 18 with 95% Wilson confidence intervals to show how precisely each rate can be estimated from 50 tasks. HEART completed 42 of the 50 tasks (84%, 95% CI 71.5–91.7%), while the strongest baseline, Claude-4.6-Sonnet with ToolFace, completed 16 (32%, 95% CI 20.8–45.8%). The table below reports results for all systems. Rows marked “+ ToolFace” correspond to Table 18, and the remaining rows correspond to Table 8.

SystemCompletion rate (%)95% CI
HEART84[71.5, 91.7]
GPT-5.420[11.2, 33.0]
Claude-4.6-Sonnet24[14.3, 37.4]
Gemini-3.1-Pro22[12.8, 35.2]
GPT-5.4 + ToolFace26[15.9, 39.6]
Claude-4.6-Sonnet + ToolFace32[20.8, 45.8]
Gemini-3.1-Pro + ToolFace22[12.8, 35.2]

HEART completed 26 more tasks than the strongest baseline, Claude-4.6-Sonnet with ToolFace. Because the systems ran the same tasks, we also examined the most conservative pairing consistent with these counts. Even if Claude succeeded on all eight tasks that HEART missed, 34 tasks would still have been completed only by HEART and eight only by Claude. A two-sided exact McNemar test gives p<0.0001p<0.0001 even under this pairing.

To address the concern about author curation, we evaluated 50 additional tasks adapted from real users’ public posts. We screened posts from 38 Reddit and Stack Exchange communities covering the same five domains. Within each domain, we reviewed posts from newest to oldest and selected the first 10 that met our criteria. An eligible post had to request either an action on a live service, such as starting a booking or requesting a quote, or a comparison of current information from at least two services. It also had to state enough constraints to judge success and require no personal information beyond what the user had provided. We excluded posts primarily seeking opinions, diagnoses, or legal advice.

We removed usernames, names, and post titles and checked each adapted task against its source post to preserve the user’s goal and constraints. We derived the success criteria from the original request. For transactional tasks, the required stopping point was before payment or final confirmation. We ran each system once per task, with the frontier models using their product interfaces as in Table 8. We applied the same human evaluation protocol. The table below reports completion rates (%) on the new tasks.

DomainGPT-5.4Claude-4.6-SonnetGemini-3.1-ProHEART
Travel Planning30302080
Healthcare20302070
E-commerce30403080
Finance20302080
Local Services20202070
Overall24302276

HEART completed 38 of the 50 tasks adapted from public posts (76%), compared with 15 (30%) for the strongest baseline, Claude-4.6-Sonnet. Its completion rate was lower than on the author-written tasks (84%), while its lead over all three frontier models held in each of the five domains.

We summarize both analyses in Sec. 4.4 and report the statistical analysis in Appendix H.3 (Table 26) and the public-post evaluation in Appendix H.4 (Table 27). The anonymous repository includes the original tasks and tool lists, along with the 50 new tasks, their source links, their success criteria, and the order in which we screened the posts.

W3. The latency of the proposed method is indeed much higher than the compared baselines, which might restricts it to be a practical option for a real system.
R3. Thank you for raising this concern. We consider HEART’s latency overhead reasonable for the multi-step tasks evaluated here, given its higher task success rates and lower API cost. In Table 3, HEART takes 8–16% longer per task than GPT-5.4 and 13–33% longer than Claude-4.6-Sonnet. It achieves higher Pass¹ and Pass⁴ scores than both models in all three domains, and its per-task API cost is 13–14% of GPT-5.4’s.

For applications where response time matters more, we developed HEART-Fast as an optional execution mode. It uses the same model and prompts as HEART, runs independent tool calls in parallel, skips the validation call when a read-only step returns a result that satisfies its schema, and caches static prompt prefixes. We evaluated HEART-Fast under the same τ²-Bench protocol as Table 3. The table below reports per-task latency in seconds.

MethodRetailAirlineTelecom
GPT-5.416.2418.4235.11
Claude-4.6-Sonnet14.1216.5233.48
Gemini-3.1-Pro9.1510.8228.46
HEART18.7221.3437.95
HEART-Fast13.6415.8733.79

HEART-Fast reduces per-task latency by 11–27% relative to HEART. It is faster than GPT-5.4 in all three domains and faster than Claude-4.6-Sonnet on Retail and Airline. Gemini-3.1-Pro remains the fastest system shown. HEART-Fast achieves Pass¹ / Pass⁴ scores of 0.81 / 0.74 on Retail, 0.73 / 0.61 on Airline, and 0.62 / 0.50 on Telecom, all within 0.02 of the corresponding HEART scores.

We also timed every LLM call and tool execution on Retail and grouped the time by stage. The table below reports the per-task latency of each stage in seconds.

StageHEARTHEART-Fast
Decomposition3.122.71
Value resolution3.382.36
Tool Primitives4.573.08
Validation2.961.24
Revision rounds1.931.62
User simulator and environment2.762.63
Total18.7213.64

No single stage dominates HEART’s latency. HEART-Fast saves the most time in validation, the Tool Primitives, and value resolution. On Retail, 57% of validation calls are skipped, and steps run in parallel in 41% of plans. Appendix F.12 (Tables 22 and 23) reports the full results and a per-stage latency breakdown.

Q1. Would it be possible to replace Tool Primitive interface with conventional schema-based function calling?

A1. Yes. The “w/o ToolFace & Tool Primitives” variant in Table 6 uses conventional schema-based function calling within HEART. In this variant, value resolution reads the raw API schemas, resolves the arguments, and dispatches each call directly. It scores 16.1 on ToolBench Avg. Pass, compared with 75.1 for full HEART.

Inspired by your question, we ran an additional experiment comparing the Tool Primitive interface with direct schema-based invocation. We retained ToolFace and the semantic retriever and replayed HEART’s first-pass plan with the same five candidate tools at each step. Revised steps could query the retriever again. Across all steps, 94.6% of candidate lists matched HEART’s exactly. We tested schema-based invocation both with and without one additional LLM call per step to refine the arguments before execution. The self-refinement variant matches HEART’s LLM call budget.

VariantToolBench Avg. PassNESTFUL Full Acc.
HEART75.10.44
HEART w/ schema-based invocation33.90.13
HEART w/ schema-based invocation + self-refinement (budget-matched)44.60.17
w/o ToolFace & Tool Primitives (Table 6)16.10.06

Self-refinement improves schema-based invocation on both benchmarks. At the same LLM call budget, HEART achieves 75.1 on ToolBench Avg. Pass and 0.44 on NESTFUL Full Accuracy, compared with 44.6 and 0.17 for schema-based invocation with self-refinement. We report this comparison in Appendix F.10, Table 19.

Reviewer_WRgsOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes HEART, a harness engineering framework via agent-native, reusable tool primitives. The system introduces Tool Primitives, which replace traditional rigid schema-based tool invocation with natural language interfaces, where each tool is wrapped in an LLM responsible for internal schema resolution and execution. The system stores tools in ToolFace (a repository containing 25,519 functions) and retrieves only relevant tools at inference time. The framework coordinates execution through three stages: request decomposition, value resolution, and result validation with feedback-driven revision.

Critical strengths
  1. Wrapping tools with LLM-based Tool Primitives to process natural language requests instead of requiring callers to emit strict schema-compliant arguments is a compelling paradigm shift. It neatly isolates schema binding and successfully avoids prompt injection vulnerabilities at the tool-selection stage.

  2. The evaluation is exceptionally broad, spanning five prominent tool-use benchmarks (ToolBench, NESTFUL, τ2\tau^2-Bench, ACEBench, and BFCLv4) as well as a curated suite of 50 real-world tasks, consistently demonstrating performance advantages over several frontier commercial models (such as GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro).

Critical weaknesses

1.Do the ablation experiments truly decouple the causal value brought by natural-language Tool Primitives? Sections 4.3 and Table 6 directly attribute the dramatic drop in ToolBench Pass from 75.1 to 16.1 in the "w/o ToolFace & Tool Primitives" variant to the natural-language tool interface. However, this variant simultaneously removes the centralized ToolFace registry and replaces it with raw schema exposure. Does this not simultaneously alter tool retrieval behavior, visible context, and the interface itself? Even with a budget-matched control, how can it prove that the remaining performance gap stems from "natural-language invocation" rather than "altered retrieved context or registry access"?

2.Can we simply and crudely interpret PasskPass^k in τ2\tau^2-Bench as evidence of "sustained performance over extended multi-turn interactions"? Section 4.2 defines PasskPass^k as the fraction of tasks for which all kk independent trials succeed. This means that the lower Pass4Pass^4 values for baselines partly reflect the compounding probability of repeated independent trial successes, which is fundamentally distinct from performance degradation within a single longer dialogue. Can we reasonably deduce the claim that "iterative revision sustains task completion under extended multi-turn interactions" solely from the data in Table 3? Why not narrow this interpretation to "overall reliability across repeated independent trials," or alternatively provide conditional analyses based on actual dialogue depth, turn counts, and failure/recovery trajectories?

Questions for authors

For the "w/o ToolFace & Tool Primitives" variant and the budget-matched variant in Table 6, were the number and identity of retrieved candidate tools, the total schema tokens visible to the caller, and the retrieval procedure held strictly identical to the full HEART framework? In Table 3, can you report performance as a function of actual dialogue depth or tool execution turns rather than relying on independent trial PasskPass^k? For the 50-task study in Table 8, what was the precise human evaluation protocol?

5 comments

We thank the reviewer for the constructive comments. We appreciate the recognition that Tool Primitives change how tools are called, isolate schema binding and avoid prompt injection at the tool-selection stage, as well as the acknowledgment that the evaluation is broad, with five tool-use benchmarks and 50 real-world tasks. We have uploaded a revised version of the paper, with all revisions highlighted in blue. We address each of the raised weaknesses and questions below.


W1. Do the ablation experiments truly decouple the causal value brought by natural-language Tool Primitives? Sections 4.3 and Table 6 directly attribute the dramatic drop in ToolBench Pass from 75.1 to 16.1 in the "w/o ToolFace & Tool Primitives" variant to the natural-language tool interface. However, this variant simultaneously removes the centralized ToolFace registry and replaces it with raw schema exposure. Does this not simultaneously alter tool retrieval behavior, visible context, and the interface itself? Even with a budget-matched control, how can it prove that the remaining performance gap stems from "natural-language invocation" rather than "altered retrieved context or registry access"?

Q1. For the "w/o ToolFace & Tool Primitives" variant and the budget-matched variant in Table 6, were the number and identity of retrieved candidate tools, the total schema tokens visible to the caller, and the retrieval procedure held strictly identical to the full HEART framework?

R1/A1. Thank you for raising this question. The "w/o ToolFace & Tool Primitives" ablation in Table 6 measures the joint contribution of these two components. We removed the ToolFace registry and exposed raw API schemas to value resolution, which generated the arguments and called the tools directly. ToolBench Avg. Pass fell from 75.1 to 16.1. The budget-matched variant in Appendix F.8 (Table 17) used the same setup but reallocated the Tool Primitive call budget to argument refinement, bringing the score to 34.2. Table 17 therefore matches HEART's LLM call budget. Both original variants change the retrieval procedure and the caller's access to raw schemas, and neither holds candidate count, candidate identity, or total visible schema tokens fixed to full HEART.

Inspired by your suggestion, we ran an additional experiment on ToolBench and NESTFUL to examine schema exposure and invocation separately. All three new variants retained ToolFace and used HEART's semantic retriever, backbone, decomposition and validation prompts, and revision budget (B = 3). Each variant replayed HEART's first-pass invocation plan with exactly the same five candidate tools at each step. Revised steps could query the retriever again. Across all steps, 94.6% of candidate lists matched those in full HEART exactly.

  • In HEART w/ raw API schemas, value resolution could read the retrieved tools' raw schemas and continued to send natural-language requests to the Tool Primitives.
  • In HEART w/ schema-based invocation, value resolution read the raw schemas, generated schema-compliant arguments, and called the tools directly. These two variants received identical schema text whenever their candidate lists matched. Since revised steps could retrieve different candidates, total schema-token counts could vary across complete trajectories.
  • Finally, the budget-matched variant used one additional LLM call per step to refine the arguments before execution, matching HEART's call budget.

The table reports the mean of three runs per setting.

VariantToolBench Avg. PassNESTFUL Full Acc.
HEART75.10.44
HEART w/ raw API schemas75.00.43
HEART w/ schema-based invocation33.90.13
HEART w/ schema-based invocation + self-refinement (budget-matched)44.60.17
w/o ToolFace & Tool Primitives (Table 6)16.10.06

Giving value resolution access to raw schemas had little effect while calls still went through Tool Primitives. With schemas visible in both settings, switching to direct schema-based calls reduced ToolBench Avg. Pass from 75.0 to 33.9 and NESTFUL Full Accuracy from 0.43 to 0.13. Self-refinement recovered part of this loss, reaching 44.6 and 0.17 at HEART's call budget. These results support a contribution from the invocation interface under a shared retrieval procedure and identical first-pass candidates. We report the new comparison in Appendix F.10, Table 19, and have revised Sec. 4.3 to distinguish this finding from the combined effect measured in Table 6.

W2. Can we simply and crudely interpret Pass^k in τ²-Bench as evidence of "sustained performance over extended multi-turn interactions"? Section 4.2 defines Pass^k as the fraction of tasks for which all k independent trials succeed. This means that the lower Pass^4 values for baselines partly reflect the compounding probability of repeated independent trial successes, which is fundamentally distinct from performance degradation within a single longer dialogue.
R2 (1/3). Yes, Pass^k provides evidence of reliable task completion across repeated executions of multi-turn tasks. The metric was originally introduced in τ-Bench (Yao et al., 2024) and subsequently adopted by τ²-Bench (Barres et al., 2025) to evaluate how consistently an agent completes a task across k independent trials. Since each trial involves multi-turn interactions, a higher Pass^k indicates that the agent can reliably complete such tasks across repeated executions. We have revised Sec. 4.2 to state that Pass^k measures reliability across k independent trials, which is distinct from performance degradation within a single longer dialogue.

W2 (2/3). Can we reasonably deduce the claim that "iterative revision sustains task completion under extended multi-turn interactions" solely from the data in Table 3? Why not narrow this interpretation to "overall reliability across repeated independent trials,"
R2 (2/3). Yes, we have narrowed the claim. The results in Table 3 provide evidence that HEART achieves more reliable task completion across repeated multi-turn interactions, although they do not isolate the contribution of iterative revision on their own. This contribution is further supported by the revision-budget ablation in Table 7, where ToolBench Pass increases from 47.6 to 75.1 as the revision budget grows from B=0 to B=3. The additional analyses in R2 (3/3) also show that HEART's advantage becomes more pronounced on tasks requiring longer action sequences and in trials involving execution failures. Following your suggestion, we have narrowed the wording in Sec. 4.2 and Sec. 4.5 to emphasize overall reliability across repeated independent trials.

W2 (3/3)/Q2. or alternatively provide conditional analyses based on actual dialogue depth, turn counts, and failure/recovery trajectories? In Table 3, can you report performance as a function of actual dialogue depth or tool execution turns rather than relying on independent trial Pass^k?
R2 (3/3)/A2. Yes, we further analyzed the trials underlying Table 3 to examine performance across different task lengths and execution failures. We grouped the 278 tasks, each evaluated over eight independent trials, by the number of actions in their gold solutions: ≤3 (97 tasks), 4–6 (108 tasks), and ≥7 (73 tasks). We use gold-solution actions as a task-level measure of required interaction length rather than observed execution turns, which may be inflated by failed calls and retries.

Gold actionsHEARTClaude-4.6-SonnetGPT-5.4
≤30.840.860.79
4-60.740.710.62
≥70.570.470.43

HEART performs comparably to Claude-4.6-Sonnet on shorter tasks (0.84 vs. 0.86), while its advantage is largest on tasks requiring at least seven actions (0.57 vs. 0.47).

We also analyze trials containing at least one failed tool call, defined as a call that returns an error from the τ²-Bench environment, providing a consistent failure signal across methods.

MethodTrials with a failed call (%)Success with a failed callSuccess without a failed call
HEART340.580.80
Claude-4.6-Sonnet280.310.85
GPT-5.4360.270.84

Failed calls occur at comparable rates across methods, but HEART completes 58% of trials containing such errors, compared with 31% for Claude-4.6-Sonnet and 27% for GPT-5.4, suggesting greater robustness to intermediate execution failures.

Together, these analyses complement the repeated-trial reliability measured by Pass^k, showing that HEART's advantage becomes more pronounced on tasks requiring longer action sequences and in trials involving execution failures. We have added both analyses to Appendix F.11 (Tables 20 and 21).

Q3. For the 50-task study in Table 8, what was the precise human evaluation protocol?

A3. Thanks for asking about the protocol. The original submission briefly described task completion in Appendix F.9 and noted the use of human annotators in the Table 8 caption. Before grading, we reviewed each task’s written success criteria, which are available in the anonymous repository. Two annotators independently reviewed each trajectory and assigned a binary completion label. They applied the same criteria to every system and counted a run as successful only if it met every condition. For a hotel booking, for example, they checked the final account state for a confirmed reservation matching the requested location, dates, room type, and guest details. Finding a suitable hotel alone counted as a failure. Table 8 covers 200 runs, one per task for HEART and each of the three frontier models. Table 18 adds 150 runs from the same three models with access to ToolFace. The annotators agreed on 328 of the 350 runs (93.7%) and resolved the remaining 22 by discussion.

As an additional check, Claude-4.6-Sonnet and GPT-5.4 each labeled all 350 trajectories using the task criteria, the trajectory, and the final account state.

ComparisonAgreementCohen's κ
Annotator 1 vs. Annotator 2328/350 (93.7%)0.86
Claude-4.6-Sonnet331/350 (94.6%)0.88
GPT-5.4327/350 (93.4%)0.85

Both LLM judges gave HEART the same completion rate as the human evaluation, 84% (42/50). For each frontier model, their completion rates were within 2 percentage points of the human rates.

We added the full description of the evaluation protocol to Appendix H.1, with references in Sec. 4.4 and the Table 8 caption, and report judge agreement in Appendix H.2, Table 25. The tasks, their criteria, and their tool lists are in the anonymous repository.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.