acceptodds
Withdrawn from ICLR 2027

Tool Exposure Is Not Free: Admission and Execution of Tool Calls in LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-augmented large language model (LLM) agents are evaluated as though offering a tool were free, and work on tool overuse refines that price only in tokens. However, on a mixed workload of reasoning and exact-computation requests, merely making a Python tool available costs a large share of the accuracy on requests that never needed one, and nearly every tool-exposing method we measure pays some version of that price. Exposure is therefore a two-sided decision, not a resource to ration. Two facts follow. Benefiting from a tool implies needing one but not conversely, so even a perfect necessity predictor over-exposes, and the headroom of any admission policy equals the rate at which exposure breaks requests, which makes the value of admission measurable per workload. In this paper, we propose AgentRT, a runtime control plane that decides exposure from serving state before the model generates a token, so the decision costs no forward pass, and that constructs and executes the payload itself for request classes admitting a typed plan. Against ten baselines, AgentRT is the only policy that keeps accuracy on the requests that need no tool intact while solving every request that needs one. It leads every baseline on both accuracy and energy per correct answer, by up to 28 accuracy points and 69% less energy, and by 11 points against the strongest, and the lead holds on paraphrased requests and across 7B-class models. On newer models whose tool policy breaks almost nothing, admission has no headroom, as the analysis predicts, and AgentRT remains the cheapest policy per correct answer on every one of them.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
tool-augmented LLM agents, tool use, tool calling, admission control, LLM serving, runtime systems, energy efficiency, agentic systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_er5hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies tool exposure as a decision that can both improve and reduce an LLM agent's accuracy. They proposes AgentRT which decides whether to expose tools using inexpensive prompt features, directly executes supported requests through a typed parser, and constrains the remaining tool interactions. Experiments report accuracy and energy improvements on a mixture of mathematical reasoning and synthetic computation tasks. Further evaluations examine paraphrase robustness, parser coverage, and differences across models.

Critical strengths
  • The paper makes a useful empirical observation that exposing tools can damage accuracy, while withholding them can increase computation when the model struggles to answer directly. Measuring both correctness and energy makes this more informative than an analysis based only on tool-call counts. The rescued/broken decomposition also gives a clear way to quantify the accuracy headroom of admission for a fixed pair of policies.
  • The evaluation explicitly examines limitations rather than reporting only favourable aggregate results. The parser ablation, paraphrase experiments, external coverage measurements, and newer-model results show that the benefits depend on both workload structure and model behaviour. In particular, acknowledging the admission gate's poor transfer helps separate the system design from the effectiveness of its current predictor.
Critical weaknesses
  • The headline improvement is largely attributable to specialised symbolic execution, and the comparison does not isolate what AgentRT adds beyond this. A crucial missing comparison is the same parser followed by ordinary no-tool generation for unsupported requests, or the same parser attached to the strongest tool baseline. The held-out paraphrases provide useful evidence within the supported task families.
  • Serving-state awareness is central to the motivation, but its benefit is not demonstrated. The main experiments use zero serving pressure, and the additional pressure settings do not establish behaviour under an actual capacity bottleneck. Moreover, raising an exposure threshold under load needs justification because the paper shows that exposure often reduces serving cost. A score threshold without a request-specific cost estimate does not establish that the resulting policy improves the accuracy-latency trade-off under pressure.
Questions for authors
  1. How does AgentRT compare with the same frozen parser followed by either no-tool generation or the strongest existing tool policy?
  2. Can you demonstrate the serving-state mechanism under measured load, reporting throughput, tail latency, and accuracy? How should the gate treat requests for which exposing a tool reduces cost?
Reviewer_1Sb2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present an evlauation that shows that giving an LLM agent access to a tool is not "free" on a mixed workload of Math500 and exact-computation requests. Merely exposing a Python tool costs accuracy on the half of requests that never needed it, and every existing tool exposure method pays some version of this price. The authors propose, AGENTRT, a runtime control plane sitting between a modified vLLM and the tool executor, with three parts. First, an admission gate that scores exposure from cheap prompt statistics and serving pressure before any token is generated. Then a runtime owned typed execution, which parses certain request classes into a typed plan and executes it directly, bypassing the model entirely so the payload is well-formed by construction. Finally, enforcement mechanisms (trajectory budget, output truncation, payload repair) that bound what admission cannot predict. Against 10 baselines on a Qwen2.5-7B-served mixed workload, AGENTRT keeps the tool-negative half intact while solving the tool-positive outperforming baselines.

Critical strengths

[+] The observation that just the presence of tools, which then allows the agent to invoke them leading to reduction in performance is quite intriguing - and this is an interesting contribution to the community.

[+] The runtime-owned typed execution is a novel idea with clear empirical payoff. Guaranteeing well-formedness "by construction" rather than by detecting and repairing malformed model written calls is well motivated.

Critical weaknesses

[-] Proposition 1 is over-formalized for an definition that it proposes. Elevating this to a named, numbered proposition gives it more apparent weight than the content warrants. it reads better as a one-paragraph observation motivating. I'd suggest demoting Proposition 1 to a remark/lemma-in-prose and reserving the formal "Proposition" treatment for the ceiling/headroom result.

[-] The 4096-token context and 768-token generation cap (line 295) appear to cause substantial, policy-dependent truncation, which is a real confound. In appendix, you report that never-expose hits the 768-token cap on 32% of requests versus 13% for AGENTRT - so isn't ~3X the issues just truncation?

[-] I find it odd that only 140 of Math500's 500 problems are sampled per draw, with no clear justification for not using the full set.

[-] Given this is a discussion on when a model should call tools and the impact of false positives, I would like to see the discussion on never and relevent tool-calling benchmarks such as Berkeley Function-Calling Leaderboard, AND/OR \tau^{3}-bench.

Questions for authors

On the (relatively) newer models such as Qwen3.8-27B, exposure cost is already near zero and admission headroom nearly vanishes. With scale, and as models improve is this problem still persistent?

Reviewer_PXhYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Broadly the paper considers the problem of selective exposure of tools to LLM agents, based on the observation/argument that exposing unneeded tools in a prompt costs accuracy. This is a problem that has been looked at by much other work over the last three years. Unfortunately, the approach here seems really limited and thus uninteresting (see esp. points 6-8 below), and the results are all obtained from what looks like a really niche workload.

  1. To begin with, the prose in this paper is often hard to parse. Examples from the introduction:

    • “Benefiting from a tool implies needing one but not conversely,” is really awkward wording. What does “needing a tool does not imply benefiting from a tool” mean? (The notion of “needing” here is nebulous which is, I suspect, the problem).
    • “The implicit accounting is that a tool can only help and that its only price is tokens and latency, and work on tool overuse keeps that premise, predicting necessity and suppressing the rest”. What does “predicting necessity and suppressing the rest” mean here?
    • “. On a mixed workload of Math500 problems and exact-computation requests served by a 7B model, merely offering a Python tool costs a large share of the accuracy on the requests that never needed one, and nearly every tool-exposing baseline we measure pays some version of that price”. It unclear to me why “(Yao et al., 2023; Chen et al., 2024b; Zheng et al., 2024; Jiang et al., 2023b; Su et al., 2024; Zhang et al., 2026).” are cited in this sentence.
    • “For the requests that exposure breaks, that authorship is the failure”. What does this mean? etc.
  2. “. The first challenge is to decide exposure before generation, without a second forward pass, from information the model cannot see.” Why? And what are the tradeoffs arising from this requirement?

  3. “Exposure makes a request cheaper more often than costlier, and the requests it rescues are almost all among the cheaper ones”. The first part of this sentence is surely unsurprising - assuming a typical iterative harness having access to the correct tool would be more efficient? The significance of the second part of this sentence is lost to me. More broadly, It would help is Section 3 actually described the setup - what dataset, what type of harness - before delving into Figure 2. In the absence of dataset/harness information, Figure 2 is hard to understand the implications of.

  4. “Benefit is a strict subset of necessity.” Proposition 1 confuses me. It seems to hold only when the probabilities involved are binary (0/1) in which case it’s obvious? But why are the relevant probabilities binary?

  5. Proposition 2 is also obvious. And “building: where the tool policy breaks nothing, K is zero and always-expose is already optimal, which is what §5.3 finds on the newest served models.” doesn’t this render this paper not-useful?

  6. Algorithm 1. How does the deterministic non-LLM engine (“runtime execution”) generalize to other settings? And the algorithm only considers the binary problem of including/not-including the tool schema. As opposed to the far more interesting problem of selecting a subset of N tools. Both of these imply the algorithm seems extremely limited.

  7. “The gate scores a request from statistics readable off the prompt in microseconds, such as counts of digits, operators and LATEX markup and hits of two small pattern families,” Again this seems really limited. What percentage of real world problems need reasoning about Latex markup?

  8. “Workloads. A draw of the mixed workload samples 200 requests, 140 from Math500 and 60 synthetic exact-computation tasks spanning eight classes of exact integer and rational arithmetic, from modular powers to determinants, whose answers a 7B model cannot reliably produce by decoding alone.” Why is this an interesting workload ?

I think my biggest concern with this paper is that it does not provide evidence to show that this approach is useful for interesting real-world settings. It reads like a really niche solution to a really niche workload, rendering it uninteresting generally.

Critical strengths

Please see my review.

Critical weaknesses

Please see my review.

Questions for authors

My biggest concern is that this paper seems to present a niche idea for a niche workflow, that is not generally applicable. So my main question for the authors is, - How does the approach presented herein generalize to other tool-related tasks (the many tool benchmarks in literature, or agent harness benchmarks)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.