acceptodds
Under review as a conference paper at ICLR 2027

To Call or Not to Call: Mastering Tool Admission in Language Agents

Abstract

Large Agents increasingly call external tools, yet most safety research examines only what follows a tool call. The critical prior decision of whether to call the tool is unexamined and can heavily influence downstream safety. Across multiple open-weight models, the unweighted model-average harm rate after admitted harmful requests reaches a strikingly high level. To identify what controls this decision, we apply a factorial intervention to the input. Within our generated construction, counteracting the requested-operation component produces the largest admission contrast. A linear direction estimated from this contrast influences held-out admission in most tested models; effects on operation-neutralised requests in some models show it is not an operation-specific mechanism. We then test whether written refusal is behaviorally and representationally coupled to the same decision. Surprisingly, written refusal accompanies a substantial portion of admitted harmful calls. Natural language refusal training reduces unsafe admission in models that retain benign tool use, showing that response-level safety training can effectively transfer to the call decision. These results establish a partial, model-dependent relation between written refusal and tool admission. They motivate direct evaluation and supervision of tool admission, proving we can no longer treat text refusal as a simple proxy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VHkVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies meta-reasoning and safety associated with tool calls. The authors define "tool-admission" as the decision of a model to invoke a tool, instead of answering from its memory. From the lens of safety, the authors attempt to study what pieces of an unsafe input cause a model to call a tool. To this end, they decompose the request into its operation, topic, and modifier. To understand why the model made a decision (to call or not call), the authors also present a mechanistic study by analysing the alignment request-operation direction, text-refusal direction, and admission-outcome direction.

Critical strengths
  • Interesting problem statement, and worth studying.
Critical weaknesses
  • The paper is very badly written and reeks of AI at this point, with some technical jargon and adjectives. The AI generated contents seems to dominate, especially toward the end of the paper. The sentences and paragraphs are super disconnected, and I fail to see a logical flow. Due to this, I am very unclear about the contributions of the paper, why they are performing the experiments they mentioned, and what their true findings. Similarly, it's also very unclear what the takeaways of this work are, and the contributions mentioned at the end of the introduction don't seem to fit. Overall, I find no causal flow in the work, and I suggest a full, proper rewrite of the paper for a better presentation at this point.
  • For instance, the "factorial" technique is quite unclear, even to the point of why it is named so, or what it is testing. The title also has a heavy/overclaiming adjective "Mastering", but I don't see any formal proof, or a one-shot solution to tool admission. I urge that word be toned down. Also, for example, it's hard to understand what Eq 2 physically represents.
  • The analysis just seems to be limited to just URL-based web search. It would be good to try other tools as well. So, I feel the scope of the paper is a bit limited in that way.
  • I also believe multiple papers deal with tool call reasoning/meta-reasoning, so how is your work different from them, and what extra do you provide beyond them? I ask this as I am unsure about the novelty of the studied problem.
Questions for authors
  • Why was the GPT-4o judgement on a scale of 1-5 and cut at 3? Why not use a binary decision? won't a single binary decision also help the model as be more specific?
Reviewer_HK3iOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies tool admission, the decision of an LLM agent to call a tool before observing its output, as a safety-relevant decision separate from the written response. Using a controlled web-retrieval setting with user-supplied URLs and HarmBench-derived harmful requests across seven open-weight models, the paper evaluates harm on both the call and no-call branches. It decomposes each request into topic, requested-operation, and specificity components to identify what drives admission. It also analyzes how admission relates to written refusal at the behavioral, representational, and fine-tuning levels. The authors find that the requested operation affects admission the most, that written refusal often co-occurs with harmful tool calls, and that the relationship between text refusal and tool admission is partial and model-dependent. They argue that tool admission should be evaluated and supervised directly rather than inferred from text refusal.

Critical strengths
  1. Identifying tool admission as a distinct safety decision. The paper separates the decision to call a tool from the written response and evaluates harm on both the call and no-call branches. It shows that withholding a call does not by itself prevent harm, since some models rarely call the tool yet still answer harmfully from parametric knowledge. This is consequential because it exposes a blind spot in call-only evaluation, which can misjudge agent safety, and it offers a useful evaluation perspective for tool-using agents.

  2. A carefully controlled experimental design. The setup fixes the URL and the cached page, includes benign and topic-matched controls, evaluates interventions on held-out behaviours against random-direction controls, and validates the judge against human and multiple LLM annotators. This is consequential because it makes the behavioural findings credible and reproducible, even though the mechanistic results are less conclusive.

Critical weaknesses
  1. Limited new insight from inconclusive findings. The paper’s main conclusion, that written refusal is not a reliable proxy for tool admission, largely reproduces the refusal–call divergence already reported by [1], so the new contribution must come from the mechanistic analyses. However, these analyses are inconclusive. Removing the operation direction affects held-out admission in only four of seven models, with small effects, and it also changes operation-neutralized requests. Text-refusal steering produces effects of inconsistent sign, and the single-seed refusal fine-tuning works in only three models. The factorial finding is also confounded, because neutralizing the requested operation makes the request much less harmful overall. This weakness is decisive for my assessment because, without a consistent mechanism or a clearly new finding, the paper offers little insight beyond what is already known.

  2. Poor presentation obscures the contribution. Core concepts such as tool admission, the topic/operation/specificity decomposition, the decision token, and ADMIT are introduced without definitions, and their actual definitions appear only later in the method section and the appendix. Most findings are also wrapped in repeated qualifiers such as “partial” and “model-dependent.” This matters for my recommendation because it makes the setup hard to follow and the paper’s actual claims hard to identify and evaluate, which further weakens an already limited contribution.

[1] Mind the gap: Text safety does not transfer to tool-call safety in llm agent, Cartagena et al., Arxiv'26.

Questions for authors
  1. Beyond reproducing in open-weight models the refusal–call divergence reported by [1], what is the key new finding of this paper?

  2. Neutralizing the requested operation also makes the request less harmful overall. Can you show that the operation component affects admission beyond this reduction in harmfulness, for example by controlling for judge-rated harmfulness of each factorial variant?

[1] Mind the gap: Text safety does not transfer to tool-call safety in llm agent, Cartagena et al., Arxiv'26.

Reviewer_dTxyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an protocol for evaluating safety of tool calls, and it helpfully distinguishes between refusal messages and actual results of tool calling. They find a significant disconnect between refusal messages and harmful tool calls, and provide prescriptive training procedures based on this.

Critical strengths
  1. The distinction between the harm of a tool call and the harm of the final response is an important topic. They provide convincing evidence, such as high rates of written refusal responses being accompanied by dangerous intermediate tool calls.
  2. The presentation of the paper is well-organized and clearly written.
Critical weaknesses
  1. The idea that an intermediate tool call can be harmful and current evaluation methods are missing is intriguing, but the experimental setting in this paper is less convincing than it could be. The setup involves only a single tool call, which is a read-only web retrieval tool limited to a static set of URLs. A more realistic setting would be interesting to see.
  2. The experiment on the effect of refusal finetuning adds somewhat confusing support the argument in paper. Part of the argument is that the tool calling harm and refusal messages should be evaluated separately, and it turns out that training targeting refusal messages has some positive impact on both settings. This could be viewed as contradicting the core argument which would be interesting, but the results are limited enough that ends up just confusing. I'd be interested in ideas for larger experiments that would be more conclusive.
Questions for authors
  1. Did you notice any characteristics of the prompt that contributed to harmful tool-use or refusal messages? For example, did prompt length seem to have any impact?
  2. How directly would your relate harmful tool calls shown in these experiments to other harmful tool calls not studied here? For example, is there an additional harmful tool call that you see as closely related to web retrieval?
  3. If under additional seeds and further training the refusal finetuning results were able to extend to all seven models, what would that lead you to conclude about the idea of separating tool admission from final response?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.