acceptodds
Under review as a conference paper at ICLR 2027

ToolNeed: Selective Verification with Matched No-Tool Behavior

Abstract

Tool augmentation changes language models from generators into systems capable of initiating external actions. Before tool selection or argument generation comes a more fundamental decision: is external action warranted at all? We distinguish tool validity from behavioral necessity: a proposed action may be relevant, valid, executable, and even lead to a correct outcome, yet remain unnecessary relative to what the same model can accomplish without tools. In targeted construct analyses, matched no-tool behavior remains correct for 96.3% and 99.4% of successful valid tool paths on Qwen3.5 and DeepSeek, respectively. We introduce ToolNeed, a training-free, black-box-compatible method that selectively verifies proposed tool actions by eliciting matched no-tool behavior from the same target model and using it for contrastive arbitration. Candidate-blind routing determines when this verification is activated without conditioning on the concrete proposal. On SABEval, aggregate final-action Tool Invocation Rate falls from 91.2% to 8.1%, 43.0% to 8.5%, and 62.1% to 18.0% across three backbones. On the shared BFCL held-out Live subset and Non-Live split, ToolNeed achieves the highest Irrelevance accuracy in all six backbone–regime settings. Tool-required behavior is near-exactly preserved on Non-Live, while Live settings show modest preservation losses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool use, function calling, tool-using language models, selective verification, tool necessity, black-box inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YDeEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a large language model should abandon a tool call it has already proposed. ToolNeed first determines the task requirement without examining the specific call, and then checks whether the call matches that requirement. For samples that trigger behavioral verification, it has the same model answer again with the tool removed, and uses contrastive arbitration to decide whether to retain the tool call. The method requires no training and can be applied to black-box models.

Critical strengths
  1. The problem is practically meaningful, and the method is easy to follow. Checking whether a call is correct cannot fully answer whether a call is needed, and using the same model's actual response without tools as a reference provides more concrete information than merely asking the model to judge "whether a tool is needed."
  2. The results on BFCL are fairly consistent. Across six settings—three models on both Live and Non-Live—ToolNeed achieves higher Irrelevance accuracy than RA-PF, and the differences survive multiple-comparison correction. The authors also report separately how well normal tool calls are retained, rather than showing only a drop in call rate.
  3. The statistical analysis and ablations are done with care. SABEval uses template-level clustered statistics, BFCL uses paired tests, and the Masked-NoTool ablation hides the no-tool response while preserving the arbitration pipeline, verifying that the actual response content does provide additional information.
Critical weaknesses
  1. There is still a gap between the core claim and the main evaluation. SABEval and BFCL Irrelevance mainly measure whether a call can be avoided when the tool is inapplicable, whereas the paper emphasizes whether a relevant, valid tool call can still be omitted. The GSM8K analysis demonstrates that such cases are common, but it does not further report ToolNeed's decision performance on this sample set. The Retriever experiment that directly evaluates this capability covers only 59 filtered samples from a single model, so the evidence remains limited.
  2. The Retriever experiment cannot yet adequately verify the arbitrator's ability to distinguish necessity. Of the 42 unnecessary calls, only 10 trigger verification and are canceled, while the remaining 32 are still retained; all 17 necessary calls bypass arbitration. Thus, "all necessary calls are retained" mainly reflects the conservativeness of the routing, and it cannot be used to judge whether the arbitrator can recognize misleading erroneous no-tool responses.
  3. The quality of the substitute responses is insufficiently validated. A lower call rate does not necessarily mean that the user's request has been handled well. The paper's semantic review covers only 40 samples, 6 of which contain factual errors or omissions of key content; moreover, the evaluator and the arbitrator use the same model, DeepSeek. For a method that replaces tool calls with ordinary responses, a larger-scale, independent evaluation of response quality is necessary.
  4. The loss of normal tool calls needs more thorough analysis. On BFCL Live, five of the six Simple/Multiple comparisons show a significant drop relative to Raw. For example, Simple accuracy drops by about 3.7 percentage points for both DeepSeek and Qwen3.7. Only a fixed policy is currently reported, and it remains unclear whether ToolNeed still outperforms the feedback baseline at the same level of normal-call loss.
Questions for authors
  1. Could the full ToolNeed be run on the samples for which labels have already been obtained in the GSM8K construction analysis, reporting the cancellation rate for unnecessary calls, the false-deletion rate for necessary calls, and the final answer accuracy? This would more directly validate the paper's central claim.
  2. What would happen if behavioral arbitration were applied to all 59 valid calls in the Retriever setting, rather than only to the 10 samples selected by the routing? I would like to understand how much each of routing and arbitration contributes, and whether the arbitrator is reliable when faced with erroneous but plausible no-tool responses.
  3. In Table 2, Qwen3.5's BFCL Live results are 57.8% / 59.0% / 82.9%, whereas the ToolNeed results in the ablation discussion on page 17 are 55.96% / 59.76% / 82.40%. What exactly differ in the settings between these two experiments?
1 comment
AuthorsOpenReview ↗

Response to Reviewer YDeE

Thank you for these concrete suggestions. We ran the complete GSM8K policy test, applied arbitration to all 59 Retriever cases, and clarified the source of the two Qwen3.5 tables.

Full ToolNeed on GSM8K.

The original Full policy retains every tool call on the 1,141 qualifying cases, including all 1,099 cases where the no-tool answer is correct. Your concern is borne out on this set: the prevalence of redundant calls does not translate into suppression by the original policy.

Applying arbitration to every case removes all 1,099 redundant calls, but also removes 34 of the 42 calls whose no-tool answers are incorrect. Final correctness falls from 1,141/1,141 to 1,107/1,141. A separately frozen permissive gate gives the same result. Broader verification reaches the redundant calls, but the arbiter often accepts incorrect arithmetic answers.

Both results are now in Section 5.1 and Appendix G, Table 22. The original policy’s perfect conditional correctness follows from retaining the already-correct tool paths; it is not evidence that it successfully distinguishes these cases.

Arbitration on all Retriever cases.

Using the same arbitration outputs for the policy comparison, we obtain:

PolicyV1N0 canceledV1N1 retainedOfficial final correctness
Original Full10/4217/1759/59
Full-coverage arbitration42/4212/1754/59

You are right that the original 17/17 retention does not test the arbiter, because all 17 cases bypass it. The full-coverage experiment now includes those cases.

The five official failures involve entity-equivalent variants, such as “Gabriel García Márquez” instead of “Gabriel Garcia Marquez.” A supplementary contextual inspection finds correct entity content in 47 no-tool answers and incorrect content in 12; arbitration chooses DIRECT for all 47 and TOOL for all 12. This provides useful evidence about answer-content discrimination on the retrieval set. We retain 54/59 as the official result, since entity equivalence does not establish source or format compliance, and the inspection is not independent human evaluation. These results appear in Section 5.2, Table 4, and Appendix H.

The two Qwen3.5 tables.

The tables come from separate runs on the same 2,051 tasks, with correct counts of 126/595/683 for the main evaluation and 122/603/679 for the component analysis. Differences can arise from run-to-run variation in model outputs and auxiliary-service execution; the available records do not isolate their respective contributions here. All component comparisons use shared upstream outputs within their own run, rather than the main-table predictions. Appendix D now clarifies this distinction.

Response handling and preservation.

We expanded the response audit to 200 task–model pairs and used a method-blinded Qwen3.7 judge rather than the DeepSeek arbiter. Handling adequacy is scored separately from factual correctness and coverage. A larger no-tool budget improves Full’s handling, but the same-source comparison still favors RA-PF: among 113 common-DIRECT pairs, Full has four improvements and 26 regressions, with 80 ties and three uncertain pairs (Section 5.5 and Appendix I).

The matched component results remain positive. Relative to ToolNeed-Base, the behavioral branch gives 38 corrections/six regressions on Qwen3.7 Live and 42/eight on Qwen3.5 Live. Showing the actual no-tool answer rather than masking it improves Live Irrelevance by 4.13 and 2.91 points, subject to the control’s limitations described in Appendix E.

A broader gate is not an automatic improvement: on Qwen3.5 Live it adds 55 correct Irrelevance decisions but loses 63 correct Simple/Multiple decisions relative to Full. This does not establish superiority at matched preservation, but shows the trade-off when verification is expanded.

The completed tests support the usefulness of the behavioral comparison while showing where the current policy misses redundant calls or accepts incorrect alternatives. We have revised the paper accordingly and would be grateful if you could reconsider your score.

Reviewer_Dt9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes tool validity from behavioral necessity: a tool call can be relevant, well-formed, executable, and even correct, yet unnecessary because the same model could complete the task without tools. ToolNeed is a training-free, black-box-compatible inference-time method. First a candidate-blind router decides whether to trigger verification. When triggered, it generates a response from the same model with tools removed and uses an arbiter to decide whether to retain the proposed tool action or return the no-tool response. Across three backbones, ToolNeed substantially reduces tool invocation on SABEval and improves BFCL Irrelevance accuracy over the evaluated baselines. However, these gains come with some degradation on tool-required BFCL Live tasks, while end-to-end latency and the reliability of replacement responses remain insufficiently evaluated.

Critical strengths
  1. The authors evaluate three backbones on complementary benchmarks using shared task IDs across methods. Their statistical analysis accounts for template-level dependence in SABEval and uses paired tests with multiple-comparison correction for BFCL. They also report unfavorable results, including significant losses on tool-required Live tasks and mixed performance against RA-PF. This makes it easier to assess both the improvements and their limitations, and increases my confidence in the empirical findings.

  2. The ablations address whether the proposed behavioral comparison actually contributes to the results. ToolNeed-Base isolates the joint effect of no-tool generation and arbitration, while Masked-NoTool keeps the decision procedure fixed but hides the actual no-tool response. Revealing that response improves BFCL Live Irrelevance accuracy by 2.91–4.13 percentage points across the two Qwen backbones. This supports the value of examining the model’s alternative response, beyond simply instructing the arbiter to favor fewer tool calls.

Critical weaknesses
  1. The evidence does not yet establish how reliably ToolNeed identifies valid but unnecessary tool calls. The GSM8K analysis demonstrates that successful tool use can coexist with correct no-tool responses, but its high V1N0 rates are conditional on successful tool-assisted paths and may largely reflect the selected models’ ability on this benchmark. They do not establish how informative tool validity is about necessity across a broader task distribution. The more direct When2Tool Retriever evaluation contains only 59 qualifying cases from one backbone: ToolNeed converts 10/42 unnecessary calls to DIRECT, while all 17 necessary calls bypass arbitration. This leaves the arbiter’s ability to distinguish the two classes insufficiently tested. Moreover, gold correctness defines necessity in these diagnostics, whereas the deployed arbiter must judge the no-tool response without gold answers or external verification. The paper provides limited evidence that it can reliably distinguish a sufficient response from a plausible but incorrect one, particularly when correctness depends on current information or external state.

  2. The evaluation establishes reduced irrelevant invocation more clearly than a favorable tradeoff between task success and total inference cost. SABEval TIR and BFCL Irrelevance reward avoiding tool calls without establishing the quality of the replacement response. The semantic audit covers only 40 ToolNeed conversions from the Qwen backbones, with 34 passing, and provides no equivalent assessment of baseline responses. Meanwhile, tool-required accuracy declines significantly in five of six BFCL Live comparisons, and Table 17 shows additional model inference relative to RA-PF in most settings without an end-to-end latency estimate. A comparison at matched tool-required accuracy, together with broader response-quality evaluation, would clarify whether ToolNeed improves the tradeoff or mainly adopts a more aggressive suppression policy. Although Masked-NoTool is a useful control, the paper also lacks a simple necessity-focused prompting baseline and does not separately isolate the contributions of routing and requirement verification. These omissions make it difficult to assess how much of the pipeline’s complexity is justified.

Questions for authors
  1. Beyond GSM8K, can you report the proportions of valid, successful tool calls for which the matched no-tool response is correct (V1N0) vs incorrect (V1N1), particularly on retrieval, time-sensitive, or state-changing tasks? This would help determine whether the high V1N0 prevalence reflects a broader phenomenon or primarily tasks that the evaluated models can already solve easily without tools. For state-changing tasks, how would necessity be assessed when a textual response alone cannot complete the requested action?

  2. How does the gain in Irrelevance accuracy trade off against losses on tool-required Simple/Multiple tasks as the arbitration preference for DIRECT varies? A tradeoff curve, or aggregate task-success results under different proportions of tool-required requests, would help establish when ToolNeed is preferable to the baselines.

  3. Can you apply the same semantic-adequacy audit to no-tool responses from Raw and RA-PF, and validate the judgments through human evaluation or a judge from a different model family? This would clarify whether ToolNeed’s lower invocation rate preserves response quality relative to the baselines and whether the findings depend on using DeepSeek for both arbitration and evaluation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Dt9J

Thank you for asking for a baseline-matched response audit. We completed it. Increasing the generation budget improves Full’s handling of requests, but the matched comparison still favors RA-PF.

The new audit directly tests replacement-response handling.

We evaluated 200 frozen task–model pairs: 100 from SABEval and 100 from BFCL Live, covering all three backbones. A method-blinded Qwen3.7 judge records factual correctness, handling adequacy, and requirement coverage separately. The reported metric is handling adequacy; passing this dimension does not guarantee that the other two also pass. The judge differs from the DeepSeek arbiter, although it shares a model family with the Qwen targets.

Increasing Full’s no-tool budget from 512 to 4,096 produces 20 improvements and six regressions among 127 common-DIRECT pairs, with 95 ties and six pairs involving UNCERTAIN. In the same-source 4,096-token comparison, RA-PF has 91 handling passes among 116 DIRECT outputs, versus 71/127 for Full. Because the selected DIRECT sets differ, we also compare the 113 common-DIRECT pairs: Full has four improvements, 26 regressions, 80 ties, and three uncertain pairs relative to RA-PF.

Thus, truncation contributes to the handling failures, but increasing the budget does not close the baseline gap. Section 5.5 and Appendix I now report both results. A post-hoc check requiring all three dimensions to pass also favors RA-PF, with 70/116 versus 44/127 on the respective DIRECT sets.

The earlier 34/40 result used a different task-level rubric. Appendix I now reports it alongside the strict Correct-AND-Sufficient result of 14/40 on the same responses, separately from the new audit.

The component results support a narrower, but useful, contribution.

The model’s actual no-tool response provides information for deciding whether to retain an external action. With upstream behavior shared, adding the behavioral branch to ToolNeed-Base yields 38 corrections versus six regressions on Qwen3.7 Live, and 42 versus eight on Qwen3.5 Live.

Masked-NoTool tests access to that response more directly. Both variants share the frozen candidates, routing, no-tool responses, and return rules; the masked arbiter sees [WITHHELD] instead of the response. Showing the response improves Live Irrelevance by 4.13 and 2.91 points. Appendix E documents the limits of this control, including the placeholder input, different call times, and unconfirmed backend snapshot identity.

These comparisons support the value of behavioral evidence in call decisions. They do not imply that Full produces better replacement responses than RA-PF, or establish that every pipeline component is necessary.

The task mixture determines the net decision-level benefit.

To address your question, we held category accuracies fixed and varied the proportion of tool-required requests, preserving the observed Simple:Multiple ratio:

Tool-required proportionDeepSeek: Full−RA-PFQwen3.7Qwen3.5
25%+9.48 pp+4.26 pp+2.52 pp
50%+4.88 pp+2.22 pp+0.92 pp
75%+0.28 pp+0.17 pp−0.68 pp

Under the observed Live mixture, Full produces 63, 29, and six additional correct call decisions over RA-PF out of 2,051 tasks. These are decision-level aggregates, not end-to-end success rates. We have not established superiority at matched tool-required accuracy.

We also completed full-coverage arbitration on the diagnostic sets. On Retriever, it removes 42/42 officially unnecessary calls and retains 12/17 necessary calls, giving 54/59 official correctness. On GSM8K, it removes all 1,099 unnecessary calls but also 34/42 necessary calls; the original Full policy retains every path. These results limit the claim of reliable necessity detection across domains. For state-changing requests, a textual response alone cannot complete the required action.

The revised paper therefore distinguishes the consistent Irrelevance gains and matched component gains from the less favorable response-handling results. We believe the evidence that realized no-tool behavior improves action selection is a meaningful contribution, even though the complete policy does not offer a uniformly better quality–cost trade-off. We would appreciate your reconsideration of the score.

Thank you for the detailed response and the substantial additional experiments. The larger handling-adequacy audit, the full-coverage arbitration on GSM8K and Retriever, and the task-mixture analysis address my questions directly. I also appreciate that the revision reports results that go against the method. However, my main concerns remain, and in several places the new results strengthen them.

1. Quality of replacement responses.

The new audit (Table 6) shows that ToolNeed’s DIRECT responses are less adequate than the baseline’s:

  • 55.91% handling pass, versus 78.45% for RA-PF.
  • On the 113 common-DIRECT pairs, Full has 4 improvements and 26 regressions against RA-PF.
  • Requiring all three dimensions to pass gives 44/127 versus 70/116.
  • The earlier 34/40 result becomes 14/40 under the strict Correct-AND-Sufficient rubric.

A larger no-tool budget narrows the gap but does not close it. The paper also states that the judge shares a model family with the Qwen targets and that these are model-judge labels rather than human annotations. Given this, the lower tool invocation rate on SABEval and BFCL Irrelevance cannot be read as better handling of user requests.

2. Detecting valid but unnecessary calls.

The full-coverage results show that neither evaluated policy achieves this reliably:

  • GSM8K (Qwen3.5), original Full policy: it retains all 1,099 redundant calls.
  • GSM8K, arbitration on every case: it removes all of them, but also 34 of the 42 necessary calls, and final correctness falls to 1,107/1,141. The arbiter accepts incorrect arithmetic answers in most cases where the no-tool answer is wrong.
  • Retriever: full coverage gives 42/42 and 12/17 under official scoring. The 47/12 content-discrimination result comes from a supplementary inspection that the authors note is not an independent human evaluation.

So the original routing leaves the redundant calls in place, and wider arbitration does not reliably reject plausible but incorrect no-tool answers. That second failure is the risk I raised, since the arbiter works without gold answers. My earlier question about V1N0 versus V1N1 rates on time-sensitive and state-changing tasks also remains unanswered with data. The response addresses state-changing requests only qualitatively.

3. Overall trade-off.

The task-mixture analysis is helpful. However, when 75% of requests require tools, Full’s advantage over RA-PF is +0.28 pp (DeepSeek), +0.17 pp (Qwen3.7) and −0.68 pp (Qwen3.5). These are decision-level counts rather than end-to-end success rates.

The response provides this mixture analysis rather than the curve over the arbitration preference for DIRECT that I asked for, and it states that superiority at matched tool-required accuracy has not been established. The permissive-gate result on Qwen3.5 Live (+55 Irrelevance decisions, −63 Simple/Multiple decisions) likewise shows that expanding verification trades preservation for suppression.

Finally, the cost argument relies on a hypothetical $0.10 per external action, and the paper states that the available records do not support an aggregate latency estimate.

4. Scope of the contribution.

The matched comparisons are informative: ToolNeed-Base versus Full, and Masked-NoTool (+4.13 and +2.91 pp on Live Irrelevance). They support the authors’ narrower claim that the model’s actual no-tool response carries useful information for call decisions.

However, the title and abstract still frame the paper around separating tool validity from behavioral necessity, and the new diagnostics show that the method does not yet do this reliably. Reframing the paper around the information value of matched no-tool behavior would describe the evidence more accurately.

5. Presentation (minor).

The paper remains hard to follow. It relies on many custom labels (V1N0, C/R, D0–D4, A∅, and several ToolNeed variants), and the text is dense with statistics, so the main results are hard to pull out.

Separately, the break-even sentence in the response to Reviewer 1 appears garbled. Per Section 5.6, the $0.01097 per task figure assumes $0.10 per external action.

Overall, I value the careful and transparent experiments. At present, the revised evidence shows that the reduced tool use comes with lower handling adequacy than RA-PF and with losses on tool-required tasks, and that the current policy does not reliably identify valid but unnecessary calls. I am therefore currently inclined to keep my score, but I remain open to discussion and would consider the revisions, along with the authors’ and other reviewers’ views, in my final assessment.

AuthorsOpenReview ↗

Follow-up to Dt9J: Matched-Accuracy Comparison and Direct-Answer Review

Thank you for the follow-up. In Appendix K, we add the comparison you suggested at matched tool-required accuracy and review the resulting direct answers. On Qwen3.7, adding ToolNeed’s behavioral verifier after RA-PF leaves every Simple and Multiple correctness judgment unchanged and yields six more correct Irrelevance decisions.

Comparison with RA-PF. With conservative arbitration (C1), Qwen3.7’s Irrelevance accuracy increases from 654/824 to 660/824 (+0.73 pp). The gain did not reach statistical significance after Holm correction (p = .281). On Qwen3.5, C1 gets nine more Irrelevance examples right (649/824 to 658/824, +1.09 pp, Holm-adjusted p = .0391), with one fewer correct Multiple example and unchanged Simple accuracy. Table 31 reports both arbitration rules.

Direct-answer review. We reviewed all 18 cases in which C1 replaced an RA-PF tool call with a direct answer. In our AI-assisted author review, 11 answers were sufficient. Of the 15 cases that went from incorrect to correct under BFCL scoring, 10 yielded sufficient direct answers (5/6 on Qwen3.7 and 5/9 on Qwen3.5). The sole Multiple regression illustrates why we checked the answers separately. The model correctly calculated that 60 dollars remained, but BFCL marked the response incorrect because the example required a tool call.

We revised the abstract and introduction to reflect what these comparisons support. The model’s actual no-tool answer can help determine whether to keep a proposed call, even after RA-PF has refined it. For requests that depend on current information or require an external action, assessing fulfillment would require up-to-date evidence or confirmation that the action took place.

We also fixed the formatting issue you flagged in our response to Reviewer 1. We’d appreciate your considering the new comparison and answer review when reassessing the paper.

Thank you for the additional comparison and for revising the text. I appreciate the effort under time constraints. The new Appendix K answers my question about matched tool-required accuracy in a fair way. The revised abstract and introduction now qualify the GSM8K redundancy rates as conditional and state that suppression depends on routing and arbitration. The break-even description has also been corrected.

After reading the revision, however, my overall assessment is unchanged, for the following reasons.

Matched comparison: small add-on gain, not evidence for ToolNeed itself

The matched comparison shows a small gain from a composed variant, not from ToolNeed itself. With C1 applied after RA-PF, Irrelevance accuracy improves by 6/824 items on Qwen3.7 (+0.73 pp, Holm p=.281p = .281, not significant) and by 9/824 items on Qwen3.5 (+1.09 pp, Holm p=.0391p = .0391). This holds for two backbones on BFCL Live only. The evaluated system is RA-PF followed by a conservative behavioral verifier, not the proposed Full pipeline. The oracle reachability diagnostic further shows that Full's tool-required upper bounds (791/1227 and 733/1227) are below RA-PF's results (808/1227 and 749/1227). So the evidence supports a small add-on benefit of behavioral verification, not a better trade-off for ToolNeed as proposed.

Direct-answer quality remains the main concern

In the new review, 7 of the 18 TOOL-to-DIRECT replacements are insufficient. Among the 15 cases counted as BFCL corrections, 5 are insufficient. This review is AI-assisted and performed by the authors, and, as the appendix notes, it does not compare these answers with the results of executing RA-PF's tool calls. The broader audit in Appendix I, which the appendix itself cites as the main evidence on this question, still favors RA-PF (55.91% vs. 78.45% handling pass on the respective DIRECT sets).

Framing only partially aligned with the evidence

The title, the opening of the abstract and the contribution list still present the paper as distinguishing tool validity from behavioral necessity. Meanwhile, the original policy retains all 1,099 redundant GSM8K calls, and full-coverage arbitration removes 34 of the 42 necessary ones. The handling gap relative to RA-PF is also still not reflected in the abstract.

Clarity of presentation

The manuscript remains difficult to follow, which also makes its claims harder to assess. Concrete examples:

  • Figure 2 shows the arbiter comparing "A_base vs. A∅" and labels the raw candidate "A_raw". The caption and Section 3.3 state that the arbiter receives A_tool and A∅, and the text uses "A_tool".
  • Section 3.2 refers to "Stage B" decisions before Stage A or Stage B is introduced.
  • Section 3.4 describes routing through internal labels (Target Router, helper Router, APPLY HYBRID, PRESERVE RAW), with different procedures for the Qwen targets and DeepSeek, so the conditions under which verification runs are hard to grasp from the main text.
  • Many method variants are introduced: ToolNeed/"Full", ToolNeed-Base, ToolNeed-Helper, Masked-NoTool, full-coverage arbitration, the permissive-gate diagnostic variant, and C0/C1. "ToolNeed" and "Full" appear to be used interchangeably.
  • Results are often reported as dense strings of statistics, for example "([−3.52, +0.71], .18950, .2039)" in Section 4.3, without stating the takeaway.
  • Evidence needed to judge the central claim is spread across Appendices A–K.

A simplified method description with one consistent notation, a single table summarizing all variants, and short takeaway sentences in the results would make the paper substantially easier to evaluate.

Summary

In summary, the revised paper reports its results carefully and transparently. The matched comparisons support the narrower finding that the model's realized no-tool response carries useful decision information. However, the gains of the composed variant are small, the proposed pipeline still produces less adequate replacement answers than the baseline, and its preservation costs on tool-required tasks remain. I therefore maintain my score of weak reject. I encourage the authors to build the paper around the information value of matched no-tool behavior and the composed setting in Appendix K, and to evaluate it with independent assessment of answer quality on a broader set of tasks.

Reviewer_5zToOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates an important problem in tool-augmented LLM agents: determining whether a tool should be invoked at all, rather than merely whether a proposed tool call is valid. The authors distinguish tool validity from behavioral necessity, arguing that a tool call can be relevant, executable, and successful while remaining unnecessary if the model can accomplish the task without external tools. To address this problem, the paper proposes ToolNeed, a training-free, black-box-compatible framework that selectively verifies proposed tool actions. It consists of three main components: Candidate-blind routing, Requirement-grounded verification, Behavioral necessity verification. The authors evaluate ToolNeed on SABEval, ACEBench, BFCL, GSM8K, and a targeted When2Tool diagnostic, using three backbone models: DeepSeek-V4-Flash, Qwen3.7-Flash, and Qwen3.5-9B.

Critical strengths
  1. The GSM8K construct analysis provides interesting evidence that even successful and valid tool calls may be unnecessary relative to the model’s own capabilities.
  2. Training-free and black-box-compatible design. ToolNeed does not require modifying model parameters or accessing internal representations. This makes the framework potentially applicable to proprietary LLMs and existing agent systems where fine-tuning or hidden-state access is unavailable.
  3. Careful ablation studies and statistical analysis. The authors investigate the contribution of matched no-tool behavior using ToolNeed-Base and Masked-NoTool comparisons.
Critical weaknesses
  1. Performance improvements come with degradation in legitimate tool use. Although the authors emphasize preservation, Table 2 shows noticeable decreases on BFCL Live tool-required categories. The paper acknowledges these losses, but their practical significance deserves greater attention.
  2. The evaluation of behavioral necessity is limited in scope. The GSM8K analysis is conditioned on successful, valid tool-assisted paths, which provides useful evidence for the validity–necessity distinction but does not characterize the full distribution of tool-use decisions.
  3. The practical efficiency of the framework is unclear.
Questions for authors
  1. Definition of tool necessity. The paper defines necessity relative to the model’s matched no-tool behavior. However, tools can provide additional value through factual verification, information freshness, and reliability, even when the model can generate a correct answer without them. How does ToolNeed distinguish between a tool that is strictly necessary for answering and one that is valuable for improving answer reliability?
  2. Table 2 shows noticeable performance decreases on BFCL Live Simple and Multiple categories. Could the authors provide a more detailed failure analysis of cases where ToolNeed incorrectly suppresses necessary tool calls?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 5zTo

Thank you for the review. We have clarified the necessity criterion and added analyses of preservation failures and deployment cost.

Necessity includes freshness, verification, and execution requirements.

If a user asks for current information, verification against a specified source, or an external action, a plausible answer from memory does not fulfill the request. Section 3.1 now makes these requirements explicit. For example, describing a playlist does not fulfill a request to start playback.

ToolNeed judges whether the no-tool behavior meets the request, rather than whether it merely contains a correct statement. The arbiter estimates this without gold answers and can make mistakes. A DIRECT decision also does not establish that external verification would provide no additional reliability benefit.

We inspected the original Full policy’s preservation failures.

Across BFCL Live Simple and Multiple, Full has 0 corrections/35 regressions relative to Raw for DeepSeek, 0/17 for Qwen3.7, and 3/15 for Qwen3.5.

Appendix J.2 now gives concrete examples. In one Qwen3.5 case, Raw correctly calls ChaFod(BURGER), but behavioral arbitration selects DIRECT and returns an explanation of “Whopper.” The resulting response fails the official function-call evaluation. This is an example of a natural-language response incorrectly replacing a required call.

Not every regression has this cause. Two other inspected cases bypass behavioral arbitration, but the Full run’s initial candidate omits a required argument or duplicates a function call. We therefore distinguish system-level regressions from errors directly caused by arbitration.

The matched ToolNeed-Base comparison separates the behavioral branch from these initial-candidate differences. With upstream behavior shared, adding no-tool generation and arbitration gives 38 corrections and six regressions on Qwen3.7 Live, and 42 corrections and eight regressions on Qwen3.5 Live (Table 5). The gains mainly come from Irrelevance, while some tool-required decisions are lost. On Non-Live, tool-required category accuracy remains within one percentage point of Raw.

Paid or expensive external services provide a practical use case.

Verification is particularly relevant when external actions involve paid API requests or substantial computation. It can reduce total cost when the avoided calls cost more than the additional verification.

For example, on DeepSeek BFCL Live, Full avoids 0.1097 function calls per task relative to RA-PF. At a hypothetical price of USD 0.10 per call, this leaves USD 0.01097 per task to spend on verification before the savings are exhausted. Section 5.6 and Appendix F provide the action counts and break-even analysis. This gives a deployment criterion: use verification where external-action costs justify it and the resulting task quality meets the application’s requirements.

The main empirical result is consistent across the evaluated models: Full improves Irrelevance over RA-PF in all six backbone–regime settings. The matched behavioral-path results further show that the no-tool comparison contributes useful decisions beyond the base verifier. We have kept these gains and their preservation costs visible in the revision, and would appreciate your reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.