- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when a large language model should abandon a tool call it has already proposed. ToolNeed first determines the task requirement without examining the specific call, and then checks whether the call matches that requirement. For samples that trigger behavioral verification, it has the same model answer again with the tool removed, and uses contrastive arbitration to decide whether to retain the tool call. The method requires no training and can be applied to black-box models.
Critical strengths
- The problem is practically meaningful, and the method is easy to follow. Checking whether a call is correct cannot fully answer whether a call is needed, and using the same model's actual response without tools as a reference provides more concrete information than merely asking the model to judge "whether a tool is needed."
- The results on BFCL are fairly consistent. Across six settings—three models on both Live and Non-Live—ToolNeed achieves higher Irrelevance accuracy than RA-PF, and the differences survive multiple-comparison correction. The authors also report separately how well normal tool calls are retained, rather than showing only a drop in call rate.
- The statistical analysis and ablations are done with care. SABEval uses template-level clustered statistics, BFCL uses paired tests, and the Masked-NoTool ablation hides the no-tool response while preserving the arbitration pipeline, verifying that the actual response content does provide additional information.
Critical weaknesses
- There is still a gap between the core claim and the main evaluation. SABEval and BFCL Irrelevance mainly measure whether a call can be avoided when the tool is inapplicable, whereas the paper emphasizes whether a relevant, valid tool call can still be omitted. The GSM8K analysis demonstrates that such cases are common, but it does not further report ToolNeed's decision performance on this sample set. The Retriever experiment that directly evaluates this capability covers only 59 filtered samples from a single model, so the evidence remains limited.
- The Retriever experiment cannot yet adequately verify the arbitrator's ability to distinguish necessity. Of the 42 unnecessary calls, only 10 trigger verification and are canceled, while the remaining 32 are still retained; all 17 necessary calls bypass arbitration. Thus, "all necessary calls are retained" mainly reflects the conservativeness of the routing, and it cannot be used to judge whether the arbitrator can recognize misleading erroneous no-tool responses.
- The quality of the substitute responses is insufficiently validated. A lower call rate does not necessarily mean that the user's request has been handled well. The paper's semantic review covers only 40 samples, 6 of which contain factual errors or omissions of key content; moreover, the evaluator and the arbitrator use the same model, DeepSeek. For a method that replaces tool calls with ordinary responses, a larger-scale, independent evaluation of response quality is necessary.
- The loss of normal tool calls needs more thorough analysis. On BFCL Live, five of the six Simple/Multiple comparisons show a significant drop relative to Raw. For example, Simple accuracy drops by about 3.7 percentage points for both DeepSeek and Qwen3.7. Only a fixed policy is currently reported, and it remains unclear whether ToolNeed still outperforms the feedback baseline at the same level of normal-call loss.
Questions for authors
- Could the full ToolNeed be run on the samples for which labels have already been obtained in the GSM8K construction analysis, reporting the cancellation rate for unnecessary calls, the false-deletion rate for necessary calls, and the final answer accuracy? This would more directly validate the paper's central claim.
- What would happen if behavioral arbitration were applied to all 59 valid calls in the Retriever setting, rather than only to the 10 samples selected by the routing? I would like to understand how much each of routing and arbitration contributes, and whether the arbitrator is reliable when faced with erroneous but plausible no-tool responses.
- In Table 2, Qwen3.5's BFCL Live results are 57.8% / 59.0% / 82.9%, whereas the ToolNeed results in the ablation discussion on page 17 are 55.96% / 59.76% / 82.40%. What exactly differ in the settings between these two experiments?
Response to Reviewer YDeE
Thank you for these concrete suggestions. We ran the complete GSM8K policy test, applied arbitration to all 59 Retriever cases, and clarified the source of the two Qwen3.5 tables.
Full ToolNeed on GSM8K.
The original Full policy retains every tool call on the 1,141 qualifying cases, including all 1,099 cases where the no-tool answer is correct. Your concern is borne out on this set: the prevalence of redundant calls does not translate into suppression by the original policy.
Applying arbitration to every case removes all 1,099 redundant calls, but also removes 34 of the 42 calls whose no-tool answers are incorrect. Final correctness falls from 1,141/1,141 to 1,107/1,141. A separately frozen permissive gate gives the same result. Broader verification reaches the redundant calls, but the arbiter often accepts incorrect arithmetic answers.
Both results are now in Section 5.1 and Appendix G, Table 22. The original policy’s perfect conditional correctness follows from retaining the already-correct tool paths; it is not evidence that it successfully distinguishes these cases.
Arbitration on all Retriever cases.
Using the same arbitration outputs for the policy comparison, we obtain:
You are right that the original 17/17 retention does not test the arbiter, because all 17 cases bypass it. The full-coverage experiment now includes those cases.
The five official failures involve entity-equivalent variants, such as “Gabriel García Márquez” instead of “Gabriel Garcia Marquez.” A supplementary contextual inspection finds correct entity content in 47 no-tool answers and incorrect content in 12; arbitration chooses DIRECT for all 47 and TOOL for all 12. This provides useful evidence about answer-content discrimination on the retrieval set. We retain 54/59 as the official result, since entity equivalence does not establish source or format compliance, and the inspection is not independent human evaluation. These results appear in Section 5.2, Table 4, and Appendix H.
The two Qwen3.5 tables.
The tables come from separate runs on the same 2,051 tasks, with correct counts of 126/595/683 for the main evaluation and 122/603/679 for the component analysis. Differences can arise from run-to-run variation in model outputs and auxiliary-service execution; the available records do not isolate their respective contributions here. All component comparisons use shared upstream outputs within their own run, rather than the main-table predictions. Appendix D now clarifies this distinction.
Response handling and preservation.
We expanded the response audit to 200 task–model pairs and used a method-blinded Qwen3.7 judge rather than the DeepSeek arbiter. Handling adequacy is scored separately from factual correctness and coverage. A larger no-tool budget improves Full’s handling, but the same-source comparison still favors RA-PF: among 113 common-DIRECT pairs, Full has four improvements and 26 regressions, with 80 ties and three uncertain pairs (Section 5.5 and Appendix I).
The matched component results remain positive. Relative to ToolNeed-Base, the behavioral branch gives 38 corrections/six regressions on Qwen3.7 Live and 42/eight on Qwen3.5 Live. Showing the actual no-tool answer rather than masking it improves Live Irrelevance by 4.13 and 2.91 points, subject to the control’s limitations described in Appendix E.
A broader gate is not an automatic improvement: on Qwen3.5 Live it adds 55 correct Irrelevance decisions but loses 63 correct Simple/Multiple decisions relative to Full. This does not establish superiority at matched preservation, but shows the trade-off when verification is expanded.
The completed tests support the usefulness of the behavioral comparison while showing where the current policy misses redundant calls or accepts incorrect alternatives. We have revised the paper accordingly and would be grateful if you could reconsider your score.