acceptodds
Under review as a conference paper at ICLR 2027

ToolHallu: Mitigating Hallucination in Agentic Tool Calls via Reinforcement Learning

Abstract

Large language model (LLM)-based agents have been applied across various tasks. A key factor underpinning their powerful capabilities is the use of external tools. While LLMs provide the basic understanding required for tool invocation, their susceptibility to hallucination continues to threaten the reliability of LLM-based agents, making hallucination mitigation imperative for reliable agent deployment. In this work, we propose a novel reinforcement learning (RL) framework named ToolHallu to tackle tool-call hallucination. We first systematically analyze hallucination patterns specific to agentic tool invocation. Based on this, we decompose tool-call hallucinations into distinct error types and assign differentiated rewards to provide interpretable training signals. To further address the varying difficulty of these errors, we introduce a curriculum learning schedule to adjust the optimization process, which first guides the model to learn accurate tool selection and then progressively shifts the optimization focus towards more fine-grained parameter-level hallucinations. Extensive experiments across multiple models and benchmark datasets demonstrate that our method consistently reduces hallucination rates and improves invocation reliability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic Tool Calls, Hallucination Mitigation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CjNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ToolHallu addresses hallucinations in language model agents’ tool calls, including incorrect tool selection, invalid parameters, and repeated calls. It adds fine-grained rewards for these errors to a reinforcement learning framework. A curriculum schedule shifts the reward emphasis from selecting the correct tool to generating accurate parameters. Experiments on API-Bank and StableToolBench show lower overall tool-call hallucination rates across the evaluated models, with accuracy gains in many settings. The evaluation does not execute real APIs, so its results do not establish improvements in end-to-end task completion.

Critical strengths

1.A clearly defined problem. The paper distinguishes incorrect tool selection, parameter errors, and repeated calls, making failures easier to diagnose than evaluating final answers alone.

2.Targeted reward signals. It scores errors such as invoking nonexistent tools and using incorrect parameter types separately, providing direct guidance for improving tool calls.

Critical weaknesses

1.No actual tool execution. The results mainly show that generated calls satisfy the evaluation criteria; they do not establish success in a real API environment.

2.Task accuracy may be overstated. On StableToolBench, a reasonable first step can receive full credit even if the task remains incomplete.

3.Accuracy drops substantially in some subsets. For example, on StableToolBench G2 with Qwen3 0.6B, accuracy falls from 0.58 with GRPO to 0.36 with the proposed method.

Questions for authors

1.Could you add multi-turn experiments with actual API execution? Do the reported gains translate into higher end-to-end task completion rates?

2.How many tool calls does each method make on average, and how often does it decline to call a tool? Could part of the lower hallucination rate result from making fewer calls?

1 comment

We would like to express our gratitude for your valuable suggestions!

[Q1] No actual tool execution.

We initially align our tool-call evaluation with [1, 2] using LLM-as-a-judge. We conducted additional experiments such as BFCL's multi-turn tasks, ACE-Bench, and Tau-Bench, which use interactive API simulators. Table 1, 2 and 3 reveal that our method consistently reduces the hallucination rate.

Table 1. The multi-turn results evaluated on BFCL.

MethodMulti Turn AccMulti Turn BaseMulti Turn Miss FuncMulti Turn Miss ParamMulti Turn Long Context
Base0.62%0.50%0.00%1.00%1.00%
GRPO2.38%4.00%1.00%1.50%3.00%
ResT1.75%3.00%1.00%1.00%2.00%
Ours4.00%6.00%3.00%4.00%3.00%

Table 2. The results evaluated on ACE-Bench with Qwen3-0.6B.

MethodSummaryHallu rate
Base0.160.58
GRPO0.180.70
ResT0.170.64
Ours0.180.53

Table 3. The results evaluated on Tau-Bench with Llama-3.2-3B-Insturct.

MethodOverall pass@1Hallu rate
Base0.140.80
GRPO0.150.76
Ours0.150.72

[Q2] Task accuracy may be overstated.

We first align our evaluation protocol with API-Bank for evaluation efficiency and convenience. Since errors and failures at a given step accumulate across multi-turn interactions, localized issues (at a specific step) can reflect overall performance. Furthermore, because all methods are evaluated under this identical framework, the comparison remains fair.

We also conduct experiments running the full process on StableToolBench to see the performance. Our method still outperforms others.

Table 4. The full turn results evaluated on Qwen3-0.6B.

MethodSolve rateHallu rate
Base3.01%93.94%
GRPO6.08%88.22%
ResT5.82%87.72%
Ours8.04%80.70%

[Q3] Accuracy drops substantially on some subsets G2,Qwen3−06˙BG2, Qwen3-0\.6B.

We inadvertently misaligned the table entries, impacting four table cells. After correcting this, the updated results generally outperform the GRPO baseline. The overall accuracy and hallucination rate are correct and don't need fixing.

Table 5. Fixed table of Qwen3-0.6B on StableToolBench

Overall AccOverall HalluG1 AccG1 HalluG2 AccG2 HalluG3 AccG3 Hallu
GRPO0.660.420.720.380.580.440.540.64
Ours-mistake0.670.400.750.360.360.570.420.52
Ours-checked0.670.400.750.360.570.420.520.62

[Q4] Call statistics and abstention.

We report the tool calling times under different training methods in Table 6. The results show that the model learns to correct hallucinated tool calls instead of avoiding calling tools.

Table 6. The comparison of different training methods regarding the tool-calling times tested on API-Bank with Qwen3-1.7B as the base model.

MethodOverall HalluTool CallsTool Calls Avg
Base0.336911.16
GRPO0.216071.02
ResT0.216351.06
Ours0.166021.01

In multi-turn situations, we evaluated the BFCL multi-turn subset as shown in Table 7. Our method retained the tool-calling willingness and did a relatively better job on Multi Turn Miss Func and Multi Turn Miss Param, which are specifically designed to see whether the model can avoid unreasonable tool calls.

Table 7. The tool calls under multi-turn situations in BFCL with different training methods.

BaseLong contextMiss funcMiss param
MethodAvgAccAvgAccAvgAccAvgAcc
Base3.250.50%1.530.00%1.861.00%1.461.00%
GRPO3.404.00%3.051.00%3.461.50%3.843.00%
ResT2.593.00%2.291.00%4.671.00%4.872.00%
Ours3.786.00%3.353.00%4.004.00%4.003.00%

The Irrelevance Detection subset of BFCL evaluates the model's ability to reject unreasonable tool calls. ToolHallu out-performs other training methods, as shown in Table8, indicating our success of reducing tool-calling hallucination.

Table 8. The Irrelevance Detection results evaluated on BFCL.

Irrelevance Detection
Base84.64%
Grpo81.02%
ResT82.74%
Ours93.95%

[1] Cheng Qian et al. “ToolRL: Reward is All Tool Learning Needs.” NeurIPS 2025

[2] Zihan Lin et al. “ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models.” ICLR 2026

Reviewer_Sm4jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose an RLVR approach for reducing hallucinations in LLM-generated tool calls, called ToolHallu. It adds reward designs for tool selection, argument validity, argument values, and repeated invocations. In addition to these, they also implement a cosine-based curriculum approach which initially emphasizes tool selection and gradually shifts attention toward argument correctness.
They evaluate six models from the Qwen3 and Llama-3.2 families on API-Bank and StableToolBench, and according to the authors claim, their RLVR approach can lower overall hallucination rates with overall accuracy improvements in different settings.

Critical strengths
  • Authors focus on a timely challenge by adapting an RLVR approach for tool hallucination which can be important for the current agent community, especially in long-horizon practical applications.

  • They conduct human studies and report 95.7% agreement with the evaluator across 300 judgments, and three-seed experiments on Llama-3.2-3B show that the improvements hold steady on both benchmarks which increase reliability of the provided numbers in the current version of manuscript.

Critical weaknesses
  • The main limitation of this work is coming from novelty of the overall idea and the insufficient acknowledgment of prior work. Tool hallucination is a well-known problem in the LLM agent literature. Agent-Flan [1] first identified format and action hallucinations in tool calls (Section 4.2), and xLAM [2] discusses most of the errors mentioned in this paper, explicitly categorizing them by function name, argument value, and argument type (Section 3.3), and others also evaluated on those [3, 4, 5]. Given this context, it is unclear what algorithmic or scientific contribution the paper makes beyond prior work. It raises the concern on where the proposed method overlaps with existing approaches, is there evidence that it performs better?
  • Authors also present curriculum learning-based approach for tool use, but it is unclear how the curriculum is constructed and what progression it produces. Is there quantitative and qualitative evidence that the model learns stably and progressively through the curriculum? Moreover, although the curriculum-based reward is interesting; it is covered for tool-learning with calibration [6] and self-play [7]; similarly, given the missing previous work, how does the proposed method compare with these approaches?
  • Finally, another major limitation is the simplicity of the evaluation environments, which makes it difficult to assess how well the method generalizes to more realistic and complex settings, such as BFCL (which also includes a hallucination category), Tau-Bench, and ACE-Bench. It is highly recommended for authors to include more baselines from other tool-using agents [2, 3, 4, 5], evaluated under the same base model and training setup to provide a clearer comparison with prior work which helps reviewers to understand robustness of the approach.

References

[1] Chen et al. “Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models.” ACL 2024 Findings

[2] Zhang et al. “xLAM: A Family of Large Action Models to Empower AI Agent Systems.” NAACL 2025

[3] Lin et al. "Robust Function-Calling for On-Device Language Model via Function Masking." ICLR 2025

[4] Liu et al. “ToolACE: Winning the Points of LLM Function Calling.” ICLR 2025

[5] Acikgoz et al. “Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model.” ACL 2025

[6] Zeng et al. “iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Use”. EMNLP 2025

[7] Acikgoz et al. “Tool-r0: Self-evolving llm agents for tool-learning from zero data.” EMNLP 2026 Findings

Questions for authors
  1. How does the method perform on BFCL, Tau-Bench, and ACE-Bench in terms of general and hallucination performance, including multi-turn scenarios?

  2. Could the authors report results for Hammer, xLAM, ToolACE, and CoALM as baselines under a fair experimental setup?

  3. It would also help to see the RL training dynamics, such as reward and entropy curves.
2 comments

We thank the reviewer for the valuable insight! In the revision, we will further make an effort to clarify and justify our motivations and claims.

[Q1] Contribution beyond existing works.

Thank you for mentioning the related work! We would clarify the following aspects to address your concerns about novelty and acknowledgment of prior work.

  1. We first would like to highlight existing works that improve tool-call capabilities primarily focus on data processing strategies. [2, 4, 5] filter hallucinated data from data construction to guarantee the quality of the training data. Agent-FLAN [1] constructs negative samples via two counterfactual scenarios: queries requesting tools when none are provided, and normal conversational queries when tools are available. Hammer [3] builds negative samples by removing the ground-truth function from the tool list and relabeling it as irrelevant; it further randomizes function/parameter names and default values while keeping descriptions intact, forcing the model to infer function purpose from descriptions rather than naming conventions.

  2. Differently, we first identify the fine-grained taxonomy of tool-call hallucination and correspondingly propose a delicate hierarchical reward design to solve this problem. Moreover, we incorporate a delicate curriculum learning strategy that enhances invocation fidelity from coarse-grained tool selection to fine-grained argument correctness. In summary, we solve this issue from the perspective of algorithm-level design, instead of data-level processing.

  3. To show the better performance of our method, we provide a comparison of these methods as follows. We follow the training settings and dataset usage assigned by previous works, reporting the training gains on the same base model in Table 1. The results confirm the superior performance of our method.

Table 1. Comparison of ToolHallu and the proposed methods as baselines trained on Llama-3.2-3B-Instruct, tested on API-Bank.

BaseOursGRPOToolACEHammerxLAMCoALM
Overall Acc0.340.770.600.570.700.520.60
Overall Hallu0.620.290.390.380.410.350.34
  1. For acknowledgment of prior work, we actually have referred to [2, 3, 4] in Section 4.1. Following your suggestion, we further add [1,5] and discuss all of these works to our appendix in the revision.

[Q2] Construction and effect of the curriculum learning.

  1. Difference of our method to previous studies. iTool [6] implements curriculum learning by categorizing training samples into easy, medium, and hard subsets based on three difficulty criteria (number of candidate tools, toolset string length, and required number of tool calls), and the model is sequentially fine-tuned on these subsets.Tool-R0 [6] uses a dynamic self-play process in which the Solver calibrates the Generator's task difficulty via Solver success rates, and the generated tasks are then fed back to train the Solver in an easy-to-hard order. Both methods define curriculum as difficulty variation within a fixed task domain (e.g., different chapters of the same course). In contrast, our work defines curriculum as topic variation (e.g., different courses). We disentangle this by allocating different learning efforts to tool-name selection versus tool-parameter filling as training progresses.

  2. Table 2 presents evidence of the learning dynamics in our method by comparing training with and without curriculum learning using hallucination validation metrics. The gap of RnameR_{name} (tool name related reward, equal to -HnameH_{name}) narrows rapidly in the early stages, as we allocate most of the learning effort to tool-name selection. The RparamR_{param} of the curriculum scheme lag behind those of the fixed-weight scheme in the early stage, essentially catch up in the middle stage, and consistently lead at the final validation checkpoints. This phased pattern is consistent with the curriculum design that progressively strengthens parameter supervision, supporting the curriculum scheme's improvement in parameter-related performance at later stages.

Table 2. The change of name-related and parameter-related rewards during training on Llama-3.2-3B-Instruct. We report the gap between training without and with curriculum learning. A negative value of the gap indicates that training with curriculum learning is better.

Step102030405060708090100105
RnameR_{name}w/o Curriculum Learning0.34810.32590.33120.35320.36080.36710.34890.35530.34470.32570.3257
w Curriculum Learning0.25050.29320.31430.32150.33420.34680.34680.35110.35110.33840.3511
gap9.76E-023.27E-021.69E-023.16E-022.66E-022.03E-022.10E-034.20E-03-6.30E-03-1.27E-02-2.53E-02
RparamR_{param}w/o Curriculum Learning0.52520.92420.97710.99151.01781.03021.02681.02081.00060.99140.9963
w Curriculum Learning0.49970.87290.97340.99761.02081.03091.02491.03491.02471.00361.0426
gap2.55E-025.13E-023.70E-03-6.10E-03-3.00E-03-7.00E-041.90E-03-1.41E-02-2.41E-02-1.22E-02-4.63E-02

[Q3] More complex evaluation datasets.

We report more testing results on BFCL, ACE-Bench and Tau-Bench in their English settings, covering the scenes of multi-turn. Our method reduces the hallucination rate notably.

Table 3. The results evaluated on BFCL with Qwen3-0.6B. The Hallucination Measurement consists of relevant detection and irrelevant detection.

MethodNon-Live AST AccLive AccMulti Turn AccHallucination Measurement
Base66.44%43.52%0.62%84.15%
GRPO69.10%54.77%2.38%81.02%
ResT69.90%52.41%1.75%82.54%
Ours75.27%57.44%4.00%93.51%

Table 4. The results evaluated on ACE-Bench with Qwen3-0.6B.

MethodSummaryHallu rate
Base0.160.58
GRPO0.180.70
ResT0.170.64
Ours0.180.53

Table 5. The results evaluated on Tau-Bench with Llama-3.2-3B-Insturct.

MethodOverall pass@1Hallu rate
Base0.140.80
GRPO0.150.76
Ours0.150.72

[1] Chen et al. “Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models.” ACL 2024 Findings

[2] Zhang et al. “xLAM: A Family of Large Action Models to Empower AI Agent Systems.” NAACL 2025

[3] Lin et al. "Robust Function-Calling for On-Device Language Model via Function Masking." ICLR 2025

[4] Liu et al. “ToolACE: Winning the Points of LLM Function Calling.” ICLR 2025

[5] Acikgoz et al. “Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model.” ACL 2025

[6] Zeng et al. “iTool: Reinforced Fine-Tuning with Dynamic Deficiency Calibration for Advanced Tool Use”. EMNLP 2025

[7] Acikgoz et al. “Tool-r0: Self-evolving llm agents for tool-learning from zero data.” EMNLP 2026 Findings

Reviewer_MbsFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ToolHallu, a reinforcement learning method that reduces hallucinations in tool calls generated by LLMs. The method builds on GRPO and adds rule-based rewards for different errors, including choosing nonexistent or irrelevant tools, providing missing or incorrect arguments, and repeating the same call. It also adjusts the reward weights during training, first emphasizing correct tool selection and then gradually increasing the focus on argument correctness. Experiments with Qwen3 and Llama3.2 models on API-Bank and StableToolBench show lower overall hallucination rates and improved accuracy in most settings.

Critical strengths
  1. Useful insights. The experiments provide useful insights into tool-call hallucinations. The results show that higher tool-calling accuracy does not always lead to fewer hallucinations. The ablation studies also provide insights into the effects of different reward components and curriculum learning.
  2. Clear and easy-to-follow presentation. The paper is well-written and easy to follow. The motivation, method, and experimental setup are clearly explained.
Critical weaknesses
  1. Lack of evaluation on BFCL. The paper only evaluates ToolHallu on API-Bank and StableToolBench, without including the Berkeley Function Calling Leaderboard (BFCL), which is widely used to evaluate tool-calling capabilities. BFCL covers a broader range of scenarios, including single-turn, multi-turn, and parallel tool calls. Without results on BFCL, it is difficult to assess how well the proposed method generalizes to different tool-calling scenarios and how it compares with existing function-calling models.
  2. Limited comparison with specialized tool-calling models. Existing models such as ToolACE and Hammer 2.1 have already been specifically trained for tool calling. However, the paper does not compare with these models or investigate whether ToolHallu can further improve their performance. It would be valuable to apply ToolHallu to these already fine-tuned models and evaluate whether it brings additional improvements.
Questions for authors

Can ToolHallu further improve existing tool-calling models, such as ToolACE and Hammer 2.1, that have already been trained for tool use?

3 comments

Thank you very much for appreciating our work! We express our sincere gratitude for your positive assessment of the paper's clarity. We have added BFCL and a continued-RL experiment on released checkpoints of tool-calling models.

[Q1] Lack of evaluation on BFCL.

We add BFCL for Qwen3-0.6B under base mode, GRPO, ResT, and ToolHallu, focusing on accuracy and hallucination tasks. As shown, our method outperforms the baselines under BFCL.

Table 1. The results evaluated on BFCL with Qwen3-0.6B. The Hallucination Measurement consists of relevant detection and irrelevant detection.

MethodNon-Live AST AccLive AccMulti Turn AccHallucination Measurement
Base66.44%43.52%0.62%84.15%
GRPO69.10%54.77%2.38%81.02%
ResT69.90%52.41%1.75%82.54%
Ours75.27%57.44%4.00%93.51%

[Q2] Comparison with specialised tool-calling models ToolACE,Hammer21˙ToolACE, Hammer 2\.1.

Thank you for your valuable insight!

To verify the generalizability of our methods, we implement ToolHallu on fine-tuned tool-calling models to see whether ToolHallu can further improve their performance. The results show that ToolHallu can still improve accuracy and reduce hallucinations. This can be attributed to ToolHallu learning from the model's mistakes, with our delicate reward guidance and curriculum learning dynamic.

Table 2. Improvement gained by applying ToolHallu on Hammer-2.1 1.5B and ToolACE 8B, tested on API-Bank.

Overall AccOverall HalluLevel 1 AccLevel 1 HalluLevel 2 AccLevel 2 HalluLevel 3 AccLevel 3 Hallu
Hammer2.1-1.5b0.600.220.590.190.520.350.650.24
Hammer2.1-1.5b-Ours0.690.190.710.170.660.30.650.19
ToolACE-8B0.730.270.780.240.610.380.610.31
ToolACE-8B-Ours0.830.180.890.130.820.230.650.28

Thank you for the response. I will raise my score to 3.

Thank you! We sincerely appreciate your constructive comments and positive feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.