acceptodds
Under review as a conference paper at ICLR 2027

Agents' Overreliance on Unreliable Tools

Abstract

LLM agents use tools to access information and perform computations beyond their parametric knowledge. Existing tool-use benchmarks evaluate whether agents select and call the right tools, assuming that tool returns are reliable. However, tools can return plausible but incorrect outputs. We evaluate 14 models with three tools (web search, an LLM sub-agent, and a code executor), corrupting their returns to examine whether agents overrely on unreliable tools. Agents adopt corrupted returns at high rates, with mean adoption exceeding one third for every tool and reaching 68.0% for web search. Corrupted returns also frequently override correct answers that agents give without tools, and this reliance persists in more complex tasks and in tasks that combine multiple tools. Agents often make more tool calls under corrupted returns, and reasoning traces show them questioning a return and even stating the correct answer. Nevertheless, they still pass the corrupted content to users, rarely warning them of the conflict. We further test interventions spanning prompts, tool metadata, post-training, and activation steering. Verification prompts reduce adoption across all three tools while largely preserving accuracy with correct returns, and reliability labels provide partial mitigation in web search. The tested post-training methods offer limited gains, and the effect of activation steering depends on the model and intervention strength. Overall, our findings provide a basis for diagnosing tool overreliance and for developing agents that assess tool returns rather than assuming their reliability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, Tool-use, Knowledge Conflicts, Safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k74WOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates LLM agents' overreliance on unreliable tool outputs. The authors evaluate 14 models across three tool types: web search, LLM sub-agents, and code executors, using controlled output corruption. Results reveal substantial adoption of incorrect tool outputs, even when models can independently produce correct answers. The paper further investigates this behavior through reasoning traces and evaluates mitigation strategies. Verification-based prompting demonstrates promising improvements in reliability.

Critical strengths
  1. Important and practical problem. The paper highlights a critical reliability issue in tool-augmented LLM agents. Distinguishing tool-output adoption from the overriding of previously correct answers provides valuable insights.

  2. Extensive empirical analysis. The evaluation spans diverse models, tools, and corruption settings. The investigation of reasoning behavior and multiple mitigation strategies further strengthens the contribution.

Critical weaknesses
  1. The evaluation relies primarily on synthetically corrupted tool outputs, which may not fully capture the diversity of naturally occurring tool failures.

  2. Although the mitigation experiments are informative, the computational and tool-calling overhead introduced by additional verification remains insufficiently explored.

  3. There seems to be many API calls in the experiments, but the author did not provide any cost information.

Questions for authors
  1. Could the authors clarify how well the proposed corruption settings represent naturally occurring tool errors and whether verification overhead can be quantified?

  2. Providing detailed API cost in the experiments could help to verify the real attack strength. The authors should add a figure or a table in corresponding sections.

  3. The related work discussion could more clearly position this study alongside existing evaluations of tool-using agents, such as AgentDojo, ToolSandbox, and ToolEmu, Sponge Tool Attack and Efficiency in LLM Multi-Agent Systems: A Survey. These connections would help clarify the distinct contributions and practical implications of the proposed approach.

I will adjust my rating according to the authors' rebuttal.

Reviewer_H2GkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates LLM agents' reliance on unreliable tool outputs across three settings: web search, LLM sub-agent delegation, and code execution. The authors introduce controlled corruptions into tool responses and evaluate whether agents incorporate the corrupted information into their final answers. They propose Adoption Rate and Override Rate to quantify this behavior and examine the effects across different models, tasks, and corruption conditions.

Critical strengths

The paper provides a systematic empirical evaluation across three distinct tool categories—web search, LLM sub-agent delegation, and code execution—and reports results across multiple models and corruption conditions. This offers a useful descriptive characterization of how incorrect tool information can propagate into agents' final answers.

Critical weaknesses

1. Adopting an incorrect tool output does not, in itself, constitute "avoidable" over-reliance.

The authors interpret an agent’s adoption of an incorrect tool output as evidence of "over-reliance." However, when a tool returns a result that appears plausible—and the agent lacks an independent channel for verification—adopting that result is not necessarily unreasonable. Models rely on external tools precisely because their own parametric knowledge may be incomplete or outdated. For time-sensitive facts, prioritizing external information over internal knowledge can be a rational design choice.

2. The evaluation fails to clarify whether the injected errors could have been identified given the information available to the agent.

The paper does not adequately distinguish between "errors that an agent should be able to detect" and "errors that cannot be reliably identified based on the available evidence." This distinction is particularly critical when an erroneous tool output is logically consistent and the agent has no access to an independent source of "ground truth."

3. A more fundamental question is whether this evaluation actually measures "over-reliance," or simply the model's "tendency to trust the output of external tools." These are not the same thing: reducing trust in a tool does not necessarily improve an agent's ability to distinguish between reliable and unreliable information.

4. Discussion of related works is not thorough. There are several papers that are related to this submission yet not compared against.

  • When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models. ICML 2026
  • SafeSearch: Automated Red-Teaming of LLM-Based Search Agents. ICML 2026
  • Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions. arXiv:2607.20891
Questions for authors
  1. How do the authors distinguish incorrect adoption from unjustified overreliance? In particular, why should an agent be expected to reject a plausible external tool response merely because it could answer the same question correctly without using tools? What operational criterion establishes that trusting the tool is unreasonable in a particular case?

  2. How do the authors determine whether an injected error is detectable given the information available to the agent? Have the authors considered evaluating whether a human, given exactly the same task context and tool outputs but without access to the ground truth, could identify the incorrect information? How would the conclusions change if some errors were effectively impossible to verify?

  3. Do the proposed metrics measure overreliance, or merely a model's tendency to trust tool outputs? How do the authors distinguish genuinely improved reliability from a general reduction in willingness to use external information? In particular, how can the evaluation establish that an intervention improves the agent's ability to discriminate between correct and incorrect tool outputs rather than simply discouraging tool reliance?

5 comments

Thanks for your response and the additional experiments. The 64.6% rejection of correct answers suggests that rejecting a candidate does not imply detecting an error, so relying on tools in such cases is not necessarily unreasonable. The detectability of errors in the original tasks, especially P1 and Sub-Agent, also remains insufficiently established. I will maintain my rating.

AuthorsOpenReview ↗

Follow-up response - [1/2] Error detectability across tasks

Thank you for engaging with our response and raising this point. First, we want to clarify that our earlier check of 64.6% was conducted only on Search questions that models answered incorrectly without tools. We omitted P1 from that check because we do not claim that adopting P1 on unknown questions is overreliance. To provide a complete picture that directly addresses P1 and Sub-Agent, we have now compiled the full results across all tasks for both known and unknown items. In addition, we conducted a targeted analysis showing which corrupted returns should still be rejected and why adopting them constitutes overreliance, even when the model cannot answer on its own.

Specifically, without access to tools, each of the nine models was given the task prompt and a proposed answer (either the gold answer or the exact injected value) across all items in Search, Code Executor, and Sub-Agent, and rated it on a four-level scale (Correct, Unsure, Probably incorrect, Definitely incorrect) with a one-line rationale. Values below are averages across the nine models (%).

(1) Detectability of errors in each task

DomainItemsProposed answerCorrectUnsureProbably incorrectDefinitely incorrect
SearchAnswered correctly without toolsGold91.52.41.74.4
P18.60.97.583.0
P20.30.30.399.1
P30.10.10.699.2
Not answered correctly without toolsGold24.511.116.048.4
P127.912.515.644.1
P20.10.80.798.4
P30.61.73.594.1
Code ExecutorAnswered correctly without toolsGold96.30.00.82.8
P10.20.00.499.4
P20.00.00.0100.0
Not answered correctly without toolsGold54.61.411.732.3
P12.40.00.796.9
P21.10.00.098.9
Sub-AgentAnswered correctly without toolsGold95.31.41.71.6
P111.63.522.162.8
P27.27.020.765.1
Not answered correctly without toolsGold67.316.35.810.7
P112.413.113.161.4
P28.117.815.259.0

On items answered correctly without tools, models reliably flag corrupted values as incorrect across all tasks, including P1 and Sub-Agent, while accepting Gold answers. Because models both know the right answer and recognize that the injected return is wrong here, abandoning their own answer for the tool return (Override) reflects overreliance; at minimum, the agent is aware of the conflict and should surface it rather than presenting the return as settled.

On items missed without tools, failing to produce or recognize the exact gold answer does not prevent a model from detecting concrete flaws in a corrupted return. The sole exception is unfamiliar Search P1, where Gold and P1 are rejected at similar rates because plausible distractors on unknown facts cannot be ruled out from parametric knowledge alone. Across all other unknown settings, models overwhelmingly flag corrupted returns as incorrect (over 97% in Search P2/P3 and Code Executor, mostly Definitely incorrect, and ~74% in Sub-Agent, where 67.3% of Gold claims are recognized as Correct). Still, because models also reject a notable share of Gold answers on missed items in Search (64.4%) and Code Executor (44.0%), high rejection rates alone do not prove error detection. We therefore examine the judgment rationales in (2).

AuthorsOpenReview ↗

Follow-up response - [2/2] Rejection rationales (Gold vs. corrupted)

(2) How rejections of gold answers differ from rejections of corrupted values

To test whether models reject gold answers and corrupted returns for the same reasons, we classified the rationales for Definitely incorrect judgments into three categories: Task-internal, grounded solely in the prompt and proposed answer without external facts (e.g., entity-type mismatches, violated mathematical constraints, or direct calculation from prompt inputs); Knowledge-based, relying on recalled facts not stated in the prompt (e.g., dates, entity attributes, or general domain knowledge); and Other (vague or uninformative statements).

DomainItemsProposed answerTask-internalKnowledge-basedOther
SearchAnswered correctly without toolsGold3.684.511.9
P10.788.710.6
P277.819.82.4
P314.682.82.6
Not answered correctly without toolsGold2.193.84.1
P12.684.612.8
P288.410.61.0
P329.469.41.2
Code ExecutorAnswered correctly without toolsGold98.00.02.0
P199.90.00.1
P2100.00.00.0
Not answered correctly without toolsGold98.60.01.4
P1100.00.00.0
P2100.00.00.0
Sub-AgentAnswered correctly without toolsGold25.075.00.0
P110.589.50.0
P227.672.40.0
Not answered correctly without toolsGold5.694.40.0
P17.492.30.3
P213.285.90.9

On missed items, Gold rejections stem from faulty recall or miscalculation, whereas detectable corruptions fail on verifiable flaws or coarse domain knowledge. In missed Search P1, both Gold and P1 are rejected almost entirely via Knowledge-based guesses when recall fails, confirming why adopting P1 there is not treated as overreliance (Section 4.1). In missed Sub-Agent items (and Search P3), rejections are also mostly Knowledge-based: even when models miss the exact answer without tools, they retain enough general knowledge to spot false claims or implausible values while rarely rejecting Gold in Sub-Agent. By contrast, in Search P2 and Code Executor, errors are caught directly from the task. Nearly all missed Search P2 rejections cite Task-internal type mismatches (e.g., "11" for a person; Figure 1), versus almost none for Gold. In Code Executor, nearly all rationales are Task-internal because no external facts are involved, yet closer inspection shows a clear split: Gold rejections on unsolved problems come from failed recomputation, whereas 51.3–55.8% of P1 and P2 rejections flag violated mathematical properties (such as a negative product of positives or a remainder exceeding the divisor) that hold without knowing the exact value (versus 0.0% for Gold). Even so, agents adopt these detectable corruptions in 20.1% of missed Search P2 runs and 45.5% (P1) and 35.7% (P2) of Code Executor runs on missed problems (Appendix D.9).

In summary, across P1 and Sub-Agent on known items and implausible or constraint-violating returns on unknown items, models can detect the injected errors from prior knowledge or the task itself, yet still adopt them when returned by a tool. Moreover, because users see only the final answer, an agent facing a conflict or clear doubt should at least alert the user rather than presenting the corrupted value as settled fact (Section 4.5). We have added this analysis to Appendix D.9. If any points in A1, A3, or A4 remain unresolved, we would appreciate knowing which and would be glad to address them.

Reviewer_3nu5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to study the question whether an LLM agent relies on its tool results too much. The authors answer the question by corrupting tool results and compare whether an LLM's answer is affected. Corrupting means the study may replace a tool results with an irrelevant or misleading value. For example, the result of web search "who is the German president" may be replaced with "11" (which is irrelevant to the question), or may be replaced with "Teslan" which is a misleading wrong answer. The study find that for all tools, in more than 1/3 cases, the agent relies on the unreliable tool calls. Especially, on web search, the agent relies on 68% unreliable tool calls.

Critical strengths

None

Critical weaknesses

While this paper is targeting an interesting question, I have the following concerns:

  1. In the study setup, an agent that does not rely on tool call results will take answer from their pre-training knowledge. I am not sure this is a good behavior measurement for LLM agents. The pre-training knowledge may be outdated or inaccurate, relying on tool call results should be a better behavior pattern than ignoring tool call results, from my understanding. A better behavior might be stopping the execution or trying a different approach?

  2. The error injection is not realistic. It is not reasonable for a web search tool to return "11" for a question "who is the US president". The behavior on this case may not reflect realistic scenarios. Similarly, randomly changing the answer to a misleading person may not be realistic as well. For this question, a realistic setup might be setting the answer to a former president, and see whether the model can realize the date difference.

Questions for authors
  1. What is the expected behavior of an LLM under those conditions? Why relying on tool results are undesirable?
  2. How to realistically formulate "unreliable" in tools?
3 comments

Thank you for the detailed response.

For W1, I think I am still not convinced by the motivation. The authors mentioned "If the agent already knows the correct answer, delivering it to the user is desirable. " in the response. However, I think this is very hard to qualify: a model's internal knowledge may be outdated or vague. Trusting the output from a tool seems to be more safe than trusting the model's memory? Even if we want to test whether the model can detect inconsistencies between internal knowledge and tool results, it might be more reasonable to prompt the model to report the potential discrepancy, and evaluate how often the model can flag the issue.

For W2, I find myself hard to agree on the measurement of realism. An injected error appearing in the top-5 results does not automatically imply the error is realistic. For example, suppose that in real world, a search engine may return noisy content but never miss news happening earlier than 1 week. Then an injected error that replaces one result with an outdated result is still not realistic although the outdated result may be returned in the top-5. This is because the injected error misses the true result, which will never happen in this (hypothetical) real error distribution.

I will keep my original rating.

Thank you for the follow-up.

For W1, trusting the tool and using internal knowledge are not mutually exclusive. An agent can report the tool’s answer together with its own and tell the user they disagree, and such answers are not counted as adoption (Section 3, Appendix A.6). However, as shown in Section 4.1 agents often discard answers they already get right and adopt the corrupted value instead (P1 override of 56.4% in Search). Outdated or vague knowledge does not explain this. The override rate is 53.9% on never-changing facts (Section 4.2) and 33.5% in Code Executor, where arithmetic cannot become outdated. When asked without tools, models judge 90.5% of the Search P1 values incorrect on the items they answer correctly (Appendix D.9), so they adopt values they can recognize as wrong.

The reviewer suggests prompting the model to report discrepancies and measuring how often it flags them. Both parts are in the paper. Section 4.5 measures warnings without such a prompt (4.2% of adopting final answers), and the Disclose policy (Section 5.1) prompts the agent to report inconsistent evidence, under which Search P1 adoption only drops from 75.4% to 72.0%. Following the reviewer’s suggestion, we also computed the warning rate under Disclose from the existing runs. Among Search P1 final answers that adopt the corrupted value, agents warn the user about the tool results in 11.8% of cases under Disclose, compared with 6.9% under Plain (averaged across the same six models). Even when explicitly instructed to report conflicts, agents give no warning in nearly nine out of ten adopting responses. For W2, we agree that an injected value appearing in the top-five results does not by itself establish realism. The hypothetical assumes that real search never misses the true result, so we checked this in the unedited responses from our experiments. Averaged over questions, the gold answer is absent from 22.6% of responses, ranging from 9.8% for never-changing questions to 35.5% for fast-changing questions (Appendix A.8). In 4.9% of responses the P1 value appears without the gold answer, which is the pattern our corruption produces. Such failures are therefore not hypothetical, and they are likely more frequent with generative search tools, which motivate our study. When AI assistants answered from BBC articles they had access to, 19% of answers introduced incorrect numbers or dates (BBC, 2025), and outdated information was among the most common errors across 3,000 responses (EBU & BBC, 2025). Our corruptions are a controlled test of how an agent responds when such a value reaches it, including on answers it already knows.

References

BBC. 2025. Representation of BBC News Content in AI Assistants. BBC Responsible AI Team, February 2025. https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-into-ai-assistants.pdf

EBU and BBC. 2025. News Integrity in AI Assistants: An International PSM Study. European Broadcasting Union, October 2025. https://www.ebu.ch/research/open/report/news-integrity-in-ai-assistants

Dear reviewer 3nu5,

As the rebuttal period is coming to an end, we wanted to check in and see whether our second-round rebuttal has addressed your remaining concerns. We would be more than happy to discuss further or answer any additional questions. If our responses have resolved your concerns, we would be grateful if you could take them into account in your final assessment. Thank you very much for your time and thoughtful feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Review Summary (2/2)

The reviewers pointed out several weaknesses and suggestions, which we have carefully addressed through detailed responses and additional experiments:

  • Overreliance vs. Reasonable Tool Use (Reviewers 3nu5, H2Gk): Both reviewers noted that agents’ adoption of an incorrect but plausible tool return is not necessarily unreasonable, since parametric knowledge can be outdated. We clarified that what we take issue with is not adoption itself but (1) override, meaning abandoning an answer the agent already gets right without tools, which persists even for never-changing facts and arithmetic, and (2) agents often doubting a tool return in their reasoning without reporting it to the user. To test whether override is simply reasonable tool use, we added a human study. Humans adopted corrupted Search returns as often as models (70.0% vs. 68.2%) but rarely overrode answers they already knew (11.1% vs. 56.1%), which shows that override is distinct from the reasonable reliance the reviewers describe.
  • Detectability of Injected Errors (Reviewer H2Gk): Reviewer H2Gk noted that the evaluation does not clarify “whether the injected errors could have been identified given the information available to the agent”. We added an answer-judgment experiment showing that they could: without tools, models reliably recognize the injected values as incorrect, based on their own knowledge (e.g., 90.5% of Search P1 values on questions they answer correctly) or on flaws visible in the task. Agents thus adopt errors that they themselves can identify, so the overreliance we report cannot be attributed to unidentifiable errors. The one exception, plausible P1 values on unfamiliar Search questions, is not counted as overreliance.
  • Realism of Injected Errors (Reviewers 3nu5, k74W): Reviewer 3nu5 suggested “setting the answer to a former president” as a realistic setup. We clarified that this is our main condition: P1 replaces the correct answer with a real value that was once correct or is easily confused with it (e.g., the former chancellor in Figure 1). Reviewer k74W also asked how well our corruptions represent naturally occurring tool errors. We analyzed real search returns and found that P1 values appear naturally in 41.1% of top-5 results.
  • Reporting Conflicts to the User (Reviewer 3nu5): Reviewer 3nu5 suggested prompting agents to report discrepancies and measuring how often they flag them. We pointed to the corresponding analyses: Section 4.5 measures how often agents warn the user without such a prompt, and the Disclose policy (Section 5.1) instructs them to report inconsistent evidence. Even under Disclose, Search P1 adoption only drops from 75.4% to 72.0%, and agents warn the user in only 11.8% of adopting answers (vs. 6.9% under Plain), so they give no warning in nearly nine out of ten adopting responses even when instructed to.
  • Discrimination vs. Reduced Tool Trust (Reviewer H2Gk): Reviewer H2Gk asked whether lower adoption under interventions might simply reflect reduced trust in tools rather than a better ability to tell correct from incorrect returns. If an intervention merely reduced trust, agents would also benefit less from correct returns. We therefore measured the two effects separately: accuracy with correct returns on questions the agent cannot answer without tools, and override of answers it already gets right. Verify keeps the former unchanged (89.6% to 89.7% in Search) while lowering override by 4.1–10.2 points across the three tools, showing that it reduces harmful reliance while preserving the benefit of correct returns.
  • Verification Overhead and Cost (Reviewer k74W): We showed that Verify adds a moderate number of extra tool calls, while the other interventions add almost none. We also found that corrupted returns themselves substantially increase usage and cost, and we now report this in the paper along with the full API cost.
  • Related Work (Reviewers H2Gk, k74W): We added all eight suggested works. We clarified that ours complements them by focusing on non-adversarial tool errors and on whether agents override answers they already know.
AuthorsOpenReview ↗

Review Summary (1/2)

Dear Area Chair and Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We are grateful that Reviewer k74W confirmed that most concerns were resolved and maintained a positive assessment, and that Reviewers 3nu5 and H2Gk engaged in further discussion with us.

The following summarizes the contributions of our work as highlighted by the reviewers (with reviewers’ quotes in bold):

  • Reviewers found that our work addresses an “important and practical problem”, highlighting a “critical reliability issue in tool-augmented LLM agents” (Reviewer k74W), and targets “an interesting question” (Reviewer 3nu5).
  • Distinguishing tool-output adoption from the overriding of previously correct answers “provides valuable insights” (Reviewer k74W).
  • The work offers a “systematic empirical evaluation across three distinct tool categories” (Reviewer H2Gk) and an “extensive empirical analysis” spanning “diverse models, tools, and corruption settings” (Reviewer k74W).
  • The “investigation of reasoning behavior and multiple mitigation strategies further strengthens the contribution”, with verification-based prompting demonstrating “promising improvements in reliability” (Reviewer k74W).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.