acceptodds
Under review as a conference paper at ICLR 2027

When Tools Fail Quietly, LLM Agents State Wrong Answers as Fact

23 views

Abstract

Tool-using LLM agents sometimes state a wrong answer as fact. We ask which properties of a tool failure cause this, using QuietFail, a fault-injection testbed over a real filesystem, SQLite databases, an HTTP service and a shell, in which each of 30 multi-step tasks also runs without its fault. Across seven agents from five providers and 4,305 graded runs, what the failing call returns shapes the outcome. Access denials end most runs in open failure and other errors raise confident wrong answers by at most 10.3 points, while a tool that silently returns half the requested records raises them from 7.3% to 26.1%, and to 38.1% in the runs where it removed data. A notice that the result is truncated, added to otherwise unchanged output, cuts them by 17.2 points, and by 15.6 in same-window cells of five agents (task-level p = 0.008); in a concurrent three-agent collection the notice arm matches the fault-free rate. For Claude Haiku the notice must say something about this result, either that it is truncated or how much; a hedge or a bare flag does nothing, and false notices cost little. When the agent cannot re-fetch, frontier agents fail too: Claude Sonnet 5 states a wrong answer in 39.7% of runs and Claude Opus 5 in 23.8% (27.0% and 12.7% under a second judge); the notice stops most of this, and a cursor to the withheld rows restores the answer (Sonnet 5: 95.2% correct). An ungated frontier auditor catches 80% of the wrong answers when told to look for incompleteness and 58% when not, at low precision, and in the loop neither a targeted nor a generic check significantly helps. Of 87 audited tool surfaces, 63 withhold output by default; 8 silently cut a result and 26 return the top k without a total. Other graders find the same directions at smaller sizes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, tool use, tool failure, silent failure, fault injection, hallucination, abstention, AI oversight, agent-computer interface, benchmarks and testbeds

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gpuBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when tool failures make LLM agents give confident wrong answers. It builds a testbed (QuietFail) where each faulty run has a matching fault-free run. The main finding: errors like "permission denied" mostly lead to honest failure, but a tool that silently returns half the data leads to many confident wrong answers. Adding a short notice ("showing k of n rows") removes most of this, and a cursor to fetch the rest restores correct answers.

Critical strengths
  1. Clean design and a clear, useful message. Pairing every fault with a fault-free run, and grouping faults by what the tool returns, makes the main effect easy to see. The advice for tool builders (say when you truncate, say how much, give a way to get the rest) is simple and practical.
  2. Careful checks and real-world link. The authors test their results with a second judge, a rule-based grader and task-level bootstrap. The audit of 87 real tool surfaces shows the problem is common in shipped tools.
Critical weaknesses
  1. Small and simple tasks. Most claims rest on 21 tasks with tiny data (e.g., 3 of 6 rows dropped) and one fault per run. Two of the four main tests are not significant at task level, and the main tests were chosen after the data was analyzed. The most common real case in their audit (top-k results) is not tested.
  2. Grading and data collection issues. One LLM judge, from the same family as several agents, does most of the grading, and there is no human annotation. The rule-based grader gives effects only a third to nine-tenths as large. Seven of 30 ground-truth answers were wrong at first. For four of six agents, the notice runs were collected 3–4 weeks later, some on a different server.

The main idea (telling the agent that data is missing helps) is also somewhat expected, so the contribution is mostly the careful measurement.

Questions for authors
  1. Does the effect hold with larger data and with top-k search results?
  2. Can you add human labels for a sample of runs, to check the judge?
  3. Can you re-run the notice condition for all six payload agents in the same time window as their other conditions?
1 comment

Thank you. Each question now has new evidence: larger data and top-k (§4.7), human labels (App. M), and a same-window re-run (App. W.3).

Q1: larger data and top-k. Top-10 search with no total and a 4,000-character cut, on 14–222 records with 29–82% withheld (general comment, items 1–2). The silent limit leaves no agent correct; the page or offset parameter restores 67–89% correct for the Claude agents. The effect holds for Haiku at every share and for Sonnet 5 when half or less is withheld (App. W.2).

Q2: human labels. One author labelled 280 runs blind to agent, condition and grader: four-way agreement 94.3% with Opus 4.8 (κ=0.91) and 96.1% with GPT-6.1 (κ=0.94). Of the 24 disputed partial answers the LLM-free rule accepts, the annotator rejects 21. One annotator and an author, so not independent (general comment, item 4).

Q3: same-window re-run of the notice. Done for the four agents whose notice arm came later, now in one window on one service. GPT-4.1-mini and Gemini ran all four payload conditions, the notice included, on 10 September; their fault-free runs are the main grid's from 7 September.

  • Truncation vs no fault: +19.8pp (p=0.003).
  • Notice vs truncation: −19.4pp (p=0.002), 0.4pp above the fault-free rate under Opus 4.8; under GPT-6.1 and the Qwen3 judge −15.9 and −11.1pp, not shown equivalent.
  • Truncation vs denial: +18.3pp (p=0.011).

Other concerns.

  • Two of four confirmatory tests failed; designated after analysis. Conceded, and the paper says so. The re-run, collected after the tests were designated, repeated the payload experiment with its own fault-free runs, so it covers the first three contrasts. All three are significant at task level (Q3), truncation vs denial for the first time. The fourth comes from the Sonnet 5 arm, the one prospective test; it was not re-run, remains non-significant (p=0.12), and we no longer claim it.
  • One judge from the agents' family. GPT-6.1 re-graded all 7,958 later and new runs blind (four-way κ=0.94) and finds 1.3pp more wrong answers than Opus for Claude and other agents alike, so we see no self-preference; every pooled effect holds under it and when Opus never decides alone (item 5).
  • Rule effects one third to nine tenths as large. The rule accepts partial answers as correct, so it understates the effects: the blind annotation rejects 21 of the 24 disputed partial answers that the rule accepts.
  • Seven wrong ground truths. We found and corrected them ourselves and re-graded every arm; the correction strengthens every effect we rely on. A programmatic recomputation confirms 28 of the 30 targets; the other two are imprecise only toward leniency (App. N). The new tasks' ground truth is computed in code.
  • "Somewhat expected." That a notice helps is unsurprising. Which notice helps, and for which agents, is not: hedges and flags fail for Haiku and the Qwen3 agent, and Haiku keeps guessing under a count unless given a way to fetch the rest. Error-gated monitors never see the case: ours intervened in 81 of 378 truncation runs, each after an unrelated error.
Reviewer_DTrKOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies how silently incomplete tool outputs cause LLM agents to present incorrect answers as facts. Using QUIETFAIL's 30 manually constructed tasks, the authors evaluate tool failures and interventions involving truncation notices and retrieval cursors. They also audit output limitations in 87 tool interfaces.

Critical strengths
  1. Matched fault-free runs and comparisons using identical data with and without notices help isolate the effects of tool failures and completeness information.

  2. Distinguishing incomplete-data notices from access to missing data provides useful insights into preventing confident errors versus restoring correct answers.

Critical weaknesses
  1. Limited task diversity and connection to deployed tools.

The central truncation experiments cover only 21 manually constructed tasks with small outputs. Extensive repetition and robustness analyses do not establish generalization to broader workflows and more complex data.

The source-code audit establishes output limitations but does not measure their behavioral consequences in actual tool use. Top-k selection, which accounts for most silent limitations identified, is not experimentally evaluated.

  1. Unclear incremental contribution and presentation. Truncation notices and pagination metadata are established design practices. The additional generalizable insights remain insufficiently clear, particularly given model-dependent effects. Fragmented reporting across the main text and appendices also makes the central evidence difficult to follow.

Overall, the contribution and scope of validation are insufficient to support acceptance.

Questions for authors
  1. How do the eight silently truncating interfaces in Section 4.6 correspond to the experimental truncation mechanisms and recovery paths? Which properties are audited versus behaviorally validated?

  2. Which conclusions extend to untested top-k selection, and how does this limit the recommendations for deployed tools?

  3. What is the main new empirical insight beyond existing truncation and pagination guidance, and across which models and conditions is it supported?

1 comment

Thank you. Each of your concerns has a change: §4.7 (shipped defaults, larger data, top-k), Q3 (what is new), and Table 1 plus the claims table (App. Y) for presentation.

Connection to deployed tools. We agree the audit alone did not show behavioral impact. §4.7 now measures it on 12 generated tasks (14–222 records) at two shipped defaults reproduced on a local HTTP route (not the live tools, so ground truth stays computable): a silent limit leaves no agent correct (44–100% confident wrong answers), a count stops only the frontier agents from guessing, and a page or offset parameter restores correct answers, fully on search tasks and partly on documents. On the original tasks, App. W.2 varies the share withheld from 10% to 75%: Haiku is misled at every share, Sonnet 5 when half or less is withheld, Llama-3.1-8B at none. Open-ended workflows and compound faults remain untested (Limitations).

Q3: what is new beyond existing guidance, and for which models. The guidance says to announce truncation; it does not say whether agents need this, which parts matter, or whether tools comply. We add:

  • The risk is not the error. The same missing data ends in open failure when the call fails, in confident wrong answers when it returns silently (+18.3pp at the same injection point, re-run in one window, p=0.011).
  • Not every notice works. A hedge or a bare flag does nothing for Haiku or the Qwen3 agent; a definite or counted statement does.
  • Without a second route, a notice is not enough. It stops the guessing for Sonnet 5, Opus 5, and Haiku under persistent truncation, but only a fetch handle restores the answer; at shipped limits, Haiku keeps reporting the partial value under a count (80.6%) until it gets the page parameter.
  • Error-gated oversight never examines the case: the monitor intervened in 81 of 378 truncation runs, each after an unrelated error (§4.4).
  • The practice is not followed: 34 of the 63 surfaces that withhold output say nothing.

For which models: the notice helps every agent except the two small open-weight ones, Llama-3.1-8B and the Qwen3 agent. Table 1 lists where each finding fails, e.g. the truncation effect is not significant for Gemini in the main grid.

Q1: the 8 silent cuts. Table 30 (App. O) maps each silent surface to its limit, whether the rest is fetchable, and the experiment testing it.

  • Reproduced at the shipped default (§4.7): the 4,000-character cut of LangChain's Wikipedia and arXiv tools. PubMed (2,000) and requests_get (5,000) use the same cut at other caps; requests_get's LLM summary is untested. The file targets test the same prefix cut at small size.
  • Closest experiment: the computer-use loop drops a command's output whenever it wrote to stderr, like the empty and denial payloads.
  • Audited only: LlamaIndex's text-to-SQL (300-character cells, LLM-synthesized answer), AutoGen's web surfer, ADK's line filter.

Q2: top-k. All three conclusions extend: with the top 10 and no total, pooled correct answers on the search tasks fall from 70.8% to 0%; a count stops the frontier agents from guessing; a page parameter restores 70.8%. The recommendation covers selection: return the total when one is defined, and a handle for the rest. Only 3 of the 26 silent top-k surfaces offer one.

Presentation. Table 1 now puts the central evidence in one place: each finding, the agents for which it holds, and where it does not. The introduction states what we add to existing guidance, §4.7 follows the audit it tests, and App. Y lists every main-text claim with its evidence and status.

Reviewer_YSNtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QUIETFAIL studies how tool failures affect agent answers using matched fault-free controls. Silent truncation raises confident wrong answers from 7.3% to 26.1%; adding a completeness notice reduces them by 15.6 percentage points in same-window comparisons. The paper advocates explicit completeness metadata and recovery cursors.

Critical strengths

The strongest contribution is the controlled intervention: holding returned data fixed while changing the notice isolates an actionable interface effect. The notice ablations and persistent-truncation experiments distinguish recognizing missing information from recovering it; Tables 2–3 provide useful evidence about both mechanisms. Cross-family grading, full-text regrading, and transparent reporting of methodological limitations strengthen credibility.

Critical weaknesses

Limited external validity. The evidence rests on 30 authored tasks, with only 21 truncation targets and small payloads. Larger outputs, compound faults, open-ended tasks, and top-k selection remain untested. The tool audit establishes relevance, but not end-to-end deployment impact.

Measurement remains a concern. Effect sizes depend materially on how caveated and partially correct answers are classified. Cross-family agreement is reassuring, but the rule-assisted author review is not independent human validation. A blinded annotation study would better substantiate the central “confident wrong” outcome.

Some conclusions need narrower wording. Notice effectiveness is model-dependent: no variant significantly improves Llama-3.1-8B’s confident-wrong rate, whereas even a bare flag works for Sonnet 5. Moreover, the truncation-versus-denial and recoverable-frontier contrasts do not survive the reported task-level testing.

Questions for authors

Can the authors independently adjudicate disputed partial answers and validate the intervention on actual audited tools across varying truncation rates?

1 comment

Thank you. Two changes answer your main points: §4.7 tests the shipped limits on larger data, and two more judges and a blind annotation check the labels.

External validity. We agree the original tasks were small. §4.7 runs the two most common silent behaviors in our audit at their shipped defaults on larger data, reproduced on a local HTTP route rather than the live tools, which keeps ground truth computable.

  • Top 10 with no total: on the search tasks, no answer is correct; a page parameter returns pooled correct answers to the uncapped rate (70.8%).
  • 4,000-character cut: same pattern, but an offset parameter recovers the documents only partly; they take 2–5 requests to reassemble.
  • Share withheld: the effect holds from 10% to 75% withheld for Haiku and from 10% to 50% for Sonnet 5 (App. W.2; weaker under the other judges).
  • Still untested: compound faults, open-ended tasks, and tools that pass cut text through an LLM summarizer; see Limitations and the new mapping table.

Measurement.

  • Checks: the original GPT-4.1 audit (three-way κ=0.73, 360 runs). New in this revision: GPT-6.1 on all 7,958 later and new runs (four-way κ=0.94; every pooled effect holds) and the Qwen3 judge (general comment, item 5).
  • Blind annotation: one author, 280 runs, 94.3% agreement with Opus 4.8 (four-way κ=0.91); 21 of the 24 disputed partial answers are confident wrong answers (item 4).
  • What we concede: neither the rule-assisted review nor the blind annotation is independent; both are by an author.

Narrower wording. Agreed. Table 1 states, per finding, the agents for which it holds and does not, and the abstract now says Llama-3.1-8B heeds no notice and the Qwen3 agent only a counted sentence.

  • Ablations: Table 1 states that Sonnet 5 heeds any notice variant, Llama-3.1-8B none, and that the specific statement matters for Haiku and the Qwen3 agent.
  • Truncation vs denial: in the prospective same-window re-run it holds at task level (+18.3pp, p=0.011).
  • Recoverable frontier contrast: §4.5 states that it is not significant with tasks as the unit (p=0.12), and the introduction no longer states it as a finding.

Q: independent adjudication; audited tools across truncation rates. Adjudication is the blind annotation above, by one author. For audited tools, §4.7 applies the shipped defaults, which withhold 29–82% of the data, App. W.2 sweeps the share kept from 25% to 90%, and the new appendix table states which surfaces are tested this way and which are audited only.

Comments on the paper

1 comment

All three reviewers asked whether the effects hold beyond small authored payloads, at top-k selection and shipped limits; two asked for label checks. We ran three new collections (3,537 runs). The revision is not color-marked; new material: §4.7, App. W, Table 1, App. M (judges, blind annotation). Section, table and figure numbers refer to the revised PDF.

  1. Shipped limits on larger data (§4.7, Table 6, App. W.1; 576 runs). 12 generated tasks with ground truth computed in code, each served by a local HTTP route with a shipped tool's default limit.
    • Six searches (14–55 matches) cut to the top 10 with no total, the commonest silent top-k default; six documents (6k–16k characters) cut at 4,000, as LangChain's Wikipedia and arXiv tools do.
    • 29–82% of data withheld; 4 agents, 3 seeds.
    • Silent limit: no agent is correct; confident wrong answers +56.5pp over uncapped (CI [+44.1, +67.7]).
    • With a count: Sonnet 5 and Opus 5 stop guessing (2.8% and 0% wrong); Haiku still reports the partial value (80.6%).
    • With a page or offset: 66.7%, 69.4%, 88.9% correct (Haiku, Sonnet 5, Opus 5): full recovery on search tasks, partial on documents.
    • Llama-3.1-8B: 50.0% wrong under the limit, 36.1% with the notice, and under 10% correct even uncapped, so it cannot show recovery.
  2. Share withheld (§4.7, App. W.2, 1,701 runs; 25–90% of rows kept).
    • Haiku's wrong answers rise at every rate (+14.3 to +33.3pp); Sonnet 5's when 50%, 25% or 10% is missing (+11.1 to +20.6pp), not at 75%. The notice removes most of each rise.
    • Llama-3.1-8B shows no effect at any share. Sonnet 5's rises are not significant under the Qwen3 judge, nor at half missing under GPT-6.1; all three hold under the three judges' majority (W.4).
  3. Same-window re-run (§4.2, App. W.3, 1,260 runs). All five conditions (the four payload conditions and no fault), for the four agents whose notice arm came later.
    • Truncation vs no fault: +19.8pp (task-level p=0.003); vs a denial at the same point: +18.3pp (p=0.011; not significant originally).
    • Notice vs truncation: −19.4pp (p=0.002), ending 0.4pp above the fault-free rate. Both other judges: all three hold, the notice ending 1.6 and 3.2pp above it.
  4. Blind human labels (App. M, Table 28). One author labelled 280 runs blind to agent, condition and grader (240 stratified by condition, 40 rule–judge disputes). Four-way agreement: Opus 4.8 94.3% (κ=0.91), GPT-6.1 96.1% (κ=0.94); the LLM-free rule 75.4% (κ=0.58) on the 240 stratified runs. Of the 24 disputed partial answers the rule accepts, the annotator rejects 21. One annotator, an author: not independent.
  5. Judges (App. M, W.4). Original check: GPT-4.1, 360-run audit (three-way κ=0.73). New: GPT-6.1 re-graded all 7,958 later and new runs blind, 96.5% agreement with Opus 4.8 (four-way κ=0.94), 1.3pp more wrong answers than Opus for Claude and other agents alike; every pooled effect holds under it and under the three judges' majority. The open-weight Qwen3 judge agrees less (four-way κ=0.81, 0.83): no significant contrast changes sign, some per-agent effects lose significance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.