acceptodds
Under review as a conference paper at ICLR 2027

When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents

Abstract

Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26–39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. Human evaluation of 200 post-freeze trajectories yields 96% task-success scoring agreement, corroborating retry gains over Base and adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.

Primary area:
datasets and benchmarks
Keywords:
large language models, tool-using agents, data analysis, silent tool poisoning, agent evaluation, evidence verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YfySOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A data agent's tool can run without error and still return a plausible but wrong result. The paper calls this "silent tool poisoning" and introduces ToxicBench to measure it. A proxy runs the real tool and then edits its output, while the source tables stay fixed, so each task has a clean and a poisoned version. Besides task success, the paper defines trajectory metrics for whether the agent detects the poison, checks it with fresh evidence, adopts it blindly (Blind Compliance Rate, BCR), or adopts it after a check. The main study uses GPT-5.4-mini across LangGraph, smolagents and AutoGen. A defense study on Claude Haiku 4.5 compares a single-pass agent with a plain second attempt (Double-pass), a verification prompt and a checklist guard. Poisoning lowers success sharply. Double-pass is the best defense, and neither verification prompt beats it. Under repeated poisoning, many runs adopt the poison even after checking.

Critical strengths
  • The paired design is clean. The proxy edits only the returned observation while the source data stay fixed. A wrong answer therefore cannot be explained as faithful analysis of changed data.
  • The operator taxonomy separates corruptions that range or plausibility checks can catch from ones they cannot. Label swaps and treatment or control flips keep the value but change what it is bound to.
  • The metrics separate exposure from the response to exposure. They also separate blind adoption from adoption after a check. This is more informative than final accuracy or counts of tool calls.
  • The equal-cap Double-pass control keeps a gain from simply trying again from being credited to verification. The paper honestly reports the negative result that follows.
Critical weaknesses
  • Under one-shot poisoning with shared state, the second route is almost never exposed to poison. So the headline "ordinary retries help" mostly means the tool stopped lying on the second call. The only per-route control is a preselected subset where Base is already perfect, so it cannot test the effect. The paper does not report second-route exposure for the full defense matrix. It also has no Base arm under repeated poisoning, so no result shows that a two-route protocol reduces adoption when checks can also be corrupted.

  • Corruption that clears on re-query matches neither an adversary nor the error sources the paper cites. Stale summaries, unit-conversion bugs and label-binding errors would all recur on an identical call. No transient source, such as a flaky cache or nondeterministic retrieval, is named.

  • ADR is effectively unmeasured. The scorer almost never detects anomalies in development and predicts none after the freeze, so BCR in practice means adoption without VR. BCR recall is low in development and only moderate after the freeze, with few positives. Scorer errors also depend on the method. Most automatic task-success misses in the human evaluation are Verification-only runs. Under human labels, Verification-only ties Double-pass. The full-matrix gap where Verification-only trails Double-pass (Table 18), and its lower clean TSR, may therefore be partly scorer artifacts.

  • The lesson that verification must be compared against a plain retry closely parallels Huang et al. (ICLR 2024, "Large Language Models Cannot Self-Correct Reasoning Yet"), which is not cited. CRITIC, ToolSword, InjecAgent, the knowledge-conflict line (Xie et al., ICLR 2024; ClashEval), and data-agent benchmarks such as DA-Code, DSBench, InfiAgent-DABench and DABstep are also missing.

Questions for authors
  • In the full Haiku defense matrix, how often did the second route receive any corrupted observation? - Does Double-pass still beat Base when the second route is also exposed?
  • If VR is restricted to first-route checks or to checks that touch the corrupted value, do two-route BCR values stay at zero?
  • How often do the historical poisoned observations still contain a clean alias of the answer? How do the main TSR drops and RR change on alias-free, full-target deliveries?
  • Under human labels or a structured answer field on a larger sample, does Verification-only still trail Double-pass?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YfyS (1/2)

Thank you for the detailed questions. We audited route exposure and aliases, added a matched persistent-corruption comparison, and reanalyzed existing human labels by method. W and Q follow the original review order.

W1. Route exposure and the persistent Base comparison: In the full historical Haiku matrix, second-route exposure is 2/118 for Double-pass and 3/118 for each verification method. All 944 retained trajectories match tool-event records. No run is exposed in both routes. Conditional on first-route exposure, the counts are 0/99, 0/98, and 0/101, respectively. Historical retry gains thus include later opportunities for unmodified evidence.

We add 480 natural-tool trajectories using DeepSeek-v4-pro-ga-260813, crossing four methods with three conditions on 40 tasks fixed before outcomes. This adds a Base control beyond the earlier ceiling subset, with frozen automatic scores and recorded exposure:

MethodClean correct / 40One-shot correct / 40Persistent correct / 40Persistent route-2 exposure / 40
Base393433N/A
Double-pass38373335
Verification-only35352919
Generic Guard34353320

Base has one route. Double-pass records both-route exposure in 34/40 runs. Its persistent difference from Base is 0 points, with paired 95% interval [−12.5, +12.5] and source-table cluster interval [−10.9, +10.0]. It uses 8.60 requests and 7,189 reported tokens, versus 4.42 and 3,772 for Base. Rules certify target changes in 23 of its 35 second-route exposures; 12 remain unresolved. A separate diagnostic with an error warning, an explicitly faithful source, seeded observations, and structured answers gives both methods 40/40 under clean and persistent conditions.

Persistent poison-adoption counts are Base 6/40, Double-pass 6/40, Verification-only 4/40, and Guard 2/40. These require recorded exposure and use all 40 planned tasks as the denominator. The original PAR uses exposed runs as its denominator.


W2. Persistent and transient failure sources: Native pandas 1.3.3 and DuckDB 1.3.0 bugs reproduce across three identical calls and a new-process call. Later versions return correctly. RFC 5861 describes caches serving old content while updating, with later freshness depending on update timing. One-shot tests recovery after one wrong return; persistent poisoning tests errors recurring on eligible calls.


W3. ADR, BCR, and method-dependent scoring: Our existing human evaluation of 200 trajectories yields 96% task-success scoring agreement and confirms adoption of poisoned answers after checking.

Six of the eight answers accepted by humans but rejected by the automatic scorer are Verification-only outputs. On the 20 core poisoned tasks:

MethodAutomatic correct / 20Human correct / 20
Base1616
Double-pass2020
Verification-only1620
Generic Guard1919

Both methods gain 4/20 correct answers over Base, and all four methods are human-correct on 20/20 clean tasks. The 118-task matrix and new-model free-form comparisons use automatic scores; Q4 describes the human audit's coverage.

BCR identifies 8/10 human positives, with two false negatives and zero false positives; ADR recall is 0/1. ADR measures explicit recognition in the final answer. When ADR is zero, BCR records poison adoption with VR also zero. Q2 shows how alternative check definitions affect these counts while keeping the scorer frozen.


W4. Related work and our contribution: We added the suggested papers. Huang et al. establish fair additional-attempt comparisons, and CRITIC uses external tool feedback for correction. ToxicBench tests this principle when feedback can be corrupted, adding actual-exposure audits and a persistent Base comparison.

ToolSword includes erroneous feedback, InjecAgent studies instructions in tool returns, and Xie et al. and ClashEval study conflicts between model priors and external evidence. ToxicBench changes returned values or bindings over fixed current source tables, tracing exposure, checking, adoption, and recovery against computed references.

DA-Code, DSBench, InfiAgent-DABench, and DABstep assess broader data-analysis skills and workloads. ToxicBench complements these benchmarks with paired evidence changes for the same task and source, supported by route, alias, and human-scoring audits.

AuthorsOpenReview ↗

Response to Reviewer YfyS (2/2)

Q1. Second-route exposure and Double-pass versus Base: Historical second-route exposure is 2/118, 3/118, and 3/118; the new persistent comparison gives Base and Double-pass 33/40 each. W1 provides the full matrices, conditional denominators, and intervals. Double-pass records second-route exposure in 35/40 new runs, with both-route exposure in 34/40. The historical Haiku audit and new DeepSeek experiment are reported separately. Method comparisons use complete experimental arms, and exposure counts describe the evidence received within them.


Q2. Restricting which checks count toward VR: The check-scope audit gives explicit BCR values under each definition. Counting only first-route checks after exposure gives historical Verification-only 1/101. Target-related checks give a logical range of 0/101 to 1/101, accounting for unresolved cases. Historical Double-pass and Guard remain zero, consistent with their zero PAR.

In the new persistent matrix, restricting to first-route checks changes the original zero BCR counts to Double-pass 3/36, Verification-only 2/35, and Guard 1/36. PAR and ADR stay frozen. The target-related range is a logical bound based on unresolved checks. These diagnostics make the role of check scope explicit and distinguish check attempts from the evidence they supply.


Q3. Clean aliases and results on full-target deliveries: We confirm correctly bound clean-answer aliases in 8/694 events, find none in 452, and leave 234 unresolved. An alias is another occurrence of the correct answer with the correct binding. Confirmed full-target changes with no direct clean alias retain 437 events. Task success still falls in the resulting subsets:

AdapterFull-set TSR dropCertified-subset TSR dropFull-set RRCertified-subset RR
AutoGen31/11819/6332/10130/63
LangGraph38/11814/6633/10132/66
smolagents46/11827/6113/8710/61

TSR drop is the net decrease in correct answers divided by paired tasks. RR uses exposed runs. These estimates describe subsets with certified direct answer fields. The common defense subset contains 50/54 semantic tasks and scores near the ceiling. The matched experiment and human checks provide separate evidence for method comparisons.


Q4. Human or structured scoring on a larger sample: The available human reassessment answers this comparison on the existing 20 core poisoned tasks: Verification-only and Double-pass each score 20/20. W3 provides the paired automatic/human table and clean results. The current human comparison is this existing audit, while full-matrix Haiku results use automatic scoring. The new structured-answer experiments separately test information and cost effects on DeepSeek. Each comparison is presented with its model, scoring method, and task coverage.


Revision: The revised paper will clarify retry and scoring results in the experiments, report route and alias audits in the appendix, and expand Related Work with the suggested comparisons.

Thank you for the detailed response and additional analyses. The route-exposure audit and the new persistent-corruption comparison substantially address my concerns about missing controls. The alias audit also provides useful evidence that the task-success degradation persists in subsets with certified target changes and no direct clean-answer alias.

These results clarify the scope of the conclusions. The historical retry gains include opportunities to obtain unmodified evidence, while the new persistent comparison shows no observed advantage of Double-pass over Base, with substantial uncertainty. The existing human audit also places Verification-only and Double-pass at the same accuracy on the audited poisoned tasks.

I am increasing my overall assessment by one level because these additions strengthen the benchmark contribution and make the empirical findings more interpretable. My increased assessment assumes that these qualifications and the new results are incorporated into the revised paper.

Reviewer_ugLCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper designed ToxicBench, a benchmark with tools silently fails to provide correct tool output while the source data stays unchanged, and evaluates whether the LLM is capable of checking from the source data to verify it. The benchmark pairs clean and poisoned runs of the same task and uses metrics to separate blind adoption, adoption after checking, and recovery. In a 118-task GPT evaluation across three agent frameworks, poisoning lowers task success by 26-39 percentage points.

Critical strengths

The metrics is designed to measure beyond accuracy. The metrics distinguish adopting the poison without checking (BCR), adopting it despite checking (VPA), and recovering the clean answer (RR). This reveals more detailed agent behavior, and shows that many cases, although the agent check the results, it may still adopt the wrong answer, which means the agent trusts the tool result more than the source data.

The evaluation is across three different kind of agent scaffoldings, which is quite comprehensive.

Critical weaknesses

The validity of the problem. The current formulation of the problem of silently failing tool output, requires the agent to verify each tool output regardless of whether the result make sense or not. The core tasks use made-up CSVs about sales, campaigns, inventory and clinics. The model can't know from memory which store had the highest revenue, so its prior knowledge says nothing about whether the tool output is correct or not, the model has to verify every tool result as a consequence.

This seems impractical, as in long-horizon tool-execution, there could be hundreds of tool calls, and we cannot expect the LLM agent to verify every single one of them.

Instead of expecting the LLM agent to always verify each individual tool output, it seems to make more sense to treat the tool as a software component and design well-covered tests for the tool itself, before we integrate the tool into the harness. Therefore, it is unclear to me whether the current formulation of the problem is valid and realistic.

Questions for authors

Is the threat model realistic, and when should an agent verify? ToxicBench corrupts the output of a correctly executed tool call, giving the agent no contextual reason to suspect an error. What real-world failure does this represent, and why should it be addressed through agent-side verification rather than tool-level testing?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ugLC (1/2)

Thank you for raising the role and cost of verification. Our new experiments reproduce real faults and measure recovery alongside checking costs. W and Q follow the original review order.

W1. Task validity and whether every return needs checking: ToxicBench evaluates final correctness and checking behavior separately, allowing agents to choose when to verify. An answer can receive full credit through either direct use of a correct return or successful verification. Synthetic tables provide an inspectable reference based on current data. This setting evaluates system reliability when plausible returns are accepted, and the controls measure the recovery benefit and resource cost of additional checking.


W2. Checking cost in long workflows: An error warning improves recovery on corrupted targets and adds measurable cost on clean targets. We test this in 160 fresh Base runs using DeepSeek-v4-pro-ga-260813: 40 tasks crossed with clean/persistent targets and warning/no-warning conditions. The interface, correct source table, and budgets are fixed, with no added source-trust guarantee:

Target conditionWarning addedCorrect / 40Read source / 40Mean requestsMean reported tokens
CleanNo40161.40791.05
CleanYes40402.001115.60
Persistent corruptionNo19191.475907.20
Persistent corruptionYes40402.001191.85

The corrupted-target gain is 52.5 percentage points, with source-table cluster 98.3333% interval [34.09, 73.58]. Clean correctness is 40/40 in both arms, while source reading increases by 60 points [45.05, 80.00]. These intervals adjust for three planned outcomes, including clean correctness.

On clean targets, the warning adds 0.60 requests and 324.55 reported tokens per run. The token increase has an exploratory source-table cluster 95% interval [250.69, 421.02]. Nine questions already contain checking cues. Each run starts with a supplied target observation, followed by voluntary source reading. Zero active target re-queries occur, so this control tests prompting and source reading.

The separate 40-task natural-tool experiment tests repeated exposure. Persistent correctness is 33/40 for both Base and Double-pass, with paired 95% difference interval [−12.5, +12.5] points. Double-pass uses 8.60 requests and 7,189 reported tokens, versus 4.42 and 3,772 for Base. Together, these results quantify recovery and resource use. They provide concrete measurements for comparing checking policies in short analytical tasks and for studying their extension to longer workflows.


W3. Tool-level testing and realistic faults: Tool testing and repair are the first line of protection, complemented by checks while a fault is still present. We reproduced two native library bugs with unchanged source inputs:

LibraryAffected version and observed errorWrong / queriesLater version: correct / queries
pandas (issue #43541)1.3.3: values assigned to the wrong category4/42.2.0: 4/4
DuckDB (issue #17764)1.3.0: zero matching rows instead of two4/41.3.1: 4/4

Each version is queried three times in one process and once in a new process. All target calls return normally without warnings or exceptions. Alternative computations recover the references using object keys in the same pandas library, or raw DuckDB rows joined with standard-library code. Later versions return correctly. These results show both the value of software repair and the successful-but-wrong returns that agents can encounter before repair.

The bugs recur on identical queries. One-shot poisoning separately tests recovery opportunities after one wrong return. RFC 5861 describes a transient mechanism in which caches can serve old content while checking for updates, then serve newer content later. Recovery on a later query depends on update timing and freshness information. We report outcomes separately for these evidence conditions.

AuthorsOpenReview ↗

Response to Reviewer ugLC (2/2)

Q1. Threat model and when to verify: Our proxy isolates the effect of returned evidence, and the warning experiment measures the value of a general risk reminder. The proxy preserves source tables, the task, and agent instructions. A compromised upstream service or response wrapper could control this boundary in an adversarial setting. W3 demonstrates the same return-level failure pattern through native software defects.

The reminder states that the target answer may be incorrect and applies to both clean and corrupted conditions. It describes general tool risk rather than a detected error in a particular call. W2 shows its measured effects: improved recovery under corruption and additional source checks on clean tasks. These results support evaluating verification through available evidence, error consequences, and checking cost. Tool-level tests and runtime evidence checks address complementary parts of this reliability problem.


Revision: The revised paper will clarify the role of runtime checks in the benchmark section, add recovery and cost results to the experiments, and document native software faults in the appendix.

Thanks for the additional experiments. My question on the efficiency of this check remains. Because the possibility of such tool poisoning happening in our daily use is low, we cannot spend about double the tokens (7,189 versus 3,772 per task for Double-pass versus Base) or 0.6 more model requests per task (about 43% more, in the warning experiment) on every task in order to prevent it. If poisoning is rare, almost all of this extra cost is spent on correct tool outputs.

AuthorsOpenReview ↗

Response to Reviewer ugLC

Thank you for clarifying the concern. Selective checking targets the cost of checking healthy tasks. The model produces a Boolean checking flag within the first pass. An additional checking pass runs only for flagged tasks. The flag uses the task and observations available to the model. It requires no separate screening request. All flag-based policy costs below include the decision instruction and flag output on every task.

Alignment with the original cost comparison: The quoted Base cost of 3,772 tokens per task is the original persistent-fault result, which is 3,772.05 before rounding. The corresponding Double-pass cost is 7,188.63. The tables below retain the original Base costs and correct-answer counts for all three conditions. The uniform-checking policy includes the decision flag and has its own cost column.


Checking allocation and accuracy: Each condition contains 40 tasks. Additional checking is triggered on 1/40 healthy tasks, 8/40 one-shot tasks, and 10/40 persistent-fault tasks. Thus, 39/40 healthy tasks finish without an additional checking pass.

PolicyHealthy correct / 40One-shot correct / 40Persistent correct / 40
Base393433
Decision flag, first answer only393433
Uniform checking394040
Selective checking393939

The first-answer control has the same aggregate correct-answer counts as Base. Under one-shot corruption, selective checking recovers five answers, while uniform checking recovers six. This retains 5/6 (83.33%) of the uniform-checking gain. Under persistent corruption, selective checking recovers six answers, while uniform checking recovers seven. This retains 6/7 (85.71%) of the gain. The other triggered checks confirm correct answers: one healthy, three one-shot, and four persistent-fault answers. Selective checking preserves every initially correct answer in this evaluation.


Complete cost accounting: Flag-based policy costs include the first-pass decision on every task and all triggered checking calls, including the additional check on one healthy task. Values are measured mean tokens per task.

ConditionBase tokensDecision flag, first pass onlyUniform checking: total tokensSelective checking: total tokens
Healthy2,208.002,226.434,581.782,285.30
One-shot3,123.683,148.355,548.453,628.38
Persistent3,772.053,800.387,216.954,654.53

Low-frequency deployment scenarios: The fault frequencies below are deployment scenarios; the condition-level accuracy, checking allocation, and token costs are measured. Each row weights healthy tasks and the indicated fault condition at the stated frequency. The same weights apply to every policy. The cost formula is (1 − p) × healthy cost + p × fault-condition cost, where p is the fault frequency.

Scenario fault frequencyOne-shot: tokens over BasePersistent: tokens over BaseOne-shot: additional correct per 1,000 tasksPersistent: additional correct per 1,000 tasks
0.1%+3.52%+3.53%0.1250.15
1%+3.68%+3.84%1.251.5
5%+4.38%+5.14%6.257.5

At a 1% fault frequency, the estimated selective-checking costs are 2,298.73 tokens for the one-shot mixture and 2,308.99 tokens for the persistent mixture. The corresponding Base costs are 2,217.16 and 2,223.64 tokens. Uniform checking costs 4,591.45 and 4,608.13 tokens under the same weighting.

The efficiency target is to concentrate checking on tasks that benefit from it. Here, 39/40 healthy tasks use only the first pass. Selective checking retains 83.33% and 85.71% of the uniform-checking recovery gain, with 3.68% and 3.84% additional total tokens at a 1% fault frequency. Both token-overhead percentages include the cost of producing the flag on every task. These results provide a lower-cost alternative to checking every task in this evaluation; whether the trade-off is worthwhile in deployment depends on the actual fault frequency and the consequences of an incorrect answer.

Reviewer_M51HOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ToxicBench, a benchmark for evaluating the reliability of tool-augmented data agents when tools return plausible but incorrect results. It employs a proxy to modify tool outputs while keeping the underlying source data unchanged, and evaluates how different Model-Harness combinations adopt, verify, and recover from incorrect information. The experiments cover numerical and semantic errors across multiple models and agent frameworks.

The paper also compares several verification strategies, including ordinary retries and explicit verification prompts, under one-shot and repeated poisoning. It introduces trajectory-level metrics to distinguish blind compliance, verification, and recovery, and uses human annotations to assess the reliability of its automatic evaluation.

Critical strengths
  1. The paper focuses on a reliability issue of agent tools. Tool-augmented agents may encounter plausible but incorrect outputs during real-world operation. Investigating how such errors affect downstream decisions is meaningful, particularly for data analysis tasks where incorrect numerical values or entity bindings can lead to incorrect conclusions.

  2. The paper provides a systematic evaluation across different models and agent frameworks. The benchmark covers multiple types of numerical and semantic errors, and distinguishes between incorrect answer adoption, additional verification, and eventual recovery. The comparison between ordinary retries and explicit verification strategies is also useful for understanding the effectiveness of different approaches.

Critical weaknesses
  1. The paper does not sufficiently establish whether the problem is solvable under the agent's information constraints. When an accessible tools return incorrect information, the agent cannot reliably identify the truth. For example, the change of "Store A" or "Store B", is not recognizable. This is an information limitation rather than a model failure.

  2. The experiments conflate evidence availability with verification capability. When clean evidence is accessible, the problem is largely about selecting trustworthy information. When it is not, verification may be fundamentally impossible. Although the paper considers different poisoning conditions, it does not sufficiently separate these cases when interpreting agent reliability.

  3. The human evaluation does not establish whether the errors are identifiable. Human annotations validate the scorer, but do not test whether an observer could detect the errors under the same information constraints as the agent.

  4. Several related papers are not compared against this submission.

  • When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Models. ICML 2026
  • SafeSearch: Automated Red-Teaming of LLM-Based Search Agents. ICML 2026
  • Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions. arXiv:2607.20891
Questions for authors
  1. Can you clarify the attacker's capabilities and provide a more realistic threat model? The current setting assumes that an attacker can intercept and arbitrarily modify the feedback from successfully executed tools, effectively compromising the agent's execution environment. Under what realistic circumstances would an attacker possess such control? How does this setting differ from a compromised tool or execution environment, and why should the resulting failures be attributed to the model or harness rather than the underlying system?
  2. Whether wrong outcomes are identifiable under the agent's actual information constraints? Although the paper provides scripted recovery paths, how can the agent determine that the recovered evidence is more trustworthy than the original tool output? What is the expected behavior when all accessible observations are consistently corrupted?
  3. How to distinguish failures of evidence acquisition from failures of evidence selection? In the one-shot setting, additional tool calls may simply provide clean evidence, whereas in the repeated-poisoning setting, further checks may remain corrupted. How can these fundamentally different situations support a unified interpretation of the agent's verification capability?
4 comments
AuthorsOpenReview ↗

Response to Reviewer M51H (1/2)

Thank you for the questions about information access and realistic faults. We added recovery controls and native software reproductions, and audited what each route received. Below, W and Q follow the original review order.

W1. Solvability under the available information: Base and Double-pass each solve 40/40 tasks under clean and persistent target corruption in our recovery diagnostic. The interface supplies an error warning, an explicitly faithful source channel, a seeded target observation, and structured answers. This establishes recovery under stated information conditions. Source-table entity-value bindings allow store-label checks using current data. Q2 then isolates the effects of the warning and source description within a fixed interface.


W2. Evidence availability and verification capability: We now separate check attempts, the evidence returned, and final answer adoption. The historical audit finds zero both-route exposures under one-shot poisoning, allowing later unmodified observations. We add 480 natural-tool trajectories across four methods and three conditions on 40 fixed tasks. Persistent correct counts are Base 33/40, Double-pass 33/40, Verification-only 29/40, and Guard 33/40. Double-pass records both-route exposure in 34/40 runs. Its difference from Base is zero, with paired 95% interval [−12.5, +12.5] percentage points.

Among 101 VPA runs, seven contain enough correct evidence to recover: five in the first route and two in the final route. All seven later receive further poison. VR records a qualifying check attempt, while this audit assesses its evidence. The distinction lets ToxicBench trace where recovery can break down, from obtaining evidence to choosing an answer.


W3. What the human audit validates: Our completed human evaluation supports the main findings. Human annotations on 200 trajectories yield 96% task-success scoring agreement. On the 20 core poisoned tasks, Double-pass and Verification-only each achieve 20/20 correct answers, compared with 16/20 for Base. Human judgments also confirm adoption of poisoned answers after checking on audited trajectories.

Agreement is 192/200. The completed human evaluation consists of retrospective annotation of recorded trajectories. Its purpose is to validate scoring. Recovery paths and agent experiments separately provide evidence about executable access and recovery under stated information conditions. Same-information interactive human testing is a distinct evaluation from these completed annotations.


W4. Related work and our contribution: We have added all three suggested papers and clarified our contribution. When Search Goes Wrong studies query-level red teaming. SafeSearch inserts unreliable websites into search results. Is Deep Research Reliable? studies misleading documents and false-conclusion adoption. They establish important precedents for unreliable external evidence.

ToxicBench adds paired changes to tool returns for the same task over a fixed source table. Answers computed from that table give both runs a common reference. Numerical, label, and schema operators separate changes to values and entity bindings. Exposure, checking, adoption, and recovery measures then trace how the altered evidence affects the answer. Our route audit and persistent comparison make this diagnosis more precise by checking what later calls actually supplied.

AuthorsOpenReview ↗

Response to Reviewer M51H (2/2)

Q1. Attacker capabilities, realism, and responsibility: The threat model specifies control of selected tool-return content. A compromised upstream tool service or response wrapper could provide this access. Source tables, agent instructions, and the user goal remain fixed. This scoped intervention lets us trace how changed evidence affects the agent's response.

Software faults can produce the same interface without an attacker. We reproduced two native defects:

Software bugAffected version: wrong / queriesLater version: correct / queries
pandas values assigned to the wrong category (issue #43541)1.3.3: 4/42.2.0: 4/4
DuckDB missing two matching rows (issue #17764)1.3.0: 4/41.3.1: 4/4

Each version is queried three times in one process and once in a new process. All target calls return normally without warnings. Alternative computations recover independently calculated references: object keys within the same pandas library, and raw DuckDB rows joined with standard-library code. These cases establish persistent errors and executable alternatives.

The initial fault belongs to the tool or service. ToxicBench evaluates the model and harness response under available evidence. Tool testing and repair address the error's source, while runtime checks address decisions made before repair.


Q2. Trust in recovered evidence and fully corrupted observations: We test source information separately from access to the source itself. In 160 fresh Base trajectories using DeepSeek-v4-pro-ga-260813, the interface, correct source table, poisoned target, and budgets stay fixed. Only an error-warning sentence and a source-trust guarantee change:

Error warning addedSource-trust guarantee addedCorrect / 40
YesYes40
YesNo40
NoYes22
NoNo19

The warning effect, averaged across the two source-description conditions, is +48.75 percentage points, with source-table cluster 97.5% interval [36.53, 64.58]. The guarantee effect is +3.75 [−4.45, 12.94]. Both intervals adjust for two planned comparisons. Nine questions already contain checking cues, so these effects concern the added sentences. This directly measures how supplied information changes recovery under fixed evidence access.

Our persistent condition preserves source access, giving agents a concrete route to corrective evidence. When accessible observations are identical across different true answers, identifying the truth requires additional distinguishing evidence. If unreliability is known, an appropriate next step is to request a trusted source or communicate uncertainty. TSR records factual correctness, while the interpretation of agent decisions accounts for the evidence available to them.


Q3. Acquiring evidence versus selecting an answer: The same diagnostic steps apply across conditions, while recovery is interpreted within each condition. W2 separates exposure, checking, evidence received, and final adoption. Historical one-shot retries have later opportunities for unmodified evidence. Under persistent corruption, Base and Double-pass each score 33/40. The VPA audit further identifies what checks supplied. These measurements connect checking behavior to the evidence obtained and the answer selected.


Revision: The revised paper will clarify information access in the benchmark section, add recovery controls to the experiments and native faults to the appendix, and expand Related Work.

Thank you for the additional experiments. Our central concern remains whether models can identify errors from the available information, rather than correct them after a warning. Your response shifts the focus to recovery under specific conditions, leaving the paper’s “blind compliance” claim unsupported and requiring substantial reframing. We therefore maintain our rating.

AuthorsOpenReview ↗

Response to Reviewer M51H

Thank you for clarifying the central question. We tested it directly in 160 new runs with no error warning. The experiment separates whether the information needed to identify an error is visible from what answer the model selects.

Available information and experimental control: We used 40 fixed tasks with DeepSeek-v4-pro-ga-260813. Each task had four conditions: correct or incorrect tool output, with or without the complete source table in the input. The table included all relevant rows and column names as ordinary user-provided task data. We removed the nine existing checking cues from the questions. All conditions used the same instructions and requested an answer with a brief explanation. There was no warning, request to verify, or explicit guarantee of source reliability. After the initial tool observation, the model answered directly, with no further tool calls.

Tool outputComplete source data visibleCorrect answersIncorrect tool answer adopted
CorrectNo40/400/40
IncorrectNo1/4039/40
CorrectYes40/400/40
IncorrectYes29/4010/40

One additional output in the last row had a format error. It remains in the denominator and is separate from the 10 cases of incorrect-answer adoption. With incorrect tool output, providing the source data increased correctness by 70 percentage points, with a source-table cluster 98.33% interval of [49.21, 92.50], adjusted for three planned comparisons.


Direct evidence under identifiable inputs: The key observation is the 10 cases where the model adopted the incorrect tool answer despite having the relevant source data in its input. Two examples make the available evidence explicit:

TaskCalculation from the visible rowsTool outputModel answer
Search cost per acquisition(2000 + 1600) / (100 + 80) = 202424
Total software revenue30000 + 22500 = 525006300063000

In the first case, the model wrote: “The cost per acquisition for Search is 24.0, based on the provided target observation.” In the second, it wrote: “The total revenue for software SKUs, as returned by query_target, is $63,000.” Both paired runs with correct tool output answered correctly using the same source input.

We also scored explicit error statements separately. Among valid structured outputs, a fixed conservative automatic rule flagged none; final-answer correctness and explicit error identification are distinct measures. The evidence above concerns observable answer adoption. It does not depend on an inference about the model's internal recognition of the error.

What this establishes: These controlled observations directly support the behavioral concern behind blind compliance: an incorrect tool answer can be adopted even when the data needed to reject it are already visible and no warning is given. The four conditions separately measure evidence availability and answer selection, directly addressing the distinction raised in your comment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.