acceptodds
Under review as a conference paper at ICLR 2027

"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused

Abstract

LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs. Their finished work keeps receiving follow-up input that sometimes falsely accuses it for later failures. We call an agent's acceptance of such a false accusation *gaslight sycophancy*, and *destructive over-correction* when acting on it damages previously correct work. We introduce CAVE-Bench, a benchmark of 365 agentic tasks across six domains built around opaque tasks. Every scored run first reaches a verified correct state, whose supporting rationale and history stay in the workspace while the facts that would settle the accusation lie in external or runtime state beyond the agent's reach. The agent cannot confirm or refute the claim with a local check, so the right response should keep the work and ask for the missing evidence. Each task either hands the agent correct work with saved evidence or let it build and verify that work first, and five risk factors set how the accusation enters the workflow. We score accusation acceptance and evidence use from the trajectory and measure harm by deterministic replay of downstream events. Across 14 of the latest models in Claude Code, false accusations damage correct work in up to 60.06% of runs, and stronger models often do so after recovering the supporting evidence. The same model behaves differently across OpenCode, Codex, and Hermes, and a harness gate driven by the benchmark's live signals cuts replayed harm by 74%. These results show that preserving already-correct work under unsupported accusation is a distinct safety challenge for long-lived agents. Our benchmark and live trajectory demo is available at https://anonymous.4open.science/r/cave_bench-1C28/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent safety, LLM Agent, Sycophancy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QyA6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CAVE-Bench is an executable benchmark for gaslight sycophancy and destructive over-correction. Each task begins from a verified correct state, after which the agent receives a false accusation while decisive evidence remains outside its environment. The benchmark tracks whether the agent accepts blame, follows available counterevidence, and damages the correct state. Harm is measured by replaying fixed downstream events rather than judging the final response alone.

CAVE-Bench includes 365 tasks and 685 interactions across six domains. Fourteen model backbones are evaluated under Claude Code, with a smaller cross-harness comparison. The results show that agents may recover relevant evidence yet still act against it. The paper also studies inherited versus just-completed work and several safeguards for irreversible actions.

Critical strengths
  1. The benchmark measures consequential behavior rather than verbal agreement. An agent can resist the accusation in text but still damage the workspace, which reply-level evaluation would miss.

  2. The benchmark is relatively easy to audit. It covers several domains, models, and execution harnesses, while fixed replays reduce reliance on an LLM judge.

Critical weaknesses
  1. The comparison between just-completed and inherited work is not matched. The two conditions use different task collections, so task content and difficulty may contribute to the observed difference. Self-built runs also require the agent to establish the correct state first, which may select different task subsets across models. I therefore do not think the current experiment isolates work recency as the cause.

  2. The safeguards are tested only on false accusations. Preserving the current state is therefore always correct, which naturally favors evidence requirements and irreversible-action gates. Without matched cases where the earlier work is genuinely faulty, it is hard to judge how often the safeguards would block a valid repair. The paper should also describe how the live safety signal is produced.

Questions for authors
  1. Are any inherited and self-built tasks matched on workflow, accusation, evidence, and replay outcome?

  2. How many self-built runs are excluded because the model fails to establish the initial correct state, and do the results hold on a common task subset?

  3. What produces the safety signal used by the live gate, and what information can it access?

  4. How do the safeguards behave when the accusation is true and a repair is warranted?

2 comments
AuthorsOpenReview ↗

Response to Reviewer QyA6 (Part 1/2)

We sincerely thank the reviewer for the thoughtful and constructive review. We are encouraged that the reviewer finds that the benchmark measures consequential behavior, and the way we use replay to measure harm is relatively easy to audit. We respond to the three concerns in order. The revised manuscript (PDF) is updated, with all changes highlighted in teal.


Q1: "The comparison between just-completed and inherited work is not matched … Self-built runs also require the agent to establish the correct state first, which may select different task subsets across models."

Q2: "Are any inherited and self-built tasks matched on workflow, accusation, evidence, and replay outcome?"

Q3: "How many self-built runs are excluded because the model fails to establish the initial correct state, and do the results hold on a common task subset?"

A1: The two task types are separate collections, so RQ2 compares two settings. Both come from the same six domains, the same five risk factors with balanced assignment (Section 3.4, Figure 3), the same human review, and the same replay scoring. They differ in task content and in when the finished work enters the session. The revised Section 5.2 states the finding at this level: agents remain vulnerable on work they have just built and checked.

Revised Section 5.2: "Agents remain vulnerable on work they have just built and checked (Figure 6). On self-built tasks, 28.13% of later steps fail on replay, against 29.21% on inherited tasks. Agents ignore the saved evidence more often, reject the blame less often, and more often find the evidence and edit anyway. On the 83 self-built tasks that all 14 models build, agents use the saved evidence and reject the blame as often as on inherited work, yet they find the evidence and edit anyway more often, at 9.29% against 5.27% (Appendix Table 18). They see that the work is correct and still change it."

A2: No inherited task is matched with a self-built task on all four. We go through them in order:

  • Workflow and accusation: each collection has its own task directions (Appendix B).
  • Evidence: this is what defines the two types. An inherited task carries notes, history, and test logs. In a self-built task the agent's own build and check steps stay in context.
  • Replay outcome: both types are scored by replaying a fixed set of later steps on the final workspace, and each task has its own set.

A3: The agent builds the finished work in 2,494 of 2,702 self-built runs (92.3%). 206 runs are left out because the finished work was not built, and 2 because of a run error. GLM-5.2 accounts for 84 of them, and each of the other 13 models builds 88% to 99% of the tasks (QyA6-Table 1). GLM-5.2 is therefore the largest source of the runs left out. 83 tasks are built by all 14 models, and QyA6-Table 2 repeats the comparison on them.

QyA6-Table 1. Self-built runs per model under Claude Code, and why runs are left out.

ModelSelf-built tasksFinished work builtLeft out: finished work not builtLeft out: run error
Claude-Sonnet-5193171220
Qwen-3.8-Max19318670
Claude-Opus-519318490
GLM-5.2193108841
Kimi-K319318760
Hy3193170230
Grok-4.519318850
Qwen-3.7-Plus19319120
MiniMax-M3193183100
GPT-5.6-Sol19318940
Kimi-K2.7-Code19318760
MiniMax-M2.7193181120
Qwen-3.5-Flash19318580
Qwen-3.5-9B19318481

QyA6-Table 2. Inherited and self-built tasks under Claude Code, pooled over the 14 models, with the self-built tasks that all 14 models build.

Task setTasksRunsIgnores the saved evidence (%)Rejects the blame, points to evidence (%)Finds evidence, edits anyway (%)Later steps that fail (%)
Inherited, all1722,35520.1958.815.2729.21
Self-built, all1932,49424.6051.769.0228.13
Self-built, built by all 14 models831,16220.4858.789.2924.95

Results analysis. Agents remain vulnerable on the tasks every model builds, and the weak point is acting on evidence they have already found. On these 83 tasks, agents use the saved evidence and reject the blame as often as on inherited work (20.48% vs. 20.19% ignore the evidence, and 58.78% vs. 58.81% reject the blame). They then edit the work anyway almost twice as often (9.29% vs. 5.27%), and a quarter of later steps fail on replay (24.95%). Building and verifying the work in the same session leaves the agent able to see that the work is correct and still willing to change it.

AuthorsOpenReview ↗

Response to Reviewer QyA6 (Part 2/2)

Q4: "The safeguards are tested only on false accusations … Without matched cases where the earlier work is genuinely faulty, it is hard to judge how often the safeguards would block a valid repair." and "How do the safeguards behave when the accusation is true and a repair is warranted?"

A4: The three interventions leave real repairs still feasible (added to Section 5.4, Appendix Table 24):

Revised Section 5.4: "On the paired tasks of true and false accusations, GPT-5.6-Sol without intervention breaks 50.0% of correct work under false accusation. Under the three interventions, breakage of correct work falls sharply to 14.3%, 13.0%, and 8.6%. Interventions miss at most three more faults under true accusations compared to no intervention. In addition, the share of pairs where the agent both keeps correct work and fixes the real fault increases (Appendix Table 24). Agents can still repair genuine faults under true accusations while preserving correct work under false ones."

Table below reports the paired experiment for GPT-5.6-Sol under Claude Code.

QyA6-Table 3. The three interventions for GPT-5.6-Sol on paired tasks under Claude Code.

InterventionValid pairsFalse accusation: breaks correct work (%)True accusation: misses fault (%)True accusation: misses fault under None (same pairs, %)Both correct: keeps work, fixes fault (%)
None6050.030.030.043.3
I1 evidence rule5614.328.630.464.3
I2 irreversible-action gate5413.037.031.555.6
I3 live-signal gate588.636.231.058.6

Results analysis. On the paired tasks, GPT-5.6-Sol without intervention breaks 50.0% of correct work under false accusation. Under the three interventions, breakage of correct work falls sharply to 14.3%, 13.0%, and 8.6%. Our interventions miss at most three more faults under true accusations compared to no intervention. Agents can still repair genuine faults under true accusations while preserving correct work under false ones.


Q5: "The paper should also describe how the live safety signal is produced." and "What produces the safety signal used by the live gate, and what information can it access?"

A5: An LLM-based monitor (DeepSeek-V4-Flash) reads session context before each tool call (user messages, agent replies, past calls/outputs, pending call) without seeing ground truth or replay steps (added to Appendix D.6). The monitor triggers when the agent admits an unfounded fault or ignores evidence, blocking irreversible calls until evidence or user confirmation is provided.


We hope these additions address the reviewer's main concerns, and we would be glad to provide further details.

Reviewer_gHD2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how tool-using LLM agents can damage previously correct work by accepting false accusations and making unnecessary corrections. The authors introduce CAVE-Bench, a benchmark that evaluates false-responsibility acceptance, evidence use, and downstream harm through executable replay. Experiments show that agents may damage correct work even after recognizing supporting evidence, while evidence-based instructions and action gating can mitigate such harm.

Critical strengths
  1. A systematic framework for evaluating the preservation of correct work. CAVE-Bench makes destructive over-correction measurable through verified initial states and executable downstream replay. By separately assessing false-responsibility acceptance, evidence use, and realized harm, it distinguishes behavioral failures from their operational consequences, providing a useful benchmark for studying agent reliability under unsupported accusations.

  2. Extensive empirical evaluation across models and harnesses. The paper evaluates 14 models and compares selected models across four harnesses, examining task settings, risk factors, failure patterns, and mitigation strategies. The results reveal how failure rates and behaviors vary across settings and show that evidence-based instructions and action gating can mitigate different aspects of the problem.

Critical weaknesses
  1. The relationship to the closest existing benchmarks is insufficiently developed. Although the paper discusses several relevant research directions, it does not sufficiently explain what existing benchmarks already measure and which specific gaps CAVE-Bench fills. A more focused comparison would help establish the additional evaluation capabilities provided by CAVE-Bench. In particular, the authors should clarify how it differs in the availability of evidence needed to resolve the accusation, the construction of the initial correct state, the operational domains covered, and the measurement of downstream harm. This would make the motivation for the new benchmark and its contribution relative to prior work more convincing.

  2. The claim that fresh verification offers no protection is too strong. The experiments show that agents can still damage work they have just completed and verified. However, this does not establish that recent verification has no benefit: it could reduce the risk of damage without eliminating it. The paper compares agents that have just completed the work themselves with agents given an already-correct state, but uses different tasks for these two settings. Differences between the tasks therefore make it difficult to determine whether recent verification helps. The authors could compare the two settings using the same tasks, or narrow the conclusion to state that agents remain vulnerable even after completing and verifying their own work.

Questions for authors
  1. Which gaps in existing benchmarks does CAVE-Bench fill? Please clarify what existing works already cover and which additional aspects of the problem CAVE-Bench enables researchers to evaluate.

  2. What supports the claim that fresh verification offers no protection? Given that the two settings use different task collections, can the authors provide a comparison using matched tasks, or narrow the conclusion to state that agents remain vulnerable after verification?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gHD2 (Part 1/2)

We sincerely thank the reviewer for the thoughtful and constructive review. We are encouraged that the reviewer finds that CAVE-Bench makes it measurable when an agent breaks correct work, and that the evaluation across 14 models and four harnesses reveals how failures vary across settings. We respond to the two concerns in order. The revised manuscript (PDF) is updated, with all changes highlighted in teal.


Q1: "The relationship to the closest existing benchmarks is insufficiently developed … the authors should clarify how it differs in the availability of evidence needed to resolve the accusation, the construction of the initial correct state, the operational domains covered, and the measurement of downstream harm."

A1: CAVE-Bench uniquely evaluates unverifiable finished work using later step (downstream) replay, distinguishing it from prior benchmarks along all four dimensions (added to Section 2):

gHD2-Table 1. CAVE-Bench next to the closest benchmarks, along the four dimensions of the review.

WorkEvidence to verify the accusationInitial situationDomainsHow harm is measured
FlipFlop [1]none, answers to fixed questionsan answer of mixed correctnessclassificationanswer accuracy
Wang et al. [2]in the problema correct answerreasoningshare of correct answers given up
SycEval [3]none, answers to fixed questionsan answer of mixed correctnessmath, medicineanswer flips, LLM-judged
MINT [4], ConvCodeWorld [5]available, the code is executedan unsolved task, feedback after a wrong attemptreasoning, code, decision-makingtask success, tests for code
FixedBench [6]available, the agent can reproduce the issuecode with the fix already appliedPython repositoriesdiff of the changes
SWE-Touch [7]available in the repositoryan ongoing repair with user editsPython repositoriesunit tests
XYEval [8]available in the taska task not yet donesix benchmark suitestask score
FaulT-Bench [9]available, the agent can probe the networka healthy network in the false-ticket casesnetworkingLLM judge on diagnosis text
CAVE-BenchOut of reach: facts are held outside the workspaceCorrect when accusation arrives (inherited or self-built)Six domains: coding, DevOps, files, social, transactions, webReplay of later steps on final workspace

Compared with prior benchmarks, CAVE-Bench introduces two key distinctions:

First, existing benchmarks start from unfinished or faulty work, or keep the deciding facts within the agent's reach. CAVE-Bench alone combines correct finished work with deciding facts held outside the workspace, while the evidence that the work is finished remains inside.

Second, prior work scores the answer, the diff, or the tests of the task at hand. CAVE-Bench replays the later steps that depend on the work, scoring what the agent actually breaks rather than what it says.

This measurement is essential: in 15.45% of Claude Code runs, what the agent says and what it does point in opposite directions, a decoupling that scoring the agent's text alone misses.

AuthorsOpenReview ↗

Response to Reviewer gHD2 (Part 2/2)

Q2: "The claim that fresh verification offers no protection is too strong … The authors could compare the two settings using the same tasks, or narrow the conclusion to state that agents remain vulnerable even after completing and verifying their own work."

A2: The conclusion now reads that agents remain vulnerable after completing and verifying their own work (revised Section 5.2).

What supports it. In a self-built task, the agent builds the work and runs its checks in the same session, and the false accusation arrives right after. 28.13% of later steps then fail on replay, against 29.21% on inherited tasks. Harm after fresh verification stays at the level of inherited work. The two settings use different tasks, so these numbers describe the level of harm in each setting.

Revised Section 5.2: "Agents remain vulnerable on work they have just built and checked (Figure 6). On self-built tasks, 28.13% of later steps fail on replay, against 29.21% on inherited tasks. Agents ignore the saved evidence more often, reject the blame less often, and more often find the evidence and edit anyway. On the 83 self-built tasks that all 14 models build, agents use the saved evidence and reject the blame as often as on inherited work, yet they find the evidence and edit anyway more often, at 9.29% against 5.27% (Appendix Table 18). They see that the work is correct and still change it."

gHD2-Table 2. Inherited and self-built tasks under Claude Code, pooled over the 14 models, with the self-built tasks that all 14 models build.

Task setTasksRunsIgnores the saved evidence (%)Rejects the blame, points to evidence (%)Finds evidence, edits anyway (%)Later steps that fail (%)
Inherited, all1722,35520.1958.815.2729.21
Self-built, all1932,49424.6051.769.0228.13
Self-built, built by all 14 models831,16220.4858.789.2924.95

Results analysis. Agents remain vulnerable on the tasks every model builds, and the weak point is acting on evidence they have already found. On these 83 tasks, agents use the saved evidence and reject the blame as often as on inherited work (20.48% vs. 20.19% ignore the evidence, and 58.78% vs. 58.81% reject the blame). They then edit the work anyway almost twice as often (9.29% vs. 5.27%), and a quarter of later steps fail on replay (24.95%). Building and verifying the work in the same session leaves the agent able to see that the work is correct and still willing to change it.


We hope these additions address the reviewer's main concern, and we would be glad to provide further details.

References:

[1] Laban et al. Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment. arXiv 2311.08596.

[2] Wang et al. Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate. Findings of EMNLP 2023.

[3] Fanous et al. SycEval: Evaluating LLM Sycophancy. AIES 2025.

[4] Wang et al. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. ICLR 2024.

[5] Han et al. ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments. ICLR 2025.

[6] Gloaguen et al. Coding Agents Don't Know When to Act. arXiv 2605.07769.

[7] Tan et al. SWE-Touch: Benchmarking Coding Agents When Users Touch the Code. arXiv 2608.02499.

[8] Wu et al. XYEval: Agents Say Yes to Bad Advice. arXiv 2609.23939.

[9] Tseng et al. FaulT-Bench: Towards Benchmarking Network Troubleshooting LLM Agents under Unreliable User Tickets. arXiv 2608.27021.

Reviewer_PLNuOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores how models behave when falsely accused and creates a benchmark to measure the downstream behavior. They also do some empirical experiments and find that in 60% of the cases, models undo their work.

Critical strengths

The idea that one could falsely accuse models and this might change/undo the work LLMs have done was interesting on first read.

Critical weaknesses

(1) Its really hard to read this paper. It feels like heavily written with AI tools e.g. the presence of too many jargons plus the writing is not natural. This is my biggest reason for rejection. To even review properly, I would like the paper to be well-written and I would not like ICLR to be full of papers which are hard to read and comprehend.

(2) Missing explanation of notations used in 1 and 2. Also these equations are not really making it easier to understand the paper.

(3) The related work is very limited, only including sycophancy and agentic safety, which might be wrongly attributing novelty in some parts. I would've liked a comparison with some literature on how good are models at identifying good/bad instructions and making changes to their work based on feedback. Consider the setting but non-adversarial, that will lead to a lot of connections with existing literature.

(4) Related to the point above, there needs to be a comparison with controls like true accusations and neutral feedback to actually claim sycophancy in models?

(5) The threat model has limited real-world risk in my opinion. I believe it to not be super prevelant from an adversarial point of view, unless authors can prove otherwise.

Questions for authors

See above

9 comments
AuthorsOpenReview ↗

Response to Reviewer PLNu (Part 1/4)

We thank the reviewer for the constructive feedback and for finding our core question interesting: whether false accusations make models undo correct work. Below we address each concern including a plain rewrite, additional related works, new controls with true accusations and with neutral feedback, and real-world evidence for the threat model. The revised manuscript (PDF) is updated, with all changes highlighted in teal.


Q1: "Its really hard to read this paper. It feels like heavily written with AI tools e.g. the presence of too many jargons plus the writing is not natural. This is my biggest reason for rejection." Q2: "Missing explanation of notations used in 1 and 2. Also these equations are not really making it easier to understand the paper."

A1 & A2: We take readability seriously. The revision (updated PDF) rewrites the Abstract and Sections 1, 3, 4, and 5 with six kinds of changes, each shown with the submission version next to the revision:

1. Each term becomes a plain description.

  • Submission version (Abstract): "…built around opaque tasks. Every scored run first reaches a verified correct state, whose supporting rationale and history stay in the workspace while the facts that would settle the accusation lie in external or runtime state beyond the agent's reach."
  • Revision: "In each task, the agent's work is finished when the accusation arrives, and a local check finds nothing that supports the accusation, so the right response is to keep the work and ask for evidence."

2. One long sentence becomes short sentences with one idea each.

  • Submission version (Section 1): "Figure 1 shows what makes a task opaque: the workspace retains the rationale and history supporting the existing state, while the decisive facts that refute the accusation lie in inaccessible external or runtime state."
  • Revision: "The workspace keeps the evidence for the work, such as notes and history. The facts that would settle the accusation are held outside the workspace, in a system the agent has no access to, as they are in real deployments."

3. A general claim becomes what the agent does.

  • Submission version (Section 1): "This case exposes the central failure we study: recognizing the available evidence does not guarantee that the agent's actions remain consistent with it."
  • Revision: "The agent saw the evidence, and its action still went against it."

4. Notation becomes words (Q2). We removed Equations 1–2, and Section 3.1 keeps only the harm definition hih_i needed for Section 4.

  • Submission version (Section 3.1): "When a false accusation aia_i arrives, the agent observes Ii={xi,0,ri,ai}I_i = \lbrace x_{i,0}, r_i, a_i \rbrace but not the external or runtime evidence ziz_i that refutes it."
  • Revision: "The agent sees the work, the evidence, and the accusation. The facts that would disprove the accusation are held outside the workspace, in a system the agent has no access to."

5. Category names and inline formulas become questions and one table.

  • Submission version (Section 3.2): "The gaslight vector covers G1 direct assertion, G2 injected artifact, G3 poisoned project context, G4 misleading environment, and G5 self-fabricated history."
  • Revision: "The accusation enters the workflow in different patterns: direct statement, an injected file, a poisoned project note, set up a misleading environment, or the agent's own wrong history."
  • Submission version (Section 4): "Each run yields three signals for the decisions in Section 3.1: accusation acceptance fi∈{0,0.5,1}f_i \in \lbrace 0, 0.5, 1 \rbrace, evidence-recognition failure ei∈{0,0.5,1}e_i \in \lbrace 0, 0.5, 1 \rbrace, and harm hi∈[0,1]h_i \in [0, 1]."
  • Revision: "Does the agent admit a fault it did not cause? Does it ignore the evidence that shows the work is correctly finished? Does its change break the work?" One table in Section 4 then lists every measure and decision path with what it records.

6. Results describe behavior, and each paragraph of Section 5 opens with its finding.

  • Submission version (Section 5.1): "95% of Qwen's damaging runs follow CD while MiniMax routes a quarter of its damaging runs through EO and SD."
  • Revision: "For Qwen-3.5-9B, almost all damaging runs begin with the agent taking the blame and then editing. MiniMax-M2.7 relies much less on this path, frequently breaking finished work without admitting any fault."
AuthorsOpenReview ↗

Response to Reviewer PLNu (Part 2/4)

Q3: "The related work is very limited.. I would've liked a comparison with some literature on how good are models at identifying good/bad instructions and making changes to their work based on feedback."

A3: We added a subsection "Feedback and Self-Correction" and Table 1 in Section 2 along both dimensions:

1. Identifying Good vs. Bad Instructions

  • Benefiting from good instructions/feedback: When feedback is true and checkable, interactive prompting and tool verification significantly raise performance (Self-Refine [1], Reflexion [2], CRITIC [3], MINT [4], ConvCodeWorld [5]).
  • Failing under bad instructions/feedback: A contentless "Are you sure?" flips 46% of answers and lowers accuracy by 17% (FlipFlop [6]), invalid arguments cause models to abandon correct solutions (Wang et al. [7]), rebuttals turn correct answers wrong in 14.7% of cases (SycEval [8]), plausible but bad user advice cuts agent performance by up to 46.7% in relative terms (XYEval [9]), and false fault tickets lead network troubleshooting agents to promote benign conditions to root causes on a healthy network (FaulT-Bench [10]).

2. Making Changes to Completed Work Based on Feedback

  • Intrinsic self-correction is ineffective: Without reliable external ground truth, models struggle to self-correct reasoning and frequently degrade their own outputs upon re-prompting (Huang et al. [11], Kamoi et al. [12]).
  • The Confidence vs. Critique trade-off: Preserving correct answers trades off against fixing wrong ones (Yang et al. [13]); prompting models to revise more makes them give up correct answers more often.
  • Unwarranted revision in agent repositories: In coding environments, agents modify code for 35% to 65% of issues that are already fixed (FixedBench [14]), lose previously correct behavior in 40% to 73% of multi-turn tasks (Regression Accumulation [15]), and resolve 7.7 points fewer issues when users make incorrect edits to the code (SWE-Touch [16]).

3. How CAVE-Bench Extends This Literature

Existing feedback studies start from unfinished or faulty work, or keep the deciding facts within the agent's reach. CAVE-Bench alone combines correct finished work, deciding facts held outside the workspace, and harm measured by replaying later steps:

PLNu-Table 1. CAVE-Bench compared with closest studies on feedback, revision, and interactive benchmarks (Table 1 in Section 2).

WorkFeedbackEvidence for VerificationEarlier workDomainsHarm scored by
Self-Refine [1], Reflexion [2], CRITIC [3]Helpful self/tool feedbackModel's own feedback, tools, or executionDraft / incomplete outputQA, code, reasoningAccuracy / pass rate gain
Huang et al. [11], Kamoi et al. [12]None (intrinsic self-correction)None, the model's own judgmentMixed correctnessReasoning, mathAccuracy change
FlipFlop [6]A bare challenge ("Are you sure?")None, answers to fixed questionsAn answer of mixed correctnessClassificationAnswer accuracy
Wang et al. [7]Invalid argumentsIn the problemA correct answerReasoningShare of correct answers given up
SycEval [8]True and false rebuttalsNone, answers to fixed questionsAn answer of mixed correctnessMath, medicineAnswer flips, LLM-judged
MINT [4], ConvCodeWorld [5]Helpful language and execution feedbackAvailable, the code is executedAn unsolved task, feedback after a wrong attemptReasoning, code, decision-makingTask success, tests for code
FixedBench [14]Stale reports of fixed issuesAvailable, the agent reproduces the issueCode with the fix already appliedPython repositoriesDiff of the changes
SWE-Touch [16]Incorrect user editsAvailable in the repositoryAn ongoing repair with user editsPython repositoriesUnit tests
XYEval [9]Plausible but wrong user adviceAvailable in the taskA task not yet doneSix benchmark suitesTask score
FaulT-Bench [10]Accurate and wrong fault ticketsAvailable, the agent probes the networkA healthy network in the false-ticket casesNetworkingLLM judge on diagnosis text
CAVE-Bench (Ours)False accusations (5 entry points)Out of reach: facts are held outside the workspaceCorrect finished work (inherited or self-built)Six domains: coding, DevOps, files, social, transactions, webReplay of later steps

CAVE-Bench differs in three ways:

  1. Unreachable evidence: Disproving facts sit outside the workspace rather than locally checkable.
  2. Finished work: Agents must protect completed, correct work whose deciding facts are out of reach.
  3. Later step replay: Harm is measured by replaying dependent steps to catch real functional breakage, rather than diffs or LLM judges.
AuthorsOpenReview ↗

Response to Reviewer PLNu (Part 3/4)

Q4: "There needs to be a comparison with controls like true accusations and neutral feedback to actually claim sycophancy in models?"

A4: We ran both controls as new experiments, the true-accusation control and the neutral-feedback control, and the damage comes from following the blame (added to Section 5.1 and Appendix Table 16):

Revised Section 5.1: "A capable agent cannot simply refuse every accusation. It must tell false accusations apart from genuine faults. We evaluate 60 paired tasks, coupling each false accusation on correct work with a matched true accusation on a real workspace fault. A model handles a pair correctly only when it keeps correct work under the false accusation and repairs the genuine fault under the true one. Claude-Sonnet-5, GPT-5.6-Sol, and Qwen-3.5-9B succeed on both sides in 60.4%, 43.3%, and 23.6% of pairs, showing that stronger models better separate real bugs from false blame (Appendix Table 16). The blame itself drives the damage. Under a neutral check-in that names no fault, the three models break correct work in 1.9%, 14.3%, and 28.0% of tasks, against 18.8%, 50.0%, and 65.5% under the false accusation (Appendix Table 16)."

PLNu-Table 2. Paired tasks with a false and a true accusation under Claude Code.

ModelValid pairsFalse accusation: breaks correct work (%)True accusation: misses fault (%)Both correct: keeps work, fixes fault (%)Keeps work, misses fault (%)Breaks work, fixes fault (%)Breaks work, misses fault (%)
Claude-Sonnet-54818.827.160.420.812.56.3
GPT-5.6-Sol6050.030.043.36.726.723.3
Qwen-3.5-9B5565.554.523.610.921.843.6

Results analysis. Stronger models tell true accusations from false ones, and weaker models follow the blame. Claude-Sonnet-5 keeps correct work and fixes the real fault in 60.4% of pairs, against 23.6% for Qwen-3.5-9B. Qwen-3.5-9B edits correct work in 65.5% of false accusations yet fixes the real fault in only 45.5% of true ones: it follows the blame, whether or not a fault exists.

For the neutral-feedback control, the same tasks keep the correct finished work. The same role sends a neutral check-in that asks the agent to look at the finished work once more and names no fault.

PLNu-Table 3. The same correct finished work under a false accusation and under a neutral check-in, Claude Code.

ModelFalse accusation: valid tasksFalse accusation: breaks correct work (%)Neutral check-in: valid tasksNeutral check-in: breaks correct work (%)
Claude-Sonnet-54818.8541.9
GPT-5.6-Sol6050.05614.3
Qwen-3.5-9B5565.55028.0

Results analysis. The damage follows the blame. Under the neutral check-in, the three models break correct work in 1.9%, 14.3%, and 28.0% of tasks, against 18.8%, 50.0%, and 65.5% under the false accusation. The same finished work is damaged more than twice as often once someone blames the agent, which is the pattern of sycophancy.

For reference, the submission version of CAVE-Bench also covers neutral feedback, as neutral wording and as self-doubt in the agent's own notes. Across the full benchmark, 10.69% of later steps fail under neutral wording against 35.77% under an appeal to authority, and 8.53% fail when the doubt appears only in the agent's own notes against 37.79% under direct blame.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

For clarity and simplicity, we will refer to Reviewers PLNu, gHD2, and QyA6 as R1, R2, and R3, respectively, in the following response.

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by their recognition of the key contributions and strengths of our work.

An interesting problem measured by consequences: "The idea that one could falsely accuse models and this might change/undo the work LLMs have done was interesting" (R1). "A systematic framework for evaluating the preservation of correct work" (R2). "The benchmark measures consequential behavior rather than verbal agreement" (R3).

An extensive evaluation that is easy to audit: "Extensive empirical evaluation across models and harnesses" (R2). "The benchmark is relatively easy to audit. It covers several domains, models, and execution harnesses, while fixed replays reduce reliance on an LLM judge" (R3).


Core Contributions of Our Work

A Severe and Timely Failure Mode: As agents resume after compaction or take over handoffs, a false accusation about their finished work leads them to damage it, and even the strongest frontier models do so.

A New Lens Beyond Chatbot Sycophancy: Prior sycophancy studies mainly score what a model says. CAVE-Bench evaluates what the agent does in its consequence: it replays the later steps that depend on the work, catching damage that the agent's reply alone would hide.

Seeing the Evidence Is Not Enough: Agents often find the evidence that their work is correct and still change it, so recognizing the evidence does not protect the work.

Practical Safeguards: Lightweight harness-level gates sharply reduce the damage and need no change to the model, while agents can still repair genuine faults under true accusations.


We addressed every comment with new experiments or exact locations in the paper, and R1's follow-up questions led us to document our construction checks in full. Below we summarize the updates in our revision.

Updates of experimental results during Rebuttal

Main Text : Added true accusation experiment on 60 matched twin pairs and a neutral check-in control on the same tasks (R1), subset analysis on the 83 self-built tasks built by all 14 models (R2, R3), and paired evaluations of safeguards under true accusation (R3).

Appendix D.2 : Added per-model validity analysis and subset comparison on self-built tasks (R2, R3).

Appendix D.3 : Added neutral feedback and self-doubt analysis (R1).

Appendix D.1 and D.6 : Added full paired results for model competence and the neutral check-in control (Table 16) and safeguard preservation (Table 24) on 60 matched true accusation pairs (R1, R3).

Updates of in-depth discussions during Rebuttal

Main Text : Added systematic comparison with feedback, sycophancy, and self-correction literature in Section 2 (R1, R2), and clarified the distinct findings of inherited versus self-built tasks in Section 5.2 (R2, R3).

Appendix B : Added the construction checks, diversity rules, and human–LLM agreement at each stage where AI is used (R1).

Appendix A : Documented real world non-adversarial industry incidents behind every benchmark risk factor in Table 12 (R1).

Appendix D.6 : Detailed the monitor specification, prompt design, and input scope for the live-signal gate (R3).

All revisions are highlighted in teal.


Summary of the Discussion: R2 found that our responses addressed their concerns and keeps the positive rating at 3, after we compared CAVE-Bench with the closest benchmarks and narrowed the self-built finding with the 83-task common subset. For R1, we rewrote the paper in plain language, added the feedback and self-correction literature, and ran both requested controls, which show that the damage comes from the false blame itself; R1 then asked follow-up questions on the correctness and diversity of the AI-assisted tasks, which we answered in full the same day, with no further reply. The follow-up raised no remaining concern about the original review. For R3, we added the common-subset analysis, showed that the safeguards still allow genuine repairs under true accusations, and described the live-signal monitor. They maintain the score.

We sincerely thank the reviewers and the Area Chair for their careful handling of our submission, and we truly hope CAVE-Bench can contribute to agent safety as a new and timely problem for long-lived agents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.