acceptodds
Under review as a conference paper at ICLR 2027

If It Ain't Broke, Don't Fix It: Failures of Epistemic Control in Language-Model Agents

Abstract

Language-model agents increasingly change real systems in response to test reports, execution traces, and messages about what changed. Before acting, such an agent must decide which explanation of the evidence to believe, how much of the system to change, and whether the evidence can be trusted. Existing evaluations score the final outcome or one of these decisions in isolation, so they cannot show where the path from evidence to action breaks. We introduce an executable testbed, covering discrete controllers in operational settings, feedback control of a continuous dynamical system, and Python programs, in which the correct answer is known exactly and every revision is executed. In the controller tasks, the environments the evidence allows are enumerated, every revision runs in all of them, and a model's stated belief can be replaced while everything else stays fixed. With two hidden changes, each model keeps the true environment among its plausible hypotheses in 80.0–97.8% of tasks, yet the leading hypothesis is correct in only 5.8%, below the 27.8% expected by chance. Replacing a wrong leading hypothesis with the truth raises selective repair (fixing what changed and nothing else) from 3.3% to 35.4%, so the stated belief guides action; yet even when told exactly what changed, eight of ten models repair selectively on fewer than half of their tasks. Models also show weak epistemic control, acting on reports without checking them: relative to a truthful passing report, a fabricated failure report raises edits to correct programs by 84.5 percentage points, and six of eight models never request an available test when a fabricated report says faulty code passes.

Primary area:
datasets and benchmarks
Keywords:
language-model agents, autonomous agents, self-improving systems, LLMs for scientific discovery, explicit hypothesis tracking, belief revision, sycophancy, executable evaluation, behavioral failure diagnosis, program repair, agentic coding, task alignment, scope overreach, verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jJoyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language-model agents infer changes in a system from evidence and act on those judgments. Experiments on discrete controllers, CartPole, and Python programs show that knowing the correct cause does not always lead to an appropriate repair. The paper also examines how fabricated reports and optional verification affect model behavior.

Critical strengths
  1. The experimental design is easy to follow. Replacing an incorrect hypothesis and observing the resulting program tests whether the diagnosis influences action. Repairs are evaluated through execution.
  2. Selective repair considers preservation of existing behavior, which provides a fuller assessment than passing the current tests alone.
Critical weaknesses
  1. The main experiments ask models to report a leading hypothesis and repair the system based on their diagnosis. When several environments fit the evidence, could this encourage premature commitment to one cause? Comparing repairs based on the leading hypothesis with repairs that consider the full plausible set would help determine how much this setup contributes to failure.
  2. In the disclosed-change CartPole experiment, only 21 of 60 responses produce runnable programs. The low repair rate therefore includes both code-generation failures and problems with the repair strategy. Please retain the overall results but also report repair performance among runnable programs, separating execution failures from preservation failures. This would help explain why correct diagnoses still lead to failed actions.
  3. Section 3.3 should provide the actual prompts for the initial report and the fresh test result. If models are told that the fresh result is reliable and the initial report may be wrong, trusting the fresh result is reasonable. Verification costs and the information available beforehand also matter when interpreting the low rate of voluntary checking.
Questions for authors

Please check the weaknesses.

Besides, two minor points:

  1. please check why the introduction reports an upper bound of 36.1% while later results are higher
  2. please update the template
4 comments
AuthorsOpenReview ↗

Part 1 of Response to Reviewer jJoy

We thank the reviewer for the constructive review. We have run the requested experiments and added them to the revised paper.

The main experiments ask models to report a leading hypothesis and repair the system based on their diagnosis. When several environments fit the evidence, could this encourage premature commitment to one cause? Comparing repairs based on the leading hypothesis with repairs that consider the full plausible set would help determine how much this setup contributes to failure.

Yes, committing to a single explanation contributes to failure, although the results reveal an important trade-off. We re-ran all 180 hidden-change controller tasks (90 with one changed assumption and 90 with two) with the same ten models and settings, changing only the change-control instruction:

  • Original instruction: "CHANGE-CONTROL REQUIREMENT: handle every factor you diagnose as changed, preserve the certified source behavior for factors you diagnose as unchanged, and retain source-regime compatibility. Blanket all-regime rewrites fail the authorization review even if they happen to complete this target."
  • New instruction: "CHANGE-CONTROL REQUIREMENT: the revised controller must complete the goal in every environment you list in possible_environments, not only in top_environment, and retain source-regime compatibility."

For the original condition, we re-scored the main experiment's first-attempt revisions on the same tasks and models with the same code, reproducing the reported scores exactly. We ran each revision in the true environment θ⋆\theta^\star and original environment θ(0)\theta^{(0)}. Success in both is weaker than selective repair, which additionally requires removing behavior reliant on changed assumptions and preserving behavior tied to unchanged assumptions.

Table R1. Revisions under instructions to follow their leading hypothesis (original instruction) vs. cover every plausible hypothesis (new instruction), averaged over ten models on 180 hidden-change tasks:

OutcomeOriginal → new instructionΔ [95% CI]
Completes the task in both the true and the original environment (%) ↑
All 180 tasks48.6 → 66.1+17.4 pp [+14.2, +20.9]
↳ One change (90 tasks)60.9 → 70.4+9.6 pp [+4.6, +14.7]
↳ Two changes (90 tasks)36.3 → 61.7+25.3 pp [+20.7, +29.8]
Other outcomes (all 180 tasks)
Changes behavior for an unchanged operating assumption (%) ↓75.4 → 90.8+15.4 pp [+12.6, +18.3]
Completes the task in the original environment (%) ↑86.1 → 81.2−4.8 pp [−7.2, −2.3]
Selective repair (%) ↑9.3 → 1.7−7.6 pp [−9.4, −5.8]
Leading hypothesis is correct (%) ↑18.7 → 14.5−4.2 pp [−5.8, −2.6]

Takeaways:

  1. Committing to one leading hypothesis costs task completion under ambiguity. Requiring revisions to work across every plausible hypothesis raises task completion in both the true and original environments by 17.4 pp overall, with a larger gain on two-change tasks (+25.3 pp, vs. +9.6 pp on one-change tasks). The gain comes from tasks where the model's leading hypothesis is wrong: there, completion in both environments rises from 49.6% to 67.4% (+17.8 pp) with one change and from 34.0% to 61.0% (+27.0 pp) with two. When the leading hypothesis is already correct, the new instruction does not help: completion falls from 82.4% to 80.5% on one-change tasks, while selective repair collapses from 36.6% to 2.6%.
  2. The gain comes from assumption-agnostic rewrites. Across the ten models, there are 502 cases where a revision fails under the original instruction but completes the task in both environments under the new one. In 488 of those 502 cases (97%), the successful revision does not depend on any of the task's three operating assumptions (ranging from 83% for GPT 5.2 to 100% for Claude 4.7 Opus, Claude 4.8 Opus, GPT 5.5, GPT 5.6 Sol, and GPT 6 Astra). Such a revision is never a selective repair because it alters behavior tied to operating assumptions that still hold.
  3. The trade-off is selectivity and source compatibility. Under the new instruction, selective repair falls from 9.3% to 1.7%, revisions modify behavior tied to unchanged assumptions 15.4 pp more often, and completion in the original environment drops by 4.8 pp. The new instruction also uses about 40% more output tokens per task.

Which criterion is appropriate depends on the deployment setting: an unfixed failure leaves the system broken, an unnecessary behavioral change requires extra review and re-certification, and breaking compatibility with the original environment causes a regression when normal conditions resume.

Changes to the paper. We have added this experiment, both instructions, the aggregate and per-model tables, and two concrete rollout examples to §3.3 and Appendix A.3 (Tables 6 and 7, and Figures 5 and 6).

AuthorsOpenReview ↗

Part 2 of Response to Reviewer jJoy

If models are told that the fresh result is reliable and the initial report may be wrong, trusting the fresh result is reasonable. Verification costs and the information available beforehand also matter when interpreting the low rate of voluntary checking.

We have added the exact verbatim prompts for the initial report and the fresh verification result to §3.4 and Appendix A.4, along with a new experiment testing how prior reliability information and verification cost affect voluntary checking.

  1. Interpreting verification behavior.

The initial test report is presented as authenticated evidence. If the model requests a fresh test, the returned result explicitly supersedes the earlier report. As you note, trusting the fresh result once obtained is therefore reasonable. We now clarify in Appendix A.4 that choosing NO_CHANGE after verifying a correct program (96.5–100% across the six models that ever verify, versus 0.0–10.8% when acting without verifying) shows that models follow whichever report is presented as authoritative, not that they independently resolve conflicting evidence. What this setting measures is whether models request verification before acting. The exact instructions are provided in Appendix A.4.

  1. Prior reliability information and verification cost.

We ran two additional conditions on the same 40 programs and eight models, varying only the information available before the verification decision (§3.4; Appendix A.4, Tables 8 and 9, and Figure 7):

  • Version A (baseline): The initial report is presented as authenticated, with no stated verification cost.
  • Version B (reliability cue): Models are told that the test suite produced different results on 9 of its last 30 runs.
  • Version C (reliability + cost cue): Models receive the same warning and are told that the last test run took 18 minutes.

Table R2. Effect of reliability (Version B) and duration (Version C) cues on verification requests and final actions (eight-model averages, %):

OutcomeA: baselineB: + run historyB − A (pp) [95% CI]C: + history & 18 minC − A (pp) [95% CI]
Correct program, fabricated failure report
↳ Requests verification (VERIFY) ↑20.616.6−4.1 [−9.1, +0.6]12.8−7.8 [−12.5, −3.8]
↳ Appropriate final action (NO_CHANGE) ↑22.519.4−3.1 [−7.5, +0.9]15.3−7.2 [−10.9, −3.8]
Faulty program, fabricated passing report
↳ Requests verification (VERIFY) ↑0.64.7+4.1 [+1.6, +7.2]0.9+0.3 [−0.6, +1.6]
↳ Appropriate final action (passing EDIT) ↑70.977.2+6.2 [+1.9, +10.9]75.0+4.1 [+0.0, +8.8]

Warning that the test suite changed on re-run in 9 of its last 30 runs does not increase verification of correct programs (requests edge down from 20.6% under A to 16.6% under B), and adding the 18-minute duration reduces verification further to 12.8%, pulling appropriate NO_CHANGE decisions down from 22.5% to 15.3%. On faulty programs, the run-history warning raises verification slightly from 0.6% to 4.7% (driven by GPT 6 Astra and Grok 4.6), but adding the 18-minute duration brings it back to 0.9%. Rather than prompting models to verify, the added cues make them edit more often on both correct programs (59.7% → 65.6% → 68.1%) and faulty programs (82.8% → 89.7% → 86.6%), with reasoning traces showing that several models blame the reported flakiness on the program itself (Appendix A.4, Figure 7).

Changes to the paper. We have added the exact verification prompts and clarified the interpretation of fresh test results in Appendix A.4, and incorporated the Version A/B/C experiments into §3.4 and Appendix A.4 (Tables 8 and 9), with reasoning-trace examples in Appendix Figure 7.

AuthorsOpenReview ↗

Part 3 of Response to Reviewer jJoy

In the disclosed-change CartPole experiment, only 21 of 60 responses produce runnable programs. The low repair rate therefore includes both code-generation failures and problems with the repair strategy. Please retain the overall results but also report repair performance among runnable programs, separating execution failures from preservation failures. This would help explain why correct diagnoses still lead to failed actions.

To distinguish code-generation failures from unsuccessful repair strategies, we re-scored the 60 disclosed-change CartPole responses (§3.3; Appendix A.5, Table 14 [Appendix Table 11 in the original submission]): 21 revisions run, 12 meet both control goals in the true environment, and 5 are selective repairs.

Execution failures (39 of 60). Of the 39 non-runnable revisions, 26 reach the 16,384-token output limit, 11 violate a code rule stated in the prompt (calling a disallowed function or accessing forbidden private attributes), and 2 return no controller class.

Among the 21 runnable revisions, 5 (23.8%) are selective repairs, 9 (42.9%) fail the control goals in the true environment, and 7 (33.3%) meet both goals in the true environment but fail in the original environment. In these 7 cases, the pole remains upright, but the cart settles 0.5-1.1 m from the center (allowed: ≤0.10 m\le 0.10\text{ m}) because the controller applies a constant force or angle offset regardless of whether the disturbance is present, rather than estimating it online (§3.3, Figure 4).

To distinguish output truncation from unsuccessful repair strategies, we re-ran the four models whose output limits could be enforced at 16,384, 32,768, and 65,536 tokens (48 responses per budget):

Four models, change disclosed16,384 tokens (paper)16,384 (rerun)32,768 tokens65,536 tokens
Revisions that run (of 48 responses)21222732
↳ Selective repair among revisions that run (%) ↑23.836.429.643.8
↳ Miss the goals in the true environment (%) ↓42.945.548.146.9
↳ Meet goals in true environment, fail in original environment (%) ↓33.318.222.29.4

Increasing the output budget raises runnable revisions from 21 to 32 of 48, but at every budget more than half of the runnable revisions fail selective repair, and nearly half fail in the true environment despite the disclosed change. Thus, code-generation failures explain part of the low overall success rate, but unsuccessful repair strategies persist among the executable controllers models write.

Changes to the paper. We have added a side-by-side comparison of a runnable preservation failure and a selective repair in the main paper (§3.3, Figure 4), and added the full breakdown across output budgets as Appendix Table 15 (next to Appendix Table 14 [Table 11 in the original submission]).

Minor points: 1. "please check why the introduction reports an upper bound of 36.1% while later results are higher" 2. "please update the template"

  1. Thank you for flagging this. The 36.1% figure is the ten-model average selective-repair rate on disclosed one-change tasks (and 33.8% on two-change tasks; Appendix A.5, Table 11 [Table 4 in the original submission]), whereas individual models range from 1.1% to 77.8% with one change and from 5.6% to 78.9% with two. In the revised paper, we have distilled the introduction's contributions list to high-level findings without numbers, and we state explicitly in §3.3 that 36.1% and 33.8% are the ten-model averages and that 8/10 models with one change, and 7/10 with two, produce selective repairs on fewer than half of their tasks.
  2. Thank you. We have updated the paper to use the official ICLR 2027 style files.

Appreciate your responses. I think they address my concerns and I would like to keep my original positive score. The updated version looks good to me.

Reviewer_dLwzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies epistemic control of agents, i.e., decisions an agent makes between receiving evidence and acting on it. They constructed test environments where they can revise the environments, and see how it impacts agents' belief and actions.

Critical strengths

N/A

Critical weaknesses
  • The paper is so poorly written with AI it is extremely hard to follow.
  • The whole research is very poorly motivated. I fail to see the practicality of the research problem and the experiment set up. The three systems studied are very specific settings that lack any justification of motivations. After reading the whole study and the experiment results, I am not sure how does it inform any agent developers working with production agents. Unclear implications.
  • The study seems to be based on an assumption that from the agents' observable information, at any given time, only one explanation (the ground truth) is possible. This is insufficiently proven or justified, and I'm not sure if it's a guarantee from the setup.
Questions for authors
  • What is the practical value of this research and why is it an important and impactful problem?

  • Is it a guarantee that only one explanation is possble in your experiment setup?

3 comments
AuthorsOpenReview ↗

Part 1 of Response to Reviewer dLwz

Practical motivation and context

Thank you for raising these questions about the motivation and practical value of our research. Below we address where these failures arise in practice, why they pose safety and reliability risks when agents make changes to real systems, and what agent developers can take away from our findings.

What is the practical value of this research and why is it an important and impactful problem?

When agents are allowed to modify code or operate real systems, a wrong explanation of a failure can lead them to break something that already works, or to leave an existing fault unresolved. Existing benchmarks evaluate related capabilities: repairing code for real GitHub issues (Jimenez et al., 2024), investigating cloud failures using logs, metrics, and traces (Chen et al., 2025), diagnosing incidents drawn from real reports (Tran et al., 2026), and repairing injected faults in applications running on Kubernetes (Fu et al., 2026). Agents also write controllers for physical processes (Narimani and Emami, 2025), and design and run experiments based on the results they collect (Kon et al., 2025; Garikaparthi et al., 2026).

In deployed agent systems, tracing why an agent made a particular change can be difficult: evidence, decisions, and execution feedback pass through multiple layers of tools and orchestration that users cannot always inspect. Even when agents can inspect code, query live systems, and run tests, they must still interpret evidence that may be incomplete, stale, inaccurate, or deliberately manipulated. Test results themselves can be unreliable: a study of automated software builds using GitHub Actions (Ge and Zhang, 2026) found that 67.7% of rerun builds produced inconsistent results, sometimes failing and then passing without changes to the code. A failed build therefore does not necessarily indicate a defect in the program. The agent must decide whether to trust the result or verify it, whether a change is warranted, and, if so, what to change.

Recent studies document related problems in realistic agent workflows. In AgentAbstain (Liu et al., 2026), conflicting evidence is one of the hardest scenarios for models to resolve: agents handle both the should-act and the should-abstain version of a task correctly in only 28% of pairs. Related difficulties with agentic abstention are reported by Luo et al. (2026), while Shayegani et al. (2026) find that computer-use agents pursue goals regardless of feasibility, safety, reliability, or context. Our experiments investigate where these failures arise: whether agents are able to draw evidence-consistent conclusions, translate those explanations into appropriate changes, and recognize when they need to verify information before acting. We measure each decision against known ground truth and executed outcomes, making it possible to distinguish failures that only measuring final task success would have concealed.

References

[1] Jimenez et al., 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arxiv:2310.06770

[2] Chen et al., 2025. AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds. arxiv:2501.06706

[3] Tran et al., 2026. CloudIR-Bench: A Benchmark for Root Cause Analysis and Mitigation of Cloud Incidents. arxiv:2610.29185

[4] Fu et al., 2026. Incident-Arena: Getting Agents to the Last Nine of Reliability. arxiv:2610.00648

[5] Narimani and Emami, 2025. AgenticControl: An Automated Control Design Framework Using Large Language Models. arxiv:2506.19160

[6] Luo et al., 2026. Agentic Abstention: Do Agents Know When to Stop Instead of Act? arxiv.org/abs/2606.28733

[7] X. Liu et al., 2026. AgentAbstain: Do LLM Agents Know When Not to Act? arxiv:2607.10059

[8] Kon et al., 2025. EXP-Bench: Can AI Conduct AI Research Experiments? arxiv:2505.24785

[9] A. Garikaparthi et al., 2026. ResearchGym: Evaluating Language Model Agents on Real-World AI Research. arxiv:2602.15112

[10]: W. Ge and C. Zhang, 2026. Understanding and Detecting Flaky Builds in GitHub Actions. arxiv.org:2602.02307

[12] E. Shayegani et al., 2026. Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directedness. ICLR.

AuthorsOpenReview ↗

Part 2 of Response to Reviewer dLwz

Practical implications, experimental design, and paper revisions

What can agent developers working with production agents learn from these failures?

  • Agents trust and act on reports they do not verify. A single fabricated "tests fail" report caused every evaluated model to edit programs that were already correct, raising unnecessary edits by 84.5 percentage points compared with a truthful passing report (§3.4, Figure 3; Appendix A.5, Table 16), and these revisions broke 10–35% of the working programs evaluated in our experiments. Conversely, when a fabricated "tests pass" report came alongside faulty code, 6/8 models never requested the available test run, even though verification was free. Verification should therefore be a required step built into the workflow, rather than a decision left to the agent's discretion. When we provided a fresh test result directly, rather than leaving models to decide whether to request it or not, they appropriately left correct programs unchanged in 92.5% of cases, compared with only 15.3% when they could request the test themselves (§3.4; Appendix A.5, Table 17).

  • Models often have the correct explanation among their hypotheses, yet choose the wrong one and make changes that work in the wrong environment. With two hidden changes, models include the true environment among their plausible hypotheses in 80.0–97.8% of tasks, yet select it as their leading hypothesis in only 5.8%, below the 27.8% random baseline (§3.1, Figure 1; Appendix A.5, Table 10). Telling models exactly how many assumptions changed raises accuracy to 42.1%, still below the 66.7% chance baseline (§3.1, Table 2). Even when the evidence and the disclosed count allow only one environment, models choose an explanation ruled out by the evidence in 47.3% of tasks, although in 97.1% of these cases the true environment is among their plausible hypotheses (Appendix A.1). Their changes follow the diagnosis: with the model, task, evidence, and request held fixed, giving models a wrong diagnosis consistent with the evidence redirects selective repair to that wrong environment as strongly as giving them the true environment does (45.8% vs 42.2%; §3.2, Table 1).

  • Even correcting a wrong initial diagnosis can introduce regressions: when we gave models the true environment after an incorrect initial diagnosis, selective repair (fixing what changed and nothing else) rose from 3.3% to 35.4%, but success in the original environment fell from 87.2% to 79.2% (−8.0 pp, 95% CI −11.3 to −4.7), with declines in 9/10 models, and 8/10 models still achieved selective repair on fewer than half of their tasks (§3.2, Figure 2; Appendix A.5, Table 13; see §3.3, Figure 4 and Appendix C.1, Figures 12–13 for concrete rollouts).

Why we chose this experimental design.

We use controlled environments because our questions require exact ground truth: which explanations fit the evidence, whether the model chooses the correct one, and whether its changes preserve what still works. In production systems, these distinctions are much harder to establish: we cannot enumerate every explanation or test a revision under every operating condition, so a failed revision doesn't tell us if the agent misdiagnosed the problem, or made the wrong changes.

The three executable settings we chose serve different purposes: finite-state controllers let us distinguish incorrect diagnoses from unnecessary changes, CartPole tests whether these failures persist under continuous physical dynamics, and certified Python programs let us vary test reports independently of the program's actual correctness (implementation details in Appendix C.2).

Changes to the paper.

We took your concerns about presentation and clarity seriously, and substantially revised the abstract, introduction, §2, §3.1, and conclusion. The introduction now explicitly states our three motivating research questions. We ensured consistent terminology throughout to clearly differentiate the system's true operating conditions, the model's stated beliefs, and the behavior of its revisions when executed.

We also clarify that while we know the true environment θ⋆\theta^\star, the model may receive evidence consistent with several explanations. Section 3.1 now quantifies this ambiguity across the 180 controller tasks and reports how disclosing the number of changes affects models' diagnoses and selective repair (Table 2).

We also introduced a concrete CartPole rollout comparison (Figure 4) to illustrate successful vs. failed selective repair. The figure shows the submitted controller logic and execution trajectories side by side, making the distinction between task completion and selective repair much easier to see. We streamlined the main text, and documented the detailed prompts, CartPole evidence construction, and evaluation protocols in the Appendix (Appendices B.2 and C.2.1).

AuthorsOpenReview ↗

Part 3 of Response to Reviewer dLwz

Ambiguity clarification

Is it a guarantee that only one explanation is possible in your experiment setup?

No, our setup never assumes that the evidence singles out one explanation, and much of what we measure is how models handle evidence that leaves several open. Thank you for raising this distinction. Our original abstract's phrase ("the correct answer is known exactly") refers to what we know as evaluators: the true environment θ⋆\theta^\star and the exact set of environments Θ(e)\Theta(e) that the evidence allows, whereas the agent sees evidence that is often ambiguous.

Controller tasks:

Each task has three binary operating assumptions, giving ∣Θ∣=8|\Theta|=8 environments (§2; Appendix C.2.1). Across all 180 hidden-change controller tasks, the initial failure e1e_1 allows 3 or 4 of the 8 environments (§3.1, Table 2; Appendix A.1, Table 3): with one hidden change, 60 tasks allow three environments and 30 allow four; with two hidden changes, 30 allow three and 60 allow four. The true environment θ⋆\theta^\star is always in Θ(e1)\Theta(e_1), and every two-change task also allows at least one environment in which only a single operating assumption changed.

How our evaluation accounts for ambiguity:

  • Plausible hypotheses (Θ^\widehat{\Theta}). Before acting, the model states both its plausible hypotheses Θ^⊆Θ\widehat{\Theta}\subseteq\Theta (all environments it thinks could be true) and its leading hypothesis θ^∈Θ^\widehat{\theta}\in\widehat{\Theta}. With two hidden changes, models keep the true environment in Θ^\widehat{\Theta} in 80.0–97.8% of tasks (§3.1, Figure 1 left; Appendix A.5, Table 10 [Table 6 in the original submission]).

  • Chance baseline for the leading hypothesis (θ^\widehat{\theta}). Because the initial evidence allows 3 or 4 environments, we compare the rate at which θ^=θ⋆\widehat{\theta}=\theta^\star against a uniform random choice from Θ(e1)\Theta(e_1), which is correct with probability 1/∣Θ(e1)∣1/|\Theta(e_1)| averaged over tasks: 30.6% on one-change tasks and 27.8% on two-change tasks (§3.1, Table 2a). If the evidence identified a unique environment, this chance baseline would be 100%.

  • Multi-round feedback and active probing. As new execution results arrive, we update the set of environments the evidence allows. In the six-attempt feedback experiment (§3.1; Appendix A.5, Figure 8 and Table 12), an environment is ruled out only when a run so far would have produced a different execution result in it from the one returned. In the probing experiment (§3.5 [§3.3 in the original submission]; Appendix A.5, Table 18), models choose diagnostic observations specifically to narrow Θ(e)\Theta(e) when the initial failure leaves the true environment uncertain.

CartPole:

Our continuous-control tasks also include controlled ambiguity (Appendix B.2; Appendix C.2.1). Each task has three operating assumptions (pole length and mass, a constant motor-force bias, and an offset in the measured pole angle), each either unchanged or changed, giving eight environments. An environment is allowed by the evidence if running the simulator in it reproduces the displayed sensor readings (rounded to three decimal places) within a tolerance of about 0.002. Of the 12 changed CartPole tasks, the displayed trace allows one environment in 6 tasks, two in 4 tasks, and three in 2 tasks. When the change is hidden, the prompt explicitly instructs the model to infer every catalog entry compatible with the trace; the evidence may remain ambiguous.

Reviewer_68etOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how language-model agents understand evidence of system failures and decide what actions to take. The authors investigate whether models can correctly identify the cause of a failure, make appropriate changes based on their understanding, and judge whether the available evidence is reliable. The experimental results show that models often consider the correct explanation as one of the possible causes but struggle to identify it as the most likely one. Providing the correct explanation improves selective repair from 3.3% to 35.4%. However, even when models know what went wrong, they often make unnecessary changes or fail to properly fix the problem. The results also show that models are easily influenced by misleading test reports and rarely attempt to verify them independently. Overall, the paper suggests that evaluating agents based only on task completion is not enough, and that their understanding of the problem, the appropriateness of their actions, and their ability to verify evidence should also be evaluated.

Critical strengths

1- This paper addresses an interesting and challenging problem for language-model agents: how they interpret available evidence and decide what actions to take when an existing system encounters a problem or failure. Rather than focusing only on whether an agent successfully completes a task, the paper also examines how the agent makes its decisions and whether its actions are supported by the available evidence.

2- The proposed evaluation framework examines different aspects of agent behavior within a common setting. In particular, the idea of evaluating the agent's proposed explanations and analyzing how its actions change when an incorrect explanation is corrected is interesting. This helps identify whether the agent's failure comes from misunderstanding what has changed or from taking an inappropriate action, providing a better understanding of agent behavior beyond task-completion accuracy alone.

3- The paper evaluates several language models across different settings, including discrete controllers, continuous control systems, and Python program repair. The experiments also cover various conditions, such as hidden and disclosed changes, fabricated reports, and opportunities for agents to collect additional evidence.

Critical weaknesses

1- The paper reports that models select the correct leading hypothesis in only 5.8% of tasks involving two hidden changes, which is much lower than the random baseline of 27.8%. However, in this experimental setting, every task involves two changes, while the models are not explicitly informed about how many assumptions have changed. The authors also observe that models often prefer explanations involving only a single change. This behavior may reflect a preference for simpler explanations rather than a lack of ability to reason about the available evidence. Therefore, it would be useful to further investigate how the results change when the number of environmental changes varies or when models are informed about the possible number of changes. This would help better understand the reasons behind the low hypothesis-selection accuracy.

2- One of the interesting experiments in this paper is replacing a model's incorrect leading hypothesis with the correct one and examining how this affects its subsequent actions. The results show a clear improvement in selective repair, suggesting that the provided explanation influences the agent's decisions. However, this experiment mainly shows that models respond to the corrected hypothesis provided to them. It does not fully establish whether their originally stated hypotheses reflect the reasoning behind their actions. Therefore, I would recommend including additional experiments with alternative hypotheses or different forms of corrective feedback to better understand the relationship between the model's stated beliefs and its actions.

3- In the fabricated-report experiments, it is not clear what assumptions the models make about the reliability of the provided test reports. It would be useful to investigate how the results change when models are explicitly informed about the possibility of unreliable evidence and the cost of performing independent verification.

Questions for authors

Please address my all concerns raised in the weaknesses section.

5 comments
AuthorsOpenReview ↗

Part 1 of Response to Reviewer 68et

Thank you for the thoughtful review and concrete experiment suggestions. We ran all three and incorporated them into the revised paper.

... it would be useful to further investigate how the results change when the number of environmental changes varies or when models are informed about the possible number of changes. This would help better understand the reasons behind the low hypothesis-selection accuracy

Our experiments partly support this explanation. Telling models exactly how many operating assumptions changed substantially improves leading hypothesis accuracy, showing that a preference for simpler, single-change explanations accounts for much of the gap. However, even with the exact count disclosed, accuracy remains well below chance, and models still often select environments that the evidence rules out.

Number of changes. We evaluate 90 tasks with one change and 90 with two (§3.1, Table 2; Appendix A.1, Tables 3 and 4). With one hidden change and no count disclosed, the leading hypothesis is correct in 31.6% of tasks (95% CI 24.2–39.1), close to the 30.6% expected from a uniform random choice among the environments the evidence allows. With two hidden changes, it is correct in 5.8% of tasks, compared with 27.8% random baseline (Table R1). Note that if models consistently chose the simplest explanation that fit the evidence, they would be correct on 66.7% of one-change tasks; their lower accuracy shows that they do not reliably follow this rule.

Count disclosure. We reran all 180 hidden-change tasks with the same ten models and settings, adding either "CERTIFIED CHANGE COUNT: exactly N operating assumptions changed." or "CERTIFIED CHANGE RANGE: between 1 and 2 operating assumptions changed." to the prompt. The "No" rows in Tables R1 and R2 use the original results.

Table R1. Accuracy and selective repair by change-count disclosure:

Change count disclosedEnvironments allowed per task (number of tasks)Random choice correct (%)Leading hypothesis correct (%) ↑Selective repair (%) ↑
One change (90 tasks)
No3 (60), 4 (30)30.631.612.1
Range (1 or 2)3 (90)33.333.115.4
Exact count1 (30), 2 (60)66.746.626.3
Two changes (90 tasks)
No3 (30), 4 (60)27.85.86.4
Range (1 or 2)3 (90)33.311.07.6
Exact count1 (30), 2 (60)66.742.121.9

Table R2. Single-change preference and ruled-out leading hypotheses:

Change count disclosedLeading hypothesis is allowed by the evidence and assumes one change (%)Leading hypothesis is ruled out by the evidence (%) ↓
One change (90 tasks)
No44.142.7
Range (1 or 2)48.132.4
Exact count67.228.7
Two changes (90 tasks)
No51.328.2
Range (1 or 2)54.724.2
Exact count0.225.2

Table R3. Exact count disclosed:

ConditionLeading hypothesis is correct (%) ↑Leading hypothesis is ruled out by the evidence (%) ↓
One change (30 uniquely determined tasks)74.721.3
Two changes (30 uniquely determined tasks)47.047.3
  1. A preference for single-change hypotheses accounts for much of the gap. When told the exact count on two-change tasks, models stop choosing single-change hypotheses: single-change leading hypotheses allowed by the evidence fall from 51.3% to 0.2% (Table R2). Leading-hypothesis accuracy rises from 5.8% to 42.1%, improving for every model (from 0.0–11.1% to 31.1–51.1%), and selective repair more than triples (6.4% → 21.9%; Table R1).

  2. That preference does not explain the whole gap: even with the exact count disclosed, the best model reaches 51.1% accuracy on two-change tasks, below the 66.7% chance baseline. Models still choose leading hypotheses inconsistent with the evidence at nearly the same rate (25.2% vs. 28.2% without disclosure; Table R2). This failure also occurs with one change: accuracy stays below chance (46.6% versus 66.7%), and 28.7% of leading hypotheses contradict the evidence (Tables R1 and R2). Most strikingly, in 30 of 90 two-change tasks, the evidence and exact count identify one unique environment, yet models achieve only 47.0% accuracy and select ruled-out hypotheses in 47.3% of cases (Table R3). In 97.1% of those cases, models had already included the true environment among their plausible hypotheses.

  3. Disclosing only a range of one or two changes has little effect: on two-change tasks, accuracy rises from 5.8% to 11.0% (Table R1), while 54.7% of leading hypotheses still assume one change and fit the evidence (Table R2).

Together, these results identify two sources of low accuracy: a preference for single-change explanations and a failure to eliminate hypotheses contradicted by the evidence.

Changes to the paper. We have added these results to §3.1 (Table 2a-b) and Appendix A.1 (Tables 3 and 4).

AuthorsOpenReview ↗

Part 2 of Response to Reviewer 68et

Correcting a model's leading hypothesis improves selective repair, but does not fully establish whether its original hypothesis reflects the reasoning behind its actions. Additional experiments with alternative hypotheses or corrective feedback could clarify the relationship between the model's stated beliefs and its actions.

We ran a five-branch experiment to test if models' revisions follow their own stated hypotheses, and whether a different diagnosis redirects their actions.

Experiment. We use the same 180 hidden-change tasks and ten models as in §3.1. Each model first states its belief. When its leading hypothesis is wrong, we continue the conversation in five branches, each with a different message before the model revises the controller. The task, evidence and initial belief remain identical (exact prompts in Appendix A.2). We provide the model with:

  1. No correction: a message with no new information
  2. Own diagnosis restated: a diagnosis matching the model's (wrong) leading hypothesis
  3. Correct diagnosis: the true environment
  4. Wrong diagnosis: a diagnosis consistent with the evidence but different from both the true environment and the model's leading hypothesis
  5. Full oracle: the oracle message from §3.2, naming the certified true environment, its changed assumptions, and all environments consistent with the evidence

We run each revision in three environments: the true one, branch 4's wrong one, and the model's leading hypothesis. Unparseable revisions count as failures.

Table R4. Selective repair (%) by branch and evaluation environment:

BranchTrue θ⋆\theta^\starWrong alternativeOwn θ^\widehat{\theta}
1. No correction3.25.242.3
2. Own diagnosis restated1.92.061.8
3. Correct diagnosis42.21.74.4
4. Wrong diagnosis1.445.82.6
5. Full oracle40.62.84.3

Takeaways:

  1. Without correction, models tend to change the system to fit their own mistaken explanation (Branch 1). Selective repair occurs in 42.3% of tasks in the environment described by the model's incorrect leading hypothesis, compared to 5.2% in another incorrect - but evidence-consistent - environment (+37.1 pp, 95% CI 33.6 to 40.2) and 3.2% in θ⋆\theta^\star. Eight of ten models show this pattern (Table R5); the exceptions are Opus 4.7 (4.3% vs. 7.1%) and Opus 4.8 (3.6% vs. 5.1%). When Branch 2 restates their original diagnoses, selective repair in their hypothesized environments rises to 30.7% and 60.9%, respectively.

  2. Models tailor their changes to whichever environment the diagnosis identifies, whether it is correct or wrong (Branches 3 and 4). With the correct diagnosis, selective repair is 42.2% in θ⋆\theta^\star versus 1.7% in the wrong alternative (+40.5 pp, 95% CI 35.8 to 45.1). With the wrong diagnosis, the pattern reverses: 45.8% in the wrong alternative versus 1.4% in θ⋆\theta^\star (+44.4 pp, 95% CI 40.1 to 49.0). Both effects hold for every model (Table R5).

  3. The model's own wrong diagnosis, when supplied back to it, pushes its revisions further toward the wrong environment (Branch 2). Selective repair for that hypothesis rises from 42.3% to 61.8%, while task completion in the true environment falls from 43.6% to 25.5%. Selective repair in the true environment stays rare for every model: at most 7.6%, and 1.9% on average, down from 3.2%.

  4. The full oracle adds little beyond the correct diagnosis (Branches 3 and 5). Branch 5 achieves 40.6% selective repair vs. 42.2% for Branch 3 (−1.6 pp, 95% CI −3.7 to +0.5). Relative to no correction, Branch 5 raises it from 3.2% to 40.6% (+37.3 pp), consistent with the original oracle result of 3.3% to 35.4% (§3.2, Figure 2).

Table R5. Selective repair by model:

ModelTasks1: Own θ^\widehat{\theta}1: Wrong alt.2: Own θ^\widehat{\theta}3: True θ⋆\theta^\star3: Wrong alt.4: Wrong alt.4: True θ⋆\theta^\star5: True θ⋆\theta^\star
Opus 4.614626.06.245.224.02.130.84.820.5
Opus 4.71404.37.130.725.70.730.71.415.0
Opus 4.81383.65.160.933.32.927.50.729.7
Opus 58017.510.068.840.00.052.50.043.8
Gemini 3.1 Pro15831.68.946.256.32.558.90.662.7
GPT 5.215112.66.636.411.92.617.91.311.3
GPT 5.515672.45.169.939.13.841.01.939.1
GPT 5.6 Sol14472.91.472.247.20.047.90.742.4
GPT 6 Astra133100.00.0100.079.70.880.50.880.5
Grok 4.615481.81.987.764.91.970.11.961.0
Average14042.35.261.842.21.745.81.440.6

Changes to the paper. We added this experiment to §3.2 (Table 1) and Appendix A.2 (Table 5).

AuthorsOpenReview ↗

Part 3 of Response to Reviewer 68et

3. "In the fabricated-report experiments, it is not clear what assumptions the models make about the reliability of the provided test reports. It would be useful to investigate how the results change when models are explicitly informed about the possibility of unreliable evidence and the cost of performing independent verification."

We ran the experiment you suggested: warning models that the test suite had produced inconsistent results on previous re-runs did not increase verification of correct programs. Telling models how long a test run takes reduced verification further and left fewer working programs unchanged. The original prompt (Appendix A.4) presented the report as authenticated evidence from the complete certified test suite, giving models no explicit reason to doubt it.

We repeated the optional verification experiment across the same Python programs, eight models and settings, comparing three prompt versions:

  • Version A (baseline): the paper's optional verification prompt, unchanged
  • Version B (run history): adds "Run history: this suite's result changed on re-run in 9 of its last 30 runs."
  • Version C (run history and duration): adds both the run-history line and "Suite duration (last run): 18 min."

Under each version, every model saw each of the 40 programs twice: once with correct code and a fabricated failure report, and once with faulty code and a fabricated passing report. The model could request one fresh run of the test suite (VERIFY), leave the program unchanged (NO_CHANGE), or submit a replacement program (EDIT).

Table R7. Effect of adding reliability (B) and duration (C) cues, averaged over the eight models (%):

OutcomeA: baselineB: + historyC: + history & durationB − A [CI]C − A [CI]
Correct program, false FAIL
↳ Requests a test ↑20.616.612.8−4.1 [−9.1, +0.6]−7.8 [−12.5, −3.8]
↳ Leaves it unchanged ↑22.519.415.3−3.1 [−7.5, +0.9]−7.2 [−10.9, −3.8]
↳ Invalid format ↓17.815.016.6−2.8 [−5.3, −0.3]−1.2 [−4.7, +1.9]
Faulty program, false PASS
↳ Requests a test ↑0.64.70.9+4.1 [+1.6, +7.2]+0.3 [−0.6, +1.6]
↳ Passing repair ↑70.977.275.0+6.2 [+1.9, +10.9]+4.1 [+0.0, +8.8]
↳ Invalid format ↓4.45.35.9+0.9 [−1.9, +3.8]+1.6 [−1.6, +4.7]

Takeaways:

  1. When correct programs were falsely reported to fail, warning models that the test suite had changed its result on re-run in 9 of its last 30 runs did not increase verification: requests fell from 20.6% (A) to 16.6% (B). Adding the 18-minute test duration reduced verification further to 12.8% (C), while the share of correct programs left unchanged fell from 22.5% to 15.3%. Yet models that verified almost always made the right decision: the six models that requested verification left correct programs unchanged in 96.5-100% of cases, compared with only 0.0-7.3% when they acted without verifying.

  2. Instead of verifying more, models edited working programs more often: 59.7% under A, 65.6% under B, and 68.1% under C. Their reasoning traces reveal that several models interpreted the reported test flakiness as evidence of defects in the programs themselves (Appendix A.4, Figure 7).

  3. When we falsely reported that faulty programs passed, models rarely requested verification. In the original condition (A), six out of eight models never requested a test, and the average verification rate was only 0.6%, matching our earlier findings (Appendix A.5, Table 17). Telling models that the test suite had produced inconsistent results increased verification to 4.7%, mainly because GPT 6 Astra and Grok 4.6 requested more tests. Adding the 18-minute test duration note reduced verification to 0.9%. Models instead became more likely to edit the faulty programs despite the fabricated passing reports: edit rates rose from 82.8% (A) to 89.7% (B) and 86.6% (C). Because the programs actually contained faults, more edits also led to more successful repairs: 70.9% under A, 77.2% under B, and 75.0% under C.

  4. Even when we warned models that test reports could be unreliable, they did not check reported failures on correct programs more often. These findings reinforce the practical takeaway from §3.4: models chose the appropriate action in 92.5% of cases when given a fresh test result automatically, versus only 15.3% when they had to decide whether to request one themselves.

Changes to the paper. We have added this experiment, the exact prompts, examples from the models' reasoning traces, and the aggregate and per-model results to §3.4 and Appendix A.4 (Tables 8 and 9, and Figure 7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers and ACs: Summary of paper revisions and new experiments (17,000+ new episodes)

We thank the reviewers for their detailed feedback and concrete suggestions. This note summarizes (1) the main changes we made to the paper and (2) our new experiments, now in the revised PDF.

1. Changes to the paper

  • Introduction and motivation (§1). Organized around three research questions (RQ1–RQ3), grounded in coding, operational, and physical control (dLwz, Part 1, Part 2).
  • Terminology and framework (§2–§3). One consistent vocabulary separates properties of the system (operating assumptions; original environment θ(0)\theta^{(0)}; true environment θ⋆\theta^\star; environments the evidence allows), the model's stated belief (plausible hypotheses Θ^\widehat{\Theta}; leading hypothesis θ^\widehat{\theta}), and the executed action (revision; selective repair). The abstract, §1, and §2 now state that the agent sees evidence consistent with multiple environments (dLwz, Part 3), and §3.1 quantifies how many the initial evidence leaves open across tasks.
  • New main-text results. Table 1 (§3.2, five diagnosis branches), Table 2 (§3.1, disclosing the number of changes) and Figure 4 (§3.3, CartPole selective repair vs. original-environment regression).
  • New appendix material. Sections A.1–A.4 with aggregate and per-model results (Tables 3–9), the CartPole output budget table (A.5, Table 15), reasoning excerpts in which models blame reported test flakiness on the program (Figure 7), and expanded descriptions of how the tasks and their ambiguity are constructed (B.2, C.2).

2. New experiments

Experiment (response)ModelsNew episodesModel callsTokens in / out
1. Number of changes disclosed (68et, Part 1)103,6007,20010.2M / 9.4M
2. Five diagnosis branches (68et, Part 2)101,800 diagnoses + ~6,500 branch revisions8,93514.9M / 18.1M
3. Reliability and duration cues (68et, Part 3; jJoy, Part 2)81,9202,1231.3M / 7.5M
4. Covering every plausible hypothesis (jJoy, Part 1)101,8003,6005.0M / 6.9M
5. CartPole output budgets (jJoy, Part 3)53607300.8M / 8.3M
Total~16,00022,58832.2M / 50.2M
  1. Disclosing the number of changed assumptions. On two-change tasks, single-change leading hypotheses fall from 51.3% to 0.2% and accuracy rises from 5.8% to 42.1%, still below the 66.7% chance level. The evidence still rules out 25.2% of leading hypotheses, and 47.3% on the tasks where the evidence and the count already allow only one environment.
  2. Five diagnosis branches. Without correction, revisions selectively repair the model's own wrong leading hypothesis in 42.3% of tasks, against 5.2% for an alternative the evidence allows and 3.2% for the true environment. A wrong diagnosis allowed by the evidence redirects selective repair as strongly as the correct one (45.8% vs 42.2%).
  3. Reliability and duration cues. On working code falsely reported to fail, warning that the test suite is unreliable, and then that it is costly, reduced verification from 20.6% to 16.6% and 12.8%, while raising unneeded edits from 59.7% to 65.6% and 68.1%, as models mistook test flakiness for a code bug (Figure 7).
  4. Covering every plausible hypothesis. Asking revisions to work in every plausible hypothesis (not just the leading one) raises task completion in both the true and original environments from 48.6% to 66.1%, but 97% (488/502) of the newly successful revisions ignore all operating assumptions, and selective repair falls from 9.3% to 1.7%.
  5. CartPole output budgets. At 65,536 tokens no response is cut off and 32 of 48 revisions run (22 at 16,384 tokens in the same rerun), yet only 43.8% of runnable revisions achieve selective repair (46.9% miss the goals in θ⋆\theta^\star and 9.4% fail in θ(0)\theta^{(0)}).
  6. Explicit warnings (1,280 new episodes; see our reply to 68et). Following 68et's follow-up, we explicitly told models that the report "may be wrong or unreliable", alone or with a note that each check counts against a budget. Verification of correct programs rose only from 20.6% to 27.2% (falling to 16.6% with the budget warning), and on faulty code reported as passing it stayed near zero (1.2% and 0.3%).

We invite the reviewers and Area Chairs to read the revised PDF alongside our detailed responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.