acceptodds
Under review as a conference paper at ICLR 2027

Audit Without Verification: When LLM Accountability Layers Relay Rather Than Check

Abstract

When a multi-agent pipeline fails, an LLM auditor is asked which step let the fault in. It reads the agents' reports, whose suspected-origin field can state a conclusion ("suspected origin: step 3") beside their observations. Holding the case evidence fixed, we intervene on that conclusion alone (chain agents gpt-4.1-mini and claude-haiku-4-5, auditor qwen3-30b-a3b-instruct). The auditor relays the conclusion rather than checking it. In a registered single-run re-measurement on a pinned provider, with the reports, the conclusion and the documentation each step received all in its prompt, it finds the true origin in only 4.0% (GPT) and 2.9% (Claude) of the episodes where the agents' proposal is wrong, against 59.7% and 67.5% from the same documentation alone: the evidence goes unused on GPT chains, largely unused on Claude chains. From the reports alone, deleting only the conclusion raises accuracy where the proposal is wrong by 38.3 pp (GPT) and 24.0 pp (Claude), a gain that also appears with gpt-5, claude-sonnet-5 and gemini-2.5-flash-lite as auditors and in a second domain. Removal has a price: where the conclusion is right it costs 18.3 and 10.5 pp, less consistently across auditors, both directions holding after Holm correction whether or not naming the step just before the origin is scored correct. Without the conclusion the auditor still trails the documentation alone (post hoc). Whoever writes the field steers the audit (pre-specified). A random wrong step planted in it is adopted in 81.9% (GPT) and 64.1% (Claude) of the episodes tested, nearly as often as the agents' own proposals (0.83 of their gain over the no-field rate), and causally impossible plants still in 19.4% and 29.2% (descriptive). An instruction not to use the field removes only 24% (GPT) and 37% (Claude) of the auditor's dependence on it and costs 7 to 9 pp where the proposal is right. The mechanism reading, by a rule frozen before collection, is mixed, with a measurable numeric-anchoring component. Whether deletion pays depends on how the origin is scored, and our pre-registered responsibility-framing hypothesis is not supported.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, multi-agent systems, failure attribution, AI auditing, accountability, interface design, report schema, pre-registered study, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1nfVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an LLM auditor that attributes failures in a multi-agent pipeline actually checks the evidence or simply repeats the agents’ own conclusions. Using a synthetic six-step supply-chain pipeline with injected defects, the authors hold the case evidence fixed and intervene only on the “suspected origin” field of the agents’ reports: deleting it, replacing it with a random wrong step, or supplying the raw documentation alongside it. The results show that the auditor mostly relays the agents’ conclusions rather than verifying them.

Critical strengths
  1. This paper studies a very interesting phenomenon. It could inspire further research.
  2. The experimental design is clear. Only the single phrase "Suspected origin" in the report was modified, while the rest of the content remained identical. Thus, the effect could be clearly attributed to that specific phrase.
Critical weaknesses
  1. It only uses Qwen3-30B model to do the planted-conclusion experiment, the “do not use the field” instruction and the mechanism analysis. The more powerful models (gpt-5, claude-sonnet-5) were tested only on field deletion.
  2. The paper does not test on any real multi-agent systems, so it is unclear whether the conclusion still holds outside this constructed six-step supply chain pipeline.
  3. LLMs are likely to repeat conclusions supplied to them is well studied in work on sycophancy and LLM-judge biases, and the authors state that they claim no new bias. The contribution is mainly a careful measurement in one specific setting. So the novelty of this work is limited.
Questions for authors
  1. Since the cases are regenerable from a seed, why was the registered re-measurement not conducted on newly generated base cases?
  2. Does the phenomenon appear on existing failure-attribution benchmarks or real multi-agent systems in which agents report their own guesses?
  3. Can you test more powerful models like gpt6-astra, opus-5.5?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1nfV

Thank you. You wrote that the phenomenon is very interesting and that the design is clear. The revision adds the evidence that your review asked for. Each addition is now in the abstract, in Box 2 of the introduction and in Section 4:

| your point | what the revision adds | ||| | planted, instruction and mechanism arms on Qwen only | the same arms on claude-sonnet-5: it follows planted steps (0.79 and 0.61 of the gain that its own proposals produce) | | stronger models | claude-opus-5-5: it is also pulled toward the conclusion | | real multi-agent systems | the planted test on the 184 public Who&When logs: the auditors name the planted step more often (+32.1 pp qwen, +11.4 pp claude-sonnet-5) | | fresh base cases | 150 newly generated base cases from the same generator, frozen before the first call: the gain holds (+28.0 and +31.5 pp) | | novelty | a deletion gain does not show checking, and deletion has a price (below) |

Novelty compared with sycophancy and judge-bias work. It is known that models defer to an answer that they are given. We build on this. The first new result is that a deletion gain does not tell if an auditor checks. A deletion gain is the usual proof that a cue matters. On 150 new base cases in one execution, we cannot tell apart the gain of the primary auditor and the gain of claude-sonnet-5 (+28.0 and +31.5 pp). But with the documentation in the prompt, the first finds the origin in 7.0% of episodes and the second in 97.5%. The second new result is the price of deletion. The audited agents write the conclusion, so it is often right. In that same execution, deletion costs the primary auditor 32.4 pp where the conclusion is right. To show both results, you need a bench where only the agents' conclusion changes and the observations stay byte-identical. You also need to compare the result with a free rule and with the evidence alone. Who&When and Who&When Pro do not offer this. Our bench is released.

Stronger models. We ran claude-opus-5-5 on the 400 Table 3 episodes where the agents' proposal is wrong. It finds the origin in 11.5% (GPT chains) and 35.5% (Claude chains) with the field, and in 27.5% and 42.5% without it (+16.0 [+10.7, +21.8] and +7.0 [+1.5, +12.7] pp). So it is also pulled toward the conclusion. The planted, instruction and relabelling arms now also run on claude-sonnet-5. It adopts a random wrong step planted in the field in 73.5% of GPT-chain episodes and 21.0% of Claude-chain episodes (14.0% and 2.0% without the field). This is 0.79 and 0.61 of the gain that the agents' own proposals produce. The instruction to ignore the field removes 78% and 73% of its dependence. With the documentation in its prompt, claude-sonnet-5 uses it. So the paper now says which auditors leave the evidence unused (see our reply to Ba7S).

Fresh base cases. The registered run answers a narrower question: does the effect survive auditor randomness, changes in the served model, and analysis choices, on the cases that suggested it? Its registration says this before the run ("a re-measurement on the same corpus, not an out-of-sample replication"). New base cases are the stronger test. The seeded generator made them possible, so we ran them: 150 newly generated base cases, none identical to one of the 480 (Appendix Z10 gives the overlap check), GPT chains, protocol frozen before the first call. In one execution on the same 200 wrong-proposal episodes, deleting the field gains +28.0 [+19.3, +37.3] pp for the primary auditor and +31.5 [+22.5, +41.0] pp for claude-sonnet-5. The effect holds out of sample.

Real systems and existing benchmarks. We ran the planted test on the 184 public logs of Who&When. We froze the protocol and the analysis before the first call (Appendix Z9). We add one line, Suspected origin: step N., to each real log. N is a random wrong step. Compared with the same log without the line, the auditor names that step more often: 38.6% against 6.5% for qwen (+32.1 [+25.0, +39.1] pp), and 16.8% against 5.4% for claude-sonnet-5 (+11.4 [+7.1, +16.3] pp). So a stated origin also pulls the auditor on real multi-agent logs. This result is now in the abstract and in Section 4.5. Deletion cannot be tested on existing benchmarks, because none has a field where an agent names the origin. Who&When and Who&When Pro do not have one. The Magentic-One orchestrator writes "root cause" passages in 22 of the 58 hand-crafted logs, but these passages never give a step. Our bench fills this gap.

I appreciate the author's responses, but my assessment remains unchanged.

The model tends to follow the conclusion in the input is a well-known phenomenon, and many works on sycophancy, anchoring and LLM-judge have shown it. Even the paper itself claims no new bias. The two new points in the response follow from this. A deletion gain alone cannot show whether an auditor checks. And removing a conclusion that is usually right will cost accuracy when it is right. These points are expected, not new findings. The new results in the response also demonstrate that a stronger auditor uses the documentation (97.5%). This makes the remaining effect more like a known phenomenon that "weaker models tend to follow a given answer instead of thinking about it themselves." I have raised my score to 2 because the experiments are thorough. But the work is still a careful measurement of a known behaviour.

AuthorsOpenReview ↗

Thanks for raising score

Thanks for reading our replies and for raising your score.

One short point on the idea that only weaker models follow the conclusion. From the reports alone, the strong models are pulled too. On the 150 new cases, in a single run, deleting the conclusion raised accuracy by about the same amount for our primary auditor and for Claude Sonnet 5, by 28.0 and 31.5 points. Claude Sonnet 5 also picks a planted wrong step in 73.5% of episodes. What the stronger model adds is that it uses the documentation when the documentation is in its prompt. So the pull from the reports is not limited to weak models.

Thank you again for your time.

Reviewer_mERrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an LLM auditor is independently identifying the origin of a failure, or just following the conclusions stated in the auditing reports. The authors use synthetic supply-chain and software-delivery processes, the authors remove, relabel, or replace the reports’ suspected-origin fields while keeping the remaining evidence fixed. The auditor strongly follows these fields, including incorrect conclusions, even when supporting documentation is available. Removing the field improves accuracy when the agents’ conclusions are wrong, but can reduce accuracy when they are correct. The paper also examines stronger auditors, alternative scoring conventions, and several report modifications.

Critical strengths

S1. The controlled intervention provides useful evidence of reliance on upstream conclusions. The design of removing only the suspected-origin clause while preserving the other report content builds a clear experimental comparison. The documentation-only reference is also particularly useful, as it shows that the auditor can perform substantially better with the available documentation, yet performs poorly when misleading reports are added. This is a stronger evidence than simply observing agreement between agents and an auditor.

S2. The empirical investigation is extensive, and the reporting is transparent. The authors have examined multiple auditors, two domains, planted conclusions, alternative labels, and different scoring conventions. They have also clearly reported the unsupported original hypothesis, deviations from the registration, and limitations of the follow-up analyses. These efforts strengthen the confidence in the observed effect under the tested setting.

Critical weaknesses

W1. The additional insight beyond existing work offered by the paper is limited. Prior studies[1,2] have already shown that LLM judgments can be influenced by supplied answers, evaluation metadata, and misleading contextual information. The authors acknowledge these connections and do not claim a new bias. Their main distinction is that the conclusions are produced by the agents being audited, which allows them to measure both the benefit of correct advice and the harm of incorrect advice. This is a useful extension, but I am not sufficiently convinced that it provides a substantial advance in understanding automated auditing. The extensive experiments establish the effect across several conditions, while the broader lesson remains close to the familiar problem of relying on an imperfect upstream judgment.

W2. The broader auditing interpretation is constrained by the experimental construction. Both domains use the same six-step reporting structure, and the agents inspect and report rather than take actions whose consequences propagate through a working system. The second domain therefore broadens the subject matter more than the interaction structure. The scoring convention also materially affects the practical conclusion, that accepting the preceding step as a valid origin reverses the net benefit of deletion on both chains. In addition, the strongest documentation comparisons and mechanism probes rely on the primary auditor, while the paper does not separate misleading report content from distraction caused by a longer prompt. These limitations do not invalidate the field’s influence, however, leaving the evidence for a general failure of verification less conclusive.

W3. The writing of the paper is difficult to follow. Many sentences combine a result, several qualifications, and multiple section references. Frequent parenthetical labels and repeated summaries also interrupt the argument. A more focused account of the main experiment and its implications would make the contribution much easier to assess.

[1] Kapetanovic et al. (2026), “Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence.”

[2] Alexopoulos et al. (2026), “Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review.”

Questions for authors

Q1. Relative to the contextual-bias and judge-anchoring studies discussed in the paper, what is the main new scientific conclusion beyond demonstrating the effect with agent-generated reports? A clearer explanation of this distinction would materially help me assess the contribution.

Q2. Is there evidence from the existing experiments that separates the effect of misleading conclusions from distraction by additional report text? I recognize that deleting the field isolates its influence, but the stronger claim on "available evidence goes unused" seems to require a separate justification.

Q3. How should the intended auditing task determine whether the injection step or the preceding step is the correct attribution? Since this choice changes the net benefit of deletion, I would appreciate a clearer justification of the primary convention and which practical recommendations remain valid under both.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mERr

Q1 / W1. What is new. There are two new conclusions. First, a deletion gain does not show if an auditor checks. Second, when you remove a conclusion that the audited agents wrote, there is a price, and we measure it. Both need the design that this paper adds: only that conclusion changes, and the observations stay byte-identical.

We ran one execution on 150 newly generated base cases, with the protocol frozen before the first call (the same 200 wrong-proposal episodes, GPT chains, Appendix Z10). Deletion gains +28.0 pp for the primary auditor and +31.5 pp for claude-sonnet-5. We do not detect a difference (−3.5 [−10.9, +3.9]). With the documentation in the prompt, the first auditor reaches 7.0%, against 52.0% with the documentation alone. The second reaches 97.5%, against 99.0%. These two gaps differ detectably (−43.5 [−56.2, −31.2]). So two auditors with gains that we cannot tell apart differ by 90 points in accuracy when the documentation is in the prompt. On the 306 right-proposal episodes of that execution, deletion costs the primary auditor 32.4 pp.

The two studies that you cite measure how a cue written by the experimenter changes a judgment. They do not measure what replaces the cue, if the evidence is used apart from the cue, or what it costs to remove a correct cue. Closer work separates correctness from propagation in adversarial debates (Anghel et al., 2026), traces the per-class accuracies of judges to one propensity (Scavotto, 2026), and changes the evidence shown while the run stays fixed (Zhu and Pu, 2026). None of them changes only the conclusion of the audited agents. The introduction and Section 2 now say this.

Q2 / W2. Content or a longer prompt. For the field itself, length cannot explain the effect. Deleting the field removes 52 characters, 2.1% of the median prompt, and moves accuracy by 38.3 and 24.0 pp. A planted wrong step has exactly the length of the agents' own proposal, and the auditor adopts it in 81.9% and 64.1% of episodes. For the stronger claim, that the reports reduce the use of the documentation, we ran the control that you ask for. We froze its protocol first. We used the reports-plus-documentation prompt and replaced each report slot with neutral administrative sentences of the same length (within 10% per slot). The layout did not change. Against the documentation alone, the real reports without the field lower accuracy by 12.1 [7.6, 16.8] pp (GPT) and 39.5 [32.1, 46.5] pp (Claude). The filler does not lower it (−7.4 [−11.1, −3.5], an increase, and −4.2 [−8.7, +0.1], not detected). By the rule fixed in advance, length and layout do not explain the gap on either chain (Appendix Z6). The control leaves one question open: distraction by text that reads like a report.

Q3. Which convention. We append the defect text to the work product just before step t is called. So t is the first step whose input contains the defect, and step t−1 never sees it. For this reason the strict convention ("strict scoring" in the revised body) is primary for the question that the auditor gets: where did the defect enter? The lenient convention fits a different question: which was the last step that could have stopped it? It also doubles the chance rate (16.7% to 33.3%). Appendix U recomputes every verdict under both conventions. These results hold under both: the gain from deleting the field where the proposal is wrong (Appendix U, Appendix T episodes: +36.0 / +42.2 pp GPT, +22.4 / +6.0 pp Claude), the adoption of planted wrong steps, and, for the primary auditor, the non-use of the documentation and the weak effect of an instruction to ignore the field. One result depends on the convention: the net benefit of deletion. It is positive on GPT chains under the strict convention (margin +29.9 [+16.9, +46.3] pp, against −5.3 [−14.3, +3.9] lenient). On Claude chains the margin is −7.3 [−18.4, +4.6] strict and −63.5 [−82.3, −47.1] lenient. So the recommendations of the paper do not depend on the convention: treat an agent's stated origin as a claim to check against the evidence, and measure an instruction to ignore the field on your own auditor before you rely on it. Section 5 tells operators to fix the convention before they decide to delete the field.

W2. Structure. The second domain has the same six-step structure, as Section 6 says. The revision goes further in two directions: 150 new base cases, and the 184 real logs of Who&When, where both auditors name a wrong step more often when we add a line naming it (Appendices Z9, Z10).

W3. Writing. We rewrote the body. The introduction shows one real episode. One box defines the basic terms. A second box gives each finding with one number and its status. The text gives the numbers for one chain, the tables give both, and the text says where the two chains differ. Each results paragraph makes one point. The provenance labels moved out of the headings and sentences, into Box 2 and one status line per results subsection.

Thanks for taking time on the rebuttal, which substantially addresses my concerns through the matched-length control, clearer justification of the scoring convention, and additional experiments with stronger auditors and real logs. Although distraction from report-like content remains unresolved to me, the current evidence still suggests a solid empirical contribution, particularly in showing that similar deletion gains can coexist with very different use of supporting evidence. While the novelty and generality remain somewhat limited, in all, I would like to increase my score from weak rejection to weak acceptance.

Thank you for reading our responses so carefully and for raising your score. We are glad the new experiments helped. Thank you again for your time.

Reviewer_Ba7SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether an LLM auditor that names the step at which a defect entered a six-step synthetic pipeline verifies or relays the suspected-origin conclusion in the agents' filed reports, by deleting that clause at fixed case evidence, supplying the per-step documentation beside the reports, planting wrong steps and instructing the auditor to ignore the field.

Critical strengths

Two critical strengths are listed below.

  1. The single-clause manipulation changes only the suspected-origin clause and keeps the free text, documentation, auditor, schema and episode set byte-identical, and Appendix V ablates each of the three report clauses singly on the same 4,000 episodes (the escalation flag is ablated separately in Appendix X, where the measurement failed its validity gate). Only removing the suspected-origin clause raises accuracy where the proposal is wrong, while removing the free text lowers it in all four cells. This isolates the conclusion clause as the carrier rather than the report format as a whole, and the cost where the proposal is right is reported beside the gain.

  2. Appendix A and Table Y2 place the documentation in the same prompt as the reports. In the registered execution (Table Y2), with the field kept, accuracy stays at 4.0% and 2.9% while the same documentation alone gives 59.7% and 67.5%, against an equivalence margin fixed in advance; Appendix A gives 4.9% and 4.0% with the field in a separate pre-specified run. This separates an information deficit from non-use of the information, which is the distinction the title rests on. The provenance table (Table O1), the failed gates and the measured drift are reported rather than omitted, and the released per-episode file reproduces Table B5 exactly.

Critical weaknesses

Three critical weaknesses are listed below.

  1. The deletion gain is not decomposed by the correctness of the first-escalator heuristic, which the environment makes correct whenever step tt itself escalates, since no earlier step ever escalates on a defective episode and no model output is fed forward. From the released per-episode file, on the GPT-chain wrong-proposal stratum of Table B5, deleting the field raises accuracy from 8.4% to 96.3% on the 2,647 episodes whose first escalating step is the true origin and from 0.5% to 3.2% on the 3,217 episodes where it is not, so 96% of the gain comes from the first group (93% on Claude chains). Without the field the auditor names the first escalating step in over 90% of its answers on GPT chains and in 58 to 70% on Claude chains, and that step is the true origin in only 45.1% and 57.2% of the wrong-proposal stratum. Table 2, Table B4, Figure 1, Contribution (2) and §6 report the rule beside the field-free arm, but the abstract presents the +38.3 and +24.0 pp as recovered accuracy without mentioning the rule, neither the abstract nor the body stratifies the gain by whether the first escalating step is the true origin, and the second domain is built by the same generator (Appendix E), so it does not test pipelines in which faults propagate and are detected downstream.

  2. The claims that the evidence goes unused, that planted claims are adopted and that instructions do not help rest on one mid-tier auditor. The documentation-in-context, mechanism, planted, instructed, label and ablation arms were all run with qwen3-30b-a3b-instruct, a model the paper itself screened out of the chain-agent role for escalating on 3.7% of defective links (Table I1) and whose refusal rate on identical prompts moved by 41 pp in one day (Appendix X). The frontier check (Table 3) covers only the reports-only deletion contrast on 200 episodes per cell, and there gpt-5 gains +23.1 and +8.5 pp and answers close to uniformly without the field (Appendix D), while claude-sonnet-5 reaches 46.0% with the field. The title and abstract generalise to LLM accountability layers what has been established for one model.

  3. The confirmatory status and reproducibility are weaker than the abstract conveys. The only hypothesis in the original pre-registration is null, every headline contrast began as an exploratory analysis, and the registered re-measurement of Appendix Y was frozen on 2026-09-24, after the effect had been found, on the same 480 base cases. The supplement contains protocols, harnesses and derived data for Appendices A, Q, R, S, T, V, X and Y, but not the corpus generator (the anonpkg package imported by 26 released scripts), the raw responses, the fitting code for the GEE analysis (the released refit_independence.py imports anonpkg.stats and reads unreleased phase-2 files), or any data for the Gemini replication, the frontier auditors or the second domain, so Table 3 and Appendices D and E cannot be checked from the submission, although the Reproducibility Statement says that the frozen protocols and hashes for these replications are released.

The reporting is unusually transparent and the first two weaknesses can each be addressed with one analysis or one additional collection. I would be glad to raise my rating to 3 if the authors report the deletion contrast stratified by whether the first escalating step is the true origin and restate Contribution (2) accordingly (Question 1), and run the reports-plus-documentation arms on at least one frontier auditor (Question 2).

Questions for authors

Five questions are listed below. These questions may influence my understanding of the method and the experiment results.

  1. On the wrong-proposal stratum, what is the deletion effect on episodes where the first escalating step is not the true origin, in the registered execution of Appendix Y and in the second domain? From the released per-episode file I obtain +2.8 pp on GPT chains and +4.2 pp on Claude chains for the Table B5 collection.

  2. Can the reports-plus-documentation arms, with and without the field, be run with gpt-5 or claude-sonnet-5 on the episodes of Table 3? Does the reading "present and unused" hold for an auditor that reaches 46.0% with the field?

  3. The documentation-alone figures use the asserting prompt, which states that a defect occurred at exactly one step. What is documentation-alone accuracy under the neutral prompt with declines scored as errors?

  4. Would the cost where the proposal is right persist if the omission defect text did not state that the missing record is held at the originating site? In the second domain, what fraction of wrong proposals name t−1t-1 and what accuracy does the first-escalator rule reach?

  5. Will the corpus generator, the raw responses and the data behind Table 3, Appendices D and E and the GEE analysis be released? When was the OSF registration of Appendix Y filed relative to the Table B5 collection?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ba7S (1/2): the split by first escalating step, and the documentation arms on a frontier auditor

Thank you for your review. We could act on each point. We did the two analyses that you asked for, and both are now in the paper.

Q1 / W1. Split by the first escalating step. We get your Table B5 numbers exactly (8.4% to 96.3% on 2,647 episodes, 0.5% to 3.2% on 3,217, 96% and 93% of the gain). Here is the same split, with the frozen weights and bootstrap:

| collection | chain | first escalator = origin | first escalator ≠ origin | ||||| | Table B5 | GPT | +87.9 [+84.7, +90.6] | +2.8 [+1.9, +3.7] | | Table B5 | Claude | +43.5 [+39.5, +47.4] | +4.2 [+2.7, +5.9] | | registered run (App. Y) | GPT | +78.1 [+74.0, +81.7] | +3.8 [+2.5, +5.2] | | registered run (App. Y) | Claude | +38.4 [+34.2, +42.6] | +4.3 [+2.7, +6.1] | | second domain (App. E) | GPT | +83.0 [+77.7, +87.6] | +11.8 [+6.6, +17.2] | | second domain (App. E) | Claude | +81.2 [+73.8, +87.8] | 28 episodes, 0% in both arms |

Your split is now the second finding of the paper. In the registered run, the auditor without the field names the first escalator in 88.5% (GPT) and 55.5% (Claude) of wrong-proposal episodes. That step is the origin in only 46.4% and 57.8% of them. Box 2 in the introduction now reads "Deleting the field substitutes a positional default, at a price". It gives the gain where the first step to escalate is not the origin (+3.8 pp, post hoc) next to the overall gain (+38.3 pp). Section 4.3 says: "The gain in accuracy comes almost entirely from episodes where that step is the origin. Elsewhere it is only +3.8 pp [+2.5, +5.2], and +4.3 pp [+2.7, +6.1] on Claude chains." The abstract says that most of the rise comes from that shortcut.

The split also separates the auditors. The two numbers below are the gain where the first escalator is the origin, and the gain where it is not (post hoc, 200 episodes per cell). In the Table 3 run: claude-sonnet-5 +81.6 / +6.2 (GPT) and +45.7 / +13.7 (Claude), gpt-5 +28.7 / +19.5 and +6.7 / +10.5. In a separate execution: claude-opus-5-5 +18.4 / +14.2 and +14.3 / −1.1. The two gains differ detectably for sonnet-5 on both chains and for opus-5.5 on Claude chains (Table Z1).

The split also supports the main reading of the paper. Where the first escalator is not the origin, the documentation alone finds the origin in 47.4% (GPT) and 55.9% (Claude) of the registered-run episodes. With reports, documentation and field, the auditor finds it in 0.8% and 1.7%. The evidence is unused exactly where the shortcut cannot help.

On propagation: both domains come from the same generator. Section 6 says that no fault propagates through model outputs. The Who&When test adds real logs with a different structure (Appendix Z9).

Q2 / W2. Documentation arms on a frontier auditor. We ran them with claude-sonnet-5 on the 800 Table 3 episodes. All five arms ran in one batch. We froze the protocol first (it is in the revised supplement). Accuracy where the proposal is wrong:

| chain | reports, field | reports, deleted | documentation | reports + doc., field | reports + doc., deleted | ||||||| | GPT | 10.5 | 53.0 | 100.0 | 97.0 | 96.5 | | Claude | 40.0 | 68.5 | 99.5 | 88.0 | 90.0 |

(Execution of 6 November. In the Table 3 run, the reports with the field give 13.5% and 46.0%.)

claude-sonnet-5 uses the documentation. So "present and unused" is true for the mid-tier auditor, and the abstract and Section 4.3 now say this. The failure does not go away. It now has a clear place. With the reports alone, the auditor has nothing to check against, and the same frontier auditor is pulled toward the conclusion. Deleting the conclusion gains +42.5 and +28.5 pp in this run, and accuracy rises in every cell of Table 3. This is the setting that the title describes: an audit that reads reports and does not have the evidence to verify them.

With the documentation, the reports and the field lower accuracy by 3.0 [−5.9, −0.5] and 11.5 [−18.1, −6.1] pp against the documentation alone. Adherence to the proposal rises by 2.5 and 10.5 pp (frozen reading: overridden). When we remove the field from that prompt, accuracy changes by −0.5 [−4.8, +3.7] and +2.0 [−5.3, +9.9], not detected. So this loss comes with the reports, with or without the field (−3.5 and −9.5 pp without it).

AuthorsOpenReview ↗

Response to Reviewer Ba7S (2/2): the other arms, and Questions 3 to 5

W2, the other arms. We ran the six Part N arms of Appendix Y again with claude-sonnet-5, on 200 episodes per chain (reports only, the same prompts byte for byte, protocol frozen first):

| | GPT chains | Claude chains | |||| | adherence to the agents' proposal, field kept / deleted | 76.5 / 1.0 | 32.5 / 1.5 | | adherence to a planted wrong step, field / deleted | 73.5 / 14.0 | 21.0 / 2.0 | | planted capture relative to the agents' own (qwen, other execution: 0.83 / 0.83) | 0.79 [0.70, 0.88] | 0.61 [0.46, 0.79] | | share of the dependence that an instruction removes (qwen, other execution: 0.24 / 0.37) | 0.78 [0.71, 0.85] | 0.73 [0.61, 0.83] |

The frontier auditor also follows planted steps. The instruction works much better for it than for the mid-tier auditor (frozen reading: partial, lower bounds below 75%). When we show the same step as a random tag, most of the dependence goes away (+68.0 and +28.0 pp of +75.5 and +31.0). A random wrong step shown as such a tag still raises adoption by +9.0 [+5.0, +13.5] pp on GPT chains. On Claude chains the rise is +1.5 [+0.0, +3.4], not detected. We apply the rule frozen for Appendix Y, post hoc: the mechanism reading is mixed for this auditor too, with a clear numeric anchor on GPT chains.

The label arms and the ablation arms use the primary auditor. Its refusal rate changes between executions. The same prompt on the same 4,000 episodes gave no usable origin in 17.2% of calls in Appendix V, and in 58.5% one day later (Appendix X). For this reason we read every contrast inside one execution, we state each claim for a named auditor, and the registered run pins the provider.

Q3. Documentation alone under the neutral prompt, declines as errors. We ran this on the 8,020 core episodes of Appendix Y, in one execution with the asserting arms. Under the neutral prompt, the auditor with the documentation alone declares no defect in all 8,020 episodes. It describes the documentation as consistent and complete. Accuracy is 0.0% with declines scored as errors, against 57.8% (GPT) and 66.2% (Claude) on wrong-proposal episodes under the asserting prompt in the same run. With the reports and the field under the neutral prompt, it reaches 11.3% and 10.0%. So the documentation-alone reference measures attribution when the auditor knows that a defect occurred. This is the task that the paper studies: a pipeline has failed, and the auditor must say where. Both arms of the "present and unused" comparison share this premise. Section 4.3 and Appendix Z5 now state it.

Q4. The omission wording. The closest test in our data is the inconsistency family. Its text never says that a record is held at the originating site. On this family, the strict cost is not detected in either collection: +0.3 [−6.5, +6.6] (GPT) and −2.2 [−6.8, +2.4] (Claude, under 60 base cases) in the registered run (Table Y5), and +7.3 [−2.7, +16.2] and −1.5 [−5.4, +2.3] in the Table B5 collection. On omissions the cost is −61.0 / −14.7 and −66.9 / −12.9. This is what the wording account predicts. In this corpus the family and the wording change together. Section 5 now says that the cost falls mostly on defects whose text names an upstream site. In the second domain, 37.0% (GPT) and 75.2% (Claude) of wrong proposals name step t − 1. The first-escalator rule reaches 50.4% and 75.2% on those episodes.

Q5 / W3. Release and dates. The generator, the GEE code and the data are in the revised supplement. analysis/refit_independence.py now runs from it. analysis/recompute_released.py and analysis/recompute_appendices.py recompute Table 3 and Appendices D and E. Every per-response outcome is in the supplement, and every table is recomputed from it. The raw response texts (5.6 GB) are too large for the supplement. We will archive them on publication. We collected Table B5 between 14 and 17 August 2026. We froze the protocol of Appendix Y on 24 September 2026 and registered its Part C on OSF 14 minutes later, before its first call (Part N was pre-specified in the same protocol). Appendix Y is a registered re-measurement on the same 480 base cases, and the abstract says so. The out-of-sample test is now done. On 150 newly generated base cases (GPT chains, protocol frozen before the first call), the gain for the primary auditor is +28.0 [+19.3, +37.3] pp on 200 wrong-proposal episodes from 68 of them. It is +4.8 [+1.6, +8.7] pp where the first escalator is not the origin (Appendix Z10).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers: what the revision adds

We thank the three reviewers. For this revision we ran new experiments and new analyses. We fixed each protocol before the first call. The revised PDF and the revised supplement are uploaded.

1. Frontier auditors (Ba7S, 1nfV). With the reports alone, every auditor we tested is pulled toward the agents' conclusion. When we delete the conclusion, accuracy rises by 42.5 and 28.5 pp for claude-sonnet-5 (GPT chains and Claude chains) and by 16.0 and 7.0 pp for claude-opus-5-5. claude-sonnet-5 also follows a random wrong step that we plant in the field: 73.5% and 21.0%, against 14.0% and 2.0% without the field. With the documentation in its prompt, claude-sonnet-5 uses it. It reaches 97.0% and 88.0% with reports, documentation and field, against 100% and 99.5% with the documentation alone. So the mid-tier auditor leaves the evidence unused, and the frontier auditor uses it. The abstract and Section 4.3 now say this.

2. What replaces the conclusion (Ba7S, mERr). The split that Reviewer Ba7S proposed is now the second finding of the paper. For the primary auditor, the gain is small when the first escalating step is not the origin: +3.8 and +4.3 pp in the registered run, +2.8 and +4.2 pp in Table B5. Without the field, the auditor names that step much more often (+77.6 and +40.8 pp). This shortcut does not explain why the evidence is unused. Where the shortcut is wrong, the documentation alone still finds the origin in 47.4% and 55.9% of episodes. With reports, documentation and field, the auditor finds it in 0.8% and 1.7%.

3. New cases and real logs (1nfV, Ba7S, mERr). We generated 150 new base cases (GPT chains, one execution, Appendix Z10). The gain is +28.0 pp for the primary auditor and +31.5 pp for claude-sonnet-5. We also used the 184 public Who&When logs (Section 4.5, Appendix Z9). When we add a line naming a wrong step after a log, both auditors name that step more often (+32.1 and +11.4 pp).

4. What is new (mERr, 1nfV). A common test is to delete a cue and to read the gain as proof that the cue matters. We show that this gain does not tell if an auditor checks. On the new cases, we cannot tell the two gains apart (+28.0 and +31.5 pp). But with the documentation in the prompt, one auditor finds the origin in 7.0% of episodes and the other in 97.5% (52.0% and 99.0% with the documentation alone). The second new result is the price of deletion. The audited agents write the conclusion, so it is often right. In that execution, deletion costs the primary auditor 32.4 pp where the conclusion is right. To show this, our design changes only the agents' conclusion. The observations stay byte-identical. We compare the result with a free rule and with the evidence alone.

5. Reproducibility (Ba7S, 1nfV). The revised supplement adds what Reviewer Ba7S found missing: the generator, and the data behind Table 3, Appendices D and E and the GEE analysis. Four scripts run from a clean extraction, offline. Three scripts recompute Table 3, Table B5, Table D1, the four pre-registered GEE contrasts, the Appendix Z numbers cited in these replies, and Appendices U, D and E with Table Y5 (22, 62 and 136 checks, 0 discrepancies). The fourth script regenerates the 480 base cases and matches their SHA-256.

6. Scope (all). The main evidence comes from one six-step structure in two domains. The new cases and the real logs add to it. The documentation-alone reference tells the auditor that a defect occurred. This is the task we study: a pipeline has failed, and the auditor must say where. A control with filler text of the same length shows that prompt length and layout do not explain why the documentation is unused. The registered run uses the cases that first suggested the effect. This is why we added the new cases.

7. Presentation (all). We rewrote the body. The introduction shows one real episode. Two boxes give the terms and the findings, with one number for each finding. The text gives the numbers for one chain, and the tables give both. Each results paragraph makes one point.

AuthorsOpenReview ↗

Summary of the discussion for the Area Chair

Thank you to the Area Chair and to the three reviewers for a careful and constructive discussion.

In short, the discussion moved in a positive direction:

Reviewer mERr raised the score to weak acceptance. The reviewer found that the revision substantially addresses the concerns, and highlighted our main finding: similar deletion gains can come with very different use of the evidence.

Reviewer Ba7S had set two conditions for a higher score: a split of the deletion gain by the first escalating step, and the documentation arms on a frontier auditor. We delivered both, and Reviewer Ba7S confirmed that both conditions are addressed.

Reviewer 1nfV raised the score from 1 to 2, citing the thoroughness of the experiments.

The revision also extends the evidence beyond our original setting: 150 newly generated cases, two frontier auditors, and a test on 184 public Who&When logs from other multi-agent systems. The paper is clear about its scope.

Thank you again for your time and your work on this submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.