- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether an LLM auditor that attributes failures in a multi-agent pipeline actually checks the evidence or simply repeats the agents’ own conclusions. Using a synthetic six-step supply-chain pipeline with injected defects, the authors hold the case evidence fixed and intervene only on the “suspected origin” field of the agents’ reports: deleting it, replacing it with a random wrong step, or supplying the raw documentation alongside it. The results show that the auditor mostly relays the agents’ conclusions rather than verifying them.
Critical strengths
- This paper studies a very interesting phenomenon. It could inspire further research.
- The experimental design is clear. Only the single phrase "Suspected origin" in the report was modified, while the rest of the content remained identical. Thus, the effect could be clearly attributed to that specific phrase.
Critical weaknesses
- It only uses Qwen3-30B model to do the planted-conclusion experiment, the “do not use the field” instruction and the mechanism analysis. The more powerful models (gpt-5, claude-sonnet-5) were tested only on field deletion.
- The paper does not test on any real multi-agent systems, so it is unclear whether the conclusion still holds outside this constructed six-step supply chain pipeline.
- LLMs are likely to repeat conclusions supplied to them is well studied in work on sycophancy and LLM-judge biases, and the authors state that they claim no new bias. The contribution is mainly a careful measurement in one specific setting. So the novelty of this work is limited.
Questions for authors
- Since the cases are regenerable from a seed, why was the registered re-measurement not conducted on newly generated base cases?
- Does the phenomenon appear on existing failure-attribution benchmarks or real multi-agent systems in which agents report their own guesses?
- Can you test more powerful models like gpt6-astra, opus-5.5?
3 comments
I appreciate the author's responses, but my assessment remains unchanged.
The model tends to follow the conclusion in the input is a well-known phenomenon, and many works on sycophancy, anchoring and LLM-judge have shown it. Even the paper itself claims no new bias. The two new points in the response follow from this. A deletion gain alone cannot show whether an auditor checks. And removing a conclusion that is usually right will cost accuracy when it is right. These points are expected, not new findings. The new results in the response also demonstrate that a stronger auditor uses the documentation (97.5%). This makes the remaining effect more like a known phenomenon that "weaker models tend to follow a given answer instead of thinking about it themselves." I have raised my score to 2 because the experiments are thorough. But the work is still a careful measurement of a known behaviour.
Thanks for raising score
Thanks for reading our replies and for raising your score.
One short point on the idea that only weaker models follow the conclusion. From the reports alone, the strong models are pulled too. On the 150 new cases, in a single run, deleting the conclusion raised accuracy by about the same amount for our primary auditor and for Claude Sonnet 5, by 28.0 and 31.5 points. Claude Sonnet 5 also picks a planted wrong step in 73.5% of episodes. What the stronger model adds is that it uses the documentation when the documentation is in its prompt. So the pull from the reports is not limited to weak models.
Thank you again for your time.
Response to Reviewer 1nfV
Thank you. You wrote that the phenomenon is very interesting and that the design is clear. The revision adds the evidence that your review asked for. Each addition is now in the abstract, in Box 2 of the introduction and in Section 4:
| your point | what the revision adds | ||| | planted, instruction and mechanism arms on Qwen only | the same arms on claude-sonnet-5: it follows planted steps (0.79 and 0.61 of the gain that its own proposals produce) | | stronger models | claude-opus-5-5: it is also pulled toward the conclusion | | real multi-agent systems | the planted test on the 184 public Who&When logs: the auditors name the planted step more often (+32.1 pp qwen, +11.4 pp claude-sonnet-5) | | fresh base cases | 150 newly generated base cases from the same generator, frozen before the first call: the gain holds (+28.0 and +31.5 pp) | | novelty | a deletion gain does not show checking, and deletion has a price (below) |
Novelty compared with sycophancy and judge-bias work. It is known that models defer to an answer that they are given. We build on this. The first new result is that a deletion gain does not tell if an auditor checks. A deletion gain is the usual proof that a cue matters. On 150 new base cases in one execution, we cannot tell apart the gain of the primary auditor and the gain of claude-sonnet-5 (+28.0 and +31.5 pp). But with the documentation in the prompt, the first finds the origin in 7.0% of episodes and the second in 97.5%. The second new result is the price of deletion. The audited agents write the conclusion, so it is often right. In that same execution, deletion costs the primary auditor 32.4 pp where the conclusion is right. To show both results, you need a bench where only the agents' conclusion changes and the observations stay byte-identical. You also need to compare the result with a free rule and with the evidence alone. Who&When and Who&When Pro do not offer this. Our bench is released.
Stronger models. We ran claude-opus-5-5 on the 400 Table 3 episodes where the agents' proposal is wrong. It finds the origin in 11.5% (GPT chains) and 35.5% (Claude chains) with the field, and in 27.5% and 42.5% without it (+16.0 [+10.7, +21.8] and +7.0 [+1.5, +12.7] pp). So it is also pulled toward the conclusion. The planted, instruction and relabelling arms now also run on claude-sonnet-5. It adopts a random wrong step planted in the field in 73.5% of GPT-chain episodes and 21.0% of Claude-chain episodes (14.0% and 2.0% without the field). This is 0.79 and 0.61 of the gain that the agents' own proposals produce. The instruction to ignore the field removes 78% and 73% of its dependence. With the documentation in its prompt, claude-sonnet-5 uses it. So the paper now says which auditors leave the evidence unused (see our reply to Ba7S).
Fresh base cases. The registered run answers a narrower question: does the effect survive auditor randomness, changes in the served model, and analysis choices, on the cases that suggested it? Its registration says this before the run ("a re-measurement on the same corpus, not an out-of-sample replication"). New base cases are the stronger test. The seeded generator made them possible, so we ran them: 150 newly generated base cases, none identical to one of the 480 (Appendix Z10 gives the overlap check), GPT chains, protocol frozen before the first call. In one execution on the same 200 wrong-proposal episodes, deleting the field gains +28.0 [+19.3, +37.3] pp for the primary auditor and +31.5 [+22.5, +41.0] pp for claude-sonnet-5. The effect holds out of sample.
Real systems and existing benchmarks. We ran the planted test on the 184 public logs of Who&When. We froze the protocol and the analysis before the first call (Appendix Z9). We add one line,
Suspected origin: step N., to each real log. N is a random wrong step. Compared with the same log without the line, the auditor names that step more often: 38.6% against 6.5% for qwen (+32.1 [+25.0, +39.1] pp), and 16.8% against 5.4% for claude-sonnet-5 (+11.4 [+7.1, +16.3] pp). So a stated origin also pulls the auditor on real multi-agent logs. This result is now in the abstract and in Section 4.5. Deletion cannot be tested on existing benchmarks, because none has a field where an agent names the origin. Who&When and Who&When Pro do not have one. The Magentic-One orchestrator writes "root cause" passages in 22 of the 58 hand-crafted logs, but these passages never give a step. Our bench fills this gap.