acceptodds
Withdrawn from ICLR 2027

Right Answers, Poisoned Provenance: What Accuracy Misses in LLM-Based Multi-Agent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents increasingly answer questions as a group, exchanging answers, rationales, and cited evidence that no single agent holds in full. Task outcomes and answer persistence do not establish the reliability of the cited evidence that humans and downstream agents may reuse. We study a distinct failure, provenance poisoning: adversarially planted misinformation enters honest agents' cited trails, whether or not the group answers correctly. We define a judge-free provenance-poisoning rate (PP) computed from debate logs and known planted-source identities. One adoption can poison the trail without changing the plurality answer. Across two multi-hop QA corpora and two open-model families using original source IDs and audited misinformation, one contaminated agent moves answer-and-evidence (Joint) F1 by at most .032 in absolute terms, yet its misinformation enters honest agents' citations in 12–50% of instances. The Joint-F1 change includes a 19.4% relative decline in one setting. Correct answers can also retain poisoned citations. On MINT-CWQ with Qwen3-8B, poisoning reaches 64.3% with two contaminated agents and opaque source IDs. In a separate original-ID reuse experiment, a new reader answers from passages cited by agents matching the group output, including groups that return no answer. Across two reader families, removing a planted passage adopted from a peer significantly improves Answer F1 by 4.3 points for Qwen and 2.0 for Mistral over all 250 MuSiQue questions, and helps significantly more than removing a length-matched legitimate passage. Poisoning rates vary across models in ways answer scores cannot distinguish, and a defense that checks peer claims has model-dependent effects. Multi-agent safety evaluation should measure the provenance of cited evidence, not only the answer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, multi-agent systems, provenance poisoning, misinformation, evidence attribution, safety evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7eMiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the issue of provenance poisoning in multi-agent reasoning systems powered by LLMs. The authors point out that even when a system ultimately reaches a correct consensus, compromised evidence can continue to contaminate the chain of citations.The main contributions of this work include defining a novel evaluation metric to quantify citation contamination, revealing the limitations of conventional accuracy metrics in detecting such vulnerabilities, and demonstrating that downstream applications suffer quantifiable harm when they reuse these contaminated citations.The authors validate these findings through extensive experiments across various datasets, such as MINT CWQ and MuSiQue, using models including Qwen3 and Mistral, and they further evaluate a mitigation strategy based on evidence provenance.

Critical strengths
  1. The authors introduce and formalize the concept of provenance poisoning, highlighting a significant flaw in current security assessments of multi-agent systems. The study compellingly demonstrates that standard accuracy metrics entirely overlook the potential for contamination within chains of provenance. This finding is highly significant, as it reveals a subtle security risk: even when upstream systems produce correct consensus results, downstream applications can still suffer quantifiable harm as a consequence.
  2. This research goes beyond merely identifying poisoned citations by explicitly testing the impact of these citations on subsequent readers. Through extensive experiments conducted across multiple datasets such as MuSiQue and involving various models like Qwen3 and Mistral, the authors demonstrate that removing these planted, false pieces of evidence significantly improves performance on downstream tasks. This comprehensive validation elevates the study from a purely theoretical exploration of vulnerabilities to a compelling empirical demonstration of real-world, systemic harm.
Critical weaknesses
  1. Why does the proposed EGA defense mechanism exhibit such inconsistent performance across different models, even leading to an increase in the poisoning rate and a decline in overall answer accuracy (Answer F1) when applied to the Gemma model?
  2. The experiments rely solely on static, pre-set misinformation. How effective would this system and its defense mechanism be against adaptive attackers capable of dynamically adjusting their rhetoric based on multi-agent debates?
  3. The experiments show that the baseline poisoning rate for advanced, closed-source models like GPT-4o is already extremely low (1–5%). Does this suggest that the source poisoning threat highlighted in this study does not, in fact, pose a significant challenge for state-of-the-art large language models?
  4. Although tests using opaque IDs were conducted, they were limited to a single model and dataset (MINT/Qwen) and still retained certain header cues (e.g., "Web result"). Can we definitively rule out the possibility that the models rely on metadata cues rather than the text content itself when generating citations?
  5. The EGA defense mechanism significantly increases the number of calls required for LLMs (e.g., from 16 to 35–39 calls per question). Is such a high computational and temporal cost feasible for practical deployment in multi-agent systems?
Questions for authors

See the above weaknesses.

Reviewer_MyJwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates provenance poisoning in LLM-based multi-agent reasoning, where misinformation introduced to one agent propagates into other agents' cited evidence, potentially without affecting the correctness of the group's final answer. The authors introduce provenance-poisoning rate (PP) and winner PP to measure the propagation of known planted evidence across agents. Experiments on MINT-CWQ and MuSiQue with multiple LLM families demonstrate that substantial citation contamination can occur with relatively small changes in conventional answer-and-evidence metrics. The paper further investigates the downstream consequences of contaminated citations through controlled evidence-removal experiments. Finally, the authors propose evidence-grounded admission (EGA), a verification-based mechanism intended to mitigate misinformation propagation.

Critical strengths
  1. The paper highlights a relevant limitation of conventional answer-based evaluation in multi-agent systems, namely that correct answers do not necessarily imply reliable supporting evidence. Examining the propagation of misinformation through citation trails provides a complementary perspective on multi-agent reliability.
Critical weaknesses
  1. Limited novelty and technical contribution. The technical novelty of the paper appears relatively limited. The proposed PP metric essentially tracks whether known malicious source identifiers are adopted by other agents, which is a straightforward measurement of evidence propagation. Similarly, the proposed EGA method primarily relies on standard LLM-based claim verification through prompting, without introducing a particularly novel algorithmic mechanism. It remains unclear what substantial methodological advances the paper offers beyond existing research on misinformation propagation, citation reliability, and LLM-based verification.

  2. A central motivation of the paper is that multi-agent systems can produce correct answers while retaining poisoned evidence, but the experiments provide relatively limited evidence for this specific scenario. For example, among the 31 MuSiQue/Qwen questions answered exactly correctly in both clean and contaminated runs, only 5 have poisoned collective trails, and only 2 have poisoned winner trails (Table 3). Moreover, most contaminated cases occur when the group answer is already incorrect, including 96 of the 126 poisoned MuSiQue/Qwen cases with zero Answer F1. The downstream experiments aggregate all questions, including incorrect answers and abstentions, rather than specifically investigating the consequences of poisoned citations accompanying correct answers, leaving the paper's central motivation insufficiently substantiated.

  3. The PP metric relies on prior knowledge of planted misinformation, raising questions about its applicability to realistic settings where malicious evidence is not explicitly identified. The prevalence of provenance poisoning also varies substantially across models, with GPT-4o exhibiting much lower contamination rates than the evaluated smaller open models, making the generality and practical severity of the phenomenon unclear. Furthermore, EGA produces inconsistent results across model families, even increasing contamination in some Gemma settings, while requiring substantially more LLM calls than unverified debate. Although the paper includes several control experiments, they do not clearly isolate the mechanism responsible for EGA's improvements or establish its effectiveness under comparable computational budgets across different settings.

Questions for authors

N/A

Reviewer_QYAhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes the case that multi-agent systems should be evaluated not just on their final answer accuracy but also the cited evidence that agents include alongside their answer. The authors study this in the debate setting, planting false evidence in some of the agents' contexts and studying how it propagates through the system and whether it is referenced in the final responses of agents that were not directly compromised. They find that planted evidence gets cited by other agents in the system even though the final group answer does not change very often, potentially hurting the individual agents that cite it as well as the downstream readers that are exposed to their cited passages.

Critical strengths

The paper cover a relevant area of research, in that evaluating how often agentic systems cite false sources beyond just final answer correctness is important.

Critical weaknesses

The writing is difficult to follow; there are many phrases used throughout the paper that could be simplified and in general things can be written in a more simple and direct manner. There is also a lot of unnecessary hedging in the writing.

Overall, the novelty of this work is limited. Most of the paper's findings are ones that a reader may reasonably assume given the problem construction. The core issue of having incorrect sources for correct answers appears to be rare on the tested tasks based on the authors' reported numbers (e.g 2/31 consistently correct musique/qwen qs) If there are more agents that are not 'poisoned' then a simple majority vote would ensure that the impacted agents' response does not win, so it is unsurprising that the final answer does not change much. While the authors note that models sometimes ignore explicit warning labels (::misinfo is a fairly obvious tag) on poisoned artifacts, this is also not as informative since it is unlikely that malicious or falsified material is labeled as such in practice. Removing that label also then substantially reduces the reported benefit of the authors' proposed method to counter such poisoning (EGA).

Questions for authors

Could the authors please provide more evidence of EGA's benefits with passages that have been planted in a plausible manner (e.g. without obvious tags like ::misinfo or other identifiers that can be used to easily distinguish it from real sources)? The key question here is how well agents can verify the contents of what they are given and how they contextualize it against the rest of the available evidence in multi-agent systems.

Additionally, could the authors clarify whether this problem primarily shows up with small language models? GPT-4o shows very little effect, and source attribution in frontier models has likely improved since then.

Finally, beyond the style comparison in App K, could the authors provide deeper analysis of how planted evidence spreads through multi-agent systems? In particular, do agents recognize it as suspicious in their reasoning, and does this get corroborated among other agents, or is it just treated as another piece of evidence? What happens when agents are warned that some information may be compromised? How does a simple prompt warning like that in the original instruction compare against EGA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.