acceptodds
Under review as a conference paper at ICLR 2027

When Repetition Becomes Evidence: Semantic Evidence Overcounting in Multi-Agent RAG

Abstract

Multi-agent retrieval-augmented generation (RAG) systems may treat agreement as independent evidence even when reports share a source. We study this behavioral-level mechanism in MADAM-RAG: when content derived from a single wrong semantic unit is copied or paraphrased across multiple surface-level documents, it produces apparently independent agent reports that may be counted repeatedly. We distinguish independent wrong semantic units Kw from surface-level wrong documents Mw in a TriviaQA-based environment with controlled provenance. On Qwen2.5-7B, under No Debate and with Kw=1, increasing Mw from 1 to 8 reduces accuracy from 90% to 47% and raises target-wrong selection from 0% to 48%; holding Mw=6 and varying Kw yields no comparable monotonic trend. Conditional mutual information and Bayesian accumulation provide normative references, while a correlation-neglect model captures residual behavioral weight on copies. Matched Context and round-level states indicate that debate propagates or stabilizes an existing wrong-evidence advantage rather than inherently creating errors. Aggregation- and communication-stage interventions further support repeated exposure as part of the pathway to erroneous consensus. Together, these results support a predictive and falsifiable behavioral-level account of semantic evidence overcounting, without making claims about its internal neural implementation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent RAG, retrieval-augmented generation, multi-agent debate, semantic evidence overcounting, conflicting evidence, correlation neglect, evidence redundancy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N8LdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates semantic evidence overcounting in multi-agent RAG, where agents repeating information from the same false source can create an illusion of independent confirmation. Using a TriviaQA setup with ten document-conditioned agents, the authors separately vary the number of wrong documents and underlying wrong semantic units. A behavioral model and communication analyses suggest that correlated evidence retains excessive influence and that debate can propagate or stabilize wrong answers. Semantic grouping and novelty-based message masking improve accuracy, but matched random controls achieve nearly the same gains, supporting the importance of repeated exposure without uniquely establishing semantic-provenance recognition as the mechanism.

Critical strengths
  • The paper examines both communication and final aggregation. These ablations help assess whether improvement comes from semantic information or general changes in exposure.

  • The failure analysis found that increasing copies of wrong claims reduce accuracy significantly. This provides a strong justification for the motivation of the work.

Critical weaknesses
  • The MADAM-RAG re-implementation is different from what the original MADAM-RAG paper describes. Original MADAM-RAG feeds aggregated answers and explanations to agents and an aggregator operates each round, while this paper feeds individual messages for all nine other agents and aggregation happens only after two peer-exchange rounds. The authors should justify the change or replicate the original implementation.

  • The residual copy-weight parameter \rho is not directly estimated. The paper also does not empirically measure the conditional mutual information used in its theoretical analysis.

  • The models used in the paper are relatively old (gpt-4o-mini, qwen2.5, mistral-7b). It’s unclear whether the same findings will generalize to newer models like qwen3 or gpt5.6 luna.

Questions for authors

See weakness

Reviewer_EBkbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates semantic evidence overcounting in multi-agent RAG, where repeated documents derived from the same incorrect source may be treated as independent evidence. Using a controlled TriviaQA-based setting and the MADAM-RAG framework, the authors distinguish between surface-level incorrect documents (Mw) and independent incorrect semantic units (Kw). They further introduce a correlation-neglect model and examine the effects of repetition through debate analysis and stage-specific interventions. The results show that repeated incorrect evidence can significantly reduce answer accuracy.

Critical strengths
  1. Distinguishing surface-level document multiplicity from the number of underlying semantic sources is a meaningful research direction.
  2. The paper goes beyond final-answer accuracy by examining communication dynamics and applying interventions at both aggregation and communication stages. The inclusion of matched random controls and the authors' acknowledgment of their limitations are valuable.
Critical weaknesses
  1. After reading the abstract, I’m not clear on what the authors are trying to achieve. What issues arise from "repeatedly weighting these reports" within the system? Furthermore, after describing the experimental phenomena, the paper suddenly introduces "conditional mutual information and Bayesian accumulation", what specific problem is this intended to solve? And how does the discussion shift to "agent debate"? I’m struggling to discern the core argument of this paper.
  2. The introduction still suffers from the same issue: the writing is overly disorganized. You should first clearly define the specific problem within a particular field that you intend to address (I believe the first part of the text does this reasonably well). Next, you should articulate the solution you propose to resolve specific technical challenges—this is where the technical depth of your paper should be demonstrated, yet I found this part unclear. Finally, you should outline the experiments conducted and the conclusions drawn from them (which are currently missing). This is the structure your introduction ought to follow.
Questions for authors
  1. When increasing MwM_w, the proportion of incorrect documents also increases. How do the authors disentangle semantic evidence overcounting from simple answer-frequency or majority-voting effects?
  2. Are the differences between semantic-aware interventions and their matched random controls statistically significant under question-level paired evaluation? Additionally, how is the provenance of a final agent report determined when the agent has changed its answer during debate?
  3. How were ambiguous TriviaQA questions excluded? Appendix B.4 presents “Freedom Suite” with David S. Ware as the correct answer, although another album with the same title is associated with Sonny Rollins. Could the authors audit the dataset and provide a sensitivity analysis excluding ambiguous items?
  4. Why are the main experiments limited to 100 TriviaQA questions and a fixed 10-agent setting? How robust are the findings across different agent configurations and more realistic retrieval settings?
  5. Appendix A.3 describes different answer-classification rules across experiments. How do the authors ensure that these differences do not affect the reported comparisons?
Reviewer_Tm8zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates evidence overcounting in multi-agent retrieval-augmented generation by separately controlling the number of incorrect documents and independent semantic evidence units, examining how repeated misinformation from a shared source affects discussion and answer aggregation. The experiments show that repeated incorrect evidence increases the influence of wrong answers, while evidence grouping and message masking can mitigate this effect, although interventions based on true semantic relationships do not clearly outperform random controls.

Critical strengths
  1. The research question is practically relevant. The paper examines how repeated misinformation from a shared source affects collective judgment in multi-agent RAG, addressing an important reliability concern.

  2. The experimental design is targeted. The study separately controls the number of incorrect documents and semantic evidence units, with interventions and matched random controls at both communication and aggregation stages.

  3. The statistical analysis and interpretation are appropriately cautious. The paper uses paired bootstrap estimates to quantify uncertainty and explicitly acknowledges that semantic interventions do not consistently outperform random controls.

Critical weaknesses
  1. The proposed mechanism is not sufficiently established. True semantic grouping and semantic gating do not clearly outperform matched random controls. The results therefore mainly support a general repeated-exposure effect, without adequately distinguishing insufficient discounting of correlated evidence from alternatives such as frequency bias.

  2. The comparison between debate and no debate is confounded. These experiments use different answer-matching rules, generation limits, and prompts, so their accuracy differences cannot be attributed entirely to discussion.

  3. The construction and validation of independent semantic evidence are insufficient. Distinct semantic labels do not necessarily imply informational independence. Limited details on evidence construction, strength matching, and quality validation weaken the central comparison between new information and repeated information.

  4. The experimental setting has limited real-world representativeness. The experiments use only 100 base questions and synthetic evidence averaging approximately ten words, with fixed numbers of agents and discussion rounds. Generalization to long documents, complex source dependencies, and multiple conflicting answers remains unclear.

Questions for authors
  1. Given similar gains from semantic and random interventions, what distinguishes the proposed mechanism from general repetition effects?

  2. Does the debate–no-debate performance gap persist with identical scoring rules, generation limits, and base prompts?

  3. How do you validate information independence and control evidence strength across semantic units when K_{w}>1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.