- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates semantic evidence overcounting in multi-agent RAG, where agents repeating information from the same false source can create an illusion of independent confirmation. Using a TriviaQA setup with ten document-conditioned agents, the authors separately vary the number of wrong documents and underlying wrong semantic units. A behavioral model and communication analyses suggest that correlated evidence retains excessive influence and that debate can propagate or stabilize wrong answers. Semantic grouping and novelty-based message masking improve accuracy, but matched random controls achieve nearly the same gains, supporting the importance of repeated exposure without uniquely establishing semantic-provenance recognition as the mechanism.
Critical strengths
-
The paper examines both communication and final aggregation. These ablations help assess whether improvement comes from semantic information or general changes in exposure.
-
The failure analysis found that increasing copies of wrong claims reduce accuracy significantly. This provides a strong justification for the motivation of the work.
Critical weaknesses
-
The MADAM-RAG re-implementation is different from what the original MADAM-RAG paper describes. Original MADAM-RAG feeds aggregated answers and explanations to agents and an aggregator operates each round, while this paper feeds individual messages for all nine other agents and aggregation happens only after two peer-exchange rounds. The authors should justify the change or replicate the original implementation.
-
The residual copy-weight parameter \rho is not directly estimated. The paper also does not empirically measure the conditional mutual information used in its theoretical analysis.
-
The models used in the paper are relatively old (gpt-4o-mini, qwen2.5, mistral-7b). It’s unclear whether the same findings will generalize to newer models like qwen3 or gpt5.6 luna.
Questions for authors
See weakness