- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies coordinated evidence poisoning in retrieval-augmented generation, where individually plausible passages collectively reinforce misleading claims. It proposes SAGE-RAG, which represents retrieved evidence as a graph of atomic claims connected by support, contradiction, and relatedness relations. The framework combines Attack-Aware Trust Propagation (AATP) to assess claim trust with Maximum Consistent Reasoning Subgraph (MCRS) selection to balance utility, trust, and consistency under an evidence budget. The resulting evidence is organized into structured context for a frozen generator. Evaluation includes adversarial QA benchmarks, multiple generators, component ablations, adaptive attacks, and qualitative failure analyses.
Critical strengths
-
The paper addresses a meaningful failure mode of evidence aggregation. Individually relevant passages can collectively create misleading corroboration, making passage-level relevance insufficient for assessing evidence quality. Representing atomic claims and their relations provides a useful framework for studying this problem. This framing is consequential for RAG systems that combine evidence from multiple potentially untrusted sources.
-
The selection method balances usefulness, trust, and consistency. MCRS considers both individual claim quality and how claims support or contradict one another. This is useful for selecting evidence that works together, particularly for multi-hop questions.
-
The evaluation covers a broad range of settings. The experiments include multiple QA datasets, attack types, and generators, with comparisons against existing retrieval and defense methods. Additional robustness tests, component ablations, and case studies provide useful evidence about the method’s effectiveness, the roles of its components, and its limitations.
Critical weaknesses
-
Parameter selection and sensitivity remain unclear. The method involves multiple objective weights and detection thresholds. Although the same settings are used across datasets, their selection rationale and the sensitivity of the results remain unclear. The authors should explain how these parameters were determined and examine whether answer quality and defense effectiveness remain stable as key weights and detection thresholds vary within reasonable ranges.
-
The construction of coordinated attacks in the main benchmark is insufficiently described. Although the appendices provide attack examples and some adaptive attack strategies, it remains unclear how the malicious documents in the main experiments were generated and coordinated to reinforce the same incorrect conclusion. The authors should clarify their sources and construction procedure, and whether the samples were checked against the stated conditions of coordinated evidence poisoning. This would help establish how well the evaluation covers the intended threat model.
-
Clarification of EM and CM Evaluation. Could the authors clarify why EM exceeds CM for TrustRAG on TriviaQA in Table 6 (0.320 vs. 0.295)? Under the stated definitions, an exact match should also satisfy contains match when both metrics use the same predictions, references, and normalization. Are different evaluation subsets or aggregation procedures used?
Questions for authors
I would appreciate clarification on the issues raised in the Critical Weaknesses section.
3 comments
Response to Reviewer — continued
W2. Attack-data sources, construction, and coordination
We thank the reviewer for the helpful request for greater clarity on attack-data sources, construction, and coordination. To clarify, Section 2 defines the threat model. An attacker inserts passages into the retrieval corpus to steer the answer toward a target answer, and the defender sees only the query and the retrieved documents. This follows PoisonedRAG (Zou et al., 2025), which demonstrates that five malicious texts per target question can achieve about 90% attack success in a knowledge database with millions of texts. The benchmark instantiates this threat model with a controlled construction. It is not produced by PoisonedRAG's official attack optimizer. A subset of the passages follows PoisonedRAG's construction. For each question, we set a common incorrect answer and steer the malicious passages toward it. The attack styles are surface realizations of that target. This choice keeps the generation procedure fixed and the evidence pool identical across methods.
Construction. Questions and gold answers come from NQ, TriviaQA, HotPotQA, and MuSiQue. Benign documents come from the contexts released with each dataset. For HotPotQA, these are the supporting and distractor paragraphs. For MuSiQue, they are the supporting and non-supporting paragraphs. For TriviaQA, we use Wikipedia contexts. For NQ, we use the released answer candidates because its open release provides no contexts. For each question, we set a common incorrect answer derived from the gold answer by a controlled perturbation. We then construct at least three malicious passages that steer toward it and are designed to support one another. The passages use template-based instruction injection and controlled false factual statements. A subset follows PoisonedRAG's construction. Construction is human-guided. We fix the target answer and the coordination goal, and the passages are generated by a controlled pipeline rather than by PoisonedRAG's official attack optimizer. The passages are shuffled into the same candidate pool as the benign documents. Every method sees the same pool and the same top- retrieved set.
Coordination. Coordination is generated at the answer level. The passages are steered toward a common incorrect answer and placed in the same candidate pool, so they jointly compete with the benign evidence. The attack styles are surface realizations and are not the coordination variable. We verified the resulting document-level structure. Specifically, 42.93% of the benchmark documents are malicious documents participating in cross-document SUPPORT components. These are SUPPORT-connected components with malicious claims from at least two different malicious documents. The remaining 57.07% are benign documents. The malicious documents therefore form coordinated support structures, while the benign documents provide the competing evidence.
Validation. We checked construction automatically for schema validity, multiple malicious passages per sample generated by different mechanisms, absence of duplicate documents and meta-commentary, and the presence of instruction cues in template-generated passages. Samples were also spot-checked manually. Table 2 reports graph-level attack recall and benign retention, which quantify how much of the benchmark the structural signals capture.
W3. The TrustRAG entry in Table 6
We thank the reviewer for catching this issue. We agree that, under the stated definitions, an exact match implies a contains match when predictions, references, and normalization are the same. The TrustRAG/TriviaQA entry is therefore internally inconsistent. We checked the per-dataset results and found that the EM and F1 values had been swapped. The correct values are EM = 0.148, F1 = 0.320, and CM = 0.295. The remaining rows of Table 6 satisfy . We will correct the two cells, the sentence that cites TrustRAG's EM as 0.320, and the ranking marks. After the correction, RobustRAG has the highest EM on this split (0.202), while TrustRAG keeps the lowest ASR (0.060). This is a cell-level error in one row of Table 6, not a difference in evaluation subsets or metric definitions. We are sorry for this error and appreciate the reviewer's attention to this detail.
We hope that our responses have addressed the reviewers' concerns. The new tables and future-work notes will be included in the revision. We remain happy to clarify any remaining questions during the discussion period.
Thank you for providing the additional detail on the construction of the coordinated attack benchmark and for conducting further parameter sensitivity analyzing.
These clarification and additional experiment have adequately addressed our main concern.
Response to Reviewer
W1. Parameter selection and sensitivity
We thank the reviewer for the constructive request to justify parameter selection and characterize sensitivity. To address this point, we added a one-at-a-time analysis of the seven objective parameters α, β, γ, δ, η, λ, and κ on 2,000 questions (500 per dataset). We vary each parameter at 0.50, 0.75, 1.00, 1.25, and 1.50 times its Table 4 default while fixing the other settings. The defaults were chosen to balance adversarial exclusion and benign retention and are used unchanged across all datasets and attack types. The analysis measures how both selection metrics and answer EM change around the deployed configuration.
One-at-a-time parameter sensitivity. The analysis uses the seven objective parameters and 2,000 questions.
At the default setting, exclusion/retention/Selection F1 are 0.791/0.804/0.793. In the one-at-a-time sweep, the default is the best of the five tested values for every parameter. The parameters differ in sensitivity. λ, β, and δ have the largest effects, with maximum ∣ΔSelection F1∣ values of 0.093, 0.078, and 0.076, respectively. η, γ, and α are milder, with values of 0.039, 0.041, and 0.054, while κ is intermediate (0.063). Stronger penalties improve exclusion but cost benign retention. For example, halving β raises exclusion from 0.791 to 0.854 but lowers retention from 0.804 to 0.622. Increasing β by 50% does the reverse, with exclusion at 0.642 and retention at 0.846. The sweep also records EM. The default has the highest EM among the tested values for every parameter, and the largest EM drop is 0.054 (λ).
The defaults in Table 4 are fixed across all datasets and attack types. We will add the full five-point sweep to the supplementary appendix, with a short pointer in Section 3.3. The sweep covers the seven objective weights, with the detection thresholds fixed at their Table 4 values.