- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SeKe, a soft knowledge detoxification framework for robust RAG under retrieval pollution. SeKe expands the retrieved documents, groups semantically related documents, and applies generative semantic pooling to construct a detoxified context. Experiments across multiple datasets, attacks, retrievers, LLMs, and RAG frameworks show consistent improvements in accuracy and misleading rate.
Critical strengths
- The paper addresses a practical and important robustness problem in RAG systems.
- The evaluation is broad, covering multiple datasets, attacks, retrievers, LLMs, and RAG frameworks.
- SeKe can be integrated with existing robust RAG methods and generally improves their performance.
Critical weaknesses
-
The central assumption of SeKe, namely that polluted documents exhibit stronger semantic cohesion than benign documents, is not directly validated. The method groups documents based on semantic-neighbor overlap under the assumption that polluted documents are more mutually cohesive. However, the paper does not provide quantitative analyses of the overlap distributions or separability among benign-benign, polluted-polluted, and benign-polluted document pairs. As a result, it is unclear how reliably this signal distinguishes polluted documents from benign ones.
-
It is not sufficiently isolated whether the performance gains of SeKe actually come from the proposed detoxification mechanism. Generative Semantic Pooling combines document grouping, query-conditioned summarization, and internal memory elicitation. Knowledge Expansion also increases the amount of available information by retrieving additional documents. Therefore, it remains unclear whether the gains arise from the proposed detoxification mechanism, stronger summarization/reasoning, or simply access to more external knowledge.
-
Important control baselines are missing for validating the necessity of the proposed mechanism. For example, the paper should compare against a RAG system that directly uses the same expanded top-k+t documents, as well as one that simply summarizes the expanded context. Without such controls, it is difficult to determine whether the more complex Knowledge Convolution and grouping procedures are actually necessary.
-
The fairness of the comparison in terms of retrieval budget is unclear. SeKe uses the original k=5 retrieved documents and additionally retrieves t=5 documents, resulting in access to 10 documents in total. It is unclear whether the competing baselines are given the same retrieval budget. Therefore, the current results do not fully separate the contribution of the proposed method from the potential benefit of simply retrieving more documents.
-
The context detoxification evaluation in Figure 6 may be confounded by context length. Figure 6 counts the number of occurrences of polluted answers in the context, but SeKe explicitly compresses multiple documents into shorter representations. As a result, the raw occurrence count may naturally decrease simply because the resulting context is shorter.
-
Several important details required for evaluation and reproducibility are insufficiently specified. The exact procedures used to determine ACC and MR are unclear, and the paper does not sufficiently explain how the presence of a polluted answer is detected. In addition, the actual value and sensitivity of the key Generative Semantic Pooling hyperparameter are not reported. The paper also lacks an analysis of computational overhead introduced by the additional retrieval and LLM generation steps.
-
The document grouping procedure is not clearly defined. The text states that two documents are assigned to the same group when the overlap between their corresponding filtered yields and is at least 0.5. However, Eq. 7 instead computes the overlap between a remaining filtered yield and a candidate group . Moreover, it is unclear whether denotes a set of retrieved documents or a set constructed from filtered yields. This inconsistency makes the grouping algorithm difficult to interpret and reproduce.
-
To better characterize robustness against retrieval pollution, the paper would benefit from explicitly comparing clean RAG, polluted RAG, and polluted RAG with SeKe. The current evaluation mainly compares different defense methods, making it difficult to see how much performance is degraded by pollution itself and how much of that degradation is recovered by SeKe.
-
Since the proposed method primarily operates on retrieval and context manipulation, retrieval- or context-level metrics would better align the evaluation with the claimed mechanism.
-
Figures 4 and 5 are difficult to interpret because many experimental settings are shown simultaneously in radar plots, and small differences, particularly in MR, are hard to distinguish. In addition, TrustRAG appears to achieve low MR in some settings without correspondingly high ACC. Further analysis of this behavior would be helpful.
-
Finally, there are several grammatical issues and typographical errors throughout the paper that should be corrected.
Questions for authors
Please see the weaknesses.