- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies an interesting and increasingly relevant failure mode of RAG systems, where model-generated outputs are repeatedly re-ingested and eventually cause retrieval and answer diversity to collapse. The proposed RAGGate uses document-level novelty estimation to decide whether newly arriving documents should be admitted to the retrievable corpus.
Critical strengths
The paper is clearly motivated and the controlled experiments provide evidence that such an admission mechanism can mitigate the specific feedback-loop phenomenon.
Critical weaknesses
I have several concerns about the scope and empirical support of the proposed approach. First, the core problem is formulated at the document-admission level, which appears rather coarse for the actual goal of RAG. A document may contain both redundant and genuinely useful information, and a newly generated summary may introduce little novel information at the document level while being substantially more useful for downstream query answering. Thus, rejecting an entire document based on global novelty does not necessarily preserve the information that is most useful for retrieval and QA. More generally, information novelty is not equivalent to query-conditioned retrieval utility, and the paper does not sufficiently demonstrate that its admission decisions improve downstream QA quality beyond diversity-oriented metrics.
Second, the baseline comparison is relatively weak for the broader claim of document admission control. The evaluated baselines are largely tailored to the proposed self-feedback setting (e.g., authorship detection and NLI-based filtering), while stronger and more general alternatives such as semantic deduplication, clustering/representative selection, corpus pruning, diversity-aware retrieval, or query-aware document filtering are not sufficiently explored. It is therefore unclear whether the gains come from the proposed novelty formulation or simply from removing highly redundant documents.
Third, the experimental setting is relatively small and controlled: the main experiments use only 12 authored worlds, while the real-text evaluation involves 54 questions. This makes it difficult to assess robustness across different corpora, retrievers, generators, document distributions, and contamination levels. In addition, the novelty scores and thresholds are closely tied to the chosen embedding model or generator, but the robustness of the method to these choices is not sufficiently established.
Overall, I find the RAG collapse phenomenon interesting and the proposed idea reasonable, but the current experiments primarily establish its effectiveness in the specific self-feedback setting rather than demonstrating a generally superior approach to RAG knowledge-base filtering or admission control. Stronger baselines, larger and more realistic noisy-document QA benchmarks, and direct evaluation of downstream QA quality would substantially strengthen the paper.
Questions for authors
I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.
4 comments
The authors have addressed some of my concerns, by clarifying the scope of the retrieval units and providing further evidence of the method’s capability to retain newly introduced information. These additions have increased my confidence in the technical robustness. However, in my view, they do not fundamentally alter my assessment of the study’s overall scope or the significance of its contributions. Consequently, I have raised the score for "Soundness".
Thank you for reconsidering our rebuttal and updating your soundness assessment.
We propose an admission control layer as part of a framework for mitigating RAG collapse while preserving knowledge renewal.
We believe this provides a solid building block for evolving RAG systems.
We would appreciate it if you could clarify which concerns remain about the significance of this framework-level contribution, so that we can address them more directly.
Thank you again for your time and consideration.
Thank you for the clarification. I understand the intended positioning of the work as an admission control layer for evolving RAG systems, and I agree that it can serve as a useful building block.
My remaining concern is not about the usefulness of admission control itself, but rather the scope of the demonstrated contribution. The current experiments primarily establish its effectiveness in mitigating answer concentration and preserving newly introduced information under a particular self-feedback setting. It remains less clear how broadly these findings generalize to evolving RAG systems, or how the proposed layer improves the overall quality and reliability of the evolving knowledge base beyond diversity and renewal metrics.
That said, I clarify to the AC that these concerns are a matter of personal academic preference. I acknowledge that the manuscript meets a satisfactory standard of quality.
We thank the reviewer for the thoughtful feedback. We respond to each concern below and have revised the manuscript accordingly. Changes are highlighted in blue.
We would like to clarify that our formulation defines a document as a retrieval unit, which may be either a complete document or a chunk (please see Section 3). RAGGate can therefore be applied to chunks, not just whole documents.
We have also tested documents containing both new and existing information. Each new-information document contains one new option and two existing options (please see Section 5.1). RAGGate-Geo and RAGGate-LLM admit 93/144 and 120/144 of these documents, compared with 21/144 for NN (Table 4).
Our goal is to mitigate RAG collapse, not to improve general QA performance. We have added a QA evaluation in the Appendix C. The results are mixed and do not establish consistent QA gains. Our claim is limited to mitigating RAG collapse.
NN and NLI are general redundancy filters, rather than methods specific to self-feedback.
To broaden the comparison, we added MMR, a retrieval-time diversification baseline.
The gains are not explained simply by rejecting more derived documents. NN and RAGGate-LLM use the same calibration procedure and reject similar numbers of derived documents (702/720 vs. 708/720). Yet RAGGate-LLM admits more new-information documents (120/144 vs. 21/144) and achieves higher renewal success (0.264 vs. 0.062), with similar mean answer diversity (0.664 vs. 0.650).
In response to this concern, we have revised Section 5.2 to explicitly discuss this distinction.
We would like to clarify that we already test different embedding models and generators (Appendix F, Tables 12–15), sensitivity to alpha (Appendix E), and longer feedback loops up to 50 rounds (Appendix A.2).
The 12 controlled worlds allow us to isolate self-feedback and measure knowledge renewal. We also evaluate RAGGate on 54 real-text questions across four domains (Section 5.3).
We have made these experiments and their purpose clearer in Sections 5.1 and 5.4 of the revised paper.