acceptodds
Under review as a conference paper at ICLR 2027

RAGGate: Mitigating RAG Collapse through Admission Control

Abstract

Retrieval-augmented generation (RAG) systems can suffer from RAG collapse when their own earlier outputs are retrieved and reused, causing later answers to concentrate on fewer alternatives. We study whether document admission control can mitigate this feedback while preserving adaptation to newly arriving external information. We propose RAGGate, which determines whether an incoming document becomes eligible for retrieval using a score intended to capture information beyond previously admitted evidence. We instantiate this framework in two ways: RAGGate-Geo uses an embedding-based residual, while RAGGate-LLM asks the generator whether a document adds information. In controlled feedback loops designed to isolate repeated self-retrieval, both variants more than halve answer collision probability and improve knowledge renewal relative to admitting all incoming documents. In a complementary real-text experiment, both variants reject most derived documents while still admitting many incoming human-written pages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, RAG collapse, self-generated feedback loops, admission control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r3WiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies an interesting and increasingly relevant failure mode of RAG systems, where model-generated outputs are repeatedly re-ingested and eventually cause retrieval and answer diversity to collapse. The proposed RAGGate uses document-level novelty estimation to decide whether newly arriving documents should be admitted to the retrievable corpus.

Critical strengths

The paper is clearly motivated and the controlled experiments provide evidence that such an admission mechanism can mitigate the specific feedback-loop phenomenon.

Critical weaknesses

I have several concerns about the scope and empirical support of the proposed approach. First, the core problem is formulated at the document-admission level, which appears rather coarse for the actual goal of RAG. A document may contain both redundant and genuinely useful information, and a newly generated summary may introduce little novel information at the document level while being substantially more useful for downstream query answering. Thus, rejecting an entire document based on global novelty does not necessarily preserve the information that is most useful for retrieval and QA. More generally, information novelty is not equivalent to query-conditioned retrieval utility, and the paper does not sufficiently demonstrate that its admission decisions improve downstream QA quality beyond diversity-oriented metrics.

Second, the baseline comparison is relatively weak for the broader claim of document admission control. The evaluated baselines are largely tailored to the proposed self-feedback setting (e.g., authorship detection and NLI-based filtering), while stronger and more general alternatives such as semantic deduplication, clustering/representative selection, corpus pruning, diversity-aware retrieval, or query-aware document filtering are not sufficiently explored. It is therefore unclear whether the gains come from the proposed novelty formulation or simply from removing highly redundant documents.

Third, the experimental setting is relatively small and controlled: the main experiments use only 12 authored worlds, while the real-text evaluation involves 54 questions. This makes it difficult to assess robustness across different corpora, retrievers, generators, document distributions, and contamination levels. In addition, the novelty scores and thresholds are closely tied to the chosen embedding model or generator, but the robustness of the method to these choices is not sufficiently established.

Overall, I find the RAG collapse phenomenon interesting and the proposed idea reasonable, but the current experiments primarily establish its effectiveness in the specific self-feedback setting rather than demonstrating a generally superior approach to RAG knowledge-base filtering or admission control. Stronger baselines, larger and more realistic noisy-document QA benchmarks, and direct evaluation of downstream QA quality would substantially strengthen the paper.

Questions for authors

I encourage the authors to further clarify the above concerns and discuss how they plan to address these limitations in future revisions.

4 comments

We thank the reviewer for the thoughtful feedback. We respond to each concern below and have revised the manuscript accordingly. Changes are highlighted in blue.


W1a. document-admission level

We would like to clarify that our formulation defines a document as a retrieval unit, which may be either a complete document or a chunk (please see Section 3). RAGGate can therefore be applied to chunks, not just whole documents.


W1b. A document may contain both redundant and genuinely useful information

We have also tested documents containing both new and existing information. Each new-information document contains one new option and two existing options (please see Section 5.1). RAGGate-Geo and RAGGate-LLM admit 93/144 and 120/144 of these documents, compared with 21/144 for NN (Table 4).


W1c. the paper does not sufficiently demonstrate that its admission decisions improve downstream QA quality

Our goal is to mitigate RAG collapse, not to improve general QA performance. We have added a QA evaluation in the Appendix C. The results are mixed and do not establish consistent QA gains. Our claim is limited to mitigating RAG collapse.


W2a. The evaluated baselines are largely tailored to the proposed self-feedback setting

NN and NLI are general redundancy filters, rather than methods specific to self-feedback.

To broaden the comparison, we added MMR, a retrieval-time diversification baseline.


W2b. unclear whether the gains come from the proposed novelty formulation or simply from removing highly redundant documents

The gains are not explained simply by rejecting more derived documents. NN and RAGGate-LLM use the same calibration procedure and reject similar numbers of derived documents (702/720 vs. 708/720). Yet RAGGate-LLM admits more new-information documents (120/144 vs. 21/144) and achieves higher renewal success (0.264 vs. 0.062), with similar mean answer diversity (0.664 vs. 0.650).

In response to this concern, we have revised Section 5.2 to explicitly discuss this distinction.


W3. the robustness of the method

We would like to clarify that we already test different embedding models and generators (Appendix F, Tables 12–15), sensitivity to alpha (Appendix E), and longer feedback loops up to 50 rounds (Appendix A.2).

The 12 controlled worlds allow us to isolate self-feedback and measure knowledge renewal. We also evaluate RAGGate on 54 real-text questions across four domains (Section 5.3).

We have made these experiments and their purpose clearer in Sections 5.1 and 5.4 of the revised paper.

The authors have addressed some of my concerns, by clarifying the scope of the retrieval units and providing further evidence of the method’s capability to retain newly introduced information. These additions have increased my confidence in the technical robustness. However, in my view, they do not fundamentally alter my assessment of the study’s overall scope or the significance of its contributions. Consequently, I have raised the score for "Soundness".

Thank you for reconsidering our rebuttal and updating your soundness assessment.

We propose an admission control layer as part of a framework for mitigating RAG collapse while preserving knowledge renewal.

We believe this provides a solid building block for evolving RAG systems.

We would appreciate it if you could clarify which concerns remain about the significance of this framework-level contribution, so that we can address them more directly.

Thank you again for your time and consideration.

Thank you for the clarification. I understand the intended positioning of the work as an admission control layer for evolving RAG systems, and I agree that it can serve as a useful building block.

My remaining concern is not about the usefulness of admission control itself, but rather the scope of the demonstrated contribution. The current experiments primarily establish its effectiveness in mitigating answer concentration and preserving newly introduced information under a particular self-feedback setting. It remains less clear how broadly these findings generalize to evolving RAG systems, or how the proposed layer improves the overall quality and reliability of the evolving knowledge base beyond diversity and renewal metrics.

That said, I clarify to the AC that these concerns are a matter of personal academic preference. I acknowledge that the manuscript meets a satisfactory standard of quality.

Reviewer_VF6BOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies feedback loops in evolving RAG stores, where documents derived from earlier outputs can dominate retrieval and concentrate later answers. RAGGate separates document storage from retrieval eligibility and admits incoming documents according to estimated information beyond previously admitted evidence. It instantiates this rule using either an embedding-span residual with background removal or an LLM novelty score, with thresholds calibrated on the system’s own derived outputs. The evaluation jointly measures answer diversity and incorporation of incoming information in controlled worlds and a real-text feedback experiment.

Critical strengths
  1. The paper identifies a useful intervention point for evolving RAG systems. Admission control addresses the accumulation of repetitive evidence before retrieval, and the distinction between information novelty and machine authorship is well motivated.
  2. Evaluating renewal alongside diversity is a strong design choice: rejecting every arrival would otherwise appear successful. The controlled experiment also makes new-information documents overlap with existing evidence, avoiding a comparison based only on obvious near-duplicates.
  3. The LLM variant provides convincing evidence within the evaluated setting. On 54 real-text questions, it admits 93.8% of incoming pages while rejecting 2083/2160 derived documents. Its supplementary renewal score is 0.205 versus 0.117 for NN, with a positive paired confidence interval for the difference. The bootstrap analysis, background-removal ablation, 50-round extension, and alternative-model experiments make the empirical evaluation more informative.
Critical weaknesses
  1. The real-text results support the two implementations differently. Geo has essentially the same diversity as NN (0.448 versus 0.450), with lower mean supplementary renewal (0.098 versus 0.117). Its paired renewal improvement over Admit-all also includes zero. Geo does admit more incoming pages than NN, but retaining pages is weaker evidence than incorporating useful new information into answers. The strongest real-text renewal evidence therefore comes from the more expensive LLM variant, despite Geo being presented as the default implementation.
  2. The evaluation remains a simplified recommendation setting. Each stream begins with three documents, serves a fixed query, and receives only three external page arrivals in the main real-text experiment. Reserved pages are selected to contain an unseen entity with limited lexical overlap and are not individually checked for factual novelty. The paper is appropriately explicit about these choices, but they limit conclusions about factual updates and novelty decisions in shared, multi-query stores.
  3. The NLI comparison confounds the scoring function with the threshold procedure. NN and both proposed variants calibrate their thresholds on derived outputs, whereas NLI uses a fixed 0.5 cutoff. Since calibration is a central part of the framework, applying the same calibration procedure to an NLI novelty score would better isolate the benefit of the proposed scores.
Questions for authors
  1. Could you include a calibrated-NLI control on the existing controlled benchmark, or compare admission trade-offs at matched derived-document rejection rates? This would clarify how much of the advantage comes from the score rather than threshold selection.
  2. On the existing real-text sample, does Geo’s increased page admission translate into verified incorporation of new facts? A small manual audit of the cases admitted by Geo but rejected by NN would help explain the gap between retention and renewal.
  3. Historical calibration scores are retained even as the admitted reference set changes. Can you report how thresholds and derived-document admission rates evolve in the existing 50-round runs? This would help assess the stability of the operational calibration rule without assuming a formal coverage guarantee.
1 comment

We thank the reviewer for the thoughtful feedback. We respond to each concern below and have revised the manuscript accordingly. Changes are highlighted in blue.


W1. The strongest real-text renewal evidence comes from the more expensive LLM variant, despite Geo being presented as the default implementation.

We agree that the strongest real-text renewal evidence comes from RAGGate-LLM. Geo’s higher page admission demonstrates retention, but does not establish improved incorporation of new information over NN. Our choice of Geo as the default reflects its lower computational cost. The two implementations offer a cost–effectiveness trade-off.


W2. The evaluation remains a simplified recommendation setting.

The fixed-query setting lets us isolate self-feedback and measure the trade-off between suppressing derived content and retaining incoming information. We agree that unseen entities and limited lexical overlap do not establish factual novelty. Our results demonstrate mitigation of self-feedback in the evaluated streams. Further evaluation is needed to establish effectiveness for factual updates in shared, multi-query stores.


W3. The NLI comparison confounds the scoring function with the threshold procedure.

We agree that the original comparison does not isolate scoring from calibration. We therefore applied Geo’s threshold formula to offline NLI scores from an earlier 50-round controlled stream and examined admission across derived-document rejection rates. Calibration left new-information admission unchanged at 19.4% on the archived documents. Please see Q1 below for the results and the limitations of comparison with the in-loop policies.


Q1. Could you include a calibrated-NLI control on the existing controlled benchmark

To address this question, we conducted an offline calibrated-NLI analysis on an earlier 50-round controlled stream (12 worlds × 3 seeds), using the same NLI checkpoint and Geo’s online threshold formula (α = 0.05). This stream contains 1,800 derived documents and 144 new-information documents, with two new documents arriving in each of six worlds at rounds 10, 20, 30, and 40.

Calibration changes little: derived-document rejection is 98.8% versus 99.4% with the fixed threshold, while new-information admission remains 19.4%.

We also evaluated NLI across derived-document rejection rates:

Target derived rejectionDerived rejectedNew-information admitted
0.500.5030.708
0.800.8000.417
0.900.9000.306
0.950.9500.264
0.990.9900.215

At 95.0% derived-document rejection, NLI admits 26.4% of new-information documents. For reference, in-loop Geo achieves 95.7% rejection and 64.6% admission. Applying Geo’s calibration rule alone provides little improvement for NLI on the archived documents. The comparison with the main Geo results is descriptive because both the stream protocol and reference stores differ. Offline NLI uses all earlier documents as candidate references rather than the policy’s admitted store.


Q2. does Geo’s increased page admission translate into verified incorporation of new facts?

We conducted an automated lexical check on the earlier 12-question real-text sample across three seeds. Geo admitted pages that NN rejected in 26 page–seed cases, covering 11 distinct pages. In 13 cases (50%), a later Geo answer reused a five-word phrase from the page that was absent from the other retrieved documents. For example, an answer reused “British Indian Good Food Guide” from an additionally admitted page.

This provides evidence of textual reuse from additionally admitted pages, but does not verify factual novelty or correct use.


Q3. how thresholds and derived-document admission rates evolve in the existing 50-round runs

We examined the 50-round Geo runs from the α-sensitivity experiment (12 worlds × 3 seeds, α = 0.05). Within each world and seed, the threshold range across rounds had a median of 0.031 and a maximum of 0.098, compared with a between-world spread of approximately 0.26–0.42 at a given round. Derived-document admission remained below 5% in most rounds, with higher rates in the initial rounds.

These observations suggest limited threshold drift in typical runs. They do not establish a coverage guarantee, since historical calibration scores are retained while the reference store changes.

Reviewer_db89OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAGGate, a document admission control framework designed to mitigate RAG collapse, a phenomenon in which RAG systems repeatedly retrieve their own previously generated content, causing subsequent responses to become increasingly concentrated on a limited set of answers. To address this problem, the proposed RAGGate determines whether an incoming document should become eligible for future retrieval based on the additional information it provides relative to previously admitted documents. The authors evaluate RAGGate using controlled synthetic feedback loops and real-text experiments involving 54 questions. The results demonstrate that both variants substantially improve answer diversity and knowledge renewal compared with admitting all incoming documents.

Critical strengths
  • The studied problem is quite new. Most of the literature on RAG collapse is published in 2026. This may be an underexplored problem in RAG systems.
  • Empirical studies demonstrate improvements over multiple baselines.
Critical weaknesses
  • The scope of the studied problem is limited. It only studies whether a document should be accepted or not. This may not be suitable for ICLR.
  • The proposed method seems naive, and the performance is not significantly improved. RAGGate-LLM simply uses LLM's next token probability, and RAGGate-Geo uses the embedding distance. I think it's OK for the method to be simple, but the empirical improvement is not very significant in certain cases. For instance, on the real-text experiment, RAGGate-Geo is statistically indistinguishable from the simple nearest-neighbour baseline on both answer diversity and renewal, and is significantly worse than RAD in diversity.
Questions for authors
  • How is the scalability of this method when the number of existing documents is large?
  • Would simpler approaches like clustering based on embeddings achieve comparable results?
  • RAGGate-Geo assumes that information already contained in admitted documents can be approximated by the linear span of their embeddings. However, semantic novelty may not correspond directly to geometric novelty. Could the authors provide additional justification or analysis of this assumption? In particular, how does the method handle documents that introduce little changes such as new relationships between previously known entities?
1 comment

We thank the reviewer for the thoughtful feedback. We respond to each concern below and have revised the manuscript accordingly. Changes are highlighted in blue.


W1. "It only studies whether a document should be accepted or not."

Document admission is not the problem itself. It is our approach to mitigating RAG collapse. We introduce an admission control layer before retrieval. We show that it can mitigate RAG collapse while preserving knowledge renewal. We believe this is a solid building block for evolving RAG systems.


W2a. "RAGGate-LLM simply uses LLM's next token probability, and RAGGate-Geo uses the embedding distance."

The main contribution of our work is an admission control framework for mitigating RAG collapse in evolving RAG systems, rather than the individual scoring functions. We show that this approach can suppress self-feedback while preserving the ability to incorporate new information.

We would also like to clarify that RAGGate-Geo does not simply use embedding distance. It measures the residual outside the span of similar admitted document embeddings after removing common embedding directions (please see Section 4.1).


W2b. "the empirical improvement is not very significant in certain cases"

We would like to highlight three points.

First, NN also uses our admission framework with self-generated calibration. Its diversity gains over Admit-all in both controlled (0.650 vs. 0.261) and real-text (0.450 vs. 0.165) experiments support the value of this framework.

Second, diversity should be considered alongside renewal. Although Geo has lower real-text diversity than RAD, it achieves significantly higher renewal (0.098 vs. 0.050).

Third, LLM significantly improves renewal over NN in both settings (0.264 vs. 0.062 and 0.205 vs. 0.117). Geo offers lower computational cost and significantly improves controlled renewal, although its real-text diversity and renewal differences from NN are not statistically significant (Tables 3, 5, and 8).


Q1. How is the scalability of this method when the number of existing documents is large?

RAGGate-LLM uses at most eight references, while Geo uses all admitted references in our experiments. Geo’s computation therefore depends on the reference-set size, and LLM reference selection requires similarity computation and sorting.


Q2. Would simpler approaches like clustering based on embeddings achieve comparable results?

We compare against NN, a simple embedding-based filter, and have added MMR for retrieval diversification. RAGGate-LLM achieves higher controlled renewal success than both (0.264 vs. 0.062 and 0.181, respectively), as discussed in Section 5.2.


Q3. semantic novelty may not correspond directly to geometric novelty

Prior work supports the motivation for Geo’s novelty proxy by using components orthogonal to a context span to quantify semantic novelty [1].

We have strengthened the explanation in Section 4.1 with this supporting reference.

[1] Yang et al., Parameter-free Sentence Embedding via Orthogonal Basis, EMNLP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.