- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GE³ is a black-box attack that extracts a RAG system’s private corpus with benign queries, follow-up question prompts, and distance-based query selection. On four corpora and twelve LLMs, it covers more of the corpus than three baselines and gives enough data to copy the service.
Critical strengths
The experimental part is quite solid with many models and proper ablation, including six commercial models and three pipeline defenses, with the 2x2 ablation cleanly separating the contribution of the dual track from that of maximin selection.
Critical weaknesses
There is no limitation in the main body. One of the main contribution of the paper is EKO, while we cannot determine either the unused retrieved content was actually leaked, or the model is simply making associations based on the same topic. Besides, we cannot distinguish extracted content and retrieved content. The coverage of GE^3 do not show clear advantage as baseline (Dual-Rand and Ans-MMD).
Questions for authors
Look at critical Weakness section
3 comments
Response to Reviewer PDoU [2/2]
Critical weakness, point 3. Extracted content versus retrieved content
Retrieving a corpus record does not necessarily imply that its contents have been disclosed. Retrieval coverage (CR) measures the fraction of corpus records retrieved using evaluator-visible identifiers, whereas the attacker has access only to generated responses. The revised defines CR as a retrieval metric rather than a measure of disclosed information.
To assess the utility of information actually available to the attacker, we added a 100-question Healthcare evaluation (). Shadow RAG uses public answers from 1,000 target requests, without access to original records or internal retrieved passages. Both conditions use the same Llama-3.1-8B generator. On questions targeting extracted facts verified against original records, necessary-fact completeness increases from 18/100 (closed-book) to 92/100 (Shadow RAG). Here, necessary facts are the key pieces of information that an answer must include, established from the source record before evaluation. Completeness requires all essential source-supported facts to be correctly conveyed. Since the generator remains identical, these results demonstrate that the recovered information provides substantial factual utility beyond the generator's unaided responses, complementing the source-dependence evidence from the controlled interventions.
Critical weakness, point 4. Coverage relative to Dual-Rand and Ans-MMD
Our component analysis () reveals a coverage–fidelity–acceptance trade-off and clarifies the distinct contributions of EKO and maximin selection.
GE³ achieves average CR of 84.71%, higher than Dual-Rand (77.99%) but lower than Ans-MMD (87.65%). Maximin selection improves coverage over Dual-Rand in 23/24 configurations, while introducing EKO to Ans-MMD improves SF and ASR in all 24 configurations, despite reducing coverage in 18.
Specifically, adding EKO increases average SF from 0.6679 to 0.7365 and ASR from 90.15% to 97.57%, while decreasing CR from 87.65% to 84.71%. These results show that EKO improves extraction fidelity and answer acceptance, whereas maximin promotes broader retrieval coverage within the dual-track framework.
We have clarified this distinction in the revised , emphasizing that GE³ achieves a competitive balance across the three objectives rather than maximizing coverage alone.
We sincerely appreciate your time and effort in reviewing our paper. We have provided detailed responses to your comments, along with additional experiments and revisions to address your concerns.
As the discussion period is drawing to a close, we would be very grateful if you could take a moment to review our responses when convenient. If you have any remaining concerns or would like further clarification, we would be more than happy to address them.
Thank you again for your valuable feedback and consideration.
Response to Reviewer PDoU [1/2]
We sincerely appreciate your constructive comments and valuable feedback. Below, we provide point-by-point responses supported by additional analyses and experiments. We would be happy to provide further clarification or supporting evidence if needed.
Critical weakness, point 1. Limitations in the main body
Thank you for highlighting the importance of explicitly discussing limitations. We have substantially revised Section 5 to include a dedicated limitation discussion in the main manuscript.
Specifically, we distinguish evaluator-visible retrieval coverage from actual information disclosure, motivating future methods for directly measuring recovered knowledge from public responses. We also discuss the distinction between semantic novelty and knowledge discovery, as well as opportunities for adaptive coordination between answer-grounded exploration and EKO-driven exploitation. These directions extend the current framework toward more knowledge-aware and adaptive extraction.
Critical weakness, point 2. EKO leakage or topic association?
We appreciate the concern that EKO continuation cues may simply reflect topical associations rather than information from retrieved passages. To address this, we tested eight questions per corpus from Healthcare, TREC-COVID, SciDocs, and NFCorpus, using ten retrieved passages, BGE-M3 and Qwen3-Embedding-8B, with complete context, single-passage deletion, length-matched same-corpus replacement, and empty context (Appendix B.2).
Rather than relying on semantic similarity, we identify source-specific five-word phrases shared by a passage and continuation but absent from the query, primary answer, and other passages. We use five words to help distinguish source-specific expressions from incidental short matches while retaining matches in brief continuation questions. Deletion reduces these signals in 6/8, 5/8, and 6/8 literature-corpus questions using BGE-M3, and 7/8, 6/8, and 7/8 using Qwen (Table B.1).
A further control holds the complete-context primary answer fixed, ruling out changes in that answer as the sole explanation. For the 24 literature-corpus questions with BGE-M3, the same answer excludes 53 of 240 question–passage positions sharing a source-specific five-word phrase with it, leaving 187 per mode, fixed across interventions. Source matches fall from 48/187 to 0/187 after deletion in ordinary generation, and from 44/187 to 3/187 with the primary answer fixed; replacement introduces 35/187 and 45/187 new-source matches (Appendix B.2, “Primary-answer overlap”). These findings demonstrate that at least some EKO cues depend on retrieved information beyond the primary answer, rather than merely reflecting topical associations.