- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes E-MIA, which converts verifiable hard evidence in the target document (e.g., fine-grained details, proper nouns/technical terms, definitional statements, metadata cues, and causal/constraint relations) into an exam with four objectively gradable question types (FB/SC/MC/T/F), and uses the aggregated exam score across multiple evidence-targeted questions as the membership signal. Experiments across multiple datasets and diverse RAG configurations demonstrate the effectiveness of E-MIA.
Critical strengths
- The paper has organized experimental results.
- The paper addresses an important topic.
- The paper finds an interesting feature as the membership signal.
Critical weaknesses
-
The performance of the method is surprisingly great. They have a "117.8–142.6% improvement in TPR@FPR=0.005 over the strongest baseline." They have this record but they never explain why. And all the performance of E-MIA is above a certain value consistently. The paper should provide more discussion on why they achieve great performance and why the baselines don't.
-
The paper has very few defense details and content. They did not provide a valid solution on how to defend against this kind of method.
-
It is not clear how they differentiate members and non-members in their experiments. How do they guarantee non-members are not in the training set? It is still questionable.
Questions for authors
See comments.
We thank Reviewer D3Zh for raising the performance magnitude, the practical defense question, and the definition of a non-member. These concerns are important for interpreting what E-MIA demonstrates, and we answer them directly.
Weakness 1: The performance gap is surprisingly large and insufficiently explained.
Answer: E-MIA's performance is not a constant near-perfect plateau. Across the 12 main cells, TPR@FPR$=0.005$ spans 0.817--0.998, and Accuracy spans 0.977--0.999. Several cells fall well below saturation, including 0.817 and 0.842 on NFCorpus. The consistent result is the direction of the gap, not an identical score in every setting.
The mechanism behind the gap is visible in the score distributions. Baselines score the whole response through similarity, completion accuracy, or Yes/No correctness. These signals are smooth: a non-member can produce a plausible response from parametric knowledge or a similar retrieved passage, leaving substantial mass in the upper tail of the non-member distribution. At FPR$=0.005$, that tail is exactly where the threshold must be placed, so even a small overlap produces many false positives. E-MIA instead grades document-specific hard evidence with exact canonical answers. Its member and non-member distributions are wider and more bimodal, leaving less density near the low-FPR boundary (Fig.~2).
The format ablation isolates this design effect. At a fixed exam length, single-format exams reach only about 0.31--0.38 TPR@FPR$=0.005$, whereas the full four-format exam reaches 0.996. Changing the evidence source alone, while holding length, type mix, grading, and aggregation fixed, shifts separability by up to 0.538. The reported 117.8--142.6% figure is a relative improvement over a strongest baseline whose TPR at this strict operating point is often only about 0.2--0.5; it is therefore a ratio with a small denominator, not a claim that every E-MIA cell is perfect. The absolute TPR increase and the distributional separation are the substantive results.
Weakness 2: The defense discussion is incomplete and lacks an actionable solution.
Answer: The paper is primarily an attack-and-audit study, not a defense benchmark, and it does not claim a validated defense. Its results do, however, identify a concrete defense surface. Table~2 shows that query rewriting, response rewriting, and malicious-input guardrails do not remove the signal; these defenses act after the document has already made its evidence reproducible. A valid defense must intervene earlier, at the evidence that is exposed to the generator.
Concretely, we propose utility-aware evidence sanitization before ingestion. High-specificity facts that are not required for the supported downstream tasks are generalized to task-preserving ranges, categories, or paraphrases; exact numbers, rare identifiers, and narrow constraints are retained only when task utility requires them. At serving time, retrieved evidence can additionally be synthesized across multiple documents and rendered with randomized but semantics-preserving wording, so that no single document produces a stable exact answer key. The defense should be evaluated on the same audit metric used here, TPR@FPR$=0.005$, together with task-level QA utility, under both static and adaptive attackers. This is a proposed mechanism and evaluation protocol, not an experimentally validated result in the current paper.
Weakness 3: Membership labels and non-member exclusion remain unclear.
Answer: The inferred property is membership in the RAG index, not membership in the base model's pretraining corpus. After preprocessing, we use a fixed split: 60% of each corpus is ingested into the knowledge base and 40% is never ingested. The 1,000 members and 1,000 non-members per corpus are labeled directly by that ingestion log, not by model behavior. Non-members are therefore guaranteed absent from the RAG knowledge base, while other documents, including similar ones, remain available for retrieval.
No study using public pretrained models can guarantee that a document was absent from pretraining. If a non-member appeared in pretraining, that contamination would make it more knowledgeable and therefore more likely to be mistaken for a member; it would raise non-member scores and reduce measured separation. The reported low-FPR separation is thus conservative with respect to this confound. The paper's claim is correspondingly scoped to index membership. If these clarifications are persuasive, we would be grateful if the reviewer would reconsider the corresponding assessment.