- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates black-box, document-level membership inference against retrieval-augmented generation (RAG) systems, focusing on false positives caused by semantically similar non-member documents. It proposes QP-MIA, which generates document-specific fill-in-the-blank questions and constructs two partially overlapping joint queries by replacing selected questions and changing their order. The method evaluates the RAG system's textual answers against reference answers and combines answer correctness with repeated correctness on shared questions to produce a membership score. Experiments on TREC-COVID, HealthCareMagic, and MS-MARCO, across different generators, retrievers, and RAG configurations, report substantial AUC improvements over several baselines. Additional analyses examine query composition, retrieval behavior, threshold selection, and representative defenses. The authors conclude that query-composition perturbation provides a robust membership signal under black-box access.
Critical strengths
-
The paper addresses a meaningful challenge in RAG privacy auditing: a non-member document can still yield correct answers when related evidence is present in the knowledge base. The motivation for probing a system under different query compositions is intuitive and relevant to black-box membership inference.
-
The reported empirical gains are encouraging. Table 1 evaluates several generators, retrievers, and RAG configurations across three datasets, and Appendix C includes analyses of retrieval overlap and target retrieval rates. This breadth suggests that the approach is potentially useful beyond a single experimental configuration and warrants further investigation.
Critical weaknesses
-
The proposed cross-query consistency score does not clearly capture consistency beyond repeated answer correctness. In Section 3.3, (S_{\mathrm{stab}}=\lambda_2 C^{both}+C^{part}), with (\lambda_2=2) in Appendix B. Under this setting, (S_{\mathrm{stab}}=2C^{both}+C^{part}) is exactly the total number of correct answers to shared questions across the two queries. Consequently, two response patterns with identical correct-answer counts but substantially different question-level persistence can receive the same score; the final score can also be identical when the per-query totals and unique-question correctness are matched. While Table 8 shows that repeated correct answers are discriminative, it does not establish an additional benefit from modeling cross-query consistency separately from accumulated correctness. Similarly, although Table 3 demonstrates the benefit of the overall partially overlapping query strategy, it does not isolate whether the proposed consistency term contributes discriminative information beyond repeated answer correctness. This gap matters because such consistency is central to the paper's claimed mechanism and technical contribution.
-
The experiments do not directly establish that the reported improvements arise from resolving interference by semantically similar non-member documents, which is the central problem motivating QP-MIA. The principal comparisons use randomly sampled members and non-members and report aggregate AUC/ACC. Appendix C analyzes overlap between retrieved top-k sets, but retrieval-set overlap is not the same as the semantic similarity between a non-member target and its nearest indexed document. The paper therefore does not show whether false positives are specifically reduced on high-similarity or near-duplicate non-members. A targeted similarity-stratified evaluation would be important to substantiate the proposed explanation, rather than only the aggregate effectiveness of the scoring procedure.
Questions for authors
Q1 : With (\lambda_2=2), (S_{\mathrm{stab}}) is algebraically equal to the total number of correct shared-question responses. What additional membership information is intended to be captured by the cross-query consistency term beyond repeated correctness? Could the authors compare the proposed score with a simple correct-answer count and analyze whether (C^{both}) remains discriminative after controlling for total correctness? As a minor implementation clarification, the released MIA.py weights (C^{uni}) by 1.5, whereas Section 3.3 assigns it a coefficient of 1. Which version produced the reported results?
Q2 : Could the authors stratify non-member documents by their maximum semantic similarity to the indexed corpus and report the false-positive rates within each similarity group at matched true-positive rates, together with AUC comparisons using a common set of member samples? In particular, how does QP-MIA compare with the baselines on highly similar or near-duplicate non-members?
Q3 : Query-composition perturbation changes both the retriever input and the generator prompt, including question order. How much of the observed response variation is due to a change in retrieved evidence rather than a generator-side order/context effect? A controlled comparison with fixed retrieved passages but varied query composition, and conversely fixed evaluation questions but varied retrieved passages, would clarify the source of the signal.
Q4 : Could the authors report TPR at low FPR values (e.g., 1% and 5%) and, if possible, PR-AUC or precision under imbalanced membership prevalence? The focus on false positives makes these operating points particularly relevant. Related RAG membership-inference work such as E-MIA [R1] already reports TPR@FPR and AUC-PR. The 2:1 class-ratio ACC analysis in Appendix C is useful but does not resolve the low-FPR question.
Q5 : How much of the shared-question correctness and membership score remains when the same questions are answered without retrieved context? Although Appendix A asks the auxiliary model to avoid common-knowledge questions, that instruction alone cannot rule out correct answers from the base LLM's parametric knowledge. A no-retrieval/closed-book control, as considered in the related MEntA study [R2], would help determine whether the observed membership signal depends on external retrieval rather than being explained primarily by the base LLM's parametric knowledge.
Q6 : Appendix B describes the 80% member / 20% non-member partition as following prior RAG membership-inference studies and cites Liu et al. (2023). However, the cited paper, "Membership Inference Attacks Against Machine Learning Models via Prediction Sensitivity," concerns conventional training-data membership inference rather than RAG knowledge-base membership. Could the authors correct the reference and explain the actual source or rationale for the 80%/20% partition?
[R1] Zelin Guan, Shengda Zhuo, Zeyan Li, Jinchun He, Wangjie Qiu, Zhiming Zheng, and Shuqiang Huang. "E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems." arXiv:2605.00955, 2026.
[R2] Nguyen Linh Bao Nguyen, Wanlun Ma, Viet Vo, Alsharif Abuadbba, Minghong Fang, Jun Zhang, and Yang Xiang. "Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment." USENIX Security 2026.
6 comments
Q2: Could the authors stratify non-member documents by their maximum semantic similarity to the indexed corpus and report the false-positive rates within each similarity group at matched true-positive rates, together with AUC comparisons using a common set of member samples? In particular, how does QP-MIA compare with the baselines on highly similar or near-duplicate non-members?
Response 2 to Q2:
As suggested, we investigate how semantic similarity between non-member targets and the knowledge base affects attack performance. Specifically, we select 100 non-member target documents and divide them into four equal-sized groups (Q1–Q4) according to their similarity to the knowledge base, in ascending order, with similarity scores spanning approximately [0.85, 1.0]. We evaluate all methods using AUC and PR-AUC, where higher values indicate better performance, and FPR@90% and FPR@95%, where lower values indicate better performance at fixed true-positive rates (TPRs) of 90% and 95%, respectively. We report the results on HealthCareMagic in Table 2 while the corresponding results on TREC-COVID and MS-MARCO are provided in Appendix Section C.1, under the heading Similarity-Stratified Evaluation.
As shown in Table 2, QP-MIA consistently achieves higher AUC and PR-AUC, as well as lower FPR@90% and FPR@95%, than the competing methods across the similarity groups. As semantic similarity increases, the performance of all methods generally degrades, reflecting the increasing difficulty of distinguishing non-member targets from documents supported by semantically similar knowledge-base content. Nevertheless, QP-MIA consistently outperforms the baselines, demonstrating its advantage in mitigating similarity-induced false positives and distinguishing non-member documents even under substantial semantic overlap. We include this analysis in Appendix Section C.1, under the heading Similarity-Stratified Evaluation.
Table 2. Performance comparison across similarity quartiles on HealthCareMagic using Mistral.
| Layer | Method | AUC | PR-AUC | FPR@90% | FPR@95% |
|---|---|---|---|---|---|
| Q1 | QP-MIA | 0.9604 | 0.9905 | 0.0000 | 0.3200 |
| DCMI | 0.9100 | 0.9604 | 0.0800 | 0.5400 | |
| E-MIA-J | 0.9034 | 0.9752 | 0.4400 | 0.6000 | |
| E-MIA-I | 0.6584 | 0.9008 | 0.7600 | 0.8133 | |
| Q2 | QP-MIA | 0.9600 | 0.9901 | 0.0800 | 0.2200 |
| DCMI | 0.8300 | 0.9237 | 0.2400 | 0.6200 | |
| E-MIA-J | 0.9234 | 0.9802 | 0.3200 | 0.4800 | |
| E-MIA-I | 0.6542 | 0.8932 | 0.8800 | 0.9067 | |
| Q3 | QP-MIA | 0.9586 | 0.9895 | 0.1200 | 0.2800 |
| DCMI | 0.7700 | 0.8982 | 0.3600 | 0.6800 | |
| E-MIA-J | 0.8590 | 0.9597 | 0.5200 | 0.6400 | |
| E-MIA-I | 0.5864 | 0.8717 | 0.8400 | 0.9200 | |
| Q4 | QP-MIA | 0.9476 | 0.9873 | 0.1200 | 0.4000 |
| DCMI | 0.7900 | 0.9065 | 0.3200 | 0.6600 | |
| E-MIA-J | 0.8886 | 0.9707 | 0.4800 | 0.6400 | |
| E-MIA-I | 0.6594 | 0.9031 | 0.8000 | 0.8933 |
Q3: Query-composition perturbation changes both the retriever input and the generator prompt, including question order. How much of the observed response variation is due to a change in retrieved evidence rather than a generator-side order/context effect? A controlled comparison with fixed retrieved passages but varied query composition, and conversely fixed evaluation questions but varied retrieved passages, would clarify the source of the signal.
Response 3 to Q3:
As suggested, we conduct two controlled evaluations to separately examine the effects of query composition and retrieved evidence:
(i) Setting A (): We vary the query composition while keeping the retrieved top- passages fixed, thereby examining response variations induced by changes in question composition and ordering under the same retrieval evidence.
(ii) Setting B (): We keep the evaluation questions fixed while varying the retrieved top- passages, thereby examining the impact of changes in retrieved evidence on answer correctness.
We define the answer flip rate as the proportion of answers whose correctness status changes between the two conditions (i.e., from correct to incorrect or vice versa). The results are reported in Tables 3.
For Setting A, we observe relatively low answer flip rates for both member and non-member targets. In particular, the correctness of shared questions, which appear in both query compositions, remains largely unchanged when the query composition is perturbed. This is because the fixed top- passages, originally retrieved for one of the query compositions, continue to provide supporting evidence for these shared questions. Thus, although changes in query composition may affect the generator's input context, they do not substantially alter answer correctness when the relevant retrieval evidence is preserved.
For Setting B, replacing the original retrieved passages with randomly selected texts leads to a substantially higher answer flip rate for member targets. In particular, many answers that are initially correct become incorrect due to the removal of relevant supporting evidence. For non-member targets, although changing the retrieved passages also increases the flip rate, the increase is comparatively smaller. This is likely because the original retrieved passages provide relatively limited evidence for non-member targets, resulting in fewer initially correct answers that can become incorrect after replacing the retrieval context.
Overall, these results suggest that retrieved evidence has a more substantial influence on answer correctness than query composition changes alone. When relevant supporting passages are preserved, perturbing the query composition and ordering has a relatively limited effect on the correctness of shared questions. Conversely, removing the supporting retrieval evidence can substantially impair answer correctness, particularly for member targets. This indicates that QP-MIA's query composition perturbations do not merely introduce arbitrary changes to the generator's responses; rather, they examine whether target-relevant evidence can be consistently retrieved and continue to support correct answers under different query compositions. We include this analysis in Appendix Section C.1, under the heading Source of Response Variation.
Table 3. Controlled analysis of answer flip rates under different perturbations.
| Generator | Dataset | Sample Type | ||||
|---|---|---|---|---|---|---|
| Llama | TREC-COVID | Member | 0.1833 | 0.6520 | 0.0800 | 0.4590 |
| Non-member | 0.1533 | 0.2660 | — | — | ||
| HealthCareMagic | Member | 0.1700 | 0.5740 | 0.0933 | 0.3900 | |
| Non-member | 0.1200 | 0.2060 | — | — | ||
| Mistral | TREC-COVID | Member | 0.0733 | 0.6780 | 0.0800 | 0.4410 |
| Non-member | 0.0867 | 0.2040 | — | — | ||
| HealthCareMagic | Member | 0.0867 | 0.6140 | 0.0933 | 0.4000 | |
| Non-member | 0.1000 | 0.1860 | — | — |
Q1. With (\lambda_2=2), (S_{\mathrm{stab}}) is algebraically equal to the total number of correct shared-question responses. What additional membership information is intended to be captured by the cross-query consistency term beyond repeated correctness? Could the authors compare the proposed score with a simple correct-answer count and analyze whether (C^{both}) remains discriminative after controlling for total correctness? As a minor implementation clarification, the released MIA.py weights (C^{uni}) by 1.5, whereas Section 3.3 assigns it a coefficient of 1. Which version produced the reported results?
Response 1 to Q1:
Our accuracy score already accounts for the correctness of all questions across the original and perturbed query compositions. In addition, the cross-query consistency score assigns an extra reward to shared questions that are correctly answered under both compositions, thereby explicitly emphasizing consistent answer correctness. Our motivation is to highlight whether the same target-specific facts can be consistently recovered when the query composition changes, rather than relying solely on the correctness of individual responses. In particular, a shared question answered correctly under both compositions receives a higher reward than one answered correctly under only one composition.
To investigate the contribution of these signals, we compare QP-MIA against two variants that rely solely on the number of correctly answered questions from a single joint query: (i) a joint query covering five questions, matching the number of questions in each of our original and perturbed query compositions, denoted as 5 Questions; and (ii) a joint query covering ten questions, approximately matching the total number of questions covered by the two compositions, denoted as 10 Questions.
Table 1 reports the AUC results of all variants using Llama, Mistral, and Qwen across TREC-COVID and HealthCareMagic. The results show that neither single-query variant consistently outperforms QP-MIA. In contrast, QP-MIA achieves the highest AUC. These findings suggest that simply relying on the number of correct answers, even when increasing the number of questions in a joint query, is insufficient to achieve comparable membership inference performance. In contrast, evaluating responses across different query compositions provides complementary membership signals. By jointly leveraging answer correctness and cross-query consistency, QP-MIA achieves more effective membership inference. We add this ablation analysis to Section 4.2 of the main paper, under the heading Comparison of Different Membership Signals.
The coefficient λ1 applies to both max(Cori,Cptb) and Cuni. Thus, when λ1=1.5, the coefficient of Cuni is also 1.5. For better clarity, we have revised the formula for computing Scor in Section 3.3 of the main paper by explicitly writing out the two terms.