acceptodds
Under review as a conference paper at ICLR 2027

When Similarity Misleads: Query Perturbation for Robust Membership Inference in RAG

Abstract

Retrieval-Augmented Generation (RAG) systems enhance LLMs with external knowledge bases, but introduce new privacy risks. Membership inference attacks (MIAs) against RAG systems can exploit black-box access to infer whether a target document is contained in the knowledge base. Existing works typically rely on response accuracy, yet semantically similar non-member documents can also provide sufficient evidence for correct answers, causing false-positive inferences. To address this challenge, we propose QP-MIA, a black-box MIA based on query composition perturbation. QP-MIA constructs joint queries from target-specific fill-in-the-blank questions and perturbs their composition by replacing a subset of questions and reordering their positions, while preserving their connection to the target document. It then contrasts responses to two partially overlapping query compositions and extracts membership signals from changes in answer accuracy. This design distinguishes coherent evidence from genuine members from fragmented evidence provided by semantically similar non-members. Extensive experiments demonstrate an average AUC improvement of 20% over representative baselines across diverse RAG configurations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Membership Inference Attacks, Retrieval-Augmented Generation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ercQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates black-box, document-level membership inference against retrieval-augmented generation (RAG) systems, focusing on false positives caused by semantically similar non-member documents. It proposes QP-MIA, which generates document-specific fill-in-the-blank questions and constructs two partially overlapping joint queries by replacing selected questions and changing their order. The method evaluates the RAG system's textual answers against reference answers and combines answer correctness with repeated correctness on shared questions to produce a membership score. Experiments on TREC-COVID, HealthCareMagic, and MS-MARCO, across different generators, retrievers, and RAG configurations, report substantial AUC improvements over several baselines. Additional analyses examine query composition, retrieval behavior, threshold selection, and representative defenses. The authors conclude that query-composition perturbation provides a robust membership signal under black-box access.

Critical strengths
  • The paper addresses a meaningful challenge in RAG privacy auditing: a non-member document can still yield correct answers when related evidence is present in the knowledge base. The motivation for probing a system under different query compositions is intuitive and relevant to black-box membership inference.

  • The reported empirical gains are encouraging. Table 1 evaluates several generators, retrievers, and RAG configurations across three datasets, and Appendix C includes analyses of retrieval overlap and target retrieval rates. This breadth suggests that the approach is potentially useful beyond a single experimental configuration and warrants further investigation.

Critical weaknesses
  • The proposed cross-query consistency score does not clearly capture consistency beyond repeated answer correctness. In Section 3.3, (S_{\mathrm{stab}}=\lambda_2 C^{both}+C^{part}), with (\lambda_2=2) in Appendix B. Under this setting, (S_{\mathrm{stab}}=2C^{both}+C^{part}) is exactly the total number of correct answers to shared questions across the two queries. Consequently, two response patterns with identical correct-answer counts but substantially different question-level persistence can receive the same score; the final score can also be identical when the per-query totals and unique-question correctness are matched. While Table 8 shows that repeated correct answers are discriminative, it does not establish an additional benefit from modeling cross-query consistency separately from accumulated correctness. Similarly, although Table 3 demonstrates the benefit of the overall partially overlapping query strategy, it does not isolate whether the proposed consistency term contributes discriminative information beyond repeated answer correctness. This gap matters because such consistency is central to the paper's claimed mechanism and technical contribution.

  • The experiments do not directly establish that the reported improvements arise from resolving interference by semantically similar non-member documents, which is the central problem motivating QP-MIA. The principal comparisons use randomly sampled members and non-members and report aggregate AUC/ACC. Appendix C analyzes overlap between retrieved top-k sets, but retrieval-set overlap is not the same as the semantic similarity between a non-member target and its nearest indexed document. The paper therefore does not show whether false positives are specifically reduced on high-similarity or near-duplicate non-members. A targeted similarity-stratified evaluation would be important to substantiate the proposed explanation, rather than only the aggregate effectiveness of the scoring procedure.

Questions for authors

Q1 : With (\lambda_2=2), (S_{\mathrm{stab}}) is algebraically equal to the total number of correct shared-question responses. What additional membership information is intended to be captured by the cross-query consistency term beyond repeated correctness? Could the authors compare the proposed score with a simple correct-answer count and analyze whether (C^{both}) remains discriminative after controlling for total correctness? As a minor implementation clarification, the released MIA.py weights (C^{uni}) by 1.5, whereas Section 3.3 assigns it a coefficient of 1. Which version produced the reported results?

Q2 : Could the authors stratify non-member documents by their maximum semantic similarity to the indexed corpus and report the false-positive rates within each similarity group at matched true-positive rates, together with AUC comparisons using a common set of member samples? In particular, how does QP-MIA compare with the baselines on highly similar or near-duplicate non-members?

Q3 : Query-composition perturbation changes both the retriever input and the generator prompt, including question order. How much of the observed response variation is due to a change in retrieved evidence rather than a generator-side order/context effect? A controlled comparison with fixed retrieved passages but varied query composition, and conversely fixed evaluation questions but varied retrieved passages, would clarify the source of the signal.

Q4 : Could the authors report TPR at low FPR values (e.g., 1% and 5%) and, if possible, PR-AUC or precision under imbalanced membership prevalence? The focus on false positives makes these operating points particularly relevant. Related RAG membership-inference work such as E-MIA [R1] already reports TPR@FPR and AUC-PR. The 2:1 class-ratio ACC analysis in Appendix C is useful but does not resolve the low-FPR question.

Q5 : How much of the shared-question correctness and membership score remains when the same questions are answered without retrieved context? Although Appendix A asks the auxiliary model to avoid common-knowledge questions, that instruction alone cannot rule out correct answers from the base LLM's parametric knowledge. A no-retrieval/closed-book control, as considered in the related MEntA study [R2], would help determine whether the observed membership signal depends on external retrieval rather than being explained primarily by the base LLM's parametric knowledge.

Q6 : Appendix B describes the 80% member / 20% non-member partition as following prior RAG membership-inference studies and cites Liu et al. (2023). However, the cited paper, "Membership Inference Attacks Against Machine Learning Models via Prediction Sensitivity," concerns conventional training-data membership inference rather than RAG knowledge-base membership. Could the authors correct the reference and explain the actual source or rationale for the 80%/20% partition?

[R1] Zelin Guan, Shengda Zhuo, Zeyan Li, Jinchun He, Wangjie Qiu, Zhiming Zheng, and Shuqiang Huang. "E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems." arXiv:2605.00955, 2026.

[R2] Nguyen Linh Bao Nguyen, Wanlun Ma, Viet Vo, Alsharif Abuadbba, Minghong Fang, Jun Zhang, and Yang Xiang. "Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment." USENIX Security 2026.

6 comments

Q1. With (\lambda_2=2), (S_{\mathrm{stab}}) is algebraically equal to the total number of correct shared-question responses. What additional membership information is intended to be captured by the cross-query consistency term beyond repeated correctness? Could the authors compare the proposed score with a simple correct-answer count and analyze whether (C^{both}) remains discriminative after controlling for total correctness? As a minor implementation clarification, the released MIA.py weights (C^{uni}) by 1.5, whereas Section 3.3 assigns it a coefficient of 1. Which version produced the reported results?

Response 1 to Q1:

Our accuracy score already accounts for the correctness of all questions across the original and perturbed query compositions. In addition, the cross-query consistency score assigns an extra reward to shared questions that are correctly answered under both compositions, thereby explicitly emphasizing consistent answer correctness. Our motivation is to highlight whether the same target-specific facts can be consistently recovered when the query composition changes, rather than relying solely on the correctness of individual responses. In particular, a shared question answered correctly under both compositions receives a higher reward than one answered correctly under only one composition.

To investigate the contribution of these signals, we compare QP-MIA against two variants that rely solely on the number of correctly answered questions from a single joint query: (i) a joint query covering five questions, matching the number of questions in each of our original and perturbed query compositions, denoted as 5 Questions; and (ii) a joint query covering ten questions, approximately matching the total number of questions covered by the two compositions, denoted as 10 Questions.

Table 1 reports the AUC results of all variants using Llama, Mistral, and Qwen across TREC-COVID and HealthCareMagic. The results show that neither single-query variant consistently outperforms QP-MIA. In contrast, QP-MIA achieves the highest AUC. These findings suggest that simply relying on the number of correct answers, even when increasing the number of questions in a joint query, is insufficient to achieve comparable membership inference performance. In contrast, evaluating responses across different query compositions provides complementary membership signals. By jointly leveraging answer correctness and cross-query consistency, QP-MIA achieves more effective membership inference. We add this ablation analysis to Section 4.2 of the main paper, under the heading Comparison of Different Membership Signals.

The coefficient λ1\lambda_1 applies to both max⁡(Cori,Cptb)\max(C^{\mathrm{ori}}, C^{\mathrm{ptb}}) and CuniC^{\mathrm{uni}}. Thus, when λ1=1.5\lambda_1=1.5, the coefficient of CuniC^{\mathrm{uni}} is also 1.5. For better clarity, we have revised the formula for computing ScorS_{\mathrm{cor}} in Section 3.3 of the main paper by explicitly writing out the two terms.

Table 1. AUC comparison of different membership scoring strategies.

GeneratorDataset5 Questions10 QuestionsFinal Score SS
LlamaTREC-COVID0.89180.89120.9354
HealthCareMagic0.89190.95390.9442
MistralTREC-COVID0.89480.89870.9277
HealthCareMagic0.90190.95340.9549
QwenTREC-COVID0.90910.87580.9421
HealthCareMagic0.78650.79950.8808

Q2: Could the authors stratify non-member documents by their maximum semantic similarity to the indexed corpus and report the false-positive rates within each similarity group at matched true-positive rates, together with AUC comparisons using a common set of member samples? In particular, how does QP-MIA compare with the baselines on highly similar or near-duplicate non-members?

Response 2 to Q2:

As suggested, we investigate how semantic similarity between non-member targets and the knowledge base affects attack performance. Specifically, we select 100 non-member target documents and divide them into four equal-sized groups (Q1–Q4) according to their similarity to the knowledge base, in ascending order, with similarity scores spanning approximately [0.85, 1.0]. We evaluate all methods using AUC and PR-AUC, where higher values indicate better performance, and FPR@90% and FPR@95%, where lower values indicate better performance at fixed true-positive rates (TPRs) of 90% and 95%, respectively. We report the results on HealthCareMagic in Table 2 while the corresponding results on TREC-COVID and MS-MARCO are provided in Appendix Section C.1, under the heading Similarity-Stratified Evaluation.

As shown in Table 2, QP-MIA consistently achieves higher AUC and PR-AUC, as well as lower FPR@90% and FPR@95%, than the competing methods across the similarity groups. As semantic similarity increases, the performance of all methods generally degrades, reflecting the increasing difficulty of distinguishing non-member targets from documents supported by semantically similar knowledge-base content. Nevertheless, QP-MIA consistently outperforms the baselines, demonstrating its advantage in mitigating similarity-induced false positives and distinguishing non-member documents even under substantial semantic overlap. We include this analysis in Appendix Section C.1, under the heading Similarity-Stratified Evaluation.

Table 2. Performance comparison across similarity quartiles on HealthCareMagic using Mistral.

LayerMethodAUCPR-AUCFPR@90%FPR@95%
Q1QP-MIA0.96040.99050.00000.3200
DCMI0.91000.96040.08000.5400
E-MIA-J0.90340.97520.44000.6000
E-MIA-I0.65840.90080.76000.8133
Q2QP-MIA0.96000.99010.08000.2200
DCMI0.83000.92370.24000.6200
E-MIA-J0.92340.98020.32000.4800
E-MIA-I0.65420.89320.88000.9067
Q3QP-MIA0.95860.98950.12000.2800
DCMI0.77000.89820.36000.6800
E-MIA-J0.85900.95970.52000.6400
E-MIA-I0.58640.87170.84000.9200
Q4QP-MIA0.94760.98730.12000.4000
DCMI0.79000.90650.32000.6600
E-MIA-J0.88860.97070.48000.6400
E-MIA-I0.65940.90310.80000.8933

Q3: Query-composition perturbation changes both the retriever input and the generator prompt, including question order. How much of the observed response variation is due to a change in retrieved evidence rather than a generator-side order/context effect? A controlled comparison with fixed retrieved passages but varied query composition, and conversely fixed evaluation questions but varied retrieved passages, would clarify the source of the signal.

Response 3 to Q3:

As suggested, we conduct two controlled evaluations to separately examine the effects of query composition and retrieved evidence:

(i) Setting A (fAf_A): We vary the query composition while keeping the retrieved top-kk passages fixed, thereby examining response variations induced by changes in question composition and ordering under the same retrieval evidence.

(ii) Setting B (fBf_B): We keep the evaluation questions fixed while varying the retrieved top-kk passages, thereby examining the impact of changes in retrieved evidence on answer correctness.

We define the answer flip rate as the proportion of answers whose correctness status changes between the two conditions (i.e., from correct to incorrect or vice versa). The results are reported in Tables 3.

For Setting A, we observe relatively low answer flip rates for both member and non-member targets. In particular, the correctness of shared questions, which appear in both query compositions, remains largely unchanged when the query composition is perturbed. This is because the fixed top-kk passages, originally retrieved for one of the query compositions, continue to provide supporting evidence for these shared questions. Thus, although changes in query composition may affect the generator's input context, they do not substantially alter answer correctness when the relevant retrieval evidence is preserved.

For Setting B, replacing the original retrieved passages with randomly selected texts leads to a substantially higher answer flip rate for member targets. In particular, many answers that are initially correct become incorrect due to the removal of relevant supporting evidence. For non-member targets, although changing the retrieved passages also increases the flip rate, the increase is comparatively smaller. This is likely because the original retrieved passages provide relatively limited evidence for non-member targets, resulting in fewer initially correct answers that can become incorrect after replacing the retrieval context.

Overall, these results suggest that retrieved evidence has a more substantial influence on answer correctness than query composition changes alone. When relevant supporting passages are preserved, perturbing the query composition and ordering has a relatively limited effect on the correctness of shared questions. Conversely, removing the supporting retrieval evidence can substantially impair answer correctness, particularly for member targets. This indicates that QP-MIA's query composition perturbations do not merely introduce arbitrary changes to the generator's responses; rather, they examine whether target-relevant evidence can be consistently retrieved and continue to support correct answers under different query compositions. We include this analysis in Appendix Section C.1, under the heading Source of Response Variation.

Table 3. Controlled analysis of answer flip rates under different perturbations.

GeneratorDatasetSample TypefAf_AfBf_BfAallf_A^{all}fBallf_B^{all}
LlamaTREC-COVIDMember0.18330.65200.08000.4590
Non-member0.15330.2660——
HealthCareMagicMember0.17000.57400.09330.3900
Non-member0.12000.2060——
MistralTREC-COVIDMember0.07330.67800.08000.4410
Non-member0.08670.2040——
HealthCareMagicMember0.08670.61400.09330.4000
Non-member0.10000.1860——
Reviewer_x9tHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents membership inference attacks (MIAs) against Retrieval-Augmented Generation (RAG) systems. They propose a black-box MIA framework that uses perturbed query composition by constructing joint queries from target-specific fill-in-the-blank questions and changing the composition of these multiple queries or their positions. Then it compares responses to two partially overlapping query compositions to extract membership signals.

Critical strengths
  1. Clear identification of existing works and their comparison to the proposed framework.
  2. Baseline comparison with multiple existing works and ablation based on different components of RAG systems.
Critical weaknesses
  1. I am not convinced that the baseline comparison is done under fair configurations. The paper mentions that all methods use the same RAG configurations, but one question is whether they use comparable question/response budgets. For example, the proposed method uses a composition of multiple queries, so one API call contains significantly more information about the target document than prior work. Using the same number of API calls is therefore insufficient control.

  2. Ablation studies do not clarify whether alleged improvement from this framework stems from answer accuracy or cross-query consistency. Because of this, I am not convinced about the mechanism under which the proposed study distinguishes membership signals from those of non-members that are semantically similar to the target document.

  3. The paper proposes a MIA framework that seems to be a combination of two prior works, DCMI and E-MIA. The paper has limited novelty, and its contributions must be justified by well-established and grounded improvements, which current experiments do not support.

Questions for authors
  1. Could you clarify whether the baselines use comparable question and response budgets? If not, how would the results change under the matched question and response budgets?

  2. Could you provide ablations that isolate two components of the MIA framework (answer correctness and cross-query consistency)? Please run additional experiments to support that query composition perturbations provide an additional membership signal beyond answer correctness.

  3. How robust is this MIA framework (QP-MIA) when non-member documents contain information that overlaps substantially with member documents? For example, what happens if information x is contained in both the member and non-member documents (so that RAG can answer a question about x correctly regardless of membership)?

5 comments

Q1: Could you clarify whether the baselines use comparable question and response budgets? If not, how would the results change under the matched question and response budgets?

Response 1 to Q1:

As suggested, Table 1 evaluates the attack cost by considering the numbers of input and output tokens and by using the official GPT-4o pricing to compute the total cost per attack, instead of considering only the number of LLM-API calls.

It is observed that QP-MIA incurs 0.53 cents per target document. Its cost is approximately 16% lower than that of DCMI, 69% lower than that of MEntA and 80% lower than that of E-MIA-I. Although QP-MIA costs slightly more than RAG-MIA and E-MIA-J, its cost remains within a comparable range to these representative approaches.

Regarding input tokens, QP-MIA requires only two API calls, meaning that the retrieved top-kk passages are submitted only twice. Moreover, it constructs target-specific fill-in-the-blank questions using selected information from the target document, rather than repeatedly submitting the entire document as in DCMI and RAG-MIA. Regarding output tokens, QP-MIA asks the target model to complete specific blanks, producing concise answers instead of requiring lengthy free-form responses as in MEntA. These results demonstrate that QP-MIA achieves its improved membership inference performance without relying on a larger monetary API budget. We add this cost analysis to Section 4.1 of the main paper, under the heading Attack Cost Analysis.

Table 1: Comparison of API token consumption and monetary cost across different methods.

MethodAvg. Input TokensAvg. Output TokensPer-Document Cost (¢)
DCMI2,51520.63
RAG-MIA1,259120.33
E-MIA-I10,330292.61
E-MIA-J1,561610.45
MEntA6,2271411.70
QP-MIA (Ours)1,934500.53

Q2: Could you provide ablations that isolate two components of the MIA framework (answer correctness and cross-query consistency)? Please run additional experiments to support that query composition perturbations provide an additional membership signal beyond answer correctness.

Response 2 to Q2:

We conduct ablations that use only answer correctness or only consistency as the membership signal. Specifically, we compare against variants that rely solely on the correctness of responses to (i) a single joint query covering 5 questions (matching the number of questions covered by one of our query compositions), labeled 5 Questions, and (ii) a single joint query covering 10 questions (roughly matching the number of questions covered by two of our query compositions), labeled 10 Questions. We also compare against a variant that uses only the consistency score, labeled Consi.

Table 2 evaluates all variants with Llama, Mistral, and Qwen across TREC-COVID and HealthCareMagic. The results demonstrate that neither relying on a single query, even when increasing the number of questions it covers, nor using consistency alone outperforms our method. In contrast, the two signals derived from query composition perturbation capture complementary membership information. By combining answer correctness and cross-query consistency, QP-MIA achieves the highest AUC in five evaluated configurations. We add this ablation analysis to Section 4.2 of the main paper, under the heading Comparison of Different Membership Signals.

Table 2. AUC comparison of different membership scoring strategies.

GeneratorDataset5 Questions10 QuestionsConsiFinal Score SS
LlamaTREC-COVID0.89180.89120.91710.9354
HealthCareMagic0.89190.95390.92010.9442
MistralTREC-COVID0.89480.89870.91030.9277
HealthCareMagic0.90190.95340.94090.9549
QwenTREC-COVID0.90910.87580.91600.9421
HealthCareMagic0.78650.79950.85950.8808

Q3: How robust is this MIA framework (QP-MIA) when non-member documents contain information that overlaps substantially with member documents? For example, what happens if information x is contained in both the member and non-member documents (so that RAG can answer a question about x correctly regardless of membership)?

Response 3 to Q3:

We evaluate QP-MIA on challenging target documents from two complementary perspectives.

(i) Evaluation on challenging target documents. Specifically, we submit two groups of target-specific questions, each containing five questions, to the RAG system and select target documents for which the system correctly answers at least three questions in either group. These cases pose particular challenges to correctness-based membership inference, as even non-member targets may receive sufficient supporting evidence from semantically related documents. We then examine the distributions of the final membership score SS for members and non-members within this selected subset.

Table 3 presents the distributions of member and non-member samples across different score ranges. QP-MIA achieves a clear separation on HealthCareMagic, with 83.7% of member samples scoring at least 14, compared with 0% of non-members. On TREC-COVID, although the separation is less pronounced, 96% of members reach this score range, compared with 43.3% of non-members. Nevertheless, QP-MIA still demonstrates better discrimination than DCMI. As shown in Table 4, 97% of member samples and 79% of non-member samples receive a score of 1 under DCMI, indicating a substantially smaller gap between the two groups and thus weaker membership discrimination. These results further demonstrate QP-MIA's effectiveness, even in challenging high-correctness cases. We include this analysis in Appendix Section C.1, under the heading Evaluation on High-Correctness Cases.

Table 3. Membership score distributions of QP-MIA under Mistral.

DatasetSample Type[8,10)[10,12)[12,14)≥14
TREC-COVIDMember0.0000.0200.0200.960
Non-member0.2670.2000.1000.433
HealthCareMagicMember0.0440.0330.0860.837
Non-member0.8470.0770.0770.000

Table 4. Binary classification results of DCMI under Mistral.

DatasetSample Type01
TREC-COVIDMember0.0300.970
Non-member0.2070.793
HealthCareMagicMember0.1000.900
Non-member0.6920.308

(ii) Similarity-stratified evaluation on non-member documents. We further investigate how semantic similarity between non-member targets and the knowledge base affects attack performance. Specifically, we select 100 non-member target documents and divide them into four equal-sized groups (Q1–Q4) according to their similarity to the knowledge base, in ascending order, with similarity scores spanning approximately [0.85, 1.0]. For each group, we evaluate the attack performance using the corresponding non-member targets together with the same set of member documents, ensuring a consistent member reference across all groups. We assess all methods using AUC and PR-AUC, where higher values indicate better performance, and FPR@90% and FPR@95%, which measure the false-positive rates at fixed true-positive rates (TPRs) of 90% and 95%, respectively, with lower values indicating better performance.

Due to space constraints, we report the results on HealthCareMagic in Table 5 while the corresponding results on TREC-COVID and MS-MARCO are provided in Appendix Section C.1 under the heading Similarity-Stratified Evaluation. As shown in Table 5, QP-MIA consistently achieves higher AUC and PR-AUC, as well as lower FPR@90% and FPR@95%, than the competing methods across the similarity groups. As semantic similarity increases, the performance of all methods generally degrades, reflecting the increasing difficulty of distinguishing non-member targets from documents supported by semantically similar knowledge-base content. Nevertheless, QP-MIA consistently outperforms the baselines, demonstrating its advantage in mitigating similarity-induced false positives and distinguishing non-member documents even under substantial semantic overlap. We include this analysis in Appendix Section C.1, under the heading Similarity-Stratified Evaluation.

Table 5. Performance comparison across similarity quartiles on HealthCareMagic using Mistral.

LayerMethodAUCPR-AUCFPR@90%FPR@95%
Q1QP-MIA0.96040.99050.00000.3200
DCMI0.91000.96040.08000.5400
E-MIA-J0.90340.97520.44000.6000
E-MIA-I0.65840.90080.76000.8133
Q2QP-MIA0.96000.99010.08000.2200
DCMI0.83000.92370.24000.6200
E-MIA-J0.92340.98020.32000.4800
E-MIA-I0.65420.89320.88000.9067
Q3QP-MIA0.95860.98950.12000.2800
DCMI0.77000.89820.36000.6800
E-MIA-J0.85900.95970.52000.6400
E-MIA-I0.58640.87170.84000.9200
Q4QP-MIA0.94760.98730.12000.4000
DCMI0.79000.90650.32000.6600
E-MIA-J0.88860.97070.48000.6400
E-MIA-I0.65940.90310.80000.8933
Reviewer_Bb3YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes QP-MIA, a black-box MIA based on query composition perturbation. QP-MIA constructs joint queries from target-specific fill-in-the-blank questions and perturbs their composition by replacing a subset of questions and reordering their positions, while preserving their connection to the target document. It then contrasts responses to two partially overlapping query compositions and extracts membership signals from changes in answer accuracy.

Critical strengths
  • The paper is well motivated, and it addresses an important topic in the research field.

  • The paper includes potential defense evaluation.

Critical weaknesses
  • The paper says "MEntA (Nguyen et al., 2026) further adopts natural-language queries to enable low-cost probing with only a small number of interactions" in the background section, but never includes this one as the baseline method. The paper should make further discussion on this.

  • There is no query cost analysis. MEntA mentions that five queries are enough to launch the attack. What is the performance of this method compared to MEntA on query cost?

  • The paper mentions "80% of the documents are included in the RAG knowledge base as members, while the remaining 20% are held out as non-members and never indexed by the RAG system." Is this a practical evaluation?

Questions for authors

See comments above.

3 comments

Q1: The paper says "MEntA (Nguyen et al., 2026) further adopts natural-language queries to enable low-cost probing with only a small number of interactions" in the background section, but never includes this one as the baseline method. The paper should make further discussion on this.

Response 1 to Q1:

As suggested, we extend our evaluation to compare MEntA with QP-MIA and other baselines across different generators, with the results reported in Table 1. As shown in Table 1, MEntA achieves competitive performance on TREC-COVID with Llama and Mistral, obtaining AUCs of 0.73 and 0.83, respectively. However, its performance varies considerably across generators and datasets. In particular, its AUC drops to 0.68 with Qwen on TREC-COVID and remains relatively low across all three generators on HealthCareMagic. This may be attributed to MEntA's query strategy, which probes the RAG system with one question at a time. On datasets such as HealthCareMagic, where non-member documents may exhibit high semantic similarity to target documents, the retrieved top-kk passages may contain semantically similar documents rather than the target document itself. Such retrieval interference may consequently weaken MEntA's ability to distinguish members from non-members. In contrast, QP-MIA consistently achieves the highest AUC across all evaluated settings, demonstrating more effective and robust membership inference performance.

We further evaluate MEntA under different retrieval modules and RAG frameworks, with the results presented in Tables 2 and 3, respectively. Similar trends are observed, with QP-MIA consistently outperforming all competing methods and achieving the highest AUC across the evaluated settings.

We have also updated the related-work discussion and experimental evaluation to include MEntA, with the corresponding analysis incorporated into Section 4.1 of the revised manuscript.

Table 1: AUC of different methods under varied generators.

DatasetGeneratorRAG-MIAE-MIA-IE-MIA-JDCMIMEntAQP-MIA
TREC-COVIDLlama0.600.740.640.690.730.94
Mistral0.790.760.820.810.830.93
Qwen0.800.810.880.810.680.94
HealthCareMagicLlama0.540.610.730.750.550.94
Mistral0.730.640.890.780.580.95
Qwen0.730.570.810.710.560.88

Table 2: AUC of different methods under varied retrievers.

DatasetRetrieverRAG-MIAE-MIA-IE-MIA-JDCMIMEntAQP-MIA
TREC-COVIDE50.600.740.640.690.730.94
Contriever0.540.760.620.660.700.91
HealthCareMagicE50.540.610.730.750.550.94
Contriever0.540.620.750.750.520.87

Table 3: AUC of different methods under varied RAG frameworks.

DatasetRAG FrameworkRAG-MIAE-MIA-IE-MIA-JDCMIMEntAQP-MIA
TREC-COVIDStd. RAG0.540.760.620.660.700.91
SC-RAG0.570.690.570.610.690.84
Spring0.500.740.600.650.690.91
HealthCareMagicStd. RAG0.540.620.750.750.520.87
SC-RAG0.540.590.670.680.520.82
Spring0.500.600.750.640.540.87

Q2: There is no query cost analysis. MEntA mentions that five queries are enough to launch the attack. What is the performance of this method compared to MEntA on query cost?

Response 2 to Q2:

Intuitively, for each attack, MEntA requires five API calls, each involving one short-answer question, whereas QP-MIA requires only two API calls, each covering five fill-in-the-blank questions. Despite using fewer API calls, QP-MIA achieves higher AUC shown in Tables 1-3, indicating more effective membership inference with fewer interactions with the target LLM.

To provide a more systematic comparison of the attack costs of QP-MIA and MEntA, we further measure the numbers of input and output tokens consumed by each method and estimate the total cost per attack based on the official GPT-4o API pricing, rather than relying solely on the number of API calls. Table 4 shows that MEntA consumes approximately three times as many input tokens as QP-MIA. This is mainly because MEntA requires five separate API calls, each of which includes the retrieved top-kk passages in the LLM input, resulting in substantial repeated context overhead. MEntA also generates nearly three times as many output tokens. This is largely due to its short-answer question format, which generally requires longer responses than the fill-in-the-blank format adopted by QP-MIA.

Notably, although QP-MIA covers ten questions across its two API calls, its fill-in-the-blank format keeps the output considerably more concise than MEntA's five short-answer responses. Consequently, the total API cost of MEntA is also approximately three times that of QP-MIA. These results demonstrate that QP-MIA achieves better membership inference performance while requiring substantially fewer API interactions and a lower monetary budget. We have added this cost analysis to Section 4.1 of the revised manuscript under the heading Attack Cost Analysis.

Table 4: Comparison of API token consumption and monetary cost across different methods.

MethodAvg. Input TokensAvg. Output TokensPer-Document Cost (¢)
MEntA6,2271411.70
QP-MIA (Ours)1,934500.53

Q3: The paper mentions "80% of the documents are included in the RAG knowledge base as members, while the remaining 20% are held out as non-members and never indexed by the RAG system." Is this a practical evaluation?

Response 3 to Q3:

We have identified several prior studies on RAG knowledge-base membership inference that adopt the same 80%/20% member/non-member partition, including the baselines evaluated in our work: DCMI, MBA, RAG-MIA, and S²MIA. We have replaced the inappropriate citation with these relevant references in the revised manuscript.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper!

Comments on the paper

1 comment
Area_Chair_Bge8OpenReview ↗

Reminder to respond to the authors’ rebuttal (ASAP)

Dear Reviewers,

Thank you for your service as reviewers for ICLR 2027. As the discussion deadline approaches, please read the authors’ rebuttal and respond at your earliest convenience. A brief note indicating whether the authors have addressed your concerns would be helpful.

Thank you for your time.

Best regards,
Area Chair Bge8

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.