- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problems of existing private RAG (DP-RAG) solutions, which directly privatize document text and fail to distinguish the types of evidence required for queries, easily leading to unnecessary privacy leaks and loss of internal record correlation information. It proposes the GEPI method. This method categorizes queries into four types (K/P/CP/Case) and specifically handles P/CP group statistical problems: it pre-defines the correlation indicators to be statistically analyzed, encodes documents into bounded contribution values, and uses document-level differential privacy aggregation to obtain a global statistical summary. The LLM (Local Level Management) only answers based on this summary and no longer accesses the original private document. The paper demonstrates through dataset auditing, theoretical fidelity bounds, answer quality experiments, and record completion attack experiments that, in group queries in medical and legal scenarios, given a privacy budget, GEPI provides higher answer quality than the baseline and does not significantly increase the success rate of record completion attacks.
Critical strengths
- The paper introduces a useful taxonomy (K/P/CP/Case) to categorize queries by evidence needs, highlighting a key limitation of existing DP-RAG: retrieving raw records may expose unnecessary private information.
- Empirical audits convincingly demonstrate that relational links inside each record, not isolated terms, carry most evidence for clinical and legal population queries.
- It derives an elegant fidelity bound characterizing inherent privacy-fidelity tradeoffs for a class of Case queries under document-level DP.
- For population-style queries, GEPI achieves strong numerical fidelity and allows repeated inference without re-consuming privacy budget.
Critical weaknesses
-
The differences, advantages, and disadvantages between GEPI and DP-RAG are not clearly discussed.** The paper does not systematically compare the suitable scenarios and trade-offs of GEPI and DP-RAG. GEPI sacrifices retrieval capabilities for better privacy-utility performance in group statistical tasks; however, the paper doesn't adequately discuss the costs of this trade-off, potentially leading readers to a one-sided understanding of GEPI's advantages.
-
A significant portion revolves around RAGs, but it's essentially about DP data publishing, not RAGs.** The introduction and motivation sections extensively discuss private RAGs, claiming a re-formulation of them, giving readers the expectation of "improved RAGs." However, GEPI completely removes the retrieval module and doesn't support queries requiring single original documents. Essentially, it's a differential privacy data publishing solution for statistical question answering, creating a misalignment between the narrative and the actual system's positioning, potentially misleading readers.
-
Insufficient experimental coverage, lacking necessary ablation experiments.** All tests are conducted only on manually selected group statistical queries, completely omitting case-type queries, making it impossible to verify the complete query routing assumption. The lack of ablation makes it difficult to determine whether the performance improvement comes from preserving intra-record relationships or from preferential task settings.
-
Limited generalization ability of the conclusions.** Some empirical conclusions are based on SearchQA and SQL datasets, which differ from medical and legal natural language question answering scenarios. The theoretical fidelity relies on strong assumptions, applicable only to a small subset of cases and not directly generalizable to all private document question answering scenarios.
-
Technically low difficulty, limited overall innovation.** The core idea is essentially pre-defined statistics, document-level DP aggregation, and then LLM to generate natural language answers. Query classification and routing remain conceptual and haven't been implemented. The overall methodology and components are relatively conventional, offering limited innovative contributions to DP-RAG.
-
Privacy security experiments suffer from a significant flaw: unequal privacy budgets.** GEPI publishes statistical summaries once, consuming the privacy budget only once; however, the per-query baseline consumes the privacy budget with each query, resulting in a substantial increase in the total budget after multiple queries, making fair horizontal comparisons of leakage risks impossible. Furthermore, the attack experiments have a small sample size and use only a single type of attacker, resulting in insufficiently convincing conclusions.
Questions for authors
- Please clarify the fundamental difference between your method and standard DP-RAG, and systematically discuss their respective advantages, disadvantages and applicable scenarios. GEPI abandons retrieval entirely. Why do you frame the paper heavily around DP-RAG if it is essentially a DP data release system rather than a RAG method? How should readers understand this positioning mismatch?
- Your experiments only test manually selected population (P/CP) queries and never evaluate Case queries. Could you add ablation experiments to disentangle whether the performance gain comes from preserving relational evidence, or simply from the task setup that favors methods outputting direct aggregate statistics?
- Many of your empirical findings are based on SearchQA and ClinSQL/Korean LegalQA structured SQL tasks. Could you discuss to what extent these conclusions generalize to real-world unstructured medical and legal natural-language queries? Also, please clarify the scope of Proposition 1’s disjoint-support assumption and explain why the paper’s presentation does not overclaim this bound for all Case queries.
- The privacy leakage comparison in Table 2 suffers from unmatched privacy budgets: GEPI consumes the DP budget once for global release, while per-query DP accumulates a very large total ε across 87 queries. Can you provide a fair comparison under matched total privacy budget? Additionally, why only a single non-adaptive record-completion attacker is used? Could you add stronger adaptive attack evaluations to better assess leakage risk?
- Your main LLM-as-judge metric anchors scoring against exact aggregate ground truth. This design structurally favors methods outputting numerical statistics while penalizing retrieval-based baselines which need to perform counting over raw text. Can you report intra-judge repeatability and inter-annotator agreement metrics to validate the rubric reliability? Also, how does this metric distinguish hallucinated explanations with correct numbers from grounded answers with minor counting mistakes?
Thank you for your questions about GEPI's scope and comparisons. For concerns not addressed below, please see our common response (C1–C3).
W6 & Q4. Whether the attack comparison matches the total privacy budget
The requested matched-total-budget comparison is already in Table 2. The full 87-event raw-context transcript satisfies (3.99,1/3000)-DP and GEPI's release satisfies (4,1/3000)-DP. Both therefore fit the same (4,1/3000) bound for the outputs shown to the attacker. The separate per-query row is not this comparator. Neither matched condition shows an observed AUC increase under the specified reader (Sections 5.4 and H).
W4 & Q3. Audit generalization and Proposition 1's scope
Korean LegalQA is a legal Q&A collection, not a structured SQL task (Table 20). The audits examine evidence requirements; Table 1 evaluates answer utility on LegalM and LeakRAG text. Proposition 1 applies when adjacent corpora have disjoint correct-answer supports; the Case label alone is not its premise (Section 2.3).
W1–W2 & Q1. Why frame a statistical release around private RAG?
The paper asks what evidence a private-RAG answer needs. For P/CP requests, GEPI supplies that evidence as released cohort–outcome statistics instead of retrieved passages. The release is reusable; requests for a designated record require separately authorized source access. Sections 1 and 4.3 state this division; C2 tests relation preservation and workload sharing.