acceptodds
Under review as a conference paper at ICLR 2027

GEPI: Differentially Private Relational Evidence Release for Language Generation

Abstract

Differentially private retrieval-augmented generation (RAG) aims to answer questions using sensitive corpora while limiting the influence of any individual record on its outputs. Yet retrieved documents often expose a finer granularity of information than an answer requires. This raises a central question: *what evidence should a system access and release to support an answer?* We distinguish queries seeking non-statistical facts or rules, population patterns, case-conditioned population evidence, and specific records, including combinations of these needs. Together with access policy, these distinctions separate when an answer can rely on public evidence, when privacy-protected aggregate evidence is appropriate, and when authorized record-level access is required. In a preliminary GPT-based audit, relational representations preserve all required source-linked evidence for 88.3% of clinical and 88.8% of legal queries. Removing relationships while retaining the same terms and values reduces these rates to 30.0% and 61.3%, respectively. These results motivate GEPI (Global Evidence for Private Statistical Inference), which separates private evidence release from language generation. GEPI predefines relational measurements for a supported workload, aggregates document-local evidence into corpus-level differentially private statistics, and lets a language model answer from the released synopsis rather than source documents. The same synopsis can support repeated answers without reopening protected documents or incurring additional privacy cost. We compare GEPI with privacy-preserving release baselines at matched privacy budgets, measuring alignment with gold answers and attribute inference from released evidence. Our experiments show that preserving relationships is critical for private evidence, motivating relational statistics as the foundation of GEPI.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, retrieval-augmented generation, large language models, privacy-preserving natural language processing, differentially private statistics, attribute inference attack, question answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MLR4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problems of existing private RAG (DP-RAG) solutions, which directly privatize document text and fail to distinguish the types of evidence required for queries, easily leading to unnecessary privacy leaks and loss of internal record correlation information. It proposes the GEPI method. This method categorizes queries into four types (K/P/CP/Case) and specifically handles P/CP group statistical problems: it pre-defines the correlation indicators to be statistically analyzed, encodes documents into bounded contribution values, and uses document-level differential privacy aggregation to obtain a global statistical summary. The LLM (Local Level Management) only answers based on this summary and no longer accesses the original private document. The paper demonstrates through dataset auditing, theoretical fidelity bounds, answer quality experiments, and record completion attack experiments that, in group queries in medical and legal scenarios, given a privacy budget, GEPI provides higher answer quality than the baseline and does not significantly increase the success rate of record completion attacks.

Critical strengths
  1. The paper introduces a useful taxonomy (K/P/CP/Case) to categorize queries by evidence needs, highlighting a key limitation of existing DP-RAG: retrieving raw records may expose unnecessary private information.
  2. Empirical audits convincingly demonstrate that relational links inside each record, not isolated terms, carry most evidence for clinical and legal population queries.
  3. It derives an elegant fidelity bound characterizing inherent privacy-fidelity tradeoffs for a class of Case queries under document-level DP.
  4. For population-style queries, GEPI achieves strong numerical fidelity and allows repeated inference without re-consuming privacy budget.
Critical weaknesses
  1. The differences, advantages, and disadvantages between GEPI and DP-RAG are not clearly discussed.** The paper does not systematically compare the suitable scenarios and trade-offs of GEPI and DP-RAG. GEPI sacrifices retrieval capabilities for better privacy-utility performance in group statistical tasks; however, the paper doesn't adequately discuss the costs of this trade-off, potentially leading readers to a one-sided understanding of GEPI's advantages.

  2. A significant portion revolves around RAGs, but it's essentially about DP data publishing, not RAGs.** The introduction and motivation sections extensively discuss private RAGs, claiming a re-formulation of them, giving readers the expectation of "improved RAGs." However, GEPI completely removes the retrieval module and doesn't support queries requiring single original documents. Essentially, it's a differential privacy data publishing solution for statistical question answering, creating a misalignment between the narrative and the actual system's positioning, potentially misleading readers.

  3. Insufficient experimental coverage, lacking necessary ablation experiments.** All tests are conducted only on manually selected group statistical queries, completely omitting case-type queries, making it impossible to verify the complete query routing assumption. The lack of ablation makes it difficult to determine whether the performance improvement comes from preserving intra-record relationships or from preferential task settings.

  4. Limited generalization ability of the conclusions.** Some empirical conclusions are based on SearchQA and SQL datasets, which differ from medical and legal natural language question answering scenarios. The theoretical fidelity relies on strong assumptions, applicable only to a small subset of cases and not directly generalizable to all private document question answering scenarios.

  5. Technically low difficulty, limited overall innovation.** The core idea is essentially pre-defined statistics, document-level DP aggregation, and then LLM to generate natural language answers. Query classification and routing remain conceptual and haven't been implemented. The overall methodology and components are relatively conventional, offering limited innovative contributions to DP-RAG.

  6. Privacy security experiments suffer from a significant flaw: unequal privacy budgets.** GEPI publishes statistical summaries once, consuming the privacy budget only once; however, the per-query baseline consumes the privacy budget with each query, resulting in a substantial increase in the total budget after multiple queries, making fair horizontal comparisons of leakage risks impossible. Furthermore, the attack experiments have a small sample size and use only a single type of attacker, resulting in insufficiently convincing conclusions.

Questions for authors
  1. Please clarify the fundamental difference between your method and standard DP-RAG, and systematically discuss their respective advantages, disadvantages and applicable scenarios. GEPI abandons retrieval entirely. Why do you frame the paper heavily around DP-RAG if it is essentially a DP data release system rather than a RAG method? How should readers understand this positioning mismatch?
  2. Your experiments only test manually selected population (P/CP) queries and never evaluate Case queries. Could you add ablation experiments to disentangle whether the performance gain comes from preserving relational evidence, or simply from the task setup that favors methods outputting direct aggregate statistics?
  3. Many of your empirical findings are based on SearchQA and ClinSQL/Korean LegalQA structured SQL tasks. Could you discuss to what extent these conclusions generalize to real-world unstructured medical and legal natural-language queries? Also, please clarify the scope of Proposition 1’s disjoint-support assumption and explain why the paper’s presentation does not overclaim this bound for all Case queries.
  4. The privacy leakage comparison in Table 2 suffers from unmatched privacy budgets: GEPI consumes the DP budget once for global release, while per-query DP accumulates a very large total ε across 87 queries. Can you provide a fair comparison under matched total privacy budget? Additionally, why only a single non-adaptive record-completion attacker is used? Could you add stronger adaptive attack evaluations to better assess leakage risk?
  5. Your main LLM-as-judge metric anchors scoring against exact aggregate ground truth. This design structurally favors methods outputting numerical statistics while penalizing retrieval-based baselines which need to perform counting over raw text. Can you report intra-judge repeatability and inter-annotator agreement metrics to validate the rubric reliability? Also, how does this metric distinguish hallucinated explanations with correct numbers from grounded answers with minor counting mistakes?
1 comment

Thank you for your questions about GEPI's scope and comparisons. For concerns not addressed below, please see our common response (C1–C3).


W6 & Q4. Whether the attack comparison matches the total privacy budget

The requested matched-total-budget comparison is already in Table 2. The full 87-event raw-context transcript satisfies (3.99,1/3000)(3.99,1/3000)-DP and GEPI's release satisfies (4,1/3000)(4,1/3000)-DP. Both therefore fit the same (4,1/3000)(4,1/3000) bound for the outputs shown to the attacker. The separate per-query row is not this comparator. Neither matched condition shows an observed AUC increase under the specified reader (Sections 5.4 and H).

W4 & Q3. Audit generalization and Proposition 1's scope

Korean LegalQA is a legal Q&A collection, not a structured SQL task (Table 20). The audits examine evidence requirements; Table 1 evaluates answer utility on LegalM and LeakRAG text. Proposition 1 applies when adjacent corpora have disjoint correct-answer supports; the Case label alone is not its premise (Section 2.3).

W1–W2 & Q1. Why frame a statistical release around private RAG?

The paper asks what evidence a private-RAG answer needs. For P/CP requests, GEPI supplies that evidence as released cohort–outcome statistics instead of retrieved passages. The release is reusable; requests for a designated record require separately authorized source access. Sections 1 and 4.3 state this division; C2 tests relation preservation and workload sharing.

Reviewer_SzBDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a taxonomy of query evidence requirements: Knowledge (K), Population (P), Case–population (CP), and Case. It proposes GEPI to provide differentially private population evidence for P and CP queries. Using predeclared measurement contracts, GEPI encodes within-document relations as bounded contributions and aggregates them into a corpus-level statistical vector. It then applies the Gaussian mechanism to release a reusable synopsis. An LLM answers supported queries using evidence derived from this synopsis. The method is evaluated on population workloads constructed from LegalM and LeakRAG datasets.

Critical strengths
  1. GEPI provides explicit population evidence for P and CP queries. Using predeclared measurement contracts, it encodes within-document relations as bounded document-level contributions and aggregates them into a differentially private synopsis that supports LLM-based answer generation.
Critical weaknesses
  1. The theoretical analysis provides limited insight into GEPI’s design. Proposition 1 establishes a general limitation on privacy and answer fidelity and motivates distinguishing evidence requirements. However, it does not establish the advantages of GEPI’s relational encoding, shared release, or treatment of unknown states over other mechanisms. The relevant theoretical quantities are not instantiated in the experiments. The contribution therefore remains primarily motivational, without adequately explaining the specific design choices or observed performance gains.
  2. GEPI’s practical applicability remains insufficiently validated. The experiments primarily evaluate population workloads with supplied measurement bindings, without validating the mapping of natural P and CP requests to measurement contracts and subsequent end-to-end answering. New cohorts, outcomes, time windows, or conjunctions absent from the existing synopsis may require additional access to the protected corpus and further privacy expenditure. The paper does not quantify answerable coverage for natural requests, generalization to unseen measurement families, or the trade-offs among coverage, statistical error, and privacy cost as the workload expands.
  3. The experimental evaluation is insufficient. Comparisons with existing DP-RAG methods are limited; representative methods such as those in [1–3] are not evaluated. Evaluation on broader RAG tasks beyond population workloads is also lacking. Table 1 reports only LLM-judge scores and lacks more diverse evaluation metrics.
  4. The presentation of the core mechanism is insufficiently self-contained. The writing is unclear and difficult to follow. For example, the expression at line 184 does not explain EE, Figure 3 provides insufficient explanation of Ew(d)E_w(d) and Pw(d)P_w(d), and Section 4.1 does not clearly explain how document states are encoded into contribution vectors or what these vectors look like. The paper is also poorly organized, with redundant discussion before the method section and insufficiently concise writing.

[1] Mori J, Kakizaki K, Miyagawa T, et al. "Differentially private synthetic text generation for retrieval-augmented generation (rag)." Findings of ACL 2026.

[2] Joo S, Koh H, Jung K. "PEARL: Differentially Private and Entropy-Aware Regulated Language Generation." ICML 2026.

[3] Koga T, Wu R, Zhang Z, et al. "Privacy-preserving retrieval-augmented generation with differential privacy." arXiv preprint 2025.

Questions for authors

See weakness

3 comments

We appreciate your comments on the theory, applicability, and presentation. For concerns not addressed below, please see our common response (C1–C3).


W1. How the theory supports GEPI's design

Proposition 2 guarantees GEPI's release and reuse; C2 tests relation preservation and workload sharing. Proposition 1 motivates the distinction from document-specific answering: its fidelity bound applies when adjacent corpora require disjoint sets of correct answers. Sections D.1–D.2 give the proofs; Sections E.3 and F.3 give the sharing and relation controls.

W3. Missing DP-RAG baselines and broader evaluation

The comparison is designed to test relational evidence release under controlled conditions. Table 1 includes synthetic text (Aug-PE) and raw-context token-DP RAG (Grislain-style). Evaluating DP-SynRAG, PEARL, and Koga et al. on these declared P/CP workloads requires aligning protected units, total workload budgets, and cohort/denominator readouts. The same-state count-DP and relation-removal controls test our design choices directly (C2); the endpoint audit measures numerical fidelity independently of the judge (C1).

W4. Unclear symbols and contribution-vector encoding

The revised Section 4.1 adds the missing concrete vector example. An eligible document with two outcomes (T,U)(T,U) increments that joint-state coordinate and leaves the other eight eligible coordinates zero; two further coordinates record ineligibility and unknown eligibility. Figure 3's Ew(d)E_w(d) and Pw(d)P_w(d) denote eligibility and outcome states. The EE in Section 2.3's TV definition denotes a measurable answer event.

Thanks for the rebuttal. After reading the rebuttal and the reviews from other reviewers, my concerns remain unresolved:

  • W1: Lack of connection between Proposition 1 and the specific design of GEPI.
  • W2: Lack of end-to-end verification of natural requests; the inability to compare DP-SynRAG, PEARL, and Koga et al. precisely illustrates that GEPI's verification relies on workloads, which are not general RAG tasks and have limited practical applications.
  • W3: Lack of comparison with RAG tasks beyond population workloads.

I decide to remain my score.

Thank you for taking the time to read our rebuttal alongside the other reviews and for explaining your remaining concerns. We sincerely appreciate your continued engagement with our work and the opportunity to clarify these points.


W1. Connection between Proposition 1 and GEPI's design

Lack of connection between Proposition 1 and the specific design of GEPI.

Proposition 1 is connected to GEPI through the choice of evidence to release. It gives a DP fidelity limit when adjacent corpora require disjoint sets of correct answers. This motivates separating designated-record answering from population-evidence release—a distinction reflected in GEPI's scope: designated-record requests need separately authorized source access (Sections 2.3–2.4 and 4.3).

The Introduction accordingly presents this bound as part of the problem reformulation. The mechanism has separate support: Proposition 2 establishes its release-and-reuse privacy guarantee; Sections E.3 and F.3 test workload sharing and relation preservation; Sections 4.1 and 4.3 explain unknown states and the resulting interval. We do not consider Proposition 1's motivational role a weakness: the design has separate formal and experimental support.

W2. End-to-end verification and practical relevance

Lack of end-to-end verification of natural requests; the inability to compare DP-SynRAG, PEARL, and Koga et al. precisely illustrates that GEPI's verification relies on workloads, which are not general RAG tasks and have limited practical applications.

Our claim concerns the relational evidence-release interface, not end-to-end superiority over DP-SynRAG, PEARL, or Koga et al. Consider a defendant asking how drunk-driving cases with prior convictions were sentenced. The offense, prior conviction, and sentence must remain linked within each case, and the answer must reflect the relevant cohort, not isolated passages. Preserving this evidence under DP is a practical problem in legal consultation (Sections 1 and 4.1).

The binding specifies the evidence required, not the answer. It identifies the cohort and outcome without supplying private observations or reference answers (Section 3, Figure 4). Fixing this specification lets us compare what survives each release. The relation-removal and shared-release controls test the interface's design choices directly (Sections F.3 and E.3). Automatic request-to-contract matching is an additional component, not a prerequisite for testing this contribution.

W3. Evaluation beyond population workloads

Lack of comparison with RAG tasks beyond population workloads.

We do not consider focusing on population-evidence release a weakness in itself. Section 2.4 distinguishes answering routes: open evidence when sufficient, separately authorized access to designated records, and DP-released population evidence. GEPI develops the third route (Section 4.3). Comparisons on other RAG task types would extend task coverage; they are not a prerequisite for demonstrating a contribution to population-evidence release.

Reviewer_8RJtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The starting point is that private RAG methods privatize retrieved text without first asking what evidence a question actually needs. The authors propose four categories (knowledge, population, case-population, case), kept separate from access permission, and run three LLM-labelled audits of existing private-RAG benchmarks: medical settings are mostly population questions, SearchQA is mostly answerable from public pages (272/408272/408 fully supported), ContractNLI is all designated-record questions; and on ClinSQL / LegalQA, keeping field names and within-record links preserves most of the required evidence while a bag-of-terms control loses 60 to 75 pp of it. Proposition 1 is a TV lower bound of (1−δ)/(1+eε)(1-\delta)/(1+e^{\varepsilon}) for answers that have to depend on one document. The method, GEPI, has the data holder declare cohort/outcome "measurement contracts" up front, encodes each document as a bounded 0/1 vector x(d)x(d) with ∥x(d)∥2≤B\|x(d)\|_2 \le \sqrt{B}, releases the corpus histogram H=∑dx(d)H = \sum_d x(d) once with Gaussian noise N(0,σ2I)\mathcal{N}(0, \sigma^2 I) under document-level (ε,δ)(\varepsilon, \delta)-DP, and lets the LLM answer from the released counts through a fixed readout (Eq. 6); later questions are post-processing. Experiments use a Korean legal corpus the authors built (LegalM, 3,000 judgments) and LeakRAG medical consultations (1,200), two generators, and request-to-contract bindings that are supplied with each request. Baselines are Tab-PE, Aug-PE, per-query count DP and a Grislain-style raw-context token-DP adapter; the metric is an LLM-judged 1 to 5 "reference alignment" score. GEPI has the highest mean at every finite budget label in all four blocks, lower interval-endpoint error, and no measurable AUC gain in an 87-target record-completion attack.

Critical strengths

The question the paper asks is a good one, and the audits in Section 2 / Appendix G are the part I found useful: two thirds of SearchQA requests can be answered from public pages alone, and the gap between linked and unlinked representations on ClinSQL and LegalQA is large. That is a fair criticism of how private-RAG benchmarks have been assembled.

The design principle (declare bounded relational measurements, release once, everything afterwards is post-processing) is sensible, and the authors are unusually open about their development history, release ledger and protocol deviations (Appendices D.6, D.7, F.7, I.5 to I.7). I would rather have that than the usual polish, even though in this case what it reveals is a problem.

Critical weaknesses
  1. The evaluation is circular and the metric has not been validated. The reference answers are "exact aggregates of frozen case states" produced by GEPI's own extractor, and Appendix I.5 says in so many words that these states "are not independent expert labels". Each request comes with the binding that names the contract GEPI released (Section 3; matching free-form questions to contracts is explicitly outside the evaluation). Tab-PE and count DP are given the same frozen states; Aug-PE and raw-context have to recover the number from text. So the task is built around the method, and a text-based baseline cannot in principle hit a reference that is defined by the method's own extraction. The score is a GPT-5.4-mini rubric judge. The only human check (Appendix C.4) is one lawyer on 37 answers, and all three human-LLM rank-agreement intervals include zero, with the judge about 0.60.6 points more lenient. The Section 2 audits are also model labels with no human validation (G.2). On top of that, an earlier instrument found no separation between GEPI and Tab-PE (I.7, −0.012±0.037-0.012 \pm 0.037), and one cell of Table 1 (Legal-Qwen, ε=1\varepsilon = 1) uses a "typed reader" that was fixed after a 60-request diagnostic, which is why that column goes 3.060/2.949/3.1273.060 / 2.949 / 3.127 as ε\varepsilon increases. I do not know how to read "highest mean at every budget" after all this.

  2. The budgets are not matched and the mechanism is thin. The Table 1 labels are nominal: Appendix D.7 says the legal ε=1/4/8\varepsilon = 1/4/8 releases are really ε=1.33,5.01,9.73\varepsilon = 1.33, 5.01, 9.73 at δ=10−5\delta = 10^{-5}, and the "ε=1\varepsilon = 1" condition is the ε=4\varepsilon = 4 release with extra noise added. The Tab-PE / Aug-PE artifacts "are not formal matched-DP certificates", earlier releases on the same corpus are left out of the ledger, and the raw-context baseline's population gate "fails even without clipping" (Section 5.2, E.6), so it is not really a comparator. What GEPI does mechanically is a Gaussian histogram release with ℓ2\ell_2 sensitivity B\sqrt{B} followed by a ratio readout. Its advantage over per-query count DP (σ≈2.2\sigma \approx 2.2 vs 20.320.3 at the same budget) is the standard gain from releasing one vector instead of composing hundreds of scalar queries, and with exact counts the per-query baseline actually scores higher in three of the four blocks (4.524.52 vs 3.973.97, 4.874.87 vs 4.344.34). Proposition 1 is the usual hypothesis-testing bound for (ε,δ)(\varepsilon, \delta)-DP. The paper is also very hard to read: 47 pages, unusual prose, claims hedged with qualifiers whose meaning is only in the appendix, and a lot of "historical", "superseded" and "corrected after scoring" material.

Questions for authors
  • Can you produce reference answers that do not come from GEPI's extractor, for instance human-annotated cohort counts on a sample, and score against those?

  • Can you get human (lawyer / clinician) scores on enough answers to validate the judge, or use a judge whose agreement with humans has been established?

  • Please report all methods at matched (ε,δ)(\varepsilon, \delta) with certified accounting that includes the full development transcript on each corpus, and use a single readout policy across every cell of a row (or report both readers everywhere). The setup assumes each request arrives with the right contract binding. What happens when the needed conjunction was not declared, how often does that occur on realistic workloads, and what does the end-to-end system do with free-form questions?

  • Why does per-query count DP with exact counts beat GEPI with exact counts in three of four blocks? What does that say about the readout and the formatting?

  • What is LegalM exactly (real judgments with synthetic identifiers?) and what is its redistribution status? Does the "four variants per case" medical rendering interact with the per-document adjacency?

2 comments

Thank you for examining the evaluation and privacy accounting. For concerns not addressed below, please see our common response (C1–C3).


W1. Whether text baselines can reach an extractor-defined reference

The claim that a text baseline “cannot in principle hit” the reference is incorrect. The reference is an exact aggregate of frozen extracted states, fixed before noise and generation, not GEPI's generated answer. Every method is scored against it. When the states reflect the source text, a text method can recover the same aggregate. Tab-PE and count DP receive the same states as GEPI, holding source extraction fixed in those comparisons (Section 5.1).

W2 & Q3. Nominal labels versus matched privacy accounting

The two sets of legal ϵ\epsilon values use different values of δ\delta, not different releases. Labels 1/4/8 use δ=1/3000\delta=1/3000; 1.33/5.01/9.73 express those same releases at δ=10−5\delta=10^{-5} (Sections 5.1 and D.7). Table 1's caption calls the budget labels nominal; Section 5.1 specifies the synthetic baselines' nominal-budget status. The matched-budget utility evidence is the medical count-DP control in C2, which also fixes the states and evidence format and uses partition-aware accounting.

W1 & Q3. Cell-specific reader changes and consistency across budgets

Table R1 in the updated manuscript adds the requested single-reader comparison for Legal–Qwen. On the same 300-request roster, the typed reader scores 3.027/3.287/3.399 at labels 1/4/8 (1–5 scale; 298/296/298 scored requests); the original reader scores 2.568/2.949/3.127. Both single-reader rows increase with the budget label; the non-monotone row mixes readers. Table R1 distinguishes the September and November executions, whose provider routes and output settings differ (Section I.5).

Q4. Higher count-DP scores with exact counts

Equal exact counts do not imply identical generated answers. GEPI and count DP use different evidence packets and readouts (Section E.4). In the medical finite-budget control, GEPI's advantage survives a common evidence format and partition-aware accounting (C2). Formatting alone therefore does not explain that controlled gap.

Q5. Whether medical document variants change GEPI's protected unit

GEPI counts each original medical case once. Its protected units are the 1,200 original consultation cases, rather than the 4,800 rendered variants (Sections A.1 and D.7).

Thank you for the response and for adding Table R1. Two points are resolved: the protected unit in the medical corpus (Q5), and the relationship between the two sets of legal eps values. My main concerns remain.

W1. Reference answers and judge validity. I agree that a text method can recover the aggregate when the frozen states reflect the source. My concern is the reference itself: it aggregates states produced by GEPI’s extractor, which Appendix I.5 says “are not independent expert labels.” Tab-PE and count DP receive those states directly, while Aug-PE and raw-context must re-derive them, so the score partly measures agreement with GEPI’s extraction. The response offers neither an extractor-independent reference nor human validation of the judge. As I read it, the Table 14 audit covers only the medical setting at eps=4 and is conditional on a retained interval, and C.4 (one lawyer, 37 answers, agreement intervals including zero) remains the only human evidence.

W2/Q3. Privacy accounting. Since the legal corpus has 3,000 documents, delta = 1/3000 is delta = 1/n, a regime where the definition admits mechanisms that publish each record outright with probability delta. The delta = 10^-5 values should therefore be the headline budgets. The response also leaves three points unaddressed: the synthetic baselines carry no formal matched certificates, earlier releases on the same corpus are absent from the ledger, and the raw-context gate fails even without clipping. The medical count-DP control is thus the only matched-budget utility evidence, and the legal blocks have none.

Reader consistency (Table R1). Both single-reader rows are monotone, which I accept. However, the typed reader was fixed after a 60-request diagnostic on the evaluation roster, and its gain (up to 0.46 points) is comparable to the gaps between methods. Were the baselines given the same reader? The September and November executions also differ in provider route and output settings, which limits comparability across rows.

It is thus still unanswered that: How often do realistic workloads require undeclared conjunctions, and what happens to free-form questions? What are LegalM’s provenance and redistribution status?

I am keeping my score. I would reconsider given (i) human-annotated reference counts on a sample, (ii) human validation of the judge, and (iii) a single reader fixed before scoring, applied to all methods at δ = 10⁻⁵ with complete ledger accounting.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Shared Reviewer Concerns

We thank the reviewers for their careful reading. We address shared concerns here and specific points in the individual replies. All Section and Table references are to the revised manuscript; additions are orange.

Key evidence and corrections:

  • Existing controls: judge-independent numerical error, relation removal, and matched-budget count DP (C1–C2).
  • New comparison: one reader across all three Legal–Qwen budgets (Table R1; reply to 8RJt).
  • Budget correction: Table 2 already contains a matched-total-budget attack comparison (reply to MLR).

C1. Evidence beyond LLM scores and the rubric's criteria

(8RJt W1; SzBD W3; MLR Q5)

The utility evidence is not limited to LLM scores. The submitted paper's separate medical audit at ϵ=4\epsilon=4 reports mean maximum-endpoint error of 0.078/0.088 for GPT/Qwen, below 40% of either synthetic baseline's error, conditional on a retained interval (Section 5.3, Table 14). The rubric also scores factual correctness, scope, and limitations: correct numbers with materially false explanations do not qualify for its highest score (Section C.1, Table 6).

C2. Contribution beyond standard DP components and the design controls

(8RJt W2; SzBD W1; MLR W3/W5/Q2)

The contribution is a relational evidence interface: explicit cohort denominators, linked outcomes, and unknown states. GEPI releases these bounded measurements using Gaussian noise and post-processing (Section 6). The submitted paper tests the design with two controls:

  • Relations: two constructed corpora have identical relation-stripped summaries but true relation rates of 0.75 and 0.25; those summaries cannot distinguish the corpora. In a separate medical control with public source labels, ignoring subject and time relations adds six false positives to the 12 gold-positive pairs among 69 eligible pairs (Sections 5.3 and F.3).
  • Sharing: the medical GPT control fixes the states, evidence format, and total (ϵ,δ)=(4,10−5)(\epsilon,\delta)=(4,10^{-5}), including parallel composition across disjoint partitions for count DP. On the 1–5 reference-alignment scale, GEPI scores 3.360 versus 2.497, a paired advantage of 0.863±0.1200.863\pm0.120 (cluster 95% CI; Sections 5.2 and E.3).

C3. Supplied bindings and coverage of undeclared queries

(8RJt Q3; SzBD W2; MLR W3/W5)

Bindings hold the requested statistic fixed across methods. The evaluation measures evidence release and answer generation with supplied bindings, rather than automatic contract matching (Section 3). Supported requests reuse the synopsis without additional privacy expenditure; a missing recipe returns UNSUPPORTED. A new protected measurement requires further accounting, while designated-record retrieval requires separately authorized access (Section 4.3, Algorithm 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.