- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DPD, a differential privacy (DP)-based framework for synthesizing demonstrations for in-context learning (ICL) from private datasets. The method uses a task-aligned encoder and DP coverage-aware clustering to retrieve diverse demonstration candidates. It then applies the Joint Exponential Mechanism (JEM) to extract privatized keywords, which guide an LLM to generate synthetic demonstrations without token-level DP perturbation. The authors provide a theoretical demonstration-level DP guarantee and evaluate the method on text classification, information extraction, and mathematical reasoning tasks. Experimental results demonstrate improvements in downstream ICL performance and generation efficiency over existing DP-based synthesis methods.
Critical strengths
-
The paper addresses a practical challenge in privacy-preserving ICL by separating private information extraction from LLM-based demonstration generation. This design enables the use of external LLMs without directly exposing the original private demonstrations.
-
The experiments cover multiple downstream tasks and privacy budgets, showing improvements over existing DP-based synthesis methods. The paper also includes generator-controlled comparisons, ablation studies, and theoretical privacy analysis.
Critical weaknesses
-
Limited evidence of meaningful private information utilization. The proposed method aims to preserve useful information from private demonstrations while ensuring DP. However, Table 7 shows low candidate-hit rates and Top-T keyword recall, while replacing private keywords with randomly sampled keywords reduces average accuracy by only 2.23 percentage points (Table 11). These results raise questions about how much task-specific information is actually extracted from private data, rather than relying on public task descriptions and the LLM's generation capabilities.
-
Insufficient evaluation of sensitive information leakage. Although the paper provides demonstration-level DP guarantees and evaluates membership inference attacks, it does not directly assess whether sensitive information is exposed in the synthesized demonstrations. The exposure-style attack in Appendix G.4 relies on token overlap rather than explicitly measuring the leakage of personally identifiable information or sensitive attributes. Furthermore, the experiments use standard public benchmarks without evaluating datasets containing realistic sensitive records, limiting the practical validation of the privacy claims.
-
Limited methodological novelty. The proposed framework largely combines existing techniques, including task-aligned embeddings, DP-based statistical aggregation, JEM, and LLM-based generation. While their integration is reasonable, the individual components are not fundamentally new, and the technical contribution beyond combining these established techniques appears limited.
-
Insufficient comparison with simpler privacy-preserving synthesis approaches. The main experiments compare DPD primarily against two existing autoregressive DP synthesis methods from 2024 and 2025. Although several ablations are provided, comparisons against simpler alternatives, such as direct DP keyword extraction without the proposed coverage-aware clustering, are missing. Such comparisons would help establish whether the additional complexity of the proposed framework is necessary to achieve its reported improvements.
Questions for authors
-
Given the limited keyword recall and relatively small improvement over Random Keyword, could the authors clarify how much meaningful task-specific information is actually preserved from the private dataset?
-
Could the authors evaluate direct sensitive information leakage, such as the exposure of personal identifiers or rare private attributes, rather than relying primarily on membership inference attacks?
-
Could the authors clarify the methodological novelty of the proposed framework beyond combining existing DP mechanisms and LLM-based generation?
-
Could the authors compare DPD against simpler DP-based keyword extraction or demonstration synthesis approaches to justify the additional complexity of the proposed pipeline?
The Response to Reviewer nnp3, Welcome Any Further Discussions.
Thanks for the constructive feedback. Here are our responses:
Our key idea is that sparse DP-protected keywords can preserve task-specific information, guiding better demonstration synthesis and improving ICL utility. We support this idea from two perspectives:
(1) DPD-based keywords capture informative private signals. We provide both quantitative and qualitative evidence:
minerva, drive, small, crowned, christmas, characters, intellectual, freedom(3/8 Top-8 keyword hits), producing a coherent Christmas-film description. Random Keyword (0/8 hits) instead generates an unrelated ship-and-tourist narrative.correlated, contracts, lend, high, poll, pumps, relief, contestant(2/8 Top-8 hits), producing a report on global polling and international relief. Random Keyword (0/8 hits) instead introduces unsupported details.(2) Private information provides utility gains. Complete DPD outperforms Public Only by 9.23 percentage points (pp) across six benchmarks (Tables 10–11). Since Public Only uses the same LLM with only public task descriptions, this supports the contribution of task-specific information extracted from private demonstrations, beyond public knowledge and LLM capabilities.
We introduce a high-frequency PII stress test, injecting an identical identity-diagnosis tuple (name, email, phone, diagnosis) into every demonstration candidate record. Across three seeds at ε∈{4,8,16}, using non-DP Top-K keyword selection as a baseline. we measure Identity Exposure (approximate identifier recovery) and Identity-Diagnosis Co-exposure (joint appearance in a synthesized demonstration).
The results show that DPD substantially suppresses sensitive-information exposure in synthesized demonstrations, even when sensitive information is deliberately repeated across candidate records. At ε=4, identity exposure is only 4.17%, while identity-diagnosis co-exposure is 0%, compared with 100% for both metrics under non-DP Top-K.
DPD's methodological novelty lies in reformulating DP demonstration synthesis from token-level perturbation to privacy-preserving semantic guidance. Unlike existing methods that perturb next-token distributions, potentially suppressing informative tokens and propagating generation errors, DPD introduces two complementary design contributions:
(1) High-utility synthesis. DPD trains a task-aligned encoder to capture task-relevant semantics, uses coverage-aware retrieval to select diverse candidate demonstrations, and privately aggregates informative keywords to guide LLM synthesis. Together, these components preserve task-relevant semantic signals while enabling coherent and diverse demonstration generation.
(2) Decoupling DP perturbation from token generation. Instead of perturbing each generated token, DPD applies DP mechanisms at the private information selection stage. We derive the sensitivities of private-data-dependent quantities and calibrate DP mechanisms for candidate selection and keyword aggregation. The LLM then generates complete demonstrations solely from privatized keywords via DP post-processing, incurring no additional privacy cost.
We introduce Global JEM, which directly extracts DP keywords from the private demonstration pool without coverage-aware clustering. Results at ε=4:
Complete DPD outperforms Global JEM on all benchmarks, with a 5.35 percentage point (pp) average gain (95% CI: [1.33, 9.37]). This supports the necessity of coverage-aware candidate selection before DP keyword aggregation. We have added these results to Appendix G.7.1, highlighted in purple.