acceptodds
Under review as a conference paper at ICLR 2027

Boosting Diversity and Relevance: Privacy-Preserving Demonstration Synthesis for In-Context Learning

Abstract

Large language models (LLMs) rely on contextual information embedded in demonstrations to perform effective in-context learning (ICL), yet using demonstrations drawn from a private data pool may expose sensitive information when interacting with LLM servers. Existing demonstration synthesis methods based on differential privacy (DP) can mitigate such privacy risks, but they struggle to balance privacy and utility. Particularly, these methods often generate demonstrations with limited diversity and task relevance, while DP noise injected into token sampling distributions introduces semantic errors that accumulate during autoregressive generation. Motivated by these limitations, we propose DPD, a novel privacy-preserving algorithm to generate high-utility demonstrations under DP guarantees. To address diversity and relevance gaps, we develop a task-aligned encoder that maps private demonstrations into a task-aware embedding space, where a DP coverage-aware strategy privately releases coverage-weighted centers and retrieves candidate demonstrations for promoting semantic coverage. To mitigate error accumulation, we avoid perturbing token autoregressive generation and instead introduce the Joint Exponential Mechanism to efficiently extract DP-based keywords from candidates, guiding the advanced LLM to synthesize demonstrations with high utility. The overall DPD framework provides demonstration-level DP. Extensive experiments on standard benchmarks show that DPD outperforms token-level DP demonstration synthesis baselines while effectively mitigating membership inference attacks, providing an effective solution for privacy-preserving ICL.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
In-Context Learning, Differential Privacy, Privacy-Preserving Demonstration Synthesis;Diversity and Task Relevance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nnp3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DPD, a differential privacy (DP)-based framework for synthesizing demonstrations for in-context learning (ICL) from private datasets. The method uses a task-aligned encoder and DP coverage-aware clustering to retrieve diverse demonstration candidates. It then applies the Joint Exponential Mechanism (JEM) to extract privatized keywords, which guide an LLM to generate synthetic demonstrations without token-level DP perturbation. The authors provide a theoretical demonstration-level DP guarantee and evaluate the method on text classification, information extraction, and mathematical reasoning tasks. Experimental results demonstrate improvements in downstream ICL performance and generation efficiency over existing DP-based synthesis methods.

Critical strengths
  1. The paper addresses a practical challenge in privacy-preserving ICL by separating private information extraction from LLM-based demonstration generation. This design enables the use of external LLMs without directly exposing the original private demonstrations.

  2. The experiments cover multiple downstream tasks and privacy budgets, showing improvements over existing DP-based synthesis methods. The paper also includes generator-controlled comparisons, ablation studies, and theoretical privacy analysis.

Critical weaknesses
  1. Limited evidence of meaningful private information utilization. The proposed method aims to preserve useful information from private demonstrations while ensuring DP. However, Table 7 shows low candidate-hit rates and Top-T keyword recall, while replacing private keywords with randomly sampled keywords reduces average accuracy by only 2.23 percentage points (Table 11). These results raise questions about how much task-specific information is actually extracted from private data, rather than relying on public task descriptions and the LLM's generation capabilities.

  2. Insufficient evaluation of sensitive information leakage. Although the paper provides demonstration-level DP guarantees and evaluates membership inference attacks, it does not directly assess whether sensitive information is exposed in the synthesized demonstrations. The exposure-style attack in Appendix G.4 relies on token overlap rather than explicitly measuring the leakage of personally identifiable information or sensitive attributes. Furthermore, the experiments use standard public benchmarks without evaluating datasets containing realistic sensitive records, limiting the practical validation of the privacy claims.

  3. Limited methodological novelty. The proposed framework largely combines existing techniques, including task-aligned embeddings, DP-based statistical aggregation, JEM, and LLM-based generation. While their integration is reasonable, the individual components are not fundamentally new, and the technical contribution beyond combining these established techniques appears limited.

  4. Insufficient comparison with simpler privacy-preserving synthesis approaches. The main experiments compare DPD primarily against two existing autoregressive DP synthesis methods from 2024 and 2025. Although several ablations are provided, comparisons against simpler alternatives, such as direct DP keyword extraction without the proposed coverage-aware clustering, are missing. Such comparisons would help establish whether the additional complexity of the proposed framework is necessary to achieve its reported improvements.

Questions for authors
  1. Given the limited keyword recall and relatively small improvement over Random Keyword, could the authors clarify how much meaningful task-specific information is actually preserved from the private dataset?

  2. Could the authors evaluate direct sensitive information leakage, such as the exposure of personal identifiers or rare private attributes, rather than relying primarily on membership inference attacks?

  3. Could the authors clarify the methodological novelty of the proposed framework beyond combining existing DP mechanisms and LLM-based generation?

  4. Could the authors compare DPD against simpler DP-based keyword extraction or demonstration synthesis approaches to justify the additional complexity of the proposed pipeline?

1 comment
AuthorsOpenReview ↗

The Response to Reviewer nnp3, Welcome Any Further Discussions.

Thanks for the constructive feedback. Here are our responses:

W1/Q1: Private Information Utilization

Our key idea is that sparse DP-protected keywords can preserve task-specific information, guiding better demonstration synthesis and improving ICL utility. We support this idea from two perspectives:

(1) DPD-based keywords capture informative private signals. We provide both quantitative and qualitative evidence:

  • Quantitative: DPD-based keyword sampling achieves 1.40×, 3.71×, and 7.33× higher Top-K keyword recall than random sampling at ε=1, 4, and 8, respectively, demonstrating an increased probability of sampling statistically informative private keywords.
  • Qualitative: We trace candidate demonstrations, released keywords, and synthesized demonstrations in two examples:
    • DBPedia–Film label: Candidate demonstrations contain characters and Christmas. DPD releases minerva, drive, small, crowned, christmas, characters, intellectual, freedom (3/8 Top-8 keyword hits), producing a coherent Christmas-film description. Random Keyword (0/8 hits) instead generates an unrelated ship-and-tourist narrative.
    • AGNews–World label: Candidates contain poll and relief. DPD releases correlated, contracts, lend, high, poll, pumps, relief, contestant (2/8 Top-8 hits), producing a report on global polling and international relief. Random Keyword (0/8 hits) instead introduces unsupported details.

(2) Private information provides utility gains. Complete DPD outperforms Public Only by 9.23 percentage points (pp) across six benchmarks (Tables 10–11). Since Public Only uses the same LLM with only public task descriptions, this supports the contribution of task-specific information extracted from private demonstrations, beyond public knowledge and LLM capabilities.

W2/Q2: Sensitive Information Leakage

We introduce a high-frequency PII stress test, injecting an identical identity-diagnosis tuple (name, email, phone, diagnosis) into every demonstration candidate record. Across three seeds at ε∈{4,8,16}, using non-DP Top-K keyword selection as a baseline. we measure Identity Exposure (approximate identifier recovery) and Identity-Diagnosis Co-exposure (joint appearance in a synthesized demonstration).

MethodIdentity Exposure(%)Co-exposure(%)
DPD(ε=4)4.17±4.560.00±0.00
DPD(ε=8)6.25±6.252.08±3.61
DPD(ε=16)14.58±7.228.33±7.22
Non-DP Top-K100.00±0.00100.00±0.00

The results show that DPD substantially suppresses sensitive-information exposure in synthesized demonstrations, even when sensitive information is deliberately repeated across candidate records. At ε\varepsilon=4, identity exposure is only 4.17%, while identity-diagnosis co-exposure is 0%, compared with 100% for both metrics under non-DP Top-K.

W3/Q3: Methodological Novelty

DPD's methodological novelty lies in reformulating DP demonstration synthesis from token-level perturbation to privacy-preserving semantic guidance. Unlike existing methods that perturb next-token distributions, potentially suppressing informative tokens and propagating generation errors, DPD introduces two complementary design contributions:

(1) High-utility synthesis. DPD trains a task-aligned encoder to capture task-relevant semantics, uses coverage-aware retrieval to select diverse candidate demonstrations, and privately aggregates informative keywords to guide LLM synthesis. Together, these components preserve task-relevant semantic signals while enabling coherent and diverse demonstration generation.

(2) Decoupling DP perturbation from token generation. Instead of perturbing each generated token, DPD applies DP mechanisms at the private information selection stage. We derive the sensitivities of private-data-dependent quantities and calibrate DP mechanisms for candidate selection and keyword aggregation. The LLM then generates complete demonstrations solely from privatized keywords via DP post-processing, incurring no additional privacy cost.

W4/Q4: Necessity of the Complete Pipeline

We introduce Global JEM, which directly extracts DP keywords from the private demonstration pool without coverage-aware clustering. Results at ε=4:

DatasetComplete DPDGlobal JEM
TREC74.47±0.8161.88±7.12
AGNews85.07±3.5281.05±4.33
DBPedia79.83±1.9274.76±4.14
MIT-G64.54±1.5059.65±2.05
MIT-D93.82±1.0992.72±2.33
GSM8K87.93±1.8683.50±6.62
Macro Avg.80.9975.59
ComparisonΔAcc. (pp)95% CIW/T/LSign-test pp
DPD−Global JEM+5.30[1.33,9.37]6/0/00.0313

Complete DPD outperforms Global JEM on all benchmarks, with a 5.35 percentage point (pp) average gain (95% CI: [1.33, 9.37]). This supports the necessity of coverage-aware candidate selection before DP keyword aggregation. We have added these results to Appendix G.7.1, highlighted in purple.

Reviewer_NGfbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper aims to improve the accuracy of differentially private in-context learning (DP-ICL) using DP synthetic demonstrations from private demonstrations. Existing methods face two limitations in generating DP synthetic demonstrations: (i) limited diversity and task relevance, and (ii) semantic error accumulation from repeated token-wise DP perturbation. To address (i), DPD employs DP-coverage-aware clustering of task-aligned embeddings to retrieve diverse private demonstrations. To address (ii), DPD privately selects frequent keywords from these candidates using the Joint Exponential Mechanism to guarantee DP and generates synthetic demonstrations with an external LLM, avoiding token-wise perturbation. Empirical experiments show that DPD achieves higher accuracy for DP-ICL and faster generation of synthetic demonstrations.

Critical strengths
  1. DPD provides a well-motivated combination of prior DP-ICL ideas. DPD adopts the reusable demonstration synthesis settings from Tang et al. (2024) and incorporates JointEM-based Keyword Space Aggregation in Wu et al. (2024). These components are combined with a new DP-coverage-aware retrieval mechanism designed to enhance the diversity and task relevance of the generated demonstrations, thereby increasing accuracy for downstream tasks.

  2. DPD substantially reduces generation time compared with Tang et al. (2024) and Gao et al. (2025) by avoiding the token-wise private aggregation used in these methods.

Critical weaknesses
  1. The text-classification experiments do not include comparisons with (i) classification using cϕ∘fθc_{\phi}\circ f_\theta or (ii) existing ICL methods that select diverse and task-relevant demonstrations from DpubD_{\text{pub}}, such as Levy et al. (2023). Including these baselines would show whether DPD outperforms methods that use only DpubD_{\text{pub}}.
Questions for authors
  1. Could you compare DPD with baseline (i) in Weakness 1?

  2. Could you clarify whether Public Direct in Appendix G.7.2 considers diversity and task relevance in retrieval? If not, could you compare DPD with baseline (ii) in Weakness 1?

2 comments
AuthorsOpenReview ↗

The Response to Reviewer NGfb, Welcome Any Further Discussions.

We thank the reviewer for the insightful comments. Here are our responses.

W1/Q1: Classification using cϕ∘fθc_\phi \circ f_\theta

Following your suggestion, we additionally evaluate Public Classifier, a task-aligned BERT classifier cϕ∘fθc_\phi \circ f_\theta trained exclusively on the public dataset DpubD_{\mathrm{pub}}, against DPD at ε=2\varepsilon=2.

DatasetPublic ClassifierDPDGain (pp)
TREC69.80±0.0873.20±2.12+3.40
AGNews79.33±1.0383.50±2.52+4.17
DBPedia76.90±0.4680.33±2.52+3.43
Average75.3479.01+3.67

DPD consistently outperforms Public Classifier across all three benchmarks, achieving a 3.67 percentage point (pp) average improvement. This indicates that DPD-based demonstration synthesis can provide additional utility beyond direct classification using public data.

Notably, Public Classifier requires task-specific labeled public data, which may be difficult to obtain for open-form tasks such as information extraction and mathematical reasoning. In contrast, DPD uses a pretrained encoder without DpubD_{\mathrm{pub}} for these tasks, enabling DP-protected demonstration synthesis without task-specific public data.

W1/Q2: Public-Data ICL Baseline

Following your suggestion, we conduct additional experiments with Public ICL, which employs diversity-aware demonstration retrieval from DpubD_{\mathrm{pub}}, motivated by Levy et al. (2023). This baseline selects task-relevant public demonstrations while promoting syntactic and semantic diversity, without accessing private data.

We compare Public ICL with DPD at ε=2\varepsilon=2:

DatasetPublic-only ICLDPDGain (pp)
TREC66.38±3.6273.20±2.12+6.82
AGNews77.58±5.0483.50±2.52+5.92
DBPedia76.53±4.5280.33±2.52+3.80
Average73.5079.01+5.51

DPD consistently outperforms Public ICL across all three benchmarks, achieving a 5.51 pp average improvement. This supports the additional utility of private-data-dependent keyword guidance beyond retrieving diverse and task-relevant public demonstrations.

Notably, Public ICL requires sufficiently rich and task-relevant public demonstration pools, which may be difficult to obtain for open-form tasks such as information extraction and mathematical reasoning. In contrast, DPD uses a pretrained encoder without task-specific DpubD_{\mathrm{pub}} for these tasks, enabling DP-protected demonstration synthesis from privatized keywords.

We have added both baselines and their implementation details to the Appendix G.7.2, highlighted in purple.

I thank the authors for adding the baselines I suggested. The results show that DPD outperforms these two added baselines. This result empirically answers my question about whether DPD improves accuracy over methods that only use DpubD_{\mathrm{pub}}. I will therefore raise my soundness score to 3 while maintaining my overall rating of 3.

Reviewer_Q7PWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to synthesize reusable demonstrations for in-context learning (ICL) from a private demonstration pool while providing demonstration-level differential privacy (DP). The proposed framework, DPD, consists of three stages: constructing task-aligned embeddings using a publicly trained encoder followed by projection and clipping; releasing coverage-aware DP centers to select candidate demonstrations; and releasing keywords through the Joint Exponential Mechanism (JEM), which guide an LLM to generate complete synthetic demonstrations.

The paper provides sensitivity and composition analyses and evaluates DPD on six datasets covering text classification, information extraction, and mathematical reasoning. Experiments include generator-controlled comparisons, component ablations, public-data attribution, and membership inference attacks. DPD generally outperforms the two evaluated token-level DP synthesis baselines.

Critical strengths
  1. DPD exposes only privatized keywords and public task information to the external generator. Complete demonstration synthesis is therefore post-processing, and the resulting demonstrations can be reused without additional privacy expenditure. The generator-controlled comparisons also support a contribution beyond simply using a stronger generator. For example, on GSM8K at ε=4, DPD-Llama achieves 68.20%, outperforming both baselines using the same Llama-2-7B generator. This provides meaningful evidence that the framework itself improves utility.

  2. The experiments include Random Candidate, Random Keyword, Public Only, Public Direct, Public Prototypes, and No-noise DPD. These comparisons help distinguish the effects of candidate selection, private statistics, public data, and DP randomization. The appendix also acknowledges that private keywords make a modest contribution and that coverage-aware centers do not guarantee recovery of distinct semantic modes. These qualifications improve the credibility of the analysis.

Critical weaknesses
  1. Table 7 reports candidate-hit rates of only 9.69–15.73% for released keywords under the main TREC configuration. At ε=4, Top-T recall is only 5.01%. Thus, most released keywords do not occur in the selected candidate records. Consistently, Table 10 shows a macro-average improvement of only 2.23 percentage points over Random Keyword, substantially smaller than the gains over token-level synthesis baselines. Some individual differences are small: the improvement on MIT-G is 0.79 percentage points. Public Only also achieves strong performance on AGNews and GSM8K. The concern is therefore not the absence of relevant ablations; the authors already provide them. Rather, the current evidence does not adequately explain how the relatively sparse private keyword signal reliably improves demonstrations or why the full pipeline is necessary. The dataset-level sign test supports a consistent direction across six benchmarks, but it does not establish the reliability of individual dataset improvements using paired runs or example-level predictions. This limits the strength of the paper’s central mechanistic interpretation.

  2. Several reporting inconsistencies undermine the verifiability of the empirical conclusions. The following discrepancies require clarification:

  • AUC versus AUC* in Table 9: AUC* is defined as max(AUC, 1−AUC), so it cannot be smaller than the corresponding AUC, including when both are averaged over the same runs. However, Complete DPD on MIT-G reports AUC=0.5685 and AUC*=0.5580.
  • No-noise MIT-G results: Table 13 defines No-noise DPD as the same pipeline at ε=∞ and reports 67.09±0.59 on MIT-G. Table 1 reports 65.09±0.59 for the corresponding DPD-DeepSeek setting.
  • GSM8K evaluation granularity: The appendix states that accuracy is evaluated on 500 questions and averaged over five seeds. With one exact-match prediction per question per seed, the mean accuracy must occur in increments of 0.04 percentage points. Several reported values, such as 87.93%, do not directly match this protocol. Additional averaging could explain this, but it is not documented.
Questions for authors
  1. Experimental consistency: Please reconcile the AUC/AUC* discrepancy, the conflicting no-noise MIT-G results, and the GSM8K accuracy granularity. Can you provide per-seed results, exact correct counts, and the aggregation procedure? Do the principal conclusions remain unchanged after correction?

  2. Contribution of private keywords: Please provide paired differences and uncertainty estimates for Complete DPD versus Random Keyword across the five matched seeds, together with examples linking released keywords, candidate records, and synthesized demonstrations. Does utility improve consistently with keyword hit rate or the number of informative content words?

  3. Public-data provenance: What are the exact sources of the experimental Dpub datasets? How were the “public labeled splits” constructed and deduplicated against Dpriv, validation data, and test data? The discussion mentions Wikipedia, WordNet-derived resources, and LLM-generated examples, but the actual source used in each experiment remains unclear.

1 comment
AuthorsOpenReview ↗

The Response to Reviewer Q7PW, Welcome Any Further Discussions.

We thank the reviewer for the insightful comments. Here are our responses.

W1/Q2: Contribution of Private Keywords

Our key idea is that sparse DP-protected keywords provide useful semantic guidance for LLM synthesis, yielding modest but consistent ICL gains. We support this idea from four perspectives:

(1) Informative keyword sampling. DPD aims to preserve task-relevant semantic signals under demonstration-level differential privacy (DP), rather than recover private demonstrations or all Top-T keywords. We provide both quantitative and qualitative evidence:

  • Quantitative: Compared with uniform random sampling, DPD's JEM-based keyword sampling achieves 1.40×, 3.71×, and 7.33× higher Top-T recall at ε=1, 4, and 8, respectively, indicating preferential sampling of informative (high-frequency) keywords.

  • Qualitative: The following examples illustrate how sparse keyword cues guide synthesis:

    • DBPedia–Film category: Candidate demonstrations contain characters and Christmas. DPD releases minerva, drive, small, crowned, christmas, characters, intellectual, freedom (3/8 Top-8 keyword hits), generating a coherent Christmas-film description. Random Keyword has 0/8 hits and generates an unrelated ship-and-tourist narrative.
    • AGNews–World category: Candidates contain poll and relief. DPD releases correlated, contracts, lend, high, poll, pumps, relief, contestant (2/8 Top-8 keyword hits), generating a report about a global poll and international relief. Random Keyword has 0/8 hits and introduces unsupported details.

(2) Paired-seed reliability. We compare Complete DPD against Random Keyword over five seeds per benchmark:

DatasetMean±SD(pp)95%CI
TREC4.14±2.89[0.55,7.73]
AGNews1.20±0.95[0.02,2.38]
DBPedia3.00±0.91[1.87,4.13]
MIT-G0.79±0.31[0.40,1.18]
MIT-D1.37±0.32[0.98,1.76]

All 25 paired differences are positive, and all five paired 95% CIs exclude zero, supporting modest but consistent improvements across these benchmarks.

(3) Sparse keywords can still be useful. We conduct a test varying keyword hit rates from 5% to100% while fixing all other components, utility shows a modest positive but non-monotonic relationship, peaking at 40%, 30%, and 50% on TREC, AGNews, and DBPedia, respectively. These findings suggest that even sparse but task-relevant keyword cues can effectively guide demonstration synthesis, and the advanced LLM can leverage its task knowledge and generation capabilities.

(4) Complementary pipeline contributions. DPD combines coverage-aware retrieval, DP keyword aggregation, and LLM-based synthesis. Retrieval promotes demonstration candidates diversity, while their privatized keywords provide semantic guidance for LLM generation. Table 10 shows that complete DPD outperforms all ablated variants in average accuracy across six benchmarks, supporting complementary contributions rather than attributing all gains to private keywords.

W2/Q1: Experimental Consistency

(1) AUC/AUC*. The corrected MIT-G values in Table 9 are AUC=0.5685 and AUC*=0.5685. AUC*=max(AUC,1−AUC) is computed per seed before averaging.

(2) No-noise MIT-G. The correct value is 65.09±0.59. We will correct Table 13 to match Table 1.

(3) GSM8K accuracy. GSM8K uses three seeds, each evaluating 500 questions with exact-match accuracy:

SeedCorrectAccuracy
1427/50085.40%
2443/50088.60%
3449/50089.80%
Mean1319/150087.93%

The exact mean is 1319/1500×100=87.9333%, explaining the reported 87.93%. We correct the evaluation description in Appendix G.1.

These reporting corrections do not affect the principal conclusions. We have updated the above corrected value in the revised, highlighted in purple.

Q3: Public Data Provenance

(1) Sources and construction. The public dataset DpubD_{pub} consists of texts retrieved from Wikipedia corpus (https://huggingface.co/datasets/wikimedia/wikipedia) and Princeton WordNet 3.0 (https://wordnetcode.princeton.edu/3.0/). ChatGPT selects texts corresponding to task-specific categories and automatically assigns labels.

DatasetDpubsizeSourcesLabels
TREC1,090Wikipedia,WordNetChatGPT
AGNews24,000Wikipedia,WordNetChatGPT
DBPedia9,999Wikipedia,WordNetChatGPT

DpubD_{pub} is used solely to fine-tune the task-aligned BERT encoder for these classification tasks

(2) Data isolation. Following Gao et al. (2025), benchmark training samples are used for private demonstration construction and hyperparameter selection, while held-out test samples are reserved for evaluation. DpubD_{pub} is independently retrieved from Wikipedia and WordNet. We performed exact-match deduplication after text normalization against DprivateD_{private}, validation, and test splits, confirming no identical records. The BERT encoder is trained exclusively on DpubD_{pub} without direct access to private demonstrations.

Comments on the paper

1 comment

Thank you for the detailed response and the new experiments. The Global JEM comparison clearly shows the benefit of coverage-aware clustering, and the PII stress test is a helpful addition.

I still have a few remaining concerns. The gain over Public Only suggests that the private data is useful, but the gap to Random Keyword and the keyword recall remain relatively small, so it would be helpful to better understand how much of the gain comes from the released keywords. The PII stress test measures exposure only in the synthesized demonstrations, while the external LLM receives the released keywords, and I could not find a keyword-level evaluation. The test also injects a synthetic tuple into public benchmarks, and exposure increases at larger ε, so an evaluation on data with realistic sensitive records would make the privacy claims more convincing. Finally, I still see the method mainly as a careful combination of existing components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.