acceptodds
Under review as a conference paper at ICLR 2027

ICL-PrivEval: A Data-Dependent Framework for Privacy Leakage in In-Context Learning

Abstract

While in-context Learning (ICL) uses exemplars in the prompt to adapt pre-trained Large Language Models (LLMs) to downstream tasks without fine-tuning, these exemplars can carry sensitive information and be potentially leaked from the LLM's output. While prior works have explored this leakage through Membership-Inference Attacks (MIAs), there lacks a systematic evaluation of the factors that contribute to unintended privacy leakage from ICL. To this end, we propose ICL-PrivEval, an evaluation workflow for measuring this leakage. At its foundation is ICLInf, an exemplar-level leakage metric inspired by data-dependent differential privacy and counterfactual influence, which captures how the LLM's answer distribution shifts when each exemplar is removed. Using ICL-PrivEval, we study factors that affect ICL leakage and apply ICLInf as a privacy auditor, yielding an effective audit of sampling-based DP-ICL methods up to ε = 10.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, In-Context Learning, Large Language Models, Privacy Auditing, Data-dependent Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BTrzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The work proposes a metric, called ICLInf, to measure the privacy leakage of ICL samples and ICL-PrivEval, an evaluation workflow for quantifying this leakage. It further extends the metric to reasoning models by scoring the full reasoning trace. It evaluated using ICLInf how deployment factors (model family, model size, sample position, number of samples, calibration, output space, and instruction tuning) influence the privacy leakage on 3 text classification benchmarks. It further compares the metric with a membership inference attack and canary attribution. ICLInf is used to audit a DP-ICL mechanism for synthetic text generation.

Critical strengths
  • Interesting evaluation of the privacy risks of ICL using several model families and sizes.
  • Interesting extension to reasoning models.
Critical weaknesses

The motivation and usefulness of the metric are unclear. In particular, based on the DP definition maximizing over SS and x′x'

L=max⁡S⊂Dtrain,x′∈Dtest,(x,y)∈S,y∈V∣p−p′∣\mathcal{L} = \max_{S \subset D_{train}, x' \in D_{test}, (x, y) \in S, y \in \mathcal{V}} |p - p'|

instead of taking the expectation as in the paper's definition:

L=ES⊂Dtrain,x′∈Dtest[max⁡(x,y)∈Smax⁡y∈V∣p−p′∣]\mathcal{L} = \mathbb{E}_{S \subset D_{train}, x' \in D_{test}} [ \max_{(x, y) \in S} \max_{y \in \mathcal{V}} |p - p'| ]

would more closely follow the original DP worst case motivation. The average over the sample sets and queries is no longer ε(D)\varepsilon(D) for any dataset. Also, under the stated threat model, the adversary can issue many queries against the same prompt, so per-query losses should compose rather than be averaged. Additionally, the work states that for a fixed prompt and query ICLInf reflects exactly the data-dependent ε(D)\varepsilon(D), but doesn't show how it is derived from the DP definition. The work calls this metric privacy loss which can be confused with the standard DP privacy loss and sometimes mixes the two terms. For instance, Section 4 uses ICLInf and "privacy loss" interchangeably, while Section 5 reports ICLInf as ε\varepsilon.

The privacy auditing analysis in section 5 doesn't compare with prior work on DP auditing (e.g. [1]). In particular, prior DP auditing work provides a statistically valid lower bound on ε\varepsilon, whereas this work only provides an estimate. The paper reports in the Table 5 that εempirical\varepsilon_{empirical} is a diagnostic, not a certified bound, however Section 5 concludes that ICLInf provides a tighter privacy bound. Additionally , εempirical\varepsilon_{empirical} is obtained by multiplying the measured remove-one loss by 2m2 m which may inflate the reported tightness.

[1] Chadha, Karan, et al. "Auditing private prediction." ICML 2024

Questions for authors
  • What are the motivation and advantages of ICLInf compared with MIAs? In particular, given that ICLInf is largely uncorrelated with the MIA scores, what does a high ICLInf imply about what an adversary can actually infer?
  • In terms of DP auditing what is the advantage of ICLInf when compared with the standard DP auditing analysis?
1 comment

We thank the reviewer for the detailed feedback.

Based on the DP definition, maximizing over S and x' would more closely follow the original DP worst-case motivation. The average over the sample sets and queries is no longer epsilon(D) for any dataset.

We agree that the average in Eq. 4 is not epsilon(D) for any single dataset; instead, each per-query term is epsilon(D) for its own prompt and query, and Eq. 4 averages these terms. Taking the maximum over exemplar sets would bring back a worst case over data, whereas data-dependent DP fixes the data to measure the deployed prompt. For a fixed prompt, the worst case over queries is indeed the natural DP reading, but computing it would require a search over all possible queries; Eq. 4 instead estimates the expected leakage of a query drawn from the task distribution. We have clarified this distinction in Section 3.

Under the stated threat model, the adversary can issue many queries against the same prompt, so per-query losses should compose rather than be averaged.

We agree that an adversary can issue many queries, and the deployment's total leakage composes. ICLInf, however, is not a DP mechanism: it quantifies the leakage of a single answer, the per-query term that composition accumulates. For m queries to a fixed prompt, basic composition bounds the total by the sum of per-query terms. ICLInf reports their expected value over queries drawn from the task distribution (in Section 4 over sampled prompts); since every configuration accumulates over the same m queries, the comparisons in Section 4 are unaffected.

The work states that for a fixed prompt and query ICLInf reflects exactly the data-dependent epsilon(D), but doesn't show how it is derived from the DP definition.

We have added the derivation to Appendix B. For a fixed prompt and query, let p and p_k be the answer distributions with and without exemplar k, and

L = max_k max_y |log p(y) - log p_k(y)|.

For any set of answers E,

Pr[E | full] = sum_{y in E} p(y) <= exp(L) * sum_{y in E} p_k(y) = exp(L) * Pr[E | without k],

and symmetrically. Taking E = {y*}, the single answer attaining L, gives equality, so no smaller value holds. Hence the per-query ICLInf is exactly the data-dependent epsilon of one sampled answer.

The work calls this metric privacy loss which can be confused with the standard DP privacy loss and sometimes mixes the two terms. For instance, Section 4 uses ICLInf and "privacy loss" interchangeably, while Section 5 reports ICLInf as epsilon.

We no longer use "privacy loss" for ICLInf, including the figure axes: ICLInf now refers only to our metric, and epsilon only to DP parameters. The two are connected in one place: for a fixed prompt and query, ICLInf equals the data-dependent epsilon of Definition 2.2 for a single sampled answer (Appendix B), while Section 4 reports its average over queries, which is not itself an epsilon. In Section 5, the theoretical epsilon comes from the mechanism's privacy analysis, and the empirical epsilon applies the same leave-one-out comparison as ICLInf to that mechanism.

What are the motivation and advantages of ICLInf compared with MIAs? In particular, given that ICLInf is largely uncorrelated with the MIA scores, what does a high ICLInf imply about what an adversary can actually infer?

An MIA measures how well one attack performs, while ICLInf measures how much each exemplar can move the answer, which relates to what any attack can infer from it. A low ICLInf thus suggests limited leakage regardless of the attack, whereas a high ICLInf signals potential leakage that a specific attack may not fully exploit. That the attack and ICLInf rank exemplars differently is therefore not unexpected, since the MIA reads only the true label, whereas the largest distribution shift can fall on other labels.

In terms of DP auditing what is the advantage of ICLInf when compared with the standard DP auditing analysis?

Standard DP auditing (e.g., Chadha et al., 2024) instantiates an adversary and turns its observations into a statistically valid lower bound on epsilon, so its result depends on the strength of the attack. Our audit instead computes the leave-one-out divergences exactly at each generated token, so it requires no attack design and characterizes the leakage of the generation actually produced by the mechanism. It also requires no canaries since the mechanism runs unchanged on the actual private examples being protected, instead of on inserted or crafted worst-case records. The two approaches are therefore complementary.

Reviewer_eo9rOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ICL-PrivEval, a data-dependent framework for privacy leakage in in-context-learning. Compared to Membership-Inference Attacks (MIAs), the proposed method conducts a systematic evaluation of the factors contributing to unintended privacy leakage from in-context learning. To be specific, it leverages an exemplar-level leakage metric, ICLInf, to calculate the model predictions' change when some of the examples are removed. This paper leverages ICL-PrivEval to conduct experiments on a variety of models and datasets, and the results study factors that affect ICL leakage. Additionally, the experiment results also demonstrate that the proposed method is an effective privacy auditor in in-context learning.

Critical strengths
  1. This paper is well written and technically sound.
  2. It analyzes the factors affecting privacy leakage. Compare to AUC or Accuracy in MIA, the proposed method can help find the key factors affect the extent of the privacy leaks.
Critical weaknesses
  1. This novelty of this paper is limited. From my point of view, this paper mainly give an intuition formulation to measure the llm leakage in in-context learning situation. And then use this formulations to measure several factors will cause the leakages, such as the model family, model size, and number of exemplars could affect leakage. It is not interesting and novel.
  2. The experiment setting also has some problems. The task is easy and the number of demonstrations is very small. Usually, the setting will not be that ideal and experiments on more complex situation and much more demonstrations are needed.
  3. Experiments on some sota models, especially some MOE models such as GPT or Qwen3.5-397B can be also be conducted to fully evaluate the effectiveness of the proposed method.
Questions for authors

Please refer to the weakness. My main concern is the novelty and contribution of this paper is limited.

1 comment

We thank the reviewer for finding the paper technically sound and pointing out weaknesses.

Limited novelty: a formulation used to measure several factors.

Our contributions are twofold. (1) ICLInf, a new leakage measure grounded in differential privacy, counterfactual influence, and causal estimation: it compares the output with and without each exemplar and presents the data-dependent epsilon of a sampled answer for a fixed prompt and query. (2) ICL-PrivEval, a framework that operationalizes ICLInf to measure leakage of ICL demonstrations, which LLM providers can use to estimate privacy leakage risk before deployment.

The tasks are easy and the number of demonstrations is small.

We follow the benchmarks of prior ICL privacy work (e.g., Wen et al., 2024), and they are not saturated for current LLMs. Accuracy ranges from 0.69 to 0.91 (Table 1), and on TREC it keeps rising with more exemplars (0.62 to 0.84 for Qwen3-8B from 1 to 32 shots, Appendix D.2). The models therefore rely on the demonstrations, which is exactly when they can leak them. Beyond 4-shot classification, we evaluate up to 32 exemplars (RQ4), free generation over the full vocabulary (RQ6), reasoning traces (Section 4.4), and free-form text generation (Section 5).

Evaluate SOTA MoE models such as GPT or Qwen3.5-397B.

ICLInf applies directly to MoE models with logit access. We newly evaluate Qwen3.5-35B-A3B (MoE) and the dense Qwen3.5-9B from the same family under the settings of Section 4.

AG NewsDBpedia-14TREC
MoE (Qwen3.5-35B-A3B)2.05 (0.84)2.27 (0.91)1.97 (0.79)
Dense (Qwen3.5-9B)2.20 (0.83)2.53 (0.91)1.91 (0.72)

ICLInf is averaged over 500 queries, with accuracy in parentheses. In our experiment, the MoE model does not leak more than its dense counterpart, at equal or higher accuracy. MoE models carry additional numerical noise, which tends to inflate ICLInf (Appendix D.4), so this comparison is even more conservative.

Closed models such as GPT expose at most a truncated list of top-token log-probabilities rather than the full distribution ICLInf requires. Qwen3.5-397B requires roughly 800 GB of GPU memory, and we currently do not have the resources to host and run such a large model.

Reviewer_ECRuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privacy leakage from private demonstrations used in in-context learning (ICL). The authors propose ICLInf, a data-dependent metric that measures the extent to which a model's output distribution changes when an individual demonstration is removed from the prompt. Based on this metric, they introduce ICL-PrivEval, an evaluation workflow for analyzing privacy leakage and comparing different deployment configurations. The authors conduct experiments on three text classification datasets and several open-source LLMs, examining factors such as model family, model size, demonstration position, number of shots, calibration, and instruction tuning. The authors also compare ICLInf with membership inference attacks and canary-based evaluation, extend the metric to reasoning traces, and apply a Rényi-divergence-based analysis to audit a sampling-based DP-ICL mechanism.

Critical strengths
  1. This paper addresses an important privacy issue in ICL that is not fully captured by existing attack-based evaluations. I like the idea of measuring how individual demonstrations affect the output distribution rather than relying solely on the performance of a particular membership inference attack. The leave-one-out formulation is straightforward and provides a more fine-grained view of which demonstrations contribute to the measured leakage. It could be a useful diagnostic tool for practitioners who have access to model output probabilities.

  2. This work examines several factors that may influence privacy leakage across different models and datasets. Some findings are interesting, particularly the non-monotonic relationship between model size and leakage, as well as the observation that adding more demonstrations can reduce the influence of an individual example. I also appreciate the extension to reasoning models, which highlights that looking only at the final answer may miss information exposed through the reasoning trace. These results provide useful insights into how deployment choices affect the proposed leakage metric.

Critical weaknesses
  1. My main concern is whether ICLInf accurately reflects privacy risk under the paper's threat model. In Section 2.1, the adversary is allowed to observe either the output distribution or a sampled output, and the paper also assumes access to logits. However, ICLInf measures the maximum log-probability difference between two output distributions. This has a clear privacy-loss interpretation when the released output is sampled from the distribution, but the interpretation is different if the adversary can directly observe the probability vector. Furthermore, Eq. (4) averages the maximum leakage over sampled prompts and test queries. While this provides a useful empirical measure, it does not characterize the worst-case leakage over arbitrary adversarial queries or repeated interactions with the model. The distinction matters because the paper frequently refers to ICLInf as privacy loss and uses it to guide privacy-related deployment decisions. I think the metric is potentially useful, but authors should be more precise about what privacy risk it captures and under which observation model its interpretation is valid.

  2. From my point of view, the privacy auditing empirical results do not fully support the claimed tightness of the empirical privacy bounds. Section 5 presents ICLInf as a tool for auditing a sampling-based DP-ICL mechanism. However, the empirical Rényi privacy loss is evaluated along sampled generation prefixes rather than over all possible output histories. As acknowledged in Table 5, the resulting empirical epsilon is a diagnostic of the sampled run, not a certified privacy bound. This makes it difficult to interpret the claim that ICLInf provides a tighter privacy bound or closely characterizes the realized privacy loss of the mechanism. In particular, it is not clear whether the observed agreement with the theoretical epsilon for small privacy budgets generalizes beyond the sampled trajectories. A stronger justification of the connection between the sampled-prefix analysis and the privacy guarantee of the full generation mechanism would make this contribution more convincing. Alternatively, I think it may be good that the authors frame these results more explicitly as an empirical diagnostic rather than a tighter privacy bound.

Questions for authors
  1. Could the authors clarify whether the adversary is assumed to observe sampled outputs or the full probability distribution? If the latter is available, how should the privacy-loss interpretation of ICLInf be understood, given that the released object is the probability vector itself?

  2. In Section 4.1, the demonstrations are independently resampled for each query, while the threat model describes a deployment using a fixed set of private demonstrations. How sensitive are the results to this difference? In particular, would the conclusions still hold when evaluating a fixed prompt under targeted or adversarially selected queries?

  3. Could the authors clarify what guarantees, if any, can be drawn from the empirical epsilon in Section 5? Since the evaluation uses sampled generation prefixes, how does this quantity relate to the privacy loss of the complete output distribution? A comparison with an attack-based empirical lower bound on privacy would also help establish the audit's practical significance.

1 comment

We thank the reviewer for the careful reading and valuable comments.

Could the authors clarify whether the adversary is assumed to observe sampled outputs or the full probability distribution? If the latter is available, how should the privacy-loss interpretation of ICLInf be understood, given that the released object is the probability vector itself?

In this work, the adversary is only assumed to observe the sampled outputs. In our setup, the text sampled by the LLM is released, while the output probability distribution stays on the server side, where the deployer uses it to compute ICLInf before release. We have revised the threat model (Section 2.1) to make this explicit.

In Section 4.1, the demonstrations are independently resampled for each query, while the threat model describes a deployment using a fixed set of private demonstrations. How sensitive are the results to this difference? In particular, would the conclusions still hold when evaluating a fixed prompt under targeted or adversarially selected queries?

Each per-query term of our estimator is a fixed-prompt measurement, and resampling averages it over prompts. We resample because Section 4 compares configurations (model, model size, etc.), and a single fixed prompt would limit each finding to one particular choice of exemplars. To measure the sensitivity, we newly evaluated three different fixed prompts (drawn with three seeds) on AG News and TREC (Qwen3-8B). Each fixed prompt is scored on the same 500 queries as the resampled run with the same seed. Averaged over the 500 queries, the results are shown below:

ICLInf (mean over 500 queries)Seed 42Seed 43Seed 44
AG News, fixed prompt4.505.435.40
AG News, resampled4.964.804.99
TREC, fixed prompt4.665.728.22
TREC, resampled5.805.655.80

The resampled estimate reflects the expected loss over prompts, while an individual deployed prompt can leak noticeably more or less, which is why ICL-PrivEval measures the deployer's own prompt. We have added this experiment to Appendix D.3.

For targeted queries, our canary experiment offers a partial indication: when the query is the canary's own text, the per-exemplar scores locate the planted exemplar in 72-100% of queries for Qwen3 and Ministral (chance 25%), more often than on ordinary queries. A fuller evaluation under adversarially selected queries could be an important direction for future work.

Could the authors clarify what guarantees, if any, can be drawn from the empirical epsilon in Section 5? Since the evaluation uses sampled generation prefixes, how does this quantity relate to the privacy loss of the complete output distribution? A comparison with an attack-based empirical lower bound on privacy would also help establish the audit's practical significance.

We were interested in whether ICLInf's leave-one-out comparison could provide a convenient, attack-free empirical lower bound on the theoretical epsilon of a DP-ICL mechanism. The guarantee is given by the theoretical epsilon, which accounts for the complete output distribution; the empirical epsilon evaluates the tokens the mechanism actually generates, so it never exceeds the theoretical value. For theoretical epsilon up to 10, the empirical epsilon is a tight lower bound on the theoretical epsilon, while the gap grows for larger budgets. This approach is not meant to replace attack-based audits (e.g., Chadha et al., 2024) but to complement them. We have revised Section 5 to clarify this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.