- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The work proposes a metric, called ICLInf, to measure the privacy leakage of ICL samples and ICL-PrivEval, an evaluation workflow for quantifying this leakage. It further extends the metric to reasoning models by scoring the full reasoning trace. It evaluated using ICLInf how deployment factors (model family, model size, sample position, number of samples, calibration, output space, and instruction tuning) influence the privacy leakage on 3 text classification benchmarks. It further compares the metric with a membership inference attack and canary attribution. ICLInf is used to audit a DP-ICL mechanism for synthetic text generation.
Critical strengths
- Interesting evaluation of the privacy risks of ICL using several model families and sizes.
- Interesting extension to reasoning models.
Critical weaknesses
The motivation and usefulness of the metric are unclear. In particular, based on the DP definition maximizing over and
instead of taking the expectation as in the paper's definition:
would more closely follow the original DP worst case motivation. The average over the sample sets and queries is no longer for any dataset. Also, under the stated threat model, the adversary can issue many queries against the same prompt, so per-query losses should compose rather than be averaged. Additionally, the work states that for a fixed prompt and query ICLInf reflects exactly the data-dependent , but doesn't show how it is derived from the DP definition. The work calls this metric privacy loss which can be confused with the standard DP privacy loss and sometimes mixes the two terms. For instance, Section 4 uses ICLInf and "privacy loss" interchangeably, while Section 5 reports ICLInf as .
The privacy auditing analysis in section 5 doesn't compare with prior work on DP auditing (e.g. [1]). In particular, prior DP auditing work provides a statistically valid lower bound on , whereas this work only provides an estimate. The paper reports in the Table 5 that is a diagnostic, not a certified bound, however Section 5 concludes that ICLInf provides a tighter privacy bound. Additionally , is obtained by multiplying the measured remove-one loss by which may inflate the reported tightness.
[1] Chadha, Karan, et al. "Auditing private prediction." ICML 2024
Questions for authors
- What are the motivation and advantages of ICLInf compared with MIAs? In particular, given that ICLInf is largely uncorrelated with the MIA scores, what does a high ICLInf imply about what an adversary can actually infer?
- In terms of DP auditing what is the advantage of ICLInf when compared with the standard DP auditing analysis?
We thank the reviewer for the detailed feedback.
We agree that the average in Eq. 4 is not epsilon(D) for any single dataset; instead, each per-query term is epsilon(D) for its own prompt and query, and Eq. 4 averages these terms. Taking the maximum over exemplar sets would bring back a worst case over data, whereas data-dependent DP fixes the data to measure the deployed prompt. For a fixed prompt, the worst case over queries is indeed the natural DP reading, but computing it would require a search over all possible queries; Eq. 4 instead estimates the expected leakage of a query drawn from the task distribution. We have clarified this distinction in Section 3.
We agree that an adversary can issue many queries, and the deployment's total leakage composes. ICLInf, however, is not a DP mechanism: it quantifies the leakage of a single answer, the per-query term that composition accumulates. For m queries to a fixed prompt, basic composition bounds the total by the sum of per-query terms. ICLInf reports their expected value over queries drawn from the task distribution (in Section 4 over sampled prompts); since every configuration accumulates over the same m queries, the comparisons in Section 4 are unaffected.
We have added the derivation to Appendix B. For a fixed prompt and query, let p and p_k be the answer distributions with and without exemplar k, and
L = max_k max_y |log p(y) - log p_k(y)|.
For any set of answers E,
Pr[E | full] = sum_{y in E} p(y) <= exp(L) * sum_{y in E} p_k(y) = exp(L) * Pr[E | without k],
and symmetrically. Taking E = {y*}, the single answer attaining L, gives equality, so no smaller value holds. Hence the per-query ICLInf is exactly the data-dependent epsilon of one sampled answer.
We no longer use "privacy loss" for ICLInf, including the figure axes: ICLInf now refers only to our metric, and epsilon only to DP parameters. The two are connected in one place: for a fixed prompt and query, ICLInf equals the data-dependent epsilon of Definition 2.2 for a single sampled answer (Appendix B), while Section 4 reports its average over queries, which is not itself an epsilon. In Section 5, the theoretical epsilon comes from the mechanism's privacy analysis, and the empirical epsilon applies the same leave-one-out comparison as ICLInf to that mechanism.
An MIA measures how well one attack performs, while ICLInf measures how much each exemplar can move the answer, which relates to what any attack can infer from it. A low ICLInf thus suggests limited leakage regardless of the attack, whereas a high ICLInf signals potential leakage that a specific attack may not fully exploit. That the attack and ICLInf rank exemplars differently is therefore not unexpected, since the MIA reads only the true label, whereas the largest distribution shift can fall on other labels.
Standard DP auditing (e.g., Chadha et al., 2024) instantiates an adversary and turns its observations into a statistically valid lower bound on epsilon, so its result depends on the strength of the attack. Our audit instead computes the leave-one-out divergences exactly at each generated token, so it requires no attack design and characterizes the leakage of the generation actually produced by the mechanism. It also requires no canaries since the mechanism runs unchanged on the actual private examples being protected, instead of on inserted or crafted worst-case records. The two approaches are therefore complementary.