acceptodds
Under review as a conference paper at ICLR 2027

Performative Privacy: When Differential Privacy Maximizes Utility

Abstract

Privacy-preserving learning is often motivated by the idea that protecting users' data can preserve trust and thus participation, improving utility in the long term. However, this claim has not been formalized so far. In parallel, performative learning provides a framework for studying learning systems whose deployment affects the data they later observe. In this work, we bring these two perspectives together and introduce performative privacy, where data leakage reduces future participation. We study a simple model where agents repeatedly contribute data for mean estimation but may leave the system when their data is leaked. Privacy is implemented through differentially private mechanisms, creating a trade-off between estimation noise and future participation. We show, through a theoretical study of the dynamics and numerical experiments, that a finite privacy budget can outperform non-private estimation in the long term when the feedback loop between leakage and participation is sufficiently strong. This provides first evidence that differential privacy can be optimal not only as a protection mechanism, but also from the perspective of long-term utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Performative Learning, Membership Inference Attack

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_QxKGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the long-term privacy-utility trade-off by considering how privacy leakage affects future user participation. The authors propose a performative privacy framework and theoretically characterize population dynamics and privacy noise thresholds under Bernoulli mean estimation, Gaussian mean estimation, and ridge regression. Numerical experiments on synthetic datasets support the theoretical analysis, showing that an intermediate level of privacy protection can improve long-term utility by mitigating user attrition.

Critical strengths
  1. Problem Formulation: The paper models the feedback loop between privacy leakage and future user participation, providing an interesting perspective on the long-term privacy-utility trade-off by considering realized privacy leakage rather than directly relying on privacy budgets.

  2. Theoretical Analysis: The paper analytically characterizes privacy leakage probabilities and population dynamics under different learning settings, establishing quantitative relationships between privacy noise, user retention, and long-term learning utility.

Critical weaknesses
  1. Existing studies, such as Differential Privacy: An Economic Method for Choosing Epsilon (Hsu et al., 2014) and Optimal Data Acquisition with Privacy-Aware Agents (Cummings et al., 2023), have explored the interplay between privacy protection, user participation, and estimation utility. The paper does not sufficiently clarify the additional theoretical insights offered by its leakage-driven population dynamics.

  2. The long-term utility analysis relies on mean-field approximations and surrogate objectives without establishing optimality for the original objective. Moreover, the proof of Lemma 13 appears to overlook dependencies induced by shared Gaussian noise.

  3. The Bernoulli setting treats unchanged randomized responses as privacy leakage even when the privacy budget approaches zero. Furthermore, the assumed user participation dynamics lack validation beyond synthetic simulations.

Questions for authors
  1. In the proof of Lemma 13, how does the concentration analysis account for the dependencies among users' leakage events induced by shared Gaussian noise? Can the authors provide a rigorous justification for the bounded martingale differences?

  2. What motivates defining unchanged randomized responses as privacy leakage in the Bernoulli setting, particularly when the privacy budget approaches zero? Would the main conclusions remain valid under alternative leakage definitions or more realistic user participation models?

4 comments

W2 + Q1 Mean-field and Lemma 13 We thank the reviewer for pointing out the dependence induced by shared Gaussian noise. The previous proof stated that we were neglecting this effect, but we have now fixed this. We have revised Lemma 13 and its proof in the updated manuscript to account explicitly for this dependence. Under a stated low-leakage condition, the revised result retains the central conclusion that the next-round population concentrates around its conditional expectation, with an additional term accounting for shared-noise effects. The lemma serves an auxiliary role, providing partial theoretical support for the mean-field approximation underlying our optimization proxy. The proxy's main contribution is practical: it is inexpensive to evaluate, enables efficient noise selection, and yields choices that agree reasonably well with empirical optima in our experiments, as can be seen in the figures. The revised analysis preserves this role while making the conditions and scope of the concentration guarantee precise. We believe that the mean-field approximation, by allowing for rather simple formulas that are easy to interpret, contributes to the interest of the paper for human understanding.

W3 + Q2 Leakage definition, participation model In the case of the Bernoulli output, a participant can observe only two possible outputs: either their bit has been flipped, or it remains identical. The model of the paper is that people assess from the output whether their data is leaked, rather than from an abstract definition. If what they see is their true answer, regardless of what happens in the black box, it looks like the data has been leaked, so people have a certain probability of treating it as a leakage. The fact that perceived leakage can happen even at ϵ=0\epsilon = 0 does not invalidate our model: imagine for instance that we did not consider a Y/N question but a histogram computation with a large number KK of possible outputs. Even if the output is uniform over the KK answers, which corresponds to ϵ=0\epsilon = 0, it is very likely that a user seeing their true answer returned by the mechanism will consider their data leaked: users access only one draw, not the full output distribution, which might not even make sense for a non-technical user.

W1 Relation to economic DP works We thank the reviewer for the opportunity to clarify this distinction. Both works are already discussed in Section 2, where we distinguish participation based on anticipated privacy costs and benefits from the repeated population dynamics studied here. Hsu et al. (2014) analyze privacy, compensation, and accuracy under a financial budget, including circumstances where privacy improves accuracy through increased participation. Cummings et al. (2023) characterize optimal estimator weights and noise under agents' participation constraints. These establish important connections between privacy and utility that our work builds upon. (Hsu et al.; Cummings et al.)

These works show that the question of setting ϵ\epsilon is well discussed in the community, but we believe we bring a new perspective with the following advantages:

  • Long-term perspective: most economic DP works only consider a one-round interaction, whereas DP under online or repeated interaction is recognized as an increasingly important topic.
  • Agnostic to DP interpretation: here we do not try to interpret a budget directly, but implement the privacy feedback only through data leakage, which is the more realistic feedback in practice, as it is well recognized that assuming a clear interpretation of privacy budgets by non-technical users is unrealistic.
  • Explicit link with performative learning, which we believe opens new perspectives and can also interest the performative learning community.

Thank you for the detailed response and revisions. I appreciate the correction to Lemma 13, which substantially addresses my concern about dependencies induced by shared Gaussian noise. The differences from prior economic DP studies are also clearer. However, some of my concerns remain. In particular, the Bernoulli leakage definition relies on an unvalidated behavioral assumption, and the main conclusions appear sensitive to the participation dynamics, as also suggested by the alternative model in Appendix C. Moreover, the mean-field surrogate still lacks a theoretical guarantee of optimality for the original stochastic objective. While the revisions improve the paper's technical soundness and clarity, these remaining limitations prevent me from changing my overall assessment. For now, I will maintain my original rating, but I would be open to reconsidering and potentially raising it if the authors can further address these remaining concerns.

We thank the reviewer for engaging in the discussion, and for confirming that many of the points initially raised are adequately addressed. We answer the two remaining points below.

  • Unvalidated Bernoulli assumption. We cannot deploy and validate a behavioral assumption in the time of the rebuttal, but we believe that our simple model has its own merit. We designed it not to be as plausible as possible, but as simple as possible while already exhibiting interesting behavior. As similar behaviors are proved in Sections 4 and 5 under other tasks and data leakage models, we do not believe that the fact that some models would lead to other dynamics makes the Bernoulli setup uninteresting.
  • Bounding the error between the original objective and the mean-field surrogate. Even though reviewer 4ybJ, who also noted this point, considered it a minor concern, we had worked on bounding the error, and we are now including this bound in purple in the new revision. This new result bounds the gap between the original objective evaluated at the surrogate’s optimal γ\gamma and its true optimum. The proof outline is to bound uniformly the difference between the true function and its surrogate, which we obtain through simple analysis tricks and by introducing a constant that depends on TT but not on N0N_0. We will further polish the presentation of this bound in the camera-ready version. This result further confirms what the numerical experiments suggested: the mean-field approximation preserves the same behavior as the exact process, and proves that the optimal γ\gamma remains finite for the original process. The updates have been highlighted in purple in pages 33-37. We hope that this update will lift the remaining concerns the reviewer had on our paper.

Thank you for the further clarifications and substantial theoretical revisions. I appreciate the authors' continued efforts to address my remaining concerns. The successive revisions have strengthened the paper and addressed most of my original concerns. I am therefore willing to raise my rating. I encourage the authors to clearly state the assumptions and scope of the new optimality-gap guarantee, as well as the limitations of the behavioral model, in the final version.

Reviewer_4ybJOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work initiates the study of "performative privacy", a novel problem setting for private data analysis that sits at the intersection of differential privacy and performative machine learning.

The basic idea is as follows: Stronger DP protections hurt utility. However, lack of sufficiently strong protection may cause leakage events, which may reduce the willingness of individuals to contribute to a dataset. This, may in turn, have negative effects on utility. Thus, a natural question for an analyst is which noise level / privacy budget should be chosen to maximize utility.

This problem is defined and analyzed in three settings: Mean estimation for Bernoulli random variables under local DP, mean estimation for real-valued distributions under centralized DP, and ridge regression under centralized DP. In each setting, leakage is defined as a successful membership inference attack through a uniformly most powerful test. Leakage causes individuals to be removed from the dataset with some probability qq, while non-leakage causes each individual to recruit a new number with another probability rr.

In all settings, the expected growth per step is analyzed in terms of qq, rr, and privacy budget / noise level. Furthermore, optimal noise levels in terms of utility are derived, albeit only under a mean-field assumption (attack success is independent across individuals), which is only an approximation in the centralized setting.

The experiments consist of simulations of the respective birth-death processes:

  • For randomized response / local DP: The evolution of the population size under different ϵ\epsilon, as well as the effect of qq and rr on optimal noise level are visualized
  • For centralized DP: The process has the same form under mean-field assumption. Thus, only the empirically (grid-search) and theoretically optimal noise levels are compard.
Critical strengths

Novel problem with strong motivation

This work formalizes a problem with clear practical relevance. The importance of the problem is also evidenced in the related work section: Various prior work has anticipated the importance of differential privacy in ensuring continued data contributions, but no one has actually bothered to formally study the problem. In particular, this submission offers a novel perspective on how privacy budget should be chosen, which is a fundamental problem that affects all DP literature.

As such, I consider novelty and potential impact very high.

Caveat: I come at this paper from a differential privacy background. I cannot speak to its novelty from a performative machine learning perspective.

Appropriate modeling choices

Modeling data contribution over time as a birth-death process (essentially Galton-Watson) appears like a natural choice. Using likelihood ratio tests / Neyman-Pearson lemma is a good, well-founded approach to conducting a worst-case analyses. The considered use cases (localized and centralized mean estimation, ridge regression) are practically relevant. The choice of mechanisms (randomized response, clipping + additive Gaussian) is natural.

While I only checked some of the proofs for correctness, these well-founded modeling choices then appear to have been soundly translated into formal guarantees.

Overall, soundness appears good, except for one potential error (see below).

Critical weaknesses

Strong independence assumptions

This work makes various independence assumptions about data generation, private mechanisms, adversaries, and data owners: The number of records evolves over time, but data is re-drawn at each step.
The private mechanism only provides per-step privacy guarantees, rather than considering continual release / composition.
The adversary targets each record and time step independently and does not, e.g., perform differencing attacks.
Leakage only prevents recruitment of new data owners by the leaked individual, rather than having long-term population-level effects.
Overall the simplicity of the analyzed process may not capture certain real-world dynamics.

Beyond this, the authors only analyze the joint distribution of leakage events when deriving the expected count distribution via law of total expectation, but presumably treat them as independent afterwards ("(inverse) mean-field approximation"). Since the mean-field assumption is only an approximation, it would have been better to provide error bounds instead of only empirically measuring the gap. Furthermore, no part of the main text or appendix actually explains what exact mean-field approximation is, which made later parts of Sections 4 and 5 hard to follow.

Nevertheless, it seems like an excellent starting point for a novel Problem and I would argue that this weakness is significantly outweighed by novelty and potential impact on future work, which can focus on analyzing more complicated processes.

Small error with potentially severe effects

In Eq. 5, the variance is chosen as γ2Nt\frac{\gamma^2}{N_t}. Unless I am completely misreading something, it should be γ2Nt2\frac{\gamma^2}{N_t^2} because the L2 sensitivty is RN\frac{R}{N}.

This may cause various downstream issues and may invalidate the correctness of various Theorems and figures.
For instance, the denominators in the proof of Lemma 7 (l.1182-1185) will no longer match for precisely this reason.

Questions for authors
  • Please comment on the correct of Eq. 5 (noise scale potentially missing a factor of 1/Nt1 \mathbin{/} N_t).
  • What is the mean-field approximation you make? Presumably the Λ\Lambda are treated as independent, but this is never explained
  • What is the "inverse mean-field approximation"? Presumably this is 1/E[Nt]1 \mathbin{/} E[N_t], where the expectation is approximated via mean-field. But this is never explained.
  • Can you maybe come up with some (potentially loose) bounds for the error induced by the mean field approximation (either for number of leakage events, expected population size, optimal noise multiplier, or something similar?)

If the first three are resolved, I will increase my score The fourth one would be nice to have. There's no need to respond to my entire "strong independence assumptions" paragraph. What you have is good enough, I don't expect you to rewrite the entire paper.


Other comments (please respond to these)

  • As you state in ll. 062-064, the term "performative privacy" was already coined by prior work in a completely different field (Skinner-Thompson, 2017 [1]). This overloading of terms may cause confusion for future readers from both fields. In particular, performative machine learning and this submission use "performative" in the traditional dictionary sense of "having the effect of performing an action", whereas the other work uses performative in the modern sense of "intended to impress someone". Concretely, they consider the use of privacy mechanisms as a means to communicate political views on surveillance ("expressive resistance against an ever-pervasive surveillance society ). While I am not a legal scholar, I do not believe that ll 062-064 properly communicate these differences. This part of the related work section should be improved.

Other comments (no need to respond to each one individually, but please consider implementing)

  • In Section 3, I was initially confused why "report true value" is treated as a leakage event. Afterall, the whole point of DP is that it offers plausible deniability. I believe you could make the narrative much stronger / easier to follow if you already introduced the Neyman-Pearson lemma here, and then stated that the threshold τ\tau is chosen such that we are never in the Λ=τ\Lambda = \tau case where the optimal test would be non-deterministic.
  • Abstract: Maybe replace "mean estimation" with "mean estimation under local and centralized DP", so that the results from Section 3 are also mentioned
  • l. 324: Maybe explain where the subscript in z‾1\overline{z}_1 comes from (linearity of expecation and independence of mechanism noise, so it doesn't matter which index we choose).
  • Conversely: The subscript seems to be missing in the proof in l. 1430
  • Lemma 1: Please specifiy that this result holds for fixed ZtZ_t (is clear in the appendix, but not in the main text)
  • Please consider adding forward pointers to the respective proof for all Theorems / Lemmata. E.g., Lemma 2 and Lemma 3 do not have forward pointers.

Typos

  • l. 362: theorem is lowercase. Maybe you accidentally used \ref instead of \cref
  • Lemma 1 in l.1188: I believe it should maybe be ZtZ_{t} instead of Zt−1Z_{t-1}

My references

[1] Skinner-Thompson. Performative Privacy. 2017

3 comments

Q1 Noise scale in Eq. 5 We thank the reviewer for catching this error. When moving from the local model (Sec. 3) to the central model (Sec. 5), Section 4, which applies to both settings, was written in the central DP model but with the noise scaled as in the local model. We have corrected the noise scale and re-run all experiments. Sec. 4 now presents both the local and central variants. The theoretical results for the central variant, obtained by extending the analysis of the local model, are provided in the appendix. The corrected experiments show the same qualitative behavior.

Q2 + Q3: Definition of mean-field approximation We study the leakage events Λi\Lambda_i conditionally on ZtZ_t, which carries the main source of correlation between them. Conditionally on ZtZ_t, we treat the individual leakage events as independent. This is realistic for the range of NtN_t we consider, and consistent with standard practice in membership inference attacks, where membership of each sample is tested individually. We have made this explicit in the revision. We then use that NtN_t is large enough with small variance from a round to another to approximate E(1/Nt)\mathbb{E}(1/N_t) by 1/E(Nt)1/ \mathbb{E}(N_t). This yields tractable, interpretable formulas, and our numerical experiments validate the approximation. We have added references to prior work that uses this approximation more formally.

Q4 Bounds on mean-field error We did not find a bound tight enough to be informative. Since the approximation is validated experimentally and concerns only a subset of our results, we believe, as the reviewer suggests, that the paper can do without such a bound.

Strong independence assumptions We fully agree that more complex dynamics could be at work in real-world scenarios, ranging from possible model collapse to epidemic propagation above a given leakage level, or increased unfairness across subpopulations. We see as a contribution of the paper the fact that we abstracted a minimal setting where we can already exhibit a finite privacy budget as optimizing long-term utility, without introducing arbitrary additional parameters. We are now working on more complex scenarios for a different paper. So far, we always find that the takeaway of an optimal finite privacy budget, with a size that depends on the strength of the feedback loop, remains true; there is indeed room for interesting phenomenology, but that is beyond a single conference paper. We also refer the reviewer to our response to Reviewer vxkT regarding the fact that we do not consider cumulative loss across iterations.

Overloaded term performative privacy We understand the reviewer's concern that the term performative privacy has already been used in another domain. We believe this overloading is fine, as it is a consequence of the polysemy of the word performative. As machine learning has adopted the term performative learning, it seemed a natural and rational choice to coin performative privacy, and we have positioned ourselves with respect to the existing work. We remain open to alternative suggestions.

Presentation suggestions and typos We thank the reviewer for their careful assessment of our work. The fixes made thanks to this feedback are highlighted in cyan, so that you can quickly check our changes. Note that for Lemma 1 we do not need a fixed ZtZ_t.

Thank you, my primary concerns have been adequately addressed. In particular, my concern around correctness has been resolved.

I have increased my score as promised.

One last thing: I do not think that the overloaded terminology is a problem. I just meant to say that it would be nice if you could clarify the difference between the two notions of "performative" in l.63 or somewhere in Section 2.

AuthorsOpenReview ↗

Reply to Official Comment by Reviewer 4ybJ

We thank the reviewer for recognizing the potential and novelty of our work by raising their score and supporting acceptance. We also appreciate the suggested clarification and have added a sentence in lines 64-66, highlighted in magenta, to distinguish the two meanings of “performative”.

Reviewer_vxkTOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Drawing on the concept of performative learning, this paper introduces performative privacy, a perspective that considers the long-term utility of privacy protection. Under the mechanism of differential privacy, the optimal privacy budget is the one that maximizes long-term utility. This paper proposes a model and demonstrates through calculations across three types of scenarios that a finite privacy budget can achieve optimality, thereby maximizing long-term utility.

Critical strengths

This paper introduces the concept of performative privacy and is the first to examine the long-term utility of privacy protection, with rigorous theoretical proofs provided for the three investigated tasks, namely Bernoulli parameter estimation, mean estimation and ridge regression.

Critical weaknesses
  1. The theoretical model relies on fairly strong assumptions. For instance, it assumes a fixed distribution and does not account for cumulative privacy loss under multi-round stochastic DP interactions, nor settings where users retain memory and depart only after repeated leakage events.
  2. All experiments use synthetic data and lack real user behavior data. Since user behavioral feedback forms the core closed loop of this work, it remains unclear whether simulations alone can validate the existence of such feedback loops in real-world settings.
Questions for authors

see "Critical Weaknesses"

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.