acceptodds
Under review as a conference paper at ICLR 2027

Privacy Without Privatization: Inherent Membership Privacy in Ridge Regression

Abstract

Privacy in machine learning is commonly obtained by modifying the learning algorithm, for example to satisfy differential privacy (DP). We ask a complementary question: how much privacy may already be present in an ordinary, unmodified model? We study this in the context of membership inference risk in ridge regression models. To this end, we employ a recent information theoretic notion called pointwise maximal cost (PMC), which quantifies how much observing a particular released model can improve an adversary's inference relative to prior knowledge. In particular, we derive an exact characterization of membership inference risk under the PMC interpretation for ridge regression. In regimes where the numerical estimates are reliable, leakage is nonzero but generally small, typically decreases as the number of training records grows relative to dimension, and varies across released models and membership inference targets. The same qualitative behavior persists under a feature population derived from real-world data. Furthermore, at matched privacy levels, we show that two existing formal DP regression methods incur substantially greater prediction error than ordinary ridge regression. Thus, our results suggest that PMC can serve as a release-evaluation tool for quantifying inherent privacy before deciding whether additional privatization is necessary.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
information theory, privacy, regression models, membership inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zWLnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces studies the pointwise maximal cost (PMC) notion in the ridge regression setting. It uses PMC to evaluate the inherent privacy of the dataset itself. This paper applies it to the ridge regression setting, defines the corresponding terms, and compares with the standard differential privacy notion on simulated data sets.

A big difference between PMC and differential privacy is that PMC requires an underlying model, and DP guarantees the privacy whatever the dataset is. The current matching between the two notions are not persuasive.

Critical strengths

Strength: considers the inherent privacy of a dataset, and studies a notion other than the standard differential privacy. I think this diversity is good for the community.

Critical weaknesses
  1. The discussion of PMC (or the inherent privacy) is quite rare. With this new notion, I would wonder
  • What is the new understanding of the privacy problem?
  • How does the new notion connect with the classical differential privacy? For example, the new notion requires model assumptions while differential privacy does not need it, then how this difference affects the final result? To find ϵmatch\epsilon_{match}, how the definition of DP affects this matching? What is the fair part and what is the unfair part? For the unfair part, how do we visualize it in the comparisons?
  • Now it requries model assumptions. How do we solve this problem in real data applications?
  1. Limited numerical analysis, especially on the real datasets.

  2. Theoretical results are not very deep (not critical).

Questions for authors

See the weakness.

8 comments
AuthorsOpenReview ↗

Responses to the first two bullet points

Response to the first bullet point: We thank the reviewer for this question. The central insight of our work is that privacy need not be viewed exclusively as a property enforced by modifying a learning algorithm; it can also be evaluated for an individual model, trained via a standard noise-free training mechanism. Using PMC, we quantify how much observing a released model improves an adversary's inference about a protected secret, independently of any particular attack strategy. Our results demonstrate that ordinary ridge regression can exhibit relatively small privacy leakage even without any explicit privatization, with leakage varying across released models, protected targets, and training sample sizes.

This perspective is motivated by potential application in privacy certification of models (not of training algorithms), where a specific model trained inside a secure enclave is evaluated to determine whether it is safe to release from that enclave. We believe our framework provides a useful foundation for future investigations into the inherent privacy of other machine learning models.

Response to the second bullet point: We thank the reviewer for these helpful questions. PMC and DP provide different types of privacy guarantees. PMC measures the information disclosed by a particular trained model about a specified secret under an assumed data distribution, whereas DP provides a uniform guarantee over neighboring datasets without requiring assumptions about the data distribution. The narrower scope of PMC allows us to evaluate the inherent privacy of an unmodified model, potentially certifying it as sufficiently private, or "safe to release," without introducing additional noise. In contrast, we are not aware of any DP guarantees which can be made without introducing additional noise. As our experiments illustrate, avoiding such noise preserves substantially greater predictive utility.

Our choice of ϵmatch\epsilon_{\mathrm{match}} follows from the connection between PMC and log-likelihood ratios established in Section 5.2 and Appendix A.4. We first measure the privacy leakage of a released ridge regression model using PMC. We then determine the value of ϵmatch\epsilon_{\mathrm{match}} for which the upper bound on PMC implied by a symmetric log-likelihood-ratio constraint equals the observed leakage. We use ϵmatch\epsilon_{\mathrm{match}} in (ϵmatch,δ)(\epsilon_{\mathrm{match}},\delta)-DP algorithms when comparing predictive utility, which guarantees an identical bound on log likelihood, as shown in the standard lemma (see response to reviewer 1) which we have included in a revision of the manuscript. This provides a principled way of comparing DP and PMC using identical bounds on the log likelihood, despite the fact that these are vastly different notions. Moreover, our approach to PMC does not involve any noise during training, and hence does not guarantee any form of DP.

We acknowledge that the comparison does not establish equivalent privacy protection. The comparable aspect is the predictive utility of the algorithms evaluated at privacy parameters which establish a common likelihood-ratio interpretation. The difference is that DP provides a broader, distribution-independent guarantee, whereas our PMC evaluation concerns a specific secret, population, and realized model. Another crucial difference is that PMC establishes guarantees for a single realized model trained without any privatization mechanism, whereas to the best of our knowledge such guarantees cannot be made via DP. We have made this distinction clearer in the revised manuscript and explicitly identified the scope of each guarantee when presenting the experimental comparisons.

AuthorsOpenReview ↗

Responses to the remaining points

Response to the third bullet point: We thank the reviewer for raising this practical limitation. Although our primary experiments assume a known Gaussian feature distribution, the general characterization is not restricted to Gaussian populations, as the secret-conditioning procedure only requires i.i.d. features. To illustrate this flexibility, Section 6.3 already considers a non-Gaussian feature population constructed from real-world measurements in the WDBC dataset, where the same qualitative privacy behavior is observed.

In practical applications, one could estimate the feature distribution using representative data and evaluate PMC under this estimated population model. However, the resulting privacy assessment would be conditional on the accuracy of that estimate, and lack of distribution specification could affect the calculated leakage. Developing robust PMC guarantees that account explicitly for uncertainty in the underlying distribution is therefore an interesting direction for future work.

Response to comments on numerical analysis and theoretical depth: We thank the reviewer for these comments. Regarding the numerical analysis, we would like to emphasize that our experiments cover a range of settings, including different training sample sizes, feature dimensions, membership priors, and protected target locations. In addition to the synthetic Gaussian experiments, Section 6.3 investigates a non-Gaussian feature distribution derived from the real-world WDBC dataset, where we observe similar privacy behavior. Further experimental results and numerical analyses are provided in Appendix D.

Regarding the theoretical contribution, we respectfully point the reviewer to the detailed proofs in Appendix B, which establish the exact conditional distribution of the ridge estimator, characterize the feature distributions under proximity membership and non-membership, and derive the corresponding secret-conditioned output distributions. These results require careful probabilistic conditioning and integration over the possible training datasets, rather than simply applying the definition of PMC. We believe that the technical derivations and the resulting exact characterization provide a meaningful theoretical foundation for evaluating the inherent privacy of ridge regression.

Thank you for the response. The current response is consistent with the paper's tune, but it does not address my concerns.

  1. PMC itself, raises another notion of privacy. However, this notion is not from the current paper. Hence, what is the new discovery in this paper? The current statement is quite general, like "how much observing a released model improves an adversary's inference about a protected secret". What is the general understanding of this problem? How does this work pushes this understanding to its boundary?

  2. The narrower scope of PMC is a big concern in the concept of DP. I don't think real data aligns with such a good model. Under this narrower assumption, do we really get new guidance on privacy?

  3. While this reponse aims to respond for the first two bullets, it only discusses Bullet 1. Still, there is limited data analysis.

Reviewer_UJbTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors show that standard ridge regression is inherently private under a notion of privacy related to pointwise maximal cost (PMC), which measures how much observing the model coefficients can change an adversary's knowledge of which data were used to train the model compared to their prior knowledge. They give algorithms for estimating the model-dependent PMC value. Since the ridge models inherently have privacy-preserving characteristics, a higher-utility model can be released compared to enforcing privacy via differential privacy, and the authors confirm this empirically.

Critical strengths

The paper is very clearly written and easy to follow. Most of the theoretical results/intuitions for different quantities are explained in a way which makes the intuition clear.

The notion of "intrinsic privacy" is also interesting. A relaxed notion of privacy which assumes that the attacker does not have arbitrary side information may actually be more realistic in practice, and offers the opportunity to preserve more model utility while still providing a rigorous defintion of privacy.

Finally, a model-dependent measure of privacy is also quite interesting. It make sense that such a quantity could exist, e.g., models which generalize could intuitively leak much less information than models which generalize. This seems like a valuable idea for the literature.

Critical weaknesses

A bit more discussion of how the notion of privacy considered in the paper compares to existing notions (specifically DP) would be useful to discuss. My understanding is that the privacy notion in this paper is (1) relative to a (known) underlying data distribution, and (2) does not assume arbitrary side information. In particular, the posterior probabilities considered in this work assume that none of the training features are known to the attacker. If the adversary knew all but one datapoint in the training data and observed the model, it is possible that they would be able to recover that datapoint exactly. These limitations/differences should be discussed explicitly.

Why the chosen value of ϵmatch(w)\epsilon_{\textrm{match}}(w) is a reasonable level at which to compare PMC and DP is not clear. My guess is that this is the largest value of ϵ\epsilon for which ϵ\epsilon-DP implies PMC at the observed level, but this should be stated explicitly if so. Appendix A.4 essentially just repeats the exposition in Section 5.2. It is very critical that this connection is made clear, otherwise the comparison of standard ridge regression vs. DP-enforced ridge can't be considered meaningful.

Finally, there are some existing works which define privacy notions in terms of an attacker's ability to perform membership inference which are closely connected to the present work. These should be discussed:

Questions for authors

Why is ϵmatch(w)\epsilon_{\textrm{match}}(w) is a reasonable level at which to compare PMC and DP/what is the precise connection to DP?

Some of the ESS values reported for the experiments are quite high. How many Monte Carlo samples were used for these experiments?

PMC is measured with respect to a known underlying data distribution. In practice, we won't know the exact distribution of the data. How would you measure the privacy leakage given an unknown feature distribution, and does anything break if e.g. we use an estimate for the underlying distribution which is not exactly equal to the true distribution?

This work shows that ridge regression inherently limits an attacker's ability to recover features of the training data. Does a similar result hold for the labels yy?

Finally, I understand that this is primarily a theoretical paper, but it would be interesting to see whether these results have any implications for the success rate of practical membership inference attacks. Experiments in this direction would strengthen the practical relevance of the paper but I wouldn't consider them necessary.

3 comments
AuthorsOpenReview ↗

Responses to the weaknesses

We thank the reviewer for their helpful suggestions. The reviewer's understanding is correct: our privacy characterization is distribution-dependent and assumes that the adversary's prior knowledge is captured by the specified data-generating distribution and protected secret, without conditioning on arbitrary additional information about the training dataset. We agree that this is an important distinction from DP, which provides uniform protection over neighboring datasets even in the presence of arbitrary side information. Our objective is not to replace DP, but to investigate the complementary question of how much privacy is already present in an ordinary, unmodified model under a specified threat model, crucially, without altering or adding any noise to the training mechanism. We acknowledge that stronger adversarial knowledge, such as access to all but one training record, could lead to substantially DP leakage. We have expanded the discussion in a revision to explicitly state these assumptions, limitations, and distinctions from DP.

We thank the reviewer for pointing out that the motivation for our matching procedure was not sufficiently clear, and we apologize for this lack of clarity. As discussed in Section 5.2 and Appendix A.4, our choice of ϵmatch\epsilon_{\mathrm{match}} is based on the relationship between PMC and bounds on the log-likelihood ratio. The corresponding connection between differential privacy and log-likelihood ratios is also well established, but we should have explained it more explicitly in the original submission. For the reviewer's convenience, we provide a short lemma (see response to reviewer 1) establishing this connection, which we have also includes in the revision.

The reasoning behind our choice is straightforward. We first measure the privacy leakage of a released ridge regression model using PMC. We then determine the value of ϵmatch\epsilon_{\mathrm{match}} for which the upper bound on PMC implied by a symmetric log-likelihood-ratio constraint equals the observed leakage. We use ϵmatch\epsilon_{\mathrm{match}} in (ϵmatch,δ)(\epsilon_{\mathrm{match}},\delta)-DP algorithms when comparing predictive utility, which guarantees an identical bound on log likelihood via the lemma we provide (see response to reviewer 1). This provides a principled way of comparing DP and PMC using identical bounds on the log likelihood, in spite of the fact that these are vastly different notions. Moreover, our approach to PMC does not involve any noise during training, and hence does not guarantee any form of DP.

We emphasize that this comparison does not mean PMC and DP offer equivalent privacy guarantees, since they protect against different types of disclosure under different assumptions. We have made both the reasoning behind our choice and this distinction clearer in the revised manuscript.

We also thank the reviewer for bringing relevant works to our attention. While Izzo et al. (2024) and Leemann et al. (2023) study membership inference privacy and its relationship with differential privacy, we would like to emphasize that the central contribution of our paper is the characterization of inherent privacy in ordinary ridge regression without any explicit privatization. Unlike these works, our focus is not on designing or analyzing privacy mechanisms to enforce membership privacy, but on quantifying how much privacy is already present in an individual model trained via a standard noise-free algorithm. Specifically, we use PMC to evaluate the information disclosed by released ridge coefficients about a proximity-membership secret, independently of any particular attack. This release-specific evaluation of inherent privacy is the main distinction and novelty of our work. Nevertheless, we appreciate the reviewer's suggestions and have incorporated both references and further clarified these distinctions in the revised manuscript.

AuthorsOpenReview ↗

Responses to the questions

Regarding ESS, as specified in Appendix D.1–D.2, we use 60 independently trained ridge regression models per experimental setting. For each released model, the secret-conditioned densities are estimated using M=10000 Monte Carlo samples per conditional distribution when p=10. For p=20, we use M=50000, 30000, and 15000 for n=50, 100, and 200, respectively. The reported ESS values are therefore within the corresponding Monte Carlo budgets. We emphasize that ESS measures the concentration of likelihood contributions rather than the statistical accuracy of the resulting PMC estimate. We have made these sample sizes more prominent in the revised manuscript.

In addition, our current analysis assumes a known feature distribution, but the general characterization is not restricted to Gaussian features. In practice, one could estimate the feature distribution from representative data and apply the same framework using this estimated distribution to construct the secret-conditioned output densities. The resulting PMC would then quantify privacy relative to the estimated population model. Quantifying the sensitivity of PMC to distribution estimation errors and developing robust guarantees under distributional uncertainty are interesting directions for future work.

Regarding the question on labels, while our current analysis focuses on proximity membership defined through the training features, the general PMC framework can also be applied to secrets involving the training labels. In particular, one could define an analogous label-based proximity-membership event and characterize the distributions of the released ridge coefficients conditioned on the presence or absence of such an event. The resulting likelihood ratio would again determine PMC through Theorem 1. However, the specific conditional distributions would differ from those derived in our paper, and our current results do not directly establish comparable privacy guarantees for labels. We view extending the analysis to label-based secrets as a natural direction for future work and have clarified the scope of our present results in the revision.

Finally, while our work focuses on an attack-independent, information-theoretic characterization of membership privacy, we agree that investigating its implications for practical membership inference attacks would be valuable. In particular, comparing PMC with the empirical success rates of existing attacks could help bridge the gap between theoretical privacy guarantees and practical adversarial performance. We have highlighted this direction for future work in the revised manuscript.

Thanks to the authors for deeply engaging with the review. My questions have been answered and I recommend accepting the paper.

I would also like to add a note in favor of the authors with regard to the discussion with Reviewer zoar. The example provided by this reviewer is interesting and shows that some simple forms of aggregation with Gaussian data already give some PMC guarantees. However, I don't think that (1) it's immediately obvious how this general fact would imply the result for linear regression, or (2) that the existence of this general fact makes studying the inherent privacy of linear regression uninteresting; it seems to be mostly orthogonal.

Reviewer_zoarOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies privacy leakage of a specified secret from ridge regression under the hypothesis that training data is sampled i.i.d from some distribution, rather than the usual the worst-case adjacent datasets setting of DP. In this setting, the authors leverage the Pointwise Maximal Cost (PMC) and derive an exact representation for PMC under this setting. It is estimated numerically through a Monte Carlo simulation, and is compared to DP guarantees. The experiments report small leakage and compare utility against two DP methods at numerically corrsponding parameters.

Critical strengths

The setting is sound. In this paper, data is sampled i.i.d from a specified distribution, and the secret depends on the data. In full generality, this corresponds to a Pufferfish-style privacy instantiation, which is legitimate. The condional distribution construction allows to evaluate the privacy leakage related to a release without focusing on a particular attack.

Critical weaknesses
  • The characterisation obtained in Theorem 1 does not provide any insight about what ridge regression contributes to privacy beyond a simple aggregation mechanism. Theorem 1 is a generic Bayes rule applied to the PMC. For any binary secret with prior π=P(S=1)\pi = P(S = 1), conditional distributions g0,g1g_0,g_1 and L=log⁡(g1/g0)L = \log(g_1/g_0), Bayes rule yields qr(w)=πeL(w)1−π+πeL(w)q_r(w) = \frac{\pi e^{L(w)}}{1-\pi + \pi e^{L(w)}}. Substituting this formula into the definition of PMC yields Theorem 1. This identity is not specific to ridge regression. More importantly, it is unclear whether the privacy guarantees provided by this characterization go beyond those already provided by a simple aggregation of the Xi∼N(0,Ip)X_i \sim \mathcal N(0, I_p). Consider T(X)=∑i=1nXiT(X) = \sum_{i=1}^n X_i. Fix p,z,π∈(0,1)p,z, \pi \in (0,1) and choose rnr_n such that arn=P(Xi∈B(z,rn))=1−(1−π)1/n=O(n−1)a_{r_n} = P(X_i \in B(z,r_n)) = 1-(1-\pi)^{1/n} = O(n^{-1}). Also, conditioning on Srn=0S_{r_n} = 0 leaves the XiX_i independent of distribution Qrn=P(Xi∣Xi∉B(z,rn))Q_{r_n} = P(X_i | X_i \notin B(z,r_n)), and W1(Qrn,N(0,I))=O(an)W_1(Q_{r_n}, \mathcal N (0,I)) = O(a_n). We note Xi′∼QrnX_i' \sim Q_{r_n}. Therefore, since ∑i=k+1nXi\sum_{i=k+1}^n X_i is Gaussian, we have: TV(∑i=1kXi′+∑i=k+1nXi,∑i=1k−1Xi′+∑i=knXi)≤W1(QrnN(0,I))2n−k.TV\Big(\sum_{i=1}^k X_i' + \sum_{i=k+1}^n X_i, \sum_{i=1}^{k-1} X_i' + \sum_{i=k}^n X_i\Big) \leq \frac{W_1(Q_{r_n} \mathcal N (0,I))}{2\sqrt{n-k}}. For the final replacement, we have: TV(Xn+∑i=1n−1Xi′,∑i=1nXi′)≤TV(Qr,P)=arTV(X_n + \sum_{i=1}^{n-1} X_i', \sum_{i=1}^{n} X_i') \leq TV(Q_r, P) = a_r. By the triangle inequality, TV(T∣Srn=0,T)≤O(ar)(1+∑i=1ni−1/2/2)≤O(n−1/2).TV(T | S_{r_n} = 0, T) \leq O(a_r)\Big(1 + \sum_{i=1}^n i^{-1/2}/2\Big) \leq O(n^{-1/2}). Since PT=(1−π)PT∣Srn=0+πPT∣Srn=1P_{T} = (1-\pi)P_{T | S_{r_n}=0} + \pi P_{T|S_{r_n}=1}, we obtain TV(T∣Srn=0,T∣Srn=1)≤O(n−1/2)TV(T | S_{r_n}=0, T|S_{r_n}=1) \leq O(n^{-1/2}), which for the balanced prior also implies E[1−e−Λ(T)]=O(n−1/2)\mathbb E [1 - e^{-\Lambda(T)}] = O(n^{-1/2}). Therefore, for this choice of data-generating distribution and secret, PMC converges to 00 in probability trough averaging alone, without regression or regularization. Since ridge regression itself relies on empirical averages, this raises the question of what additional privacy, if any, is specific to ridge regression.
  • The privacy–utility comparison does not establish matched protection. The quantification obtained in this paper is obtained through a specific modelization of the data and the secret, by comparing output distributions conditioned on Kr=0K_r=0 and Kr≥1K_r\geq 1. In contrast, DP upper bounds the privacy leakage uniformly over adjacent datasets. Equality between numerical parameters in different frameworks does not imply equivalent privacy guarantees, as the adjacency relationship, the data-generating distribution and the measure of privacy loss differ. Section 5.2 acknowledges that the matching is only numerical, but this qualification does not establish that the methods provide comparable protection, as suggested by the abstract’s phrase “at matched privacy levels.” The experiments therefore show different utility at some chosen numerical parameters rather than a privacy/utility trade-off.
Questions for authors
  • The chosen privacy instantiation, through the choice of i.i.d Gaussian distribution for the dataset and the specific choice of secret, already allows to obtain privacy bounds through averaging the elements of the dataset. Does the ridge regression analysis contribute beyond this consideration? Do the authors identify an effect which is due to the specific form of the parameters?
  • Could the authors tune the privacy guarantees of the DP algorithm under the same adjacent secrets, data generating distribution and PMC as a measure of privacy loss? This would allow to compare privacy/utility trade-offs in a more suitable way.
3 comments
AuthorsOpenReview ↗

Responses to the two critical weaknesses and a relevant lemma with complete proof

Response to first weakness: We thank the reviewer for this observation. However, we do not see its relevance to the central objective of our paper, which is to characterize and evaluate the inherent privacy of ordinary ridge regression. Our analysis specifically considers the information disclosed by the released ridge coefficients, whose distribution depends on the structure of the ridge estimator. The proposed aggregation example, while interesting in its own right, does not appear to serve the main purpose of our paper. We are therefore unsure of why demonstrating privacy through simple aggregation would diminish the significance of evaluating the inherent privacy of ridge regression. Could the reviewer clarify what connection they see between our stated objective and the privacy properties of averaging the data?

Response to second weakness: We thank the reviewer for raising this concern and apologize for not explaining our matching procedure more clearly. We agree that PMC and DP do not provide equivalent privacy guarantees. However, both DP and PMC imply bounds on log-likelihood ratios, which motivates our comparison. Specifically, our paper establishes how PMC bounds the log-likelihood ratio between the output distributions conditioned on different values of the secret. On the other hand, DP also provides a bound on the log-likelihood ratio between the output distributions of neighboring datasets with different values of the secret. The former claim about PMC is established in Section 5.2 and Appendix A.4. The latter claim about DP is formalized by a standard lemma, which we inadvertently omitted from the original submission and provide below for the reviewer's convenience. Our choice of ϵmatch\epsilon_{\mathrm{match}} is motivated by this common likelihood-ratio interpretation. In particular, ϵmatch\epsilon_{\mathrm{match}} answers the following question: For a given bound on the log-likelihood ratio that is given by our PMC analysis, how large should epsilon be so that an (ϵ,δ)(\epsilon,\delta)-DP mechanism induces the same bound on the log-likelihood ratio? We have included the lemma and clarified this distinction in a revision.

Lemma. Let MM be an (ϵ,δ)(\epsilon, \delta)-differentially private mechanism, and let S∈0,1S \in \\{0,1\\} be a binary secret with any prior distribution. Let D0D_0 and D1D_1 be neighboring datasets such that S(D0)=0S(D_0)=0 and S(D1)=1S(D_1)=1, and let W=M(Ds)W = M(D_s). Let PiP_i denote the conditional distribution of WW given S=iS = i, equivalently the output distribution of M(Di)M(D_i), and let gig_i be its density.

Define the log-likelihood ratio L(w)=log⁡(g1(w)/g0(w))L(w) = \log(g_1(w)/g_0(w)), allowing infinite values when exactly one density is zero. Where both densities vanish, define LL arbitrarily. Then, for every t>0t > 0 we have:

Pr⁡(∣L(W)∣>ϵ+t)≤min⁡1,(1+exp⁡(−ϵ−t))δ/(1−exp⁡(−t))\Pr(|L(W)| > \epsilon + t) \le \min\\{1, (1 + \exp(−\epsilon−t))\delta / (1 − \exp(−t))\\}. The probability includes both the randomness of SS and the randomness of MM. Consequently, an (ϵ,δ)(\epsilon,\delta)-DP mechanism provides a bound of epsilon on the log-likelihood ratio of the secret with high probability.

Proof. By DP, for every measurable event AA:

P1(A)≤exp⁡(ϵ)P0(A)+δP_1(A) \le \exp(\epsilon)P_0(A) + \delta, P0(A)≤exp⁡(ϵ)P1(A)+δP_0(A) \le \exp(\epsilon)P_1(A) + \delta.

Fix t>0t > 0, set a=ϵ+ta = \epsilon + t, and define A+=w:L(w)>aA_+ = \\{w : L(w) > a\\} and A−=w:L(w)<−aA_- = \\{w : L(w) < −a\\}. On A+A_+, we have g0≤exp⁡(−a)g1g_0 \le \exp(−a)g_1. Integrating gives P0(A+)≤exp⁡(−a)P1(A+)P_0(A_+) \le \exp(−a)P_1(A_+). Combining this with DP:

P1(A+)≤exp⁡(ϵ)P0(A+)+δ≤exp⁡(−t)P1(A+)+δP_1(A_+) \le \exp(\epsilon)P_0(A_+) + \delta \le \exp(−t)P_1(A_+) + \delta.

P1(A+)≤δ/(1−exp⁡(−t))P_1(A_+) \le \delta / (1 − \exp(−t)).

Swapping the indices gives:

P0(A−)≤δ/(1−exp⁡(−t))P_0(A_-) \le \delta / (1 − \exp(−t)).

On A−A_-, we also have g1≤exp⁡(−a)g0g_1 \le \exp(−a)g_0, so P1(A−)≤exp⁡(−a)P0(A−)P_1(A_-) \le \exp(−a)P_0(A_-). Adding the two tails gives:

P1(A+∪A−)≤(1+exp⁡(−a))δ/(1−exp⁡(−t))P_1(A_+ \cup A_-) \le (1 + \exp(−a))\delta / (1 − \exp(−t)).

By symmetry, the same bound holds under P0P_0. Averaging over SS gives:

Pr⁡(∣L(W)∣>a)=Pr⁡(S=0)P0(A+∪A−)+Pr⁡(S=1)P1(A+∪A−)≤(1+exp⁡(−a))δ/(1−exp⁡(−t))\Pr(|L(W)| > a) = \Pr(S = 0)P_0(A_+ \cup A_-) + \Pr(S = 1)P_1(A_+ \cup A_-) \le (1 + \exp(−a))\delta / (1 − \exp(−t)). Substituting a=ϵ+ta = \epsilon + t and using that probabilities are at most 11 proves the claim.

AuthorsOpenReview ↗

Responses to the two questions

Response to the first question: We thank the reviewer for this observation. As the title and stated objective of our paper make clear, our goal is to characterize the inherent privacy of ordinary ridge regression, not to establish its superiority over aggregation mechanisms. While the reviewer's example is interesting, it appears to address a different question from the one investigated in our work. We are therefore unsure of the claimed connection between the two. Could the reviewer clarify why the privacy properties of simple averaging are relevant to our characterization of the inherent privacy of ridge regression?

Response to the second question: We thank the reviewer for this suggestion. Indeed, we tune the privacy guarantees by choosing ϵmatch\epsilon_{\mathrm{match}} as explained above. We have clarified this choice in the revised manuscript and included the aforementioned lemma to further justify our matching procedure. Could the reviewer clarify whether they are suggesting a different calibration procedure, beyond our choice of ϵmatch\epsilon_{\mathrm{match}}, to “tune the privacy guarantees”?

Thank you for the response. My main criticism is about what the paper teaches us. The example was not meant to suggest studying aggregation instead of ridge regression, but to illustrate that the phenomenon may have little to do with ridge regression itself. The privacy definition chosen in this paper is a specific Pufferfish privacy instantiation defined through two secrets and a dataset drawn from a population distribution, which is legitimate. However, when doing this, the paper must distinguish what is obtained by the privacy instantiation and what is specific to ridge regression.

In DP, a canonical example of a non-private query is a simple noiseless aggregation query. Yet, under the chosen population distribution and secret, such a query has a vanishing PMC. This suggests that low privacy leakage may be expected because of the privacy setting, independently of ridge regression. Therefore, in my opinion, merely observing that ridge regression is private here is not, by itself, an informative finding. The main theorem (Theorem 1) is a direct application of the Bayes rule that is not specific to ridge regression and provides no explicit privacy guarantees. I would expect the paper to prove a specific effect of ridge regression, or establish quantitative bounds for example.

My objection is not that ridge regression should be compared to aggregation, but that the paper should explain what the study of ridge regression adds beyond the privacy setting itself.

Comments on the paper

1 comment

Dear Reviewers,

We would like to kindly follow up on our responses to your comments. We have carefully addressed the concerns raised and incorporated the suggested clarifications and improvements.

As the discussion period concludes tomorrow, we would greatly appreciate it if you could take another look at our responses and revisions and let us know if you have any remaining questions or concerns.

Thank you again for your time, thoughtful feedback, and consideration of our work.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.