Privacy Without Privatization: Inherent Membership Privacy in Ridge Regression

Privacy in machine learning is commonly obtained by modifying the learning algorithm, for example to satisfy differential privacy (DP). We ask a complementary question: how much privacy may already be present in an ordinary, unmodified model? We study this in the context of membership inference risk in ridge regression models. To this end, we employ a recent information theoretic notion called pointwise maximal cost (PMC), which quantifies how much observing a particular released model can improve an adversary's inference relative to prior knowledge. In particular, we derive an exact characterization of membership inference risk under the PMC interpretation for ridge regression. In regimes where the numerical estimates are reliable, leakage is nonzero but generally small, typically decreases as the number of training records grows relative to dimension, and varies across released models and membership inference targets. The same qualitative behavior persists under a feature population derived from real-world data. Furthermore, at matched privacy levels, we show that two existing formal DP regression methods incur substantially greater prediction error than ordinary ridge regression. Thus, our results suggest that PMC can serve as a release-evaluation tool for quantifying inherent privacy before deciding whether additional privatization is necessary.