- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors formalize this leakage as predictive exposure. Given a portion of a dataset generated by a stochastic process, predictive exposure measures how well an observer can predict statistics about sensitive properties of the population of unknown individuals, before and after observing the algorithm’s output. They show that when the leaked portion of the dataset is generated by a stationary, ergodic, mixing process, they study the asymptotic predictive exposure of an algorithm using a classic tool, the generalized method of moments.
Critical strengths
-
The paper investigates the leakage of sensitive aggregate properties of an unknown population, rather than focusing solely on individual records. This provides an interesting perspective on privacy risks that are not directly captured by traditional individual-level privacy notions.
-
The connection between predictive exposure and efficient GMM estimation is theoretically interesting. In particular, characterizing information leakage through canonical correlation provides an interpretable relationship between sensitive queries and the moment conditions induced by learning algorithms.
Critical weaknesses
-
The main theoretical results rely on GMM regularity conditions, asymptotic efficiency, and assumptions such as strict convexity for ERM. However, the deep learning experiment involves a nonconvex Llama model, for which the authors explicitly acknowledge that the GMM assumptions do not hold. This raises concerns about the applicability of the theoretical guarantees to practical deep learning settings.
-
The empirical evaluation is limited and does not directly measure the proposed predictive exposure metric. Instead, the authors use the accuracy of an MLP trained to predict author gender as a proxy for information leakage. However, gender prediction accuracy does not directly quantify the reduction in an observer's prediction risk after observing the released model, as defined by predictive exposure.
-
The noise covariance in Theorem 7 involves Hessian-related and gradient covariance terms, which may be computationally expensive for large-scale neural networks. Although the method is evaluated on a 1B-parameter language model, the computational overhead and scalability of the covariance approximation remain unclear. Further implementation details and computational cost analysis would strengthen the practical relevance of the proposed mechanism.
Questions for authors
-
Although Proposition 2 establishes that predictive exposure and differential privacy are theoretically incomparable, could the authors further clarify the practical motivation for introducing predictive exposure, particularly in comparison with existing population-level privacy frameworks? In which realistic scenarios does the proposed notion offer guarantees that existing approaches cannot adequately provide?
-
How is the calibrated noise covariance in Theorem 7 computed or approximated for large neural networks, and how do the computational and memory costs scale with the number of model parameters? Could the authors provide runtime and memory comparisons with isotropic noise?
-
The TOFU experiment uses gender prediction accuracy as a proxy for predictive exposure. Could the authors directly evaluate the reduction in an observer's prediction risk before and after observing the released model, as defined in Definition 3?