acceptodds
Under review as a conference paper at ICLR 2027

On Limiting the Predictive Exposure of Learned Models

Abstract

The outputs of machine learning algorithms can reveal properties of the underlying populations in the dataset. Some of these aggregate properties of the dataset may be considered sensitive or proprietary (such as the overall customer base and marketing strategy of a business, rather than a particular transaction). Thus, we want to limit the information leaked from a learning algorithm about a sensitive aggregate property of the unknown portion of the dataset. We formalize this leakage as predictive exposure. Given a portion of a dataset generated by a stochastic process, predictive exposure measures how well an observer can predict statistics about sensitive properties of the population of unknown individuals, before and after observing the algorithm's output. We show that when the leaked portion of the dataset is generated by a stationary, ergodic, mixing process, we can study the asymptotic predictive exposure of an algorithm using a classic tool, the generalized method of moments hansen1982. Our analysis shows how learning algorithms can be modified to limit predictive exposure. Using our analysis, we derive an exposure-limiting output perturbation scheme for ERM.

Primary area:
learning theory
Keywords:
predictive exposure, population statistics, information leakage, parameter estimation, generalized method of moments

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kW9MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors formalize this leakage as predictive exposure. Given a portion of a dataset generated by a stochastic process, predictive exposure measures how well an observer can predict statistics about sensitive properties of the population of unknown individuals, before and after observing the algorithm’s output. They show that when the leaked portion of the dataset is generated by a stationary, ergodic, mixing process, they study the asymptotic predictive exposure of an algorithm using a classic tool, the generalized method of moments.

Critical strengths
  1. The paper investigates the leakage of sensitive aggregate properties of an unknown population, rather than focusing solely on individual records. This provides an interesting perspective on privacy risks that are not directly captured by traditional individual-level privacy notions.

  2. The connection between predictive exposure and efficient GMM estimation is theoretically interesting. In particular, characterizing information leakage through canonical correlation provides an interpretable relationship between sensitive queries and the moment conditions induced by learning algorithms.

Critical weaknesses
  1. The main theoretical results rely on GMM regularity conditions, asymptotic efficiency, and assumptions such as strict convexity for ERM. However, the deep learning experiment involves a nonconvex Llama model, for which the authors explicitly acknowledge that the GMM assumptions do not hold. This raises concerns about the applicability of the theoretical guarantees to practical deep learning settings.

  2. The empirical evaluation is limited and does not directly measure the proposed predictive exposure metric. Instead, the authors use the accuracy of an MLP trained to predict author gender as a proxy for information leakage. However, gender prediction accuracy does not directly quantify the reduction in an observer's prediction risk after observing the released model, as defined by predictive exposure.

  3. The noise covariance in Theorem 7 involves Hessian-related and gradient covariance terms, which may be computationally expensive for large-scale neural networks. Although the method is evaluated on a 1B-parameter language model, the computational overhead and scalability of the covariance approximation remain unclear. Further implementation details and computational cost analysis would strengthen the practical relevance of the proposed mechanism.

Questions for authors
  1. Although Proposition 2 establishes that predictive exposure and differential privacy are theoretically incomparable, could the authors further clarify the practical motivation for introducing predictive exposure, particularly in comparison with existing population-level privacy frameworks? In which realistic scenarios does the proposed notion offer guarantees that existing approaches cannot adequately provide?

  2. How is the calibrated noise covariance in Theorem 7 computed or approximated for large neural networks, and how do the computational and memory costs scale with the number of model parameters? Could the authors provide runtime and memory comparisons with isotropic noise?

  3. The TOFU experiment uses gender prediction accuracy as a proxy for predictive exposure. Could the authors directly evaluate the reduction in an observer's prediction risk before and after observing the released model, as defined in Definition 3?

Reviewer_F59KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript studies the additional information that a released model reveals about sensitive attributes of the unobserved population. The authors introduce predictive exposure to measure the reduction in an attacker's prediction risk after observing the model. They use GMM to analyze its asymptotic behavior and develop an output perturbation mechanism for ERM. The experiments compare this mechanism with isotropic noise in terms of utility and accuracy.

Critical strengths
  1. The proposed method accounts for the partial dataset known to the attacker and measures the additional predictive ability gained from the model release. This aspect has received less attention in previous work, and I think it's a novel concept.

  2. This manuscript provides theoretical bounds for predictive exposure in several common settings (Thms. 4-7, 9-10), and the experiments show the potential advantages of calibrated noise over isotropic noise at lower utility accuracy.

Critical weaknesses
  1. In Fig. 1, the difference between calibrated noise and isotropic noise is significant at lower utility accuracy. As utility accuracy increases, the difference becomes less clear.

  2. Both Thm. 4 and Cor. 1 are large-sample results, it does not show whether these formulas accurately describe the reduction in prediction error for smaller samples. Theoretical or numerical results for this case would be helpful.

Questions for authors
  1. In Def. 3, predictive exposure measures how much an attacker's predictions improve after observing the model. Fig. 1 only reports gender prediction accuracy after observing the model. I would like the authors to also report results using only the known data, without access to the model, to show how much additional information the model provides.

  2. Def. 3 averages the randomness of the known data C(S)C(S) and the algorithm AA. Is it possible that the average predictive exposure is very small, yet the attacker’s predictive ability is very good? Can the authors clarify whether this metric provides a worst-case guarantee?

  3. Lines 164-168 state that no prior distribution over datasets needs to be specified, but lines 254-255 use a posterior distribution and assume that the Bernstein–von Mises theorem holds. It is unclear to me what prior is used here. Could the authors clarify this?

Reviewer_krWoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces predictive exposure as a framework to measure how much information a learned model reveals about sensitive properties of an unknown population.

Using the generalized method of moments, the authors derive theoretical bounds on predictive exposure and connect them to canonical correlation.

The authors propose a calibrated noise mechanism for ERM and validate it theoretically for linear regression and, also nicely include, experiments on the TOFU dataset using Llama-3.2-1B.

Critical strengths

I particularly appreciate the formalization of predictive exposure. I find the problem is interesting and non-trivial.

The connection with GMM and canonical correlation is interesting.

The resulting calibrated noise mechanism gives a concrete application of the framework.

Critical weaknesses

I have a limited familiarity with this literature but it is not entirely clear to me which parts of the theoretical are genuinely new compared to previous works. I would appreciate a more explicit comparison.

The experimental section is limited, with only one main figure for the TOFU experiment. Moreover, the improvement of calibrated noise seems most visible when model utility is already relatively low. This is a serious limitation of the practicality of the method.

Questions for authors

In Definition 2, could the authors clarify the role of the loss function in defining the Bayes-optimal estimator? In my idea a BO estimator should only depend on the process generating the data and not on the choice of loss functions

Section 5 feels rather compressed. Could the authors expand the explanation of how the theoretical framework leads to the calibrated noise mechanism?

Regarding Figure 1, the advantage of calibrated noise seems to appear mainly at high noise levels, when model utility is already low. Could the authors comment on the practical relevance of this regime?

Could the authors provide additional plots showing both utility accuracy and gender classification accuracy against the noise scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.