acceptodds
Under review as a conference paper at ICLR 2027

Releasing Predicted Probabilities: Sharp Limits on Excess Disclosure

Abstract

Releasing predicted class probabilities can reveal sensitive information beyond what optimal prediction requires. We show that perfectly calibrated forecasts can approach the optimal expected log loss while revealing a sensitive bit independent of the true conditional class probabilities. We measure this excess disclosure as the largest reduction in optimal expected loss from observing the release instead of the true probabilities, over decisions about other attributes with losses in . We then determine the prediction cost of limiting this disclosure through randomization. For binary forecasts with log-loss regret at most , the smallest worst-case bound on both prediction regret after release and excess disclosure is as . Uniform noise after an arcsine square-root transformation attains this bound, and we prove a matching lower bound for all randomized releases. For any fixed number of classes, releasing counts of labels sampled independently from each forecast achieves the same cube-root rate. In experiments on retinal images and chest radiographs, this sampling method makes images from the same patient harder to link, at the cost of higher average prediction loss when each task uses only its own released probabilities.

Primary area:
learning theory
Keywords:
statistical learning theory, privacy-utility tradeoff, probabilistic prediction, information leakage, minimax analysis, proper scoring rules

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Uh31OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper measures information disclosed by a forecast (which is the predicted probability distribution) beyond the population posterior (the true probabilities) needed for optimal prediction. This disclosure is measured by excess disclosure which the largest improvement in the best expected payoff from observing the released probabilities over the true ones.

A construction shows that it is possible to infer sensitive information from a model that is perfectly calibrated and almost Bayes-optimal. The paper proves a sharp privacy-utility-tradeoff (excess disclosure vs log-loss regret), and provides a method that attains this bound.

Critical strengths
  • The paper demonstrates that perfectly calibrated, near Bayes-optimal probabilities can reveal sensitive information.
  • The paper provides a framework to measure excess disclosure and proves a sharp bound on the achievable privacy-utility tradeoff.
  • The paper proposes a randomized mechanism that attains the sharp asymptotic bound and provides empirical evidence supporting the effectiveness of randomized releases in reducing sensitive information disclosure.
Critical weaknesses
  • The population Bayes risk is uknown in practice, and at N=256N = 256, certifying disclosure at most 0.050.05 requires a summed task-regret bound of approximately 1.95e−51.95e-5 nats (Discussion; Section 7). The experiments provide no such bound, leaving release-size selection disconnected from certification.
  • The empirical evaluation is narrower than the theoretical framework. The medical experiments measure reductions in patient-verification AUROC for a selected family of classifiers, rather than the optimal recipient's excess disclosure relative to the population posterior.
Questions for authors
  • What observable quantity could guide release size when Bayes regret is unknown? Can the required regret bound be estimated empirically?
  • Could the authors compare the empirical excess disclosure observed in their controlled experiments with the theoretical worst-case bounds across different regret levels and release mechanisms?
2 comments

Thank you for the review.

1. Observable guidance when Bayes regret is unknown. At n = 256, the guarantee needs a summed task-regret certificate of about 1.95×10−51.95\times10^{-5} nats to bound disclosure by 0.05. Validation loss estimates forecast risk. A useful regret bound also needs information about Bayes risk or model approximation. Appendix A.21 constructs a certificate for a fixed finite predictor class, a probability floor, independent target-population samples, and known approximation and optimization errors.

Proposition A23 supplies an observable alternative: average total-variation distance between a forecast's release law and a record-independent reference bounds excess disclosure. We apply it to selective full-precision release with an erasure fallback. Average acceptance at most 0.05 certifies disclosure at most 0.05, including information from the acceptance flag.

With per-record acceptance capped at 0.25, this policy retains 15.02% and 13.76% of the full-precision logistic recipient's log-loss gain over the prior on BRSET and NIH. The certificate covers all tasks jointly for one uniformly sampled image from the fixed cohort and one fixed model (Section 6.6; Appendix G.1; Table A29). Appendix A.23 gives a prospective version using fixed policies and independent deployment forecasts.

2. Controlled disclosure compared with the bounds. Figure 1 compares optimal bit-inference gain with the finite bounds at six regret levels for counts and angular noise. Count gains lie well below their bounds. Angular-noise gains follow their bounds more closely. Appendix B.2 compares measured gains with the bounds at fixed parameter settings.

Appendix B.3 evaluates the full bounded-decision criterion on a four-state calibrated population. At n = 256, sampled counts give excess disclosure 0.44651 under the full criterion. Restricting the decision to classification of the example's sensitive bit gives an accuracy gain of 0.24969 over the posterior baseline. The count-release upper bound is 1. The optimal incremental prediction cost is 0.001121 nats. Thus, the selected bit understates disclosure, and the worst-case bound is loose on this population.

The medical experiments measure fitted verification AUROC and prediction loss. The unknown posterior may itself permit linkage, so these measurements do not identify excess disclosure or population-optimal prediction cost (Section 6).

Thank you for the detailed response and revisions.

You have answered my question by introducing a new approach that avoids the unknown Bayes risk. However, your original sampled count method still cannot provide practical privacy certification without knowing the model's regret.

My assessment of the theoretical contributions remains positive, although practical certification remains an important limitation. I therefore maintain my original rating.

Reviewer_y4fqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies disclosure from predicted probabilities relative to the information contained in the true task posterior. It shows that calibrated, nearly optimal forecasts can still reveal additional sensitive information and presents a sharp asymptotic minimax tradeoff for binary log loss. The paper proposes angular-noise and sampled-count releases, extends the analysis to categorical and repeated releases, and includes exploratory medical patient-linkage experiments.

Critical strengths
  1. The paper provides a clear decision-theoretic distinction between information required for optimal prediction and additional disclosure. The calibrated counterexample shows that accuracy and calibration alone do not control this disclosure, establishing a meaningful conceptual foundation for studying the privacy implications of releasing predicted probabilities.
  2. The theoretical analysis goes beyond a rate bound by presenting a sharp binary asymptotic constant and an explicit mechanism attaining it. The sampled-count construction and the extensions to categorical predictions and repeated releases broaden the relevance of the result beyond a single noise mechanism.
Critical weaknesses

1.The practical application of disclosure guarantees is limited by unknown Bayesian risks. The practical application of the disclosure guarantee relies on a trusted upper bound of the predicted regret relative to the unknown Bayesian risk. Although the proposed mechanism only operates on the prediction results, to select the appropriate mechanism strength to reach the specified disclosure level still requires a quantity that cannot be determined through medical experiments. This leads to a significant gap between theoretical guarantees with conditional constraints and verifiable information release strategies in real prediction tasks.

  1. It is difficult to distinguish the utility loss of empirical prediction from the impact of downstream decoder fitting and model selection. The paper reasonably takes into account various prediction rules of the information recipient, but at the same time points out the problem of mismatch in prediction distribution and the significant sensitivity of the experimental results to the model selection in the development stage. These factors make it difficult to directly attribute the observed changes in loss to the publishing mechanism itself, thereby limiting the support of medical experiments for the conclusions related to the inherent utility loss of this mechanism.
Questions for authors

1.Under what realistic and feasible assumptions can data holders obtain a prediction of the upper bound of regret with sufficient information, thereby selecting appropriate release mechanism parameters for the expected disclosure level? 2. If joint cross-fitting is performed for the prediction and calibration stages and a prediction distribution consistent with the actual deployment stage is adopted in the downstream model fitting process, will the utility comparison results in the medical experiment still remain consistent? This will help distinguish the utility loss caused by the release mechanism itself from the impact brought about by decoder estimation and model selection. 3. When the overall distribution class is restricted to predictions with perfect calibration, does the sharp cube-root lower bound still hold? Clarifying this issue will help establish a more direct connection between counterexamples under calibration conditions and the results of minimax optimality.

1 comment

Thank you for the review.

1. Selecting release strength when Bayes regret is unknown. A useful regret bound requires information about Bayes risk or model approximation beyond validation loss. Appendix A.21 constructs a certificate for a fixed finite predictor class, a probability floor, independent target-population samples, and known approximation and optimization errors. The medical experiments do not establish these assumptions.

Proposition A23 provides an alternative using forecasts: average total-variation distance between the release law and a record-independent reference bounds excess disclosure. For a policy that selectively releases the full forecast and otherwise returns an erasure, average acceptance at most 0.05 certifies disclosure at most 0.05, including the acceptance flag.

With per-record acceptance capped at 0.25, this policy retains 15.02% and 13.76% of the full-precision logistic recipient's log-loss gain over the prior on BRSET and NIH. The certificate covers all tasks jointly for one uniformly sampled image from the fixed cohort and one fixed model (Section 6.6; Appendix G.1; Table A29). Appendix A.23 gives a prospective version under independent sampling and fixed policies.

2. Jointly nested source and calibration fitting. The logistic recipient's prediction cost remains positive on both cohorts. We ran all nine tasks and five seeds, splitting training patients into two outer folds and swapping their roles. One half fits the predictor and calibrator, using inner folds for calibration inputs. The other fits the prespecified logistic decoders. The same final predictor and calibrator supply forecasts for decoder fitting and evaluation. An overlap control allows decoder-training patients to influence calibration inputs (Section 6.5; Appendix F.1).

At n = 256, counts increase task-averaged raw log loss relative to full precision:

CohortLogistic loss increase (nats)Conditional 95% patient interval
BRSET+0.002707[0.001867, 0.003531]
NIH+0.001345[0.001021, 0.001632]

The increase is positive for every source seed. For direct prediction on NIH, switching from full precision to counts changes raw log loss by -0.000102 nats (95% patient-bootstrap interval [-0.000728, +0.000397]). The bootstrap resamples evaluation patients and holds the fitted models and splits fixed.

Errors in the fitted decoders and differences between the training and evaluation cohorts can still affect these costs. For logistic recipients, the intervals comparing count-release costs under nested fitting and the overlap control include zero in both cohorts. They do not establish equivalence between the protocols. This is a retrospective analysis of the existing evaluation cohorts.

3. Sharp bound under perfect calibration. Yes. For input log-loss regret at most ϵ\epsilon, the minimax value of the larger of disclosure and output log-loss regret remains (1+o(1))(ϵ/12)1/3(1+o(1))(\epsilon/12)^{1/3} as ϵ→0\epsilon\to0 (Corollary 3; Appendix A.8). This holds even for classification of a fair sensitive bit independent of the posterior. The recipient may use the optimal prediction decoder for its population. One release channel must serve the population class. Forecast differences can encode the sensitive bit in one calibrated population and true task risk in another, so the common channel must handle both.

Reviewer_BnyoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a threat model, where they measure the excess information leakage on top of a "population posterior" -- perfectly calibrated oracle, conditioned on the input. Authors claim that in certian situations even near-perfect loss leads to high information leakage. Authors then address this information leakage by randomization.

Critical strengths

None

Critical weaknesses

Clarity

The paper is very hard to follow and understand. It reads more like an agent-generated experiment report rather than coherent research paper. The paper starts without explaining the setup of the threat model, non-standard terms often used without definition, experiments are poorly motivated and often seem unconnected to the claims made. Even if the paper has scientific merit, it is very hard to assess for a human reader. Few examples:

  • The paper uses quite unusual threat model, which might at a first glance look like differential privacy, but it's not. I would expect a good paper to explain where it deviates for a common setup and walk the reader through the motivations for the proposed threat model. Here, however, authors never explain the setup or the threat model anywhere in the abstract or the introduction.
  • Many unusual terms introduced mid-sentence without explanation or claims made without any supporting intuition, e.g.:
    • "Any release that permits the same expected loss must allow the recipient to recover them" (L36)
    • "learned forecast" (L37), "exact forecast" (L56, L162)
    • "forecast-only channels" (L145)
    • "Deterministic rounding replaces each probability vector with a nearby vector on the same 1/n frequency grid" (L316) -- what exactly is a nearby vector and what is the grid over.
    • The word "resolution" used 3 times across the paper, but it's unclear what it refers to
  • The experimental section seems to be quite disconnected from the theoretical claims. In medical experiments authors do not establish population posterior, and therefor do not measure excess risk.

Threat model

The threat model adopted by the paper considers an excess risk from releasing class probabilities compared to the "optimal" conditional class probabilities. It is unclear in which scenario such a threat model would be relevant, given that true conditional class probabilities ("population posterior") are never known.

Questions for authors

AI use statement claims that AI has been used to "polish the writing". Can you please clarify to which extent AI was used to write the paper, and how much human written text or other drafts has been provided as an input.

2 comments

Thank you for the review.

Posterior benchmark and threat model. The posterior serves as a theoretical reference, as Bayes risk does in prediction regret (Reid and Williamson, 2011, Section 4). We compare inference from the release with inference from the true task probabilities. Inference-gain privacy measures have precedent (Calmon and Fawaz, 2012, Section II.B).

The revised Introduction starts with a holder who shares predicted disease probabilities while withholding patient images. The recipient knows the model and release rule. Model probabilities may reveal age beyond what follows from true disease risk. Section 2.1 distinguishes this record-inference setting from differential privacy's comparisons across neighboring datasets or inputs.

Terminology. "Forecast" is the standard term for a predictive distribution (Gneiting and Raftery, 2007, Introduction). We now define a learned forecast as the model's predicted distribution and an exact forecast as the posterior. A forecast-only channel uses the model output and private randomness. Full precision means releasing the model output unchanged.

Recovery claim at line 36. The recovery claim assumes that the recipient attains Bayes-optimal log loss. Excess loss equals expected Kullback-Leibler divergence from the posterior, so zero excess requires recovering it almost surely. This follows from strict propriety of the logarithmic score (Gneiting and Raftery, 2007, Section 3.1, Example 3).

Rounding and "resolution." Section 6.2 gives the rounding rule, tie-breaking, and a numerical example. At n = 4, (0.61, 0.29, 0.10) gives counts (3, 1, 0), or frequencies (0.75, 0.25, 0). We replace "resolution" with the parameter it refers to.

What the experiments establish. The known-posterior experiments measure optimal prediction and excess disclosure, including the full bounded-decision criterion (Appendix B.3). The medical experiments measure fitted prediction loss and patient-verification AUROC (area under the receiver operating characteristic curve). They do not establish excess disclosure relative to the unknown medical posterior. Section 6 now makes this distinction explicit. The new selective-release certificates cover a declared fixed-cohort sampling law (Section 6.6; Appendix G.1).

AI use. For the original submission, we supplied rough notes in simple English and our native language, paragraph plans, and handwritten proofs. Codex helped organize these materials into English prose and LaTeX and assisted with literature search, code, proof checking, and research and implementation choices. We checked the text against our mathematical arguments and results and revised the writing. The authors take responsibility for the paper.

I thank the authors for clarifying their position, for updating the paper, and for the transparency in their AI use.

  • On the writing clarity, I recognize the improvement to the specific issues in the introduction, but overall my criticism still holds: the paper is hard to follow throughout, and shows clear signs of heavy AI writing. I recognize that ideas and proofs might be human, and AI use is not bad in itself, but in this particular case it has a clear negative impact on clarity and readability.
  • On the threat model, I still see it as impractical: the guarantee requires a bound on regret relative to the unknown Bayes risk, which cannot be established for realistic models

I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.