acceptodds
Under review as a conference paper at ICLR 2027

Let's Ask Gauss: Improved One-Run Privacy Auditing

15 views

Abstract

Privacy auditing provides an important safeguard by estimating the actual information leaked by a model, thus ensuring that theoretical privacy guarantees hold in practice. We study empirical privacy auditing for differentially private (DP) machine learning, focusing on efficient one-run methods for mechanisms such as DP-SGD. Prior one-run approaches threshold training examples or “canaries” into binary membership guesses, which discards useful information. We show that, in the white-box DP-SGD setting, canary-aligned signals naturally form a sequence of random variables whose normalized sum is asymptotically Gaussian. We show that this gaussian perspective can also be naturally extended to other composition mechanisms that satisfy a classical, triangular-array, or martingale central limit theorem. Leveraging this distributional perspective, we develop a DP-auditing framework that leads to tighter privacy lower bounds from a single training run.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, empirical auditing, one training run

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ugnbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The goal of privacy auditing is to estimate the actual information leaked by a model. Ideally, this leakage would match the theoretical privacy guarantees. The authors study privacy auditing for differentially private (DP) machine learning, focusing on efficient one-run methods for mechanisms (specifically, DP-SGD). Prior one-run approaches threshold training examples (i.e., ``canaries'') into binary membership guesses, which discards useful information. The authors show that, in the white-box DP-SGD setting, canary-aligned signals naturally form a sequence of random variables whose normalized sum is asymptotically Gaussian. Leveraging this distributional perspective, they then develop an auditing framework that leads to tighter privacy lower bounds from a single training run.

The authors of the paper propose a scalable privacy auditing method for training algorithms that add Gaussian noise, with a main focus on DP-SGD and an application to DP-FTRL. Unlike existing scalable auditing methods, the proposed approach uses the distributional properties of the Gaussian noise added during training. The authors model the distribution of the audit scores under idealized assumptions and construct a confidence region for the noise parameters for privacy auditing.

Critical strengths

(1) The method shows high practical utility in auditing DP-SGD, as shown in the experiments on the CIFAR-10 dataset.

(2) The paper provides a nice theoretical analysis for the proposed method, including CLT error analysis with the justification of the Gaussian approximation. Specifically, they show, under the idealized model, the CLT approximation error is dominated by the optimization error. While the paper relies on the ideal modeling assumption, the authors provide a justifying experiment.

Critical weaknesses

(1) The finite-sample confidence guarantee does not yet cover the implemented procedure under Model 1: Algorithm 1 is presented as an auditor under Model 1, but the finite-TT present score in that model is a binomial–Gaussian mixture. Proposition 3 assumes Gaussian score populations and a confidence region with the required coverage. The implementation instead uses an asymptotically justified bootstrap ellipsoid. Theorem 1 and Corollary 2 retain an (unquantified) O(1/T)O(1/T) remainder and the concrete numerical calculation on page 20 uses only the leading term. Neither approximation error nor background-model error is propagated into the hockey-stick divergence and confidence guarantee.

Consider the observation mechanism specified by Model 1 with $T=2500,\quad q=10^{-9},\quad C=1,\quad \sigma=10^{-6},\quad \delta=10^{-5}.$ Coupling the absent and present observation sequences with identical randomness shows that they differ only if the canary is sampled. Hence $\operatorname{TV}(P_0,P_1) \le 1-(1-q)^T \approx 2.5\times10^{-6}<\delta,$ so this observation mechanism satisfies (0,δ)(0,\delta)-DP. Its moment-matched Gaussian pair instead has means 00 and 5×10−85\times10^{-8}, and variances 10−1210^{-12} and 10−12+10−9(1−10−9)10^{-12}+10^{-9}(1-10^{-9}). The Gaussian pair’s hockey-stick divergence at ε=0\varepsilon=0 is approximately 0.925. With sufficiently many independent canaries, parameter-estimation uncertainty can shrink around this incorrect surrogate without repairing the approximation error.

(2) The DP-FTRL transcript and recovery argument are not specified consistently: Section 4.3 describes access to y=Cx+Zy=Cx+Z, while Appendix F claims that yy can be recovered from decoded releases because BB is invertible. For the full binary-tree encoder described there, CC has 2T−12T-1 rows and the prefix decoder BB has TT rows, so BB cannot be invertible. Observing all noisy tree nodes can justify the proposed statistic, but that is different from observing only decoded prefix/model updates. This affects the scope of the extension and whether the baseline comparison grants equivalent access.

Questions for authors

(1) Auditor calibration: What quantitative guarantee applies to Algorithm 1 with finite TT, the implemented bootstrap confidence region, and Gaussian approximation error? Can an explicit correction be incorporated into the returned bound? Repeated simulations from the inexpensive exact Model 1 generator could lead to empirical coverage and false-rejection rates over a range of sampling and noise regimes.

(2) Applicability conditions: What checkable restrictions exclude sparse-sampling/small-noise cases such as the example above? How is any nonzero background contribution bounded or included in the guarantee, rather than assessed only through a Q–Q plot?

(3) DP-FTRL access: Does the auditor observe all noisy tree-node releases or only decoded prefix/model updates? Please specify the dimensions of BB and CC, justify any claimed recovery, and clarify whether the comparator receives exactly the same transcript.

1 comment

We thank the reviewer for the feedback and for recognizing the practical utility and theoretical contributions of our method. We have addressed the main comment by incorporating an explicit finite-T Gaussian approximation correction into Algorithm 1 (see also response to Rev nbcR). Importantly, our updated experiments confirm that the proposed auditor continues to outperform existing one-run methods, retaining its advantages in both auditing tightness and detecting privacy violations. We elaborate on these revisions and address other points below.

Q1. Finite-T Correction and Auditor Calibration

We have revised Algorithm 1 to explicitly incorporate the finite-T Gaussian approximation error into the returned bound (Section 3.1, Theorem 1, and Appendix E of the updated manuscript).

Specifically, the correction uses an explicit, non-asymptotic Kolmogorov-distance bound between the true binomial–Gaussian mixture and its Gaussian approximation under Model 1. Unlike our previous analysis, this bound holds for finite T without an unquantified remainder. Since the absent-canary distribution is exactly Gaussian and the optimal distinguishing region for two one-dimensional Gaussians has at most two boundary points, the newly derived Kolmogorov-distance bound implies that the true directional hockey-stick divergence is at least the Gaussian divergence minus r=2ηTr=2\eta_T. Accordingly, Algorithm 1 now evaluates the privacy lower bound at δ+r\delta+r instead of δ\delta, where r is chosen uniformly over the applicable parameter region. Consequently, under Model 1, the revised auditor yields an asymptotically valid 95% privacy lower confidence bound, while explicitly controlling the Gaussian approximation error for finite T.

We rerun the experiments (updated Figure 1 and Table 1) using the revised auditor. The experiments show that our method continues to outperform previous one-run auditors, achieving an empirical privacy lower bound of 6.234 at the theoretical ϵ=8\epsilon=8, compared with 4.697 for Mahloujifar et al. and 3.330 for Steinke et al. Furthermore, our revised auditor still detects all three tested DP-SGD implementation errors. These results still support our central message: exploiting the distributional information in real-valued canary scores, rather than reducing them to binary membership guesses, yields substantially tighter privacy lower bounds.

Q1 (Cont’d) : Empirical Coverage and False-Rejection Rates

We have provided the empirical evidence the reviewer requested confirming our original results are correct: Across 500 exact Model 1 simulations at our ε = 8 experimental setting, the nominal 95% bootstrap ellipsoid covered the true parameters in 474 runs (94.8%, 95% Wilson CI 92.49%–96.43%). At the rounded-up Model 1 score threshold of ε = 7.517485, the corrected auditor falsely rejected in 0 of 500 runs (95% Wilson CI 0%–0.76%).

Q2. Applicability conditions The revised auditor addresses the reviewer's concern regarding sparse-sampling and small-noise regimes. Specifically, the correction r is computed uniformly over a DP-SGD parameter region (Section 3.1 and Appendix E). If this region is sufficiently broad to include such extreme configurations, the correction can become large enough that the revised auditor returns a vacuous privacy lower bound. In particular, for the reviewer's counterexample, the correction renders the lower bound vacuous, preventing an unjustified privacy violation claim. Conversely, for more realistic configurations, the correction remains small, as quantified in Appendix E, Table 2, allowing the auditor to retain meaningful privacy lower bounds. The theoretical guarantee is established under Model 1, which assumes exact background suppression. We provide empirical evidence supporting this assumption in the evaluated DP-SGD settings through Q–Q plots.

Q3. DP-FTRL and DP-MF-FTRL: Our MF mechanism factors A=BCA=BC, computes Cx+ZCx+Z with i.i.d. N(0,σ2)\mathcal{N}(0,\sigma^2) entries in ZZ, releases g=Byg=\mathbf{B}y, and decodes By=Ax+BZBy=Ax+BZ. The auditor observes the noisy prefix-sums gg.

What are the dimensions of Encoder/Decoder?

Our streaming DP-FTRL implementation generates one noisy dyadic block per round, so its encoder CC and decoder BB are both T×TT \times T. The noisy node created in round nn is yn=gn−gn−ℓny_n=g_n-g_{n-\ell_n}, where ℓn\ell_n is the largest power of two dividing nn and g0=0g_0=0. In DP-MF-FTRL, the square decoder BB is invertible, so the auditor can recover yy by solving for By=gBy=g. The per-canary statistic of Andrew et al. is the round-wise max cosine s̃ᵢ = max over t ∈ [0, T) of ⟨e[cᵢ], gₜ⟩ / ‖gₜ‖₂, where gₜ is the same g described above. Given the new page limit of 10, we have added a Section 4 to introduce the mechanisms, the auditors view on the canary scores and the privacy relevant score distribution before the experimental section and added implementation details in Section 5.3.

Reviewer_nbcROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a method for auditing DP-SGD in the white-box setting. Under the assumption that the normalized canary scores are normal, they design a statistical test to estimate a lower bound on the privacy parameter based on a formula for the hockey-stick divergence between two univariate Gaussian distributions. Experimental results are encouraging.

Critical strengths

Good practical performance

Critical weaknesses

My main concern is the validity of the lower bounds generated by Algorithm 1. As I understand it, the algorithm only provides valid privacy lower bounds if the canary scores are Gaussian and we can accurately estimate the parameters of their distributions. However, I see two issues:

  1. The canary scores are not necessarily Gaussian, and even if they are approximately Gaussian, the approximation error is not accounted for in the confidence bound. Shouldn't the coverage guarantee be of the form 1−α−β1-\alpha-\beta, where α\alpha accounts for finite-sample estimation error and β\beta accounts for the Gaussian approximation error? Perhaps β\beta could be derived from the Kolmogorov-distance bound, although it is unclear whether that bound is sufficient.

  2. Even if the canary scores were exactly Gaussian, wouldn't we still need them to be i.i.d., or satisfy suitable weak-dependence assumptions, for the bootstrap confidence ellipsoid to be valid? This seems problematic since all canaries depend on the same SGD trajectory, so their scores are not necessarily independent.

Correcting these issues (e.g., accounting for β\beta above) would likely weaken the privacy lower bounds obtained by the method, and it is unclear by how much. Consequently, it is unclear whether the claimed improvements in auditing tightness would persist once these issues are addressed (if they can be addressed).

Questions for authors

Can the issues mentioned above be addressed?

1 comment

We are thankful for the insightful comments and constructive feedback. We have addressed them by incorporating an explicit finite-T Gaussian approximation correction into Algorithm 1 and providing an argument for the joint independence of canary scores under Model 1. Importantly, our updated experiments confirm that the proposed auditor continues to outperform existing one-run methods, retaining its advantages in both auditing tightness and detecting privacy violations. We elaborate on these revisions and address other points below.

Gaussian Approximation Correction in the Returned Bound

The reviewer is right that our original Algorithm 1 did not explicitly account for the finite-T Gaussian approximation error in the returned privacy lower bound. To fix it, instead of introducing an additional failure probability , we can incorporate this approximation error directly into the privacy bound through a deterministic correction. We have revised our auditor in Algorithm 1 accordingly (Section 3.1, Theorem 1, and Appendix E of the updated manuscript).

Specifically, the correction uses an explicit, non-asymptotic Kolmogorov-distance bound between the true binomial–Gaussian mixture and its Gaussian approximation under Model 1. Unlike our previous analysis, this bound holds for finite T without an unquantified remainder. Since the absent-canary distribution is exactly Gaussian and the optimal distinguishing region for two one-dimensional Gaussians has at most two boundary points, the newly derived Kolmogorov-distance bound implies that the true directional hockey-stick divergence is at least the Gaussian divergence minus r=2ηTr=2\eta_T. Accordingly, Algorithm 1 now evaluates the privacy lower bound at δ+r\delta+r instead of δ\delta, where r is chosen uniformly over the applicable parameter region.

The implementation of Algorithm 1 uses a 95% bootstrap confidence ellipsoid, which provides an asymptotic coverage guarantee. Consequently, under Model 1, the revised auditor yields an asymptotically valid 95% privacy lower confidence bound, while explicitly controlling the Gaussian approximation error for finite T.

We rerun the experiments (updated Figure 1 and Table 1) using the revised auditor. The experiments show that our method continues to outperform previous one-run auditors, achieving an empirical privacy lower bound of 6.234 at the theoretical ϵ=8\epsilon=8. Furthermore, our revised auditor still detects all three tested DP-SGD implementation errors (Table 1). These results still support our central message: exploiting the distributional information in real-valued canary scores, rather than reducing them to binary membership guesses, yields substantially tighter privacy lower bounds. The comparison is listed below.

DP-SGD audit bounds before and after the finite-T correction

Target εSteinke et al.Mahloujifar et al. (f-DP)Ours (uncorrected)Ours (corrected)
10.55190.63790.62350.6231
21.17251.42701.34741.3431
41.95622.62903.29143.2446
83.32984.69736.73876.2337

Detection of injected DP-SGD implementation bugs

ImplementationReference εSteinke et al.Mahloujifar et al. (f-DP)Ours (uncorrected)Ours (corrected)
Correct implementation8.003.3304.6976.7396.234
Noise calibrated for C/2, trained with C21.985.5429.40416.02414.123
Training steps doubled without recalibrating noise12.124.3156.6549.9119.163
Sampling rate doubled relative to the accountant18.885.5289.36516.37215.356

Discussion on the above tables: Bold marks empirical lower bounds above the claimed ε = 8. The corrected auditor detects all three injected bugs. f-DP detects two, and Steinke et al. detects none.

We have edited the paper pdf in the following places for the correction: Methodology algorithm, section 3.1 and 3.2 and proofs are in Appendix E. We have also updated the plots in the main body and appendix to reflect the correction.

Independence of Canary Scores: We have extended Proposition 5 (Appendix C) to establish joint independence of canary scores within a single training run under Model 1. Specifically, distinct Dirac canaries contribute to separate coordinates, while Gaussian noise and canary-sampling decisions are independent across coordinates and training steps. Under the background suppression assumption (which we have empirically validated, see Fig 4), each canary score, therefore, depends only on its own independent randomness, establishing joint independence. (This is also why we do not need a separate i.i.d assumption).

For practical DP-SGD, Figures 4 and 5 provide empirical evidence that the observed canary-score distributions closely match the predictions of Model 1, while Figure 7 shows no systematic variation across canary indices.

Reviewer_eBd9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a new privacy auditing scheme. In particular, they propose a new method for one run white-box auditing of DP-SGD like training algorithms where the auditor can insert canaries in the training dataset. They consider the trajectory of the model (noisy) gradient updates and how well they aligned with gradients obtained from canaries (both members and non members). Then for each canary they aggregate the sequence of canary aligned gradient observations to compute a unique canary score. The paper's main idea is that those scores asymptotically follow a Gaussian distribution. While this is easy to show for non member data,the authors demonstrate that it is also asymptotically true for members. The auditor finally computes the minimum privacy parameter ε\varepsilon compatible with the hockey-stick divergence observed between member and non member canary scores. Authors validate their proposed auditing scheme on models trained on CIFAR-10 with DP-SGD and report tighter privacy lower bound compared to previous work. They also use this experiment to empirically investigate assumptions needed for the audit.

Critical strengths

The paper introduces a new white box auditing scheme for auditing machine learning training algorithm. Their approach is well motivated and authors provide a theoretical analysis of the proposed approach.

They report measuring empirical privacy lower bound significantly tighter than previous work, providing a useful tool for privacy auditing (detecting implementation bug of DP algorithm where other auditing technique wouldn’t).

Critical weaknesses

The paper introduces several assumptions in order for the audit to be valid, in particular the assumption that canary scores for members follow a Gaussian distribution (after sufficient training steps) and that gradients of genuine non canary data should be orthogonal to the direction of the canaries’ gradients. So the proposed method is able to measure tighter ε\varepsilon lower bound but at the cost of additional assumptions.

The paper is empirically evaluated on only one dataset. It would be really interesting to have more experiments for other learning tasks and data modality. In particular the experiments are used to empirically validate the audit scheme assumptions. It would be interesting to see if those assumptions hold in practice in other scenarios.

The paper clarity could be improved, in particular the paper mentions DP-FTRL and DP-MF-FTRL but fails to properly introduce those methods, making section 4.3 difficult to follow.

Moreover the Dirac canary construction seems to be important as it plays a key role in the background suppression assumption. Both for guaranteeing canary saturation and choosing the coordinate such that genuine non canary gradients have minimal projection in this direction. However Dirac canaries are not introduced in the paper and there are no details about how they are created.

MINOR: Appendix B title appears after which make the section feel empty. In Proposition 2, presenting the dp-sgd score process in appendix C is a bit harsh.

Questions for authors

How are the Dirac canary constructed?

The proposed Gaussian pair auditing and Gaussian DP seems to have similarities. I am wondering how they relate.

1 comment

We are delighted that the reviewer recognized the novelty of our auditing approach and our auditor’s strength in catching privacy bugs that previous auditors miss. In the following we address all questions and the concerns the reviewer points out.

Q1. How are the Dirac canary constructed? We follow the same synthetic gradient canary construction as previous white box auditors (Steinke et al., Mahloujifar et al.). Dirac canaries are constructed to be one-hot vectors in the model parameters. We flatten the model parameters and sample one coordinate without replacement per canary. Each included canary contributes CejCe_j where eje_j is the unit vector at its assigned coordinate and CC is the clipping bound. Since the coordinates are chosen without replacement the canaries do not contribute to each other's direction.

Does the specific Dirac canary construction play a key role to background suppression?

No. As mentioned above the construction is not modified to improve suppression. Our training gradients are empirically validated to have negligible interference with the canary directions. We simply distill this empirically observed fact as an assumption of Model 1.

We have added these details in Section 5.1.

Q2. The proposed Gaussian pair auditing and Gaussian DP seems to have similarities. I am wondering how they relate. Gaussian DP and our auditing both use hypothesis-testing interpretation and the differential privacy distance metric; but the two are orthogonal in their goal and techniques:

Gaussian DP (GDP) provides an upper bound on privacy leakage by requiring the indistinguishability of every pair of neighboring output distributions to be lower bounded by that of two unit-variance Gaussians with a mean shift. Importantly, the output distributions themselves need not be Gaussian; only their indistinguishability must satisfy this Gaussian benchmark.

In contrast, our Gaussian-pair auditor provides an empirical lower bound on privacy leakage by modeling the canary-score distributions under neighboring datasets as a pair of approximately Gaussian distributions, potentially with unequal variances. These are not restricted to the equal-variance Gaussian pair used in GDP. Thus, GDP uses Gaussian indistinguishability of shifted mean equal variance Gaussian pair to establish an upper bound on privacy leakage, whereas our approach exploits the Gaussian structure of the observed score distributions to establish a privacy lower bound.

Assumptions: We agree with the reviewer that we get better results by adding our background suppression assumption which we show holds empirically and modeling the scores as a Gaussian distribution. As a response to the comment and suggestion by the reviewer (thank you!) we have accounted for the Gaussian approximation error in our confidence bound. We have re-evaluated our benchmarks and still observe our tight results. More details are in the response to reviewer nbcR. We believe that this is the right direction for privacy auditing where the goal is to perform tight privacy audits and catch privacy violations in practice making use of the rich score distribution available.

Datasets: Our choice to audit on CIFAR10 was because we wanted to be able to compare our method and demonstrate its superiority to the state of the art: All previous works on privacy auditing use CIFAR10 for their experiments; we did not introduce new, benchmarked experiments as there is no published baseline to evaluate against. But the reviewer has a valid point that this can be useful just to demonstrate broader applicability of our methodology. We are happy to further extend our evaluation section by adding an additional dataset, but given the tight revision schedule, the reviewer will need to take our word for it :).

DP-FTRL AND DP-MF-FTRL: We thank the reviewer for their valuable suggestion for our presentation; we straggled to fit everything in 9 pages, admittedly at the cost of a harder to follow section on DP-FTRL and DP-MF-FTRL. Given the new page limit of 10, we have added a Section 4 to introduce the mechanisms, the auditors view on the canary scores, and the privacy relevant score distribution before the experimental section and added implementation details in Section 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.