- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Gradient Uniqueness (GNQ), a per-datapoint score for auditing how much a trained model reveals about its training data. GNQ measures how different a datapoint’s gradient is from the other gradients in the training batch and links to an information-theoretic measure of membership disclosure. An efficient algorithm, BS-Ghost, is given, which computes GNQ for every datapoint during training with low overhead. Experiments show that GNQ agrees with membership inference, reconstruction, and text extraction results, and that it can also be used for data attribution.
Critical strengths
The paper studies an important and practical problem: auditing how much a trained model discloses about each training datapoint. The proposed approach aims to do this during normal training at low cost, without shadow models or changes to the training data. If the claims hold, this could be a useful tool. The experiments also cover a wide range of settings and appear to point in a consistent direction.
Critical weaknesses
Section 3, which presents the core methodology, lacks sufficient clarity and justification, making the paper’s main claims difficult to assess. The key theoretical results and their assumptions are moved to the appendix, and the main text mostly states conclusions without explaining them. GNQ is defined in Eq. 1 with no intuition for why it should measure disclosure. The steps that follow, including the bound in Eq. (2), the Gaussian model of the background gradients, and the final link to GNQ, are given with little motivation. The reader therefore cannot see why each step holds or how strong the assumptions are. The 2D example in Section 4 comes too late. It also only shows that GNQ is an outlier score, not why that relates to information disclosure. As a result, it is hard to tell whether GNQ really bounds disclosure or is just a heuristic score that happens to correlate with attacks.
Questions for authors
- Can you give an intuitive explanation of the construction of GNQ in eq (1)?
- Can you state the key assumptions clearly in the main text and justify why they are reasonable and how they play a role in deriving the theorems?
2 comments
Part 2/2
Q2: Assumptions in the main text
Change 4 (assumption block). Section 3 will contain the following labeled assumptions, each with its use and justification.
| Assumption | Used in | Justification | |
|---|---|---|---|
| A1 | ; the other datapoints are i.i.d. from , independent of . | Thm. 1, 2 | Problem setup using the standard membership-inference prior. |
| A2 | is a deterministic function of , and is independent of the data. | Thm. 1 | Holds exactly for mini-batch SGD. |
| A3 | Batch masks are independent , , independent of the data and of earlier masks. | Thm. 2 | Idealization (Poisson sampling) of shuffled fixed-size batches. |
| A4 | Given , the background is Gaussian with mean and covariance . | Thm. 2 | Modeling assumption. is a sum of gradient contributions from many datapoints drawn independently from . If these contributions remain approximately independent once is fixed, the central limit theorem applies and supports this assumption. |
| A5 | . | Thm. 2 | Modeling assumption. Since was trained on the same data set constituting the background, knowing makes the background distribution weakly dependent on whether the target was used. We assume this dependence is negligible. |
| A6 | is the regularized uncentered second moment of the non-target gradients, estimated from the current batch. | Computation | Approximation. The true covariance of the background is not available, so the auditor estimates it from the current batch. |
Change 5 (precision regarding what is proved unconditionally and what requires modeling assumptions). Immediately after Eq. (3) we will state:
Theorem 1 is exact given A1 and A2. Theorem 2 is an equality given the predictive model implied by A3 through A5. Eq. (3) bounds disclosure given A3 through A5, but it may not hold if the model is unreasonable. The experiments in Section 6 test whether the model-based score tracks disclosure by actual attacks.
We believe that the assumptions are reasonable, but we will be clear about this possible limitation in the body of the paper.
Change 6 (closed-form bound in the main text). The derivation in Appendix A.3 will be stated in Section 3 as a corollary, so that the reader sees an explicit bound:
Corollary 1.
Under A1–A5, with information measured in nats,
We believe that with these changes, the paper can be significantly improved. We'd appreciate the reviewer's thoughts.
Part 1/2
We thank the reviewer for these comments. To address them, we will make the six changes listed below. Note that none of these require changing or updating any technical results. Rather, they involve changing the order of statements, adding explanations, or clearly labeling the assumptions.
Q1: Intuition for the construction of GNQ
Change 1 (reorder Section 3 so GNQ will be derived). The new order will be: (i) give the threat model; (ii) give Theorem 1, which shows that final-model disclosure is bounded by the sum of per-iteration gradient disclosures; (iii) describe how to measure per-iteration gradient disclosure, in which the target (which the adversary is trying to detect) contributes a known shift gij/B hidden in the unknown background Wi; (iv) give the observation that the detectability of this shift is gij⊤S−1gij; (v) give Definition 1, followed by the requested intuition paragraph below; (vi) give Theorem 2.
Change 2 (intuition paragraph after Definition 1). We will add:
Change 3 (move Figure 1b and connect “outlier” to “disclosure”). Figure 1(b) and its description will be moved from Section 4 into Section 3, next to the intuition paragraph. We will explain that the ellipse shows the spread of the other gradients. This is the background distribution where the target datapoint may (or may not) be hidden. A gradient far outside the ellipse cannot be explained by that background, so its presence in the batch is easy to detect. This addresses the concern that the 2D example shows only an outlier score.