- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates how efficient are membership inference attacks against calibration datasets used in post-training quantization. The authors show that calibration statistics embedded in quantized model weights can be recovered and exploited to infer whether a given sample was used for calibration. They provide theoretical results characterizing this leakage and deriving effective attacks, particularly for maximum-based statistics. Experiments across several quantization methods demonstrate substantial privacy risks, with high TPR even at very low FPR against SmoothQuant for instance. Finally, the authors show that perturbing calibration statistics reduces exact identification but does not fully eliminate leakage, motivating the use of formally private calibration methods.
Critical strengths
Novel privacy vulnerability supported by rigorous experimental evaluation. The paper identifies a concrete and underexplored privacy vulnerability in post-training quantization. The experimental evaluation is particularly convincing, with a carefully designed protocol,. In particular, not only the attacks seem efficient, but the Figure 4 for instance shows well how well the theory and actual performance match. Additionally, I believe the connection to stock probability calibration interesting.
Strong theoretical contributions and insightful mitigation analysis. The paper provides a solid theoretical framework for understanding how calibration statistics leak membership information. Finally, the analysis of potential mitigations is interesting, showing that simple perturbations may eliminate exact matches without fully addressing the underlying privacy risks.
Critical weaknesses
Scope of the privacy risk. The practical relevance of the threat depends on calibration sets containing sensitive records, but there is no evidence that this happens in practice: the surveyed checkpoints that name their calibration data appear to use public corpora, and the authors acknowledge that this remains to be measured. Without such evidence (e.g., documented deployments calibrated on private or user data), it is unclear how often this attack would expose information that is not already public.
Link to differential privacy It would be interesting to derive empirical lower bounds on from the observed attack performance, as commonly done in DP auditing.
Questions for authors
(see questions)
We thank the reviewer for noting the agreement between our theory and experiments and for suggesting a differential privacy audit.
1. Scope of the privacy risk. The model card for a public Cosmos3-Edge release reports calibration on 198 unpublished camera frames showing identifiable people. A separate targeted search found 35 additional public quantized releases reporting non-public calibration data, including text from AI assistant interactions and private coding sessions. We also inspected CUPLR and CUPPA, two releases from the same clinical model lineage. Both retain scores from probability calibration fitted on data that included patient records with restricted access (§§2 and 4, Apps. F and H.1).
The privacy risk arises when an organization calibrates on sensitive records and shares a checkpoint that retains information about those records. In the settings we tested, someone who already has a copy of a record can use the checkpoint to test whether the organization used it for calibration. We measured attack performance on public benchmarks. Most repositories in our audit left their calibration data unspecified, so the overall prevalence of sensitive calibration remains unknown (App. H.1).
2. Empirical bounds on differential privacy. We ran the requested audit on OPT-125M. For each of three target records and each release condition, we ran 10,000 trials with the target included and 10,000 with it replaced by another record. We tested whether the recovered statistics could provide evidence for inclusion or rule it out.
Substantial leakage remained after adding 1% uniform noise to the calibration maxima. At δ=10−6, the simultaneous 95% lower bounds on ε across the three target records ranged from 6.98 to 7.28 for the unchanged maxima and 6.74 to 7.28 after adding noise. For the release procedure with 1% noise, even the lowest of the three bounds rules out a differential privacy guarantee with ε<6.74 at the stated δ and confidence level.
The strongest bounds after adding noise came from ruling out membership. For example, after adding 1% uniform noise, the exclusion test flagged 8,705 of 10,000 trials with one target record absent and zero of 10,000 trials with the same record present. Noise can obscure exact matches while leaving enough information in the released statistics to rule out membership (§5, Table 22).
3. Presentation. We revised the abstract and introduction to explain the privacy risk more directly. Both now explain how a maximum retained in the released weights can preserve a value from an individual calibration record, giving an attacker a way to test membership.