acceptodds
Under review as a conference paper at ICLR 2027

The Calibration Set Is in the Checkpoint: How Release Geometry Governs Membership Leakage in Post-Training Quantization

Abstract

Quantizing a model often uses a small calibration set after training is complete. This set can remain encoded in the released checkpoint. In SmoothQuant-style methods, comparing the released checkpoint with its public base model recovers activation statistics computed from the calibration data. A maximum can copy a calibration record's activation exactly into the checkpoint, while averaging removes this exact constraint but can keep other membership signal. We show that leakage grows on a common scale, the number of released channels per calibration record (), while the released statistic sets its shape. On OPT-125M, the reference SmoothQuant implementation and NVIDIA Model Optimizer's stock exporter each identify about 68% of calibration records at measured false-positive rates below . Changing only the statistic in a Qwen2.5-7B exporter changes detection at low false-positive rates by an order of magnitude at essentially unchanged perplexity, and the same effect appears in isotonic probability calibration. Post-training calibration can therefore itself be a privacy-relevant release mechanism.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, post-training quantization, calibration data, privacy, SmoothQuant, model compression, extreme value theory, probability calibration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H6eSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how efficient are membership inference attacks against calibration datasets used in post-training quantization. The authors show that calibration statistics embedded in quantized model weights can be recovered and exploited to infer whether a given sample was used for calibration. They provide theoretical results characterizing this leakage and deriving effective attacks, particularly for maximum-based statistics. Experiments across several quantization methods demonstrate substantial privacy risks, with high TPR even at very low FPR against SmoothQuant for instance. Finally, the authors show that perturbing calibration statistics reduces exact identification but does not fully eliminate leakage, motivating the use of formally private calibration methods.

Critical strengths

Novel privacy vulnerability supported by rigorous experimental evaluation. The paper identifies a concrete and underexplored privacy vulnerability in post-training quantization. The experimental evaluation is particularly convincing, with a carefully designed protocol,. In particular, not only the attacks seem efficient, but the Figure 4 for instance shows well how well the theory and actual performance match. Additionally, I believe the connection to stock probability calibration interesting.

Strong theoretical contributions and insightful mitigation analysis. The paper provides a solid theoretical framework for understanding how calibration statistics leak membership information. Finally, the analysis of potential mitigations is interesting, showing that simple perturbations may eliminate exact matches without fully addressing the underlying privacy risks.

Critical weaknesses

Scope of the privacy risk. The practical relevance of the threat depends on calibration sets containing sensitive records, but there is no evidence that this happens in practice: the surveyed checkpoints that name their calibration data appear to use public corpora, and the authors acknowledge that this remains to be measured. Without such evidence (e.g., documented deployments calibrated on private or user data), it is unclear how often this attack would expose information that is not already public.

Link to differential privacy It would be interesting to derive empirical lower bounds on ϵ\epsilon from the observed attack performance, as commonly done in DP auditing.

Questions for authors

(see questions)

1 comment

We thank the reviewer for noting the agreement between our theory and experiments and for suggesting a differential privacy audit.

1. Scope of the privacy risk. The model card for a public Cosmos3-Edge release reports calibration on 198 unpublished camera frames showing identifiable people. A separate targeted search found 35 additional public quantized releases reporting non-public calibration data, including text from AI assistant interactions and private coding sessions. We also inspected CUPLR and CUPPA, two releases from the same clinical model lineage. Both retain scores from probability calibration fitted on data that included patient records with restricted access (§§2 and 4, Apps. F and H.1).

The privacy risk arises when an organization calibrates on sensitive records and shares a checkpoint that retains information about those records. In the settings we tested, someone who already has a copy of a record can use the checkpoint to test whether the organization used it for calibration. We measured attack performance on public benchmarks. Most repositories in our audit left their calibration data unspecified, so the overall prevalence of sensitive calibration remains unknown (App. H.1).

2. Empirical bounds on differential privacy. We ran the requested audit on OPT-125M. For each of three target records and each release condition, we ran 10,000 trials with the target included and 10,000 with it replaced by another record. We tested whether the recovered statistics could provide evidence for inclusion or rule it out.

Substantial leakage remained after adding 1% uniform noise to the calibration maxima. At δ=10−6\delta=10^{-6}, the simultaneous 95% lower bounds on ε\varepsilon across the three target records ranged from 6.98 to 7.28 for the unchanged maxima and 6.74 to 7.28 after adding noise. For the release procedure with 1% noise, even the lowest of the three bounds rules out a differential privacy guarantee with ε<6.74\varepsilon<6.74 at the stated δ\delta and confidence level.

The strongest bounds after adding noise came from ruling out membership. For example, after adding 1% uniform noise, the exclusion test flagged 8,705 of 10,000 trials with one target record absent and zero of 10,000 trials with the same record present. Noise can obscure exact matches while leaving enough information in the released statistics to rule out membership (§5, Table 22).

3. Presentation. We revised the abstract and introduction to explain the privacy risk more directly. Both now explain how a maximum retained in the released weights can preserve a value from an individual calibration record, giving an attacker a way to test membership.

Reviewer_ZFqqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits privacy risks arising during model post-training from the perspective of membership inference attacks. It focuses on potential data leakage associated with post-training quantization (PTQ) and probability calibration. The paper reveals that such data-dependent model release procedures can implicitly encode and persist the activation statistics of calibration data in the released model checkpoint. By comparing a publicly available base model with its released checkpoint, an attacker can recover these statistics in closed form.

Critical strengths
  1. Novel research perspective: The paper goes beyond the conventional focus of membership inference attacks on training-set membership and investigates the privacy risks associated with PTQ and probability calibration datasets during model deployment. It addresses a relatively underexplored aspect of privacy evaluation in the post-training stage.

  2. Rigorous theoretical analysis: The paper introduces a geometric classification framework that distinguishes between the selector and mixture cases. Building on extreme value theory, it establishes a Poisson limit model and a scaling law for the scale parameter. It also derives mathematical conditions for reliably confirming the attack at extremely low false-positive rates and theoretical bounds on attack degradation under perturbation-based obfuscation.

  3. Comprehensive empirical evaluation: The authors audit and evaluate the attack against several widely used, production-grade PTQ frameworks and probability calibration algorithms. The experiments cover multiple models, including OPT-125M, Qwen2.5-7B, Qwen2.5-1.5B, and Llama-3.2-1B, providing a relatively broad empirical assessment of the proposed attack.

Critical weaknesses
  1. Restrictive threat model assumptions: The proposed attack relies heavily on the attacker's ability to obtain the exact publicly available full-precision base model checkpoint, as well as the specific calibration, transformation, and export configuration parameters. In certain settings, such as closed-source deployments or checkpoints that have undergone additional complex fine-tuning or alignment, accurately recovering the relevant statistics may be substantially more difficult. The paper would benefit from a clearer discussion of how these practical constraints affect the applicability of the attack.

  2. Limited discussion of the practical prevalence of sensitive calibration data: The paper assumes that PTQ calibration sets may contain highly sensitive or private data. However, in industrial deployment, developers often use publicly available general-purpose text corpora, such as Pile-10k and C4, for calibration. In such cases, the practical privacy risks and potential harm associated with membership inference may be limited. A more detailed discussion of how frequently sensitive data are used for calibration in real-world deployments, and under what circumstances the proposed attack poses a substantial privacy threat, would strengthen the paper's practical significance.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for highlighting privacy risks that arise during model calibration and for asking when the attack applies in practice.

1. Access to the original model. For the NVIDIA Model Optimizer and LLM Compressor exports we tested, the released quantized checkpoint was the only model checkpoint the attack needed when the smoothing parameter was known. On OPT-125M exported with unmodified NVIDIA Model Optimizer, we detected 40.4% of calibration members at a measured false-positive rate of 0.0038%.

We also quantized a fine-tuned Qwen2.5-1.5B model with LLM Compressor. To measure how much access to the original model helps, we ran the attack on the same quantized exports under three conditions, each targeting a false-positive rate of 1% (§4, Table 3).

Model checkpoints available to the attackerAUCCalibration members detectedMeasured false-positive rate
Released checkpoint only0.73415.4%1.12%
Released checkpoint and exact fine-tuned model0.75115.2%0.85%
Released checkpoint and public base model0.71310.7%0.96%

In this fine-tuning setup, calibration membership remained detectable when the exact model used before quantization was withheld. Our experiments assume access to the quantized weights. A deployment that exposes only model outputs through an API falls outside the setting we evaluate.

2. Knowledge of the export settings. We revised the threat model in §2 and summarized the required inputs in Table 1. For the two recovery methods below, the attacker needs the released checkpoint and the exporter’s calibration transform. The remaining inputs depend on whether the smoothing parameter is known.

SettingAdditional inputs for recovery
Tested LLM Compressor and NVIDIA Model Optimizer exportsKnown smoothing parameter
SmoothQuant with an estimated smoothing parameterOriginal model, calibration set size, and public reference data

Among the repositories we audited that released per-channel smoothing scales, 99% specified the smoothing parameter (App. H.1, Table 26). In a separate SmoothQuant experiment, we estimated the parameter using the original model, the calibration set size, and public reference data. At a target false-positive rate of 0.1%, detection rates obtained with the estimated and true parameter values differed by at most 0.4 percentage points (§4, Table 9).

3. When the risk matters in practice. The risk is clearest when an organization calibrates on sensitive records and releases the quantized weights while keeping the calibration data private. In the settings we tested, a recipient who already has a copy of a record can use the released weights to test whether the organization used the record for calibration. The privacy stakes are generally lower when developers calibrate on public datasets such as Pile-10k or C4.

The model card for Cosmos3-Edge reports calibration on 198 unpublished camera frames showing identifiable people. A separate targeted search found 35 additional quantized releases that reported using non-public calibration data. Seven used text from sources such as AI assistant interactions and private coding sessions. Most of the repositories we audited left their calibration data unspecified, so the overall prevalence of sensitive calibration remains unknown (App. H.1).

We expanded the discussion in §2 to explain when the attack poses a practical privacy risk: an organization calibrates on sensitive records and shares the quantized weights while keeping the calibration data private. Appendix H.1 documents releases that used non-public calibration data.

Thank you for the comprehensive response and additional experiments. They have addressed my concerns regarding the threat model assumptions and practical privacy risks. I appreciate your efforts and will maintain my positive score.

Reviewer_3qdXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors study the problem of membership inference on the calibration set of post-training quantization methods. The main assumption is that an attacker can have access to both the base model and the quantized version and use the diff signal between activations or LayerNorm parameters information to separate between the members and non-members of the calibration set. They observe that using max as an aggregation method can provide direct clues for membership while other methods such as top-m mean with m >= 2 can still provide enough information for ranking the candidates.

Critical strengths
  • The paper's general message is interesting as they study a problem that is mostly overlooked (membership inference for the calibration set for post-training quantization methods).

  • The paper has both empirical and theoretical results which makes the paper comprehensive.

Critical weaknesses
  • The paper is sometimes hard to read. For example, there are some long sentences that might becomes easier to read if broken into multiple sentences (e.g., the second contribution).

  • While the general message is interesting, the impact might not be very high because the calibration set is usually small. Also the model developers could use a synthetic set to calibrate which could address the privacy concerns.

Questions for authors

By per-tensor quantization, do you mean there is only one activation min and max computed for that particular layer?

1 comment

We thank the reviewer for highlighting the privacy of calibration data as an overlooked problem and for suggesting synthetic calibration.

1. Effect of calibration set size. When an exporter stores the largest activation in each channel, a calibration record can leave its exact activation value in the released checkpoint. Each record in a smaller calibration set has a greater chance of supplying a maximum. For a fixed number of channels and under the assumptions in §3, halving the calibration set doubles the expected number of maxima supplied by each record.

On OPT-125M with 512 calibration records, our attack detected 68% of members with unmodified MIT SmoothQuant and 69% with unmodified NVIDIA Model Optimizer. The measured false-positive rate was below 0.01% in both cases (§4, Table 2).

2. Synthetic calibration. Following the reviewer’s suggestion, we quantized Qwen2.5-1.5B with the same LLM Compressor settings in both conditions. We calibrated one version on 512 emails from the public Enron collection and the other on 512 synthetic emails generated without using the Enron text. In both conditions, we tested the attack on the same original Enron calibration emails and a separate group of real emails. We report averages over five draws. The flagged email rates use a threshold targeting a 1% false-positive rate (§5, App. H.2, Tables 30–31).

Calibration dataChange in test loss on real emailsAttack AUCOriginal Enron calibration emails flaggedSeparate test emails flagged
Real Enron emailsBaseline0.75414.8%0.8%
Synthetic emails−0.013 nats per token0.4840.8%0.9%

For this email workload, synthetic calibration preserved performance on real emails and brought the attack’s ability to distinguish the two groups close to chance.

3. Per-tensor quantization. Yes. For a layer with one calibrated activation tensor, the static ONNX Runtime MinMax method computes one minimum and one maximum by pooling activations across channels, tokens, and calibration records. If a layer has multiple activation tensors calibrated separately, each tensor has its own minimum and maximum. We clarified this distinction in §2.

4. Presentation. We revised the abstract and introduction to explain how a stored maximum can reveal whether an individual record was used for calibration. We also split the second contribution into shorter sentences, as suggested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.