acceptodds
Under review as a conference paper at ICLR 2027

Privacy-Aware Post-Training Quantization for Mitigating Membership Inference in Diffusion Models

Abstract

Diffusion models are vulnerable to membership inference attacks that can reveal whether a sample was used during training. Existing defenses typically rely on privacy-aware training, which adds computational cost and can degrade generation quality. Meanwhile, model quantization provides an efficient way to compress AI models and can affect their membership privacy, yet existing quantization methods are not specified to optimize diffusion models for membership leakage. In this work, we propose PAQ-Diffusion, a privacy-aware post-training quantization framework that incorporates membership privacy into quantization optimization without requiring privacy-aware retraining. PAQ-Diffusion first performs layer-wise privacy-aware reconstruction by aligning the quantization residual distributions of member and non-member samples while preserving reconstruction fidelity. It then introduces timestep-specific activation quantization scales and optimizes them with a denoising-based privacy objective, allowing privacy optimization to be performed independently across timesteps. We evaluate the proposed PAQ-Diffusion on different diffusion models against representative membership inference attacks. Experimental results demonstrate that PAQ-Diffusion reduces membership leakage (e.g., reducing SecMI-NN attack AUC by up to 20.62%) while maintaining superior generation quality, providing an effective privacy-utility trade-off for quantized diffusion models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
post training quantization, diffusion model, membership inference attack, privacy and security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vPg8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAQ-Diffusion, a post-training quantization method for reducing membership-inference leakage in diffusion models. It first aligns member and non-member quantization residuals at the layer level, then learns timestep-specific activation scales using a denoising-error privacy objective. Experiments on a DDPM and two latent diffusion models show lower attack AUC with similar FID.

Critical strengths
  1. The proposed idea is reasonable, and the two-stage privacy-aware quantization framework is well-motivated and conceptually clear, without requiring privacy-aware retraining of the full-precision model.
  2. The evaluation covers three diffusion models and multiple membership-inference attacks, with an ablation separating the layer-wise and timestep-wise stages (Table 4), and reports both AUC and TPR at 1% FPR, making the privacy evaluation more comprehensive.
Critical weaknesses
  1. Inconsistencies in dataset split and optimization objective description. The paper contains several inconsistencies in its experimental and methodological descriptions. In Table 5, the reported SD-Laion source pool contains only 2,500 members and 2,500 non-members, while the listed calibration, SecMI-NN auxiliary, and test splits require more samples if they are mutually disjoint as claimed. The authors should clarify the construction of these splits. Moreover, the reconstruction objective appears inconsistent between the main paper and appendix: Eq. (13) defines a squared L2 reconstruction loss, while Appendix A.1.4 states that activation reconstruction uses “ℓ2.4\ell_{2.4} reconstruction”. The relationship between these objectives and the actual implementation should be clarified.

  2. Insufficient ablation of the privacy-aware objectives. Although Table 4 shows improvements from both stages, it does not isolate the contribution of the proposed privacy losses from other changes, such as additional optimization and timestep-specific activation scales. In particular, an ablation with the privacy loss removed (λ=0\lambda= 0) under the same optimization settings would help clarify how much of the improvement comes from the privacy-aware objectives themselves.

Questions for authors
  1. In Table 5, the SD-Laion auxiliary and test set sizes exceed the available source pool after excluding calibration samples, despite the stated disjointness. Could the authors clarify the construction, intended roles, and overlap status of these splits?

  2. In Appendix A.1.4, the authors mention an “ℓ2.4\ell_{2.4} reconstruction” objective, while Eq. (13) uses a squared L2 loss. Is “ℓ2.4\ell_{2.4}” a notation error, or does it refer to a different reconstruction objective?

  3. Could the authors provide an ablation with the privacy loss removed (λ=0\lambda= 0), while keeping the calibration data and optimization settings unchanged?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vPg8

We thank the reviewer for the constructive feedback and for identifying inconsistencies in our experimental descriptions. We provide clarifications and additional ablation results below.

1. Dataset splits and reconstruction objective (Weakness 1 and Questions 1–2)

Dataset splits. We apologize for the typo in Table 5. The SD-Laion dataset contains 2,500 members and 2,500 non-members, divided into 100/100 calibration samples, 1,000/1,000 SecMI-NN auxiliary samples, and 1,400/1,400 test samples. These three subsets are mutually disjoint, with no overlap between calibration, auxiliary training, and attack evaluation. The reported test size of 2,400/2,400 was a typographical error, which we will correct in the revised manuscript.

Reconstruction objective. We apologize for the inconsistency between Eq. (13) and Appendix A.1.4. The implementation uses the squared L2 reconstruction loss throughout both stages. The reference to a different reconstruction objective in the appendix is a reporting error and will be corrected.

2. Privacy penalty ablation (Weakness 2 and Question 3)

To isolate the contribution of the privacy objectives, we conducted additional ablation experiments on DDPM-CIFAR10 by setting the corresponding privacy coefficient λ=0\lambda=0, while keeping the calibration data, optimization budget, and timestep settings unchanged.

Table R1. Privacy penalty ablation (AUC / TPR@1%FPR, lower is better).

Privacy penalty disabledLossPIAPIANSecMI-statSecMI-NN
Stage 1.8472 / 7.23%.9175 / 24.83%.8840 / 29.53%.8175 / 7.89%.9057 / 19.71%
Stage 2.8283 / 7.13%.9045 / 22.72%.8795 / 26.01%.7082 / 2.23%.8036 / 5.79%
Stage 1 + 2.8477 / 7.47%.9177 / 24.54%.8852 / 30.80%.8264 / 7.22%.9279 / 8.87%
None (full PAQ-Diffusion).8153 / 6.80%.8830 / 15.17%.8740 / 15.79%.6417 / 1.15%.6078 / 1.40%

As shown in Table R1, Stage 1 provides the primary privacy improvement, while Stage 2 offers additional protection. Disabling the Stage 1 privacy penalty increases SecMI-NN AUC from 0.6078 to 0.9057, while disabling Stage 2 increases it to 0.8036. When both penalties are disabled, membership leakage increases substantially, approaching the performance of conventional privacy-unaware quantization for several attacks. These results further support the effectiveness of the two-stage privacy-aware quantization framework.

Reviewer_GH99OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PAQ-Diffusion incorporates empirical membership-privacy objectives into post-training quantization of pretrained diffusion models. Its first stage optimizes weight-rounding variables and activation scales using reconstruction fidelity plus an MMD penalty between member and non-member quantization residuals. Its second stage assigns independent activation scales to sampling timesteps and aligns distributions of per-example noise-prediction errors while preserving full-precision predictions. Experiments use simulated W8A8 quantization on a CIFAR-10 DDPM and two latent-diffusion settings, with conventional quantization baselines, membership attacks, and FID/sFID measurements. The largest improvements concern SecMI attacks on DDPM; gains are more variable on latent diffusion. The method is an empirical mitigation and explicitly does not provide differential privacy.

Critical strengths
  1. The DDPM results show a potentially useful joint improvement in privacy and utility. Tables 1 and 3 report SecMI-NN AUC decreasing from 0.8140 for Q-Diffusion to 0.6078 for PAQ-Diffusion, with TPR at 1% FPR decreasing from 6.93% to 1.40%, while FID improves from 8.85 to 7.80. PAQ also improves all five reported DDPM attack AUCs. If these improvements survive stronger controls, they would establish a useful alternative to obtaining privacy gains simply by sacrificing generation quality.

  2. The intervention is operationally appealing and evaluated beyond one architecture. Sections 3.2–3.3 optimize quantization variables on a pretrained model using a relatively small labeled calibration set, and freeze the quantized weights during timestep adaptation. Evaluating both pixel-space and text-conditioned latent diffusion is valuable. The distinction between local residual alignment and timestep-specific denoising-error alignment provides a concrete, implementable design.

Critical weaknesses

Please refer to the questions below.

Questions for authors

With identical calibration data, optimization budget, and timestep scales, what changes when each privacy penalty is disabled? Does the benefit survive an attacker that selects timesteps or combines their features using separate attack-training data? Please specify the released-model access assumed and scale selection for timesteps outside the sampling grid.

1 comment
AuthorsOpenReview ↗

Response to Reviewer GH99

We thank the reviewer for the insightful questions regarding the contribution of each privacy penalty and the robustness of PAQ-Diffusion against attacks targeting different timesteps. We have conducted additional experiments to address these concerns.

1. Ablation of privacy penalties.

To investigate the contribution of each privacy penalty, we conducted additional ablation experiments on DDPM-CIFAR10 by setting the corresponding privacy coefficient λ\lambda to 0, while keeping the calibration data, optimization budget, and timestep settings unchanged. The results are presented in Table R1.

Table R1. Privacy penalty ablation on DDPM-CIFAR10 (AUC / TPR@1%FPR, lower is better).

Privacy penalty disabledLossPIAPIANSecMI-statSecMI-NN
Stage 1.8472 / 7.23%.9175 / 24.83%.8840 / 29.53%.8175 / 7.89%.9057 / 19.71%
Stage 2.8283 / 7.13%.9045 / 22.72%.8795 / 26.01%.7082 / 2.23%.8036 / 5.79%
Stage 1 + 2.8477 / 7.47%.9177 / 24.54%.8852 / 30.80%.8264 / 7.22%.9279 / 8.87%
None.8153 / 6.80%.8830 / 15.17%.8740 / 15.79%.6417 / 1.15%.6078 / 1.40%

As shown in Table R1, the privacy improvement primarily originates from Stage 1 (layer-wise privacy-aware quantization). Disabling the Stage 1 privacy penalty substantially increases membership inference attack performance. Disabling only the Stage 2 privacy penalty also weakens privacy protection, but generally has a smaller impact than disabling Stage 1. When both privacy penalties are disabled, the attack performance approaches that of conventional privacy-unaware quantization for several metrics, with substantially higher membership leakage than the complete PAQ-Diffusion framework. These results further validate the effectiveness of our privacy-aware objectives, demonstrating that Stage 1 provides the primary privacy improvement, while Stage 2 offers additional protection through timestep-specific optimization.

2. Robustness against out-of-grid timesteps and mixed-timestep features.

In PAQ-Diffusion, the timestep-specific activation scales obtained through post-reconstruction are also applied to nearby timesteps that are not explicitly included in the optimization grid. Specifically, for an out-of-grid timestep, we use the activation scales associated with the nearest optimized timestep. This design is motivated by the relatively smooth variation in the member–non-member reconstruction-error gap across neighboring timesteps, as illustrated in Figure 4(b).

We further evaluated SecMI-stat and SecMI-NN attacks using out-of-grid timesteps and mixed-timestep features. The results are presented in Table R2.

Table R2. Membership inference attacks using out-of-grid timesteps and mixed-timestep features (AUC / TPR@1%FPR, lower is better).

Attack targetSecMI-statSecMI-NN
On grid.6417 / 1.15%.6078 / 1.40%
Out of grid.6827 / 2.35%.6973 / 3.31%
Mixed features.6021 / 1.89%.6146 / 2.47%

As shown in Table R2, attacks targeting out-of-grid timesteps achieve higher AUC and TPR than those targeting optimized timesteps. For mixed-timestep features, combining information from different timesteps does not consistently improve attack performance in terms of AUC, although TPR@1%FPR increases slightly. One possible explanation is that membership features exhibit timestep-dependent distributions and decision thresholds. Directly combining features from different timesteps may therefore make it more difficult for an attack classifier to identify a decision boundary that effectively separates members from non-members across all timesteps.

Overall, these additional experiments demonstrate that the privacy benefits of PAQ-Diffusion extend beyond the optimized timestep grid and remain evident under the evaluated mixed-feature attacks, although some additional membership leakage is observed.

Reviewer_hWGjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PAQ-Diffusion is a post-training quantization framework that reduces membership leakage in pretrained diffusion models without privacy-aware retraining. Stage 1 optimizes AdaRound variables and activation scales with a reconstruction loss plus an MMD term that aligns member and non-member quantization residuals. Stage 2 learns per-timestep activation scales with an output-reconstruction loss plus an MMD term on per-sample noise-prediction errors. Under W8A8 on DDPM-CIFAR10, SD-Pokémon, and SD-Laion, PAQ achieves the lowest AUC on DDPM against all five attacks, lowering SecMI-NN AUC by 20.62 points relative to the best PTQ baseline, with FID/sFID comparable to the PTQ baselines.

Critical strengths
  1. Practical setting. The method integrates membership privacy into PTQ without privacy-aware retraining of the FP model, making it an attractive alternative to DP training or fine-tuning.
  2. Empirically motivated timestep-specific design. The member/non-member error gap peaks near t≈100t \approx 100 and becomes negligible after about t=500t = 500 (Fig. 4b). Stage 2 lowers SecMI-NN AUC from 0.7203 to 0.6078 while improving FID from 8.21 to 7.80 (Table 4).
Critical weaknesses
  1. The Stage-2 objective (Eqs. 19–20) aligns the member/non-member distributions of per-sample denoising error. This is exactly the Loss attack’s statistic and closely related to SecMI’s, so the evaluation is close to testing on the training objective. Without adaptive attacks, which the authors acknowledge are missing (A.2), the claim in §4.2 that PAQ “does not merely overfit a single attack score” is unsupported.

  2. The W8A8 AdaRound weights stay within about one quantization step of the FP weights, so the memorized information is largely retained, and privacy rests on rounding choices and activation scales. In the motivating on-device setting (§1), the weights and all Δt\Delta_t would be distributed to users, yet only query-based attacks are evaluated. Robustness to white-box MIAs (e.g., gradient-based) is unknown.

  3. Missing λ=0\lambda=0 control. The Q-Diffusion row in Table 4 is a valid control only if it equals PAQ with λ=0\lambda=0. However, the baselines follow their official procedures (§4.1): Q-Diffusion calibrates on FP-generated trajectories, while PAQ calibrates on real member/non-member images, so the two likely differ. Independently of this, Q-Diffusion has no per-timestep scales, so a Stage-2 λ=0\lambda=0 ablation is needed to separate the MMD term from the added degrees of freedom (149 × 100 scales on DDPM).

  4. Unclear role of quantization. The gains may come from extra learnable parameters plus an MMD loss on the attack statistic, rather than from quantization itself. A suitable control would apply the same per-timestep learnable activation scalings, with the same loss, to the FP model without quantization. If the results are comparable, the contribution is better framed as lightweight post-hoc privacy tuning that can be attached to PTQ. If they are clearly worse, that is evidence of synergy between quantization noise and the privacy objective, which would strengthen the paper.

  5. SD-Laion is confounded. It pairs LAION members with COCO2017-val non-members (A.1.3). Prior work (Dubiński et al., 2024; Das et al., 2024) shows that AUC in such setups largely reflects distribution shift. MMD alignment may simply remove the domain gap, which makes SD-Laion weak evidence of privacy gains.

  6. Statistical reliability is low. All results use a single seed (A.2). The SD-Pokémon SecMI test split has only 216/217 samples (Table 5), so 1% FPR corresponds to about two non-members, and the TPR difference of 0.24% vs. 0.63% (SecMI-stat) amounts to less than one member sample.

  7. The effect of λ\lambda is non-monotonic (Fig. 4a). With N=1000N=1000, SecMI-stat AUC moves from 0.6962 (λ=0.5\lambda=0.5) to 0.7918 (λ=1\lambda=1) to 0.6417 (λ=2\lambda=2). With a single seed, the “changed optimization landscape” explanation cannot rule out seed noise, so it is unclear whether the main configuration is robust.

Questions for authors
  1. The SD models use a batch size of 4 in both stages (Table 6, A.1.4), so the MMD and its median-heuristic bandwidth are estimated from 2 members vs. 2 non-members per step. How does batch size affect the MMD estimate and the resulting privacy?
  2. Is the SecMI-NN classifier retrained on each target model’s own features, or trained on the FP model and transferred? If transferred, the AUC drop may reflect feature shift rather than reduced leakage.
  3. On DDPM, PAQ (7.80) and Q-Diffusion (8.85) both achieve lower FID than FP (9.19). Why? Can precision/recall or multi-seed results show that this reflects quality rather than a change in diversity?
  4. Hyperparameter consistency: Is λ\lambda equal to 1 (A.1.4) or 2 (§4.1, Table 6)? Is weight reconstruction run for 20,000 iterations (§4.1) or 2,000 per unit (Table 6)? Were λ\lambda and NN selected on a validation attack set independent of the reported results? §4.1 suggests they were chosen from the results in §4.5.
  5. Stage 2 learns scales only for timesteps on the DDIM grid. Which scales are used for off-grid timesteps? Do all evaluated attacks query on-grid timesteps? What is the AUC when an attacker deliberately queries off-grid timesteps?
2 comments
AuthorsOpenReview ↗

Response to Reviewer hWGj

We thank the reviewer for the constructive feedback. Our responses are as follows.

1. Privacy objectives and controlled ablations (Weaknesses 1 and 3)

To evaluate the contribution of each privacy penalty, we conducted ablations on DDPM-CIFAR10 by setting the corresponding λ=0\lambda=0, while keeping all other settings unchanged.

Table R1. Privacy penalty ablation (AUC / TPR@1%FPR, lower is better).

Privacy penalty disabledLossPIAPIANSecMI-statSecMI-NN
Stage 1.8472 / 7.23%.9175 / 24.83%.8840 / 29.53%.8175 / 7.89%.9057 / 19.71%
Stage 2.8283 / 7.13%.9045 / 22.72%.8795 / 26.01%.7082 / 2.23%.8036 / 5.79%
Stage 1 + 2.8477 / 7.47%.9177 / 24.54%.8852 / 30.80%.8264 / 7.22%.9279 / 8.87%
None (full PAQ-Diffusion).8153 / 6.80%.8830 / 15.17%.8740 / 15.79%.6417 / 1.15%.6078 / 1.40%

Table R1 shows that Stage 1 provides the primary privacy improvement while Stage 2 offers additional protection. Disabling either penalty increases membership leakage, and disabling both produces similar results to no privacy protection.

2. Out-of-grid and mixed-timestep attacks (Weakness 1 and Question 5)

For out-of-grid timesteps, we apply the activation scales of the nearest optimized timestep, motivated by the relatively smooth member–non-member reconstruction-error gap across neighboring timesteps (Figure 4(b)). We further evaluated SecMI attacks using out-of-grid timesteps and mixed-timestep features.

Table R2. Out-of-grid and mixed-timestep attacks (AUC / TPR@1%FPR, lower is better).

Attack targetSecMI-statSecMI-NN
On grid.6417 / 1.15%.6078 / 1.40%
Out of grid.6827 / 2.35%.6973 / 3.31%
Mixed features.6021 / 1.89%.6146 / 2.47%

As shown in Table R2, out-of-grid attacks achieve higher AUCs, but PAQ-Diffusion retains resistance. Mixed-timestep features do not consistently improve attack AUC, possibly because membership features and decision thresholds vary across timesteps, making a shared decision boundary less effective. These results support the effectiveness of PAQ-Diffusion beyond the optimized timestep.

3. Role of quantization and full-precision control (Weakness 4)

The activation scales in PAQ-Diffusion are actually necessary quantization parameters, which are the same as traditional quantization parameters. Our goal is to mitigate MIA leakage during post-training quantization by optimizing these existing parameters.

Nevertheless, a full-precision control would help isolate the contribution of our method. We will apply lightweight MMD-based fine-tuning to the FP model before conventional quantization, and compare it with PAQ-Diffusion. We will discuss this direction in the revised version.

4. Effect of batch size on MMD estimation (Question 1)

We conducted additional experiments on SD-Pokémon with batch sizes of 2, 4, and 8, since 16 will cause OOM for GPU.

Table R3. Batch size ablation on SD-Pokémon (AUC / TPR@1%FPR, lower is better).

Batch sizeLossPIASecMI-statSecMI-NN
2.8806 / 10.12%.9524 / 30.52%.5479 / 2.31%.5547 / 2.77%
4.8556 / 7.82%.9476 / 20.58%.5425 / 0.63%.5685 / 1.95%
8.8464 / 9.12%.9662 / 18.52%.5524 / 1.85%.5504 / 2.31%

Table R3 shows no consistent relationship between batch size and privacy performance, suggesting that MMD-based optimization remains effective across different batch sizes. However, performance variations may also depend on calibration data selection, as MMD estimates rely on the sampled member and non-member distributions.

5. SecMI-NN classifier training (Question 2)

We clarify that SecMI-NN is independently trained on each target model's own features, not transferring an FP-trained classifier.

6. Generation quality and statistical reliability (Weakness 6 and Question 3)

We conducted additional FID experiments on DDPM-CIFAR10 to examine generation and calibration variability.

Table R4. Multi-seed FID evaluation (mean ± standard deviation, lower is better).

ModelFID Mean ± Std
FP9.0967 ± 0.2600
PAQ-Diffusion (generation only)8.0014 ± 0.1744
PAQ-Diffusion (calibration + generation)10.1598 ± 1.9017

With a fixed calibration dataset and quantized checkpoint, varying generation seeds produces stable FID results, outperforming FP. However, varying both calibration data selection and generation seeds increases FID variability substantially. This indicates that calibration data selection affects generation quality.

7. Hyperparameter consistency and non-monotonic behavior (Weakness 7 and Question 4)

The experiments use λ=2\lambda=2 and 20,000 weight-reconstruction iterations. The main configuration (λ=2\lambda=2, N=1000N=1000) was selected based on the observed privacy–utility trade-off. We will correct the corresponding descriptions in the manuscript.

Thank you for the additional experiments and clarifications (Q2, Q4). R1 largely resolves W3, and W4 is deferred to the revision. W1, W2, and W5–W7 remain:

R2 suggests that Stage-2 gains may not transfer off-grid (SecMI-NN AUC 0.6973, near the Layer-wise model in Table 4 at 0.7203 with on-grid queries). R4 shows large calibration variance (FID 10.16 ± 1.90 vs. 9.10 for FP), so the FID improvement over FP is not supported. Yet the privacy metrics remain single-seed, and λ and N were selected on test results.

I maintain my score. The clear gains are mainly against SecMI on DDPM, while gains elsewhere (Loss/PIA/PIAN on DDPM: ~1–3 AUC points; SD models) are small or confounded. Even these gains remain unverified against white-box/adaptive attacks and across calibration seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.