- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles the fundamental difficulty confound in diffusion model membership inference by proposing calibration across two time axes: training time and diffusion time. It proposes Reference-Model Calibration (RMC), which compares fine-tuned and pretrained model losses, and Temporal Self-Calibration (TSC), which uses a symmetric three-point contrast of log-losses across noise levels within a single model. Experiments demonstrate improved detection at low false-positive rates, while further analysis shows that the informative noise region shifts with the training-loss target.
Critical strengths
- The paper clearly identifies a key limitation of loss-based membership inference: absolute denoising loss reflects both membership and intrinsic image difficulty, allowing easy non-members to incur lower losses than difficult members.
- The paper proposes complementary calibration methods across training time (RMC) and diffusion time (TSC), supporting fine-tuning audits with a reference checkpoint and single-model audits, respectively. Both substantially improve membership detection at low false-positive rates.
Critical weaknesses
- The difficulty-calibration interpretation is not fully established. Although the log-loss ablation and variance diagnostics provide supporting evidence, they do not directly isolate image difficulty as the source of the suppressed variation. Controlled experiments using independent difficulty measures and analysis of failure cases would help clarify the scope of this interpretation.
- The baseline coverage is limited, with the main comparisons focusing on early established methods such as Naive, SecMI, and PIA. Comparisons with more recent diffusion-model MIAs under compatible access assumptions would help establish the proposed methods’ empirical advantages.
- Text-to-image experiments are limited to Stable Diffusion v1.4. Evaluation on a modern DiT-based diffusion or flow-matching model would help establish whether the benefits extend beyond the tested U-Net architectures.
- TSC selects its center λ⋆ using shadow data, but its robustness to mismatches between shadow and target settings remains unclear. Cross-dataset and cross-training-stage transfer tests, sensitivity analysis of λ⋆, and threshold calibration without closely matched shadow models would clarify its practical applicability.
- The main comparisons use different query budgets: TSC typically requires 15 queries, whereas PIA/PIAN use only two. Although matched-budget comparisons with Naive are provided, comparable evaluations against PIA/PIAN are needed to assess how much of the improvement comes from calibration rather than additional queries.
Questions for authors
See the weaknesses.
3 comments
Response to Reviewer KboM, part 2/3
W3: Architecture coverage.
To test whether the benefits extend beyond U-Net backbones, we added an unconditional CIFAR-10 DiT-S/2 experiment under the DDPM training setup. TSC achieves 27.35% TPR@1%FPR, compared with 11.21% for the strongest evaluated baseline, PIAN (Table 5 page 9). Training and checkpoint details appear in Appendix B (page 15).
DiT-S/2 on CIFAR-10 at 2K epochs. All metrics are percentages. Parentheses give query counts. Bold denotes the best result and underlining the second-best.
| Method (queries) | AUC | ASR | TPR@1%FPR |
|---|---|---|---|
| Naive (20) | 70.56 | 65.41 | 2.84 |
| SecMI (12) | 77.32 | 71.11 | 3.10 |
| PIA (2) | 71.47 | 7.53 | |
| PIAN (2) | 77.61 | ||
| SimA MC (15) | 74.10 | 68.23 | 3.60 |
| TSC (15) | 86.03 | 78.64 | 27.35 |
This provides evidence that TSC’s benefits extend to transformer-based architecture. The new experiment is an unconditional pixel-space evaluation.
W4: Shadow-target mismatch and calibration.
We have added the requested transfer and sensitivity analyses in Appendix E.2, pages 22-24. Centers are selected exclusively on first-half shadow models and then applied unchanged to independently trained second-half targets. The transfer analysis selects a center per shadow setting, separately from the shared centers used in the main benchmark tables.
3 representative results are shown below. The reference column uses a shadow matched to the target’s dataset and training stage. All metrics are percentages.
All metrics are percentages. The first three metric columns report target performance using the transferred center. The matched reference uses a shadow with the target's dataset and training stage.
| Transfer | Target AUC | Target ASR | Target TPR@1%FPR | Matched AUC | Matched ASR | Matched TPR@1%FPR |
|---|---|---|---|---|---|---|
| CIFAR-100, 2K → 4K | 96.93 | 91.42 | 63.16 | 97.22 | 92.11 | 63.25 |
| CIFAR-100, 4K → 2K | 84.35 | 77.19 | 14.92 | 85.67 | 78.39 | 16.88 |
| CIFAR-10 → Tiny ImageNet, both 4K | 78.45 | 72.01 | 10.73 | 78.49 | 71.99 | 11.21 |
| Tiny ImageNet → CIFAR-10, both 4K | 96.52 | 91.12 | 65.12 | 96.65 | 91.34 | 62.90 |
Appendix E.2 (page 22) also reports center sensitivity on CIFAR-100. AUC remains above 94% for centers from 0.50 to 1.75, while TPR@1%FPR ranges from 49.16% to 63.25%. Overall membership separation therefore remains strong across this range, with greater variation visible at the strict low-FPR operating point. TPR@1%FPR measures detection at one operating point, whereas AUC summarizes the full ROC curve. The transfer experiments above show that shadow-selected centers remain effective across the tested datasets and training stages.
We separately evaluate decision-threshold transfer while fixing . All shadows and targets are trained for 4K epochs, and all metrics below are percentages.
First, we select the ASR-maximizing threshold on the shadow and apply it unchanged to the target. Across all tested cross-dataset pairs, the resulting ASR is within 1.20 percentage points of the target-optimal ASR. Representative examples are:
| Shadow → Target | ASR: shadow-selected threshold | ASR: target-selected threshold (oracle) |
|---|---|---|
| CIFAR-10 → Tiny ImageNet | 71.19 | 72.18 |
| Tiny ImageNet → CIFAR-10 | 89.54 | 90.15 |
The target-selected threshold uses target membership labels and serves only as an oracle reference.
Second, we calibrate a threshold to 1% FPR using only shadow non-members and apply it unchanged to the target. We compare cross-dataset transfer with the usual matched-dataset shadow calibration:
| Shadow → Target | Transferred TPR | Target FPR | Matched-shadow TPR | Matched-shadow FPR |
|---|---|---|---|---|
| CIFAR-10 → Tiny ImageNet | 18.65 | 1.68 | 12.87 | 1.06 |
| Tiny ImageNet → CIFAR-10 | 53.58 | 0.63 | 62.30 | 1.00 |
The matched-shadow reference uses an independently trained shadow on the target's dataset. Both calibration procedures use shadow data only, and the actual target FPR is measured rather than fixed.
These results support effective threshold transfer in the tested settings. ASR remains close to the target-selected oracle reference, while thresholds calibrated to 1% shadow FPR produce target FPRs between 0.59% and 1.68% across the full transfer grid.
Response to Reviewer KboM, part 3/3
W5: Query-budget comparisons.
TSC requires a minimum of three queries, one at each evaluated noise level. The default 15-query configuration averages five raw losses per level before taking logarithms, reducing sampling variability in the temporal contrast. We have added a low-query comparison across eight DDPM checkpoints covering four datasets at two training stages (Appendix E.1, Table 14, page 21).
Table 14: Query efficiency across eight DDPM checkpoints. Results are equal-weight averages over four datasets at 2K and 4K epochs. All metrics are percentages.
| Method | Queries | AUC | ASR | TPR@1%FPR |
|---|---|---|---|---|
| Naive loss | 1 | 68.14 | 63.71 | 4.00 |
| SimA | 1 | 72.96 | 68.09 | 5.29 |
| PIA | 2 | 9.01 | ||
| PIAN | 2 | 71.41 | 67.47 | |
| TSC | 3 | 75.98 | 70.50 | 14.95 |
| TSC | 15 | 80.54 | 74.79 | 23.28 |
Three-query TSC achieves 14.95% mean TPR@1%FPR, compared with 9.01% for PIA and 10.93% for PIAN, each using two queries. TSC also achieves higher mean AUC and ASR. Averaging five draws per level further raises its mean TPR to 23.28%, showing that its advantage is already present without multi-draw averaging.
PIA and PIAN obtain an image-specific noise estimate from the model and use it to construct the subsequent noisy input. With the image representation, conditioning, and attack timestep fixed, repeating their original two-query procedure produces the same score and provides no additional averaging benefit. Exploiting further queries would require extending their scoring procedure, for example by aggregating across timesteps or augmentations.
We appreciate the care you have taken in reviewing our paper. Your feedback and requested experiments have helped us improve both the presentation and evaluation, and we hope the revisions provide the clarification and evidence you were looking for.
Response to Reviewer KboM, part 1/3
Thank you for the time and effort you have put into reviewing our paper! We appreciate your recognition of the difficulty confound and the complementary roles of calibration across training time and diffusion time. We have added experiments addressing the interpretation, baseline coverage, architecture coverage, shadow-target mismatch, and query budgets.
W1: Evidence for the difficulty-calibration interpretation.
We agree that the original ablations do not uniquely establish image difficulty as the source of the suppressed variation. We have added a stratified analysis on CIFAR-10 in Figure 5 (3.3.2, page 7), using a model-relative difficulty proxy obtained by averaging log reconstruction losses across several noise levels. The proxy uses noise draws disjoint from those used to compute the attack scores, avoiding reuse of the same sampling noise.
For each method, we calibrate a single global threshold targeting 1% FPR and apply it unchanged across all difficulty groups. From the easiest to the hardest group, TSC’s TPR increases from 52.51% to 67.97%, whereas PIAN’s TPR decreases from 45.77% to 5.57%. The other evaluated baselines also exhibit declining detection as difficulty increases. Figure 5(a) additionally shows how temporal calibration recovers high-loss members that absolute reconstruction loss misses, while retaining some overlap between members and non-members.
These results support the interpretation that TSC mitigates model-relative reconstruction difficulty. We distinguish this empirical evidence from identifying a unique, model-independent notion of image difficulty or establishing that every suppressed component is caused by difficulty. Appendix C likewise distinguishes the contrast’s mathematical invariances from the interpretation of the removed variation.
W2: Recent baselines.
We have added SimA MC (Rao et al., 2025) to Tables 2-4 (pages 8-9) under compatible model-access and caption assumptions, using 15 queries to match TSC’s default budget. We have also added its description to Related Work (2 page 3) and the baseline description (4.1 page 7). TSC retains the highest TPR@1%FPR across all nine prediction/loss configurations in Table 4 (page 9).
Table 2:
Table 3:
While SimA might improve AUC and ASR at some experiments, our method TSC remains highly effective for TPR at low FPR region.