acceptodds
Under review as a conference paper at ICLR 2027

Difficulty Calibration Across Time for Membership Inference in Diffusion Models

Abstract

Modern diffusion models are trained on massive datasets that often include copyrighted or private images. Membership inference attacks (MIAs) address this by testing whether a given image was in a model's training set. The standard approach scores an image by its reconstruction loss and flags low-loss images as members. This often fails because some images are simply easier to reconstruct than others, so a simple unseen image can score a lower loss than a complex training image. Difficulty calibration separates memorization from how difficult an image is to begin with. We show how to calibrate along two axes of time: training time and diffusion time. First, across training time, we introduce Reference-Model Calibration (RMC), which compares a fine-tuned model's loss against an earlier checkpoint. This cleanly isolates newly memorized data and gives near-perfect auditing, even without text captions. Since earlier checkpoints are not always available, we next present Temporal Self-Calibration (TSC), which calibrates across diffusion time using only a single model. Our key observation is that an image's difficulty persists across noise levels, whereas memorization is localized in time. By comparing the loss at a target timestep against neighboring timesteps from the same model, TSC attenuates image difficulty using the model's own trajectory. Empirically, our methods beat the state of the art by a wide margin, and unlike most existing attacks, generalize across different model parameterizations such as -, -, and -prediction models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, MIA, Loss Trajectory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KboMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the fundamental difficulty confound in diffusion model membership inference by proposing calibration across two time axes: training time and diffusion time. It proposes Reference-Model Calibration (RMC), which compares fine-tuned and pretrained model losses, and Temporal Self-Calibration (TSC), which uses a symmetric three-point contrast of log-losses across noise levels within a single model. Experiments demonstrate improved detection at low false-positive rates, while further analysis shows that the informative noise region shifts with the training-loss target.

Critical strengths
  1. The paper clearly identifies a key limitation of loss-based membership inference: absolute denoising loss reflects both membership and intrinsic image difficulty, allowing easy non-members to incur lower losses than difficult members.
  2. The paper proposes complementary calibration methods across training time (RMC) and diffusion time (TSC), supporting fine-tuning audits with a reference checkpoint and single-model audits, respectively. Both substantially improve membership detection at low false-positive rates.
Critical weaknesses
  1. The difficulty-calibration interpretation is not fully established. Although the log-loss ablation and variance diagnostics provide supporting evidence, they do not directly isolate image difficulty as the source of the suppressed variation. Controlled experiments using independent difficulty measures and analysis of failure cases would help clarify the scope of this interpretation.
  2. The baseline coverage is limited, with the main comparisons focusing on early established methods such as Naive, SecMI, and PIA. Comparisons with more recent diffusion-model MIAs under compatible access assumptions would help establish the proposed methods’ empirical advantages.
  3. Text-to-image experiments are limited to Stable Diffusion v1.4. Evaluation on a modern DiT-based diffusion or flow-matching model would help establish whether the benefits extend beyond the tested U-Net architectures.
  4. TSC selects its center λ⋆ using shadow data, but its robustness to mismatches between shadow and target settings remains unclear. Cross-dataset and cross-training-stage transfer tests, sensitivity analysis of λ⋆, and threshold calibration without closely matched shadow models would clarify its practical applicability.
  5. The main comparisons use different query budgets: TSC typically requires 15 queries, whereas PIA/PIAN use only two. Although matched-budget comparisons with Naive are provided, comparable evaluations against PIA/PIAN are needed to assess how much of the improvement comes from calibration rather than additional queries.
Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KboM, part 1/3

Thank you for the time and effort you have put into reviewing our paper! We appreciate your recognition of the difficulty confound and the complementary roles of calibration across training time and diffusion time. We have added experiments addressing the interpretation, baseline coverage, architecture coverage, shadow-target mismatch, and query budgets.

W1: Evidence for the difficulty-calibration interpretation.

We agree that the original ablations do not uniquely establish image difficulty as the source of the suppressed variation. We have added a stratified analysis on CIFAR-10 in Figure 5 (3.3.2, page 7), using a model-relative difficulty proxy obtained by averaging log reconstruction losses across several noise levels. The proxy uses noise draws disjoint from those used to compute the attack scores, avoiding reuse of the same sampling noise.

For each method, we calibrate a single global threshold targeting 1% FPR and apply it unchanged across all difficulty groups. From the easiest to the hardest group, TSC’s TPR increases from 52.51% to 67.97%, whereas PIAN’s TPR decreases from 45.77% to 5.57%. The other evaluated baselines also exhibit declining detection as difficulty increases. Figure 5(a) additionally shows how temporal calibration recovers high-loss members that absolute reconstruction loss misses, while retaining some overlap between members and non-members.

These results support the interpretation that TSC mitigates model-relative reconstruction difficulty. We distinguish this empirical evidence from identifying a unique, model-independent notion of image difficulty or establishing that every suppressed component is caused by difficulty. Appendix C likewise distinguishes the contrast’s mathematical invariances from the interpretation of the removed variation.

W2: Recent baselines.

We have added SimA MC (Rao et al., 2025) to Tables 2-4 (pages 8-9) under compatible model-access and caption assumptions, using 15 queries to match TSC’s default budget. We have also added its description to Related Work (2 page 3) and the baseline description (4.1 page 7). TSC retains the highest TPR@1%FPR across all nine prediction/loss configurations in Table 4 (page 9).

Table 2:

Method (queries)CIFAR-10 AUCASRTPRTiny ImageNet AUCASRTPRSTL-10-U AUCASRTPR
Naive (20)85.3978.217.3564.3760.522.9364.1560.502.51
SecMI (12)88.2480.9110.2066.3761.882.9168.1263.333.03
PIA (2)91.90‾\underline{91.90}84.88‾\underline{84.88}25.8068.99‾\underline{68.99}64.25‾\underline{64.25}4.4369.46‾\underline{69.46}64.51‾\underline{64.51}4.08
PIAN (2)89.2683.3230.90‾\underline{30.90}67.0463.446.10‾\underline{6.10}66.7263.364.32‾\underline{4.32}
SimA MC (15)90.2683.0613.2267.0162.542.9667.2962.872.92
TSC (15)95.9190.1562.3178.6372.1812.2777.8471.299.57

Table 3:

Method (queries)Pokémon AUCASRTPRMS-COCO AUCASRTPRFlickr AUCASRTPR
Naive (10)84.8477.4328.3776.6169.927.1677.9671.6610.08
SecMI (12)83.5976.4715.1478.3371.466.8481.1474.247.40
PIA, t=500 (2)69.8666.1319.7154.0853.962.7654.7353.241.96
PIA, optimal t (2)94.5087.8861.3075.6170.9017.36‾\underline{17.36}74.1869.5817.12‾\underline{17.12}
PIAN (2)53.3454.151.9269.2064.0813.8869.1565.1012.40
SimA MC (15)78.7071.5531.7380.55‾\underline{80.55}72.96‾\underline{72.96}11.2082.86‾\underline{82.86}76.40‾\underline{76.40}13.04
TSC (15)91.37‾\underline{91.37}84.03‾\underline{84.03}55.77‾\underline{55.77}85.5578.4229.1684.6077.9830.28

While SimA might improve AUC and ASR at some experiments, our method TSC remains highly effective for TPR at low FPR region.

AuthorsOpenReview ↗

Response to Reviewer KboM, part 2/3

W3: Architecture coverage.

To test whether the benefits extend beyond U-Net backbones, we added an unconditional CIFAR-10 DiT-S/2 experiment under the DDPM training setup. TSC achieves 27.35% TPR@1%FPR, compared with 11.21% for the strongest evaluated baseline, PIAN (Table 5 page 9). Training and checkpoint details appear in Appendix B (page 15).

DiT-S/2 on CIFAR-10 at 2K epochs. All metrics are percentages. Parentheses give query counts. Bold denotes the best result and underlining the second-best.

Method (queries)AUCASRTPR@1%FPR
Naive (20)70.5665.412.84
SecMI (12)77.3271.113.10
PIA (2)78.18‾\underline{78.18}71.477.53
PIAN (2)77.6171.77‾\underline{71.77}11.21‾\underline{11.21}
SimA MC (15)74.1068.233.60
TSC (15)86.0378.6427.35

This provides evidence that TSC’s benefits extend to transformer-based architecture. The new experiment is an unconditional pixel-space evaluation.

W4: Shadow-target mismatch and calibration.

We have added the requested transfer and sensitivity analyses in Appendix E.2, pages 22-24. Centers are selected exclusively on first-half shadow models and then applied unchanged to independently trained second-half targets. The transfer analysis selects a center per shadow setting, separately from the shared centers used in the main benchmark tables.

3 representative results are shown below. The reference column uses a shadow matched to the target’s dataset and training stage. All metrics are percentages.

All metrics are percentages. The first three metric columns report target performance using the transferred center. The matched reference uses a shadow with the target's dataset and training stage.

TransferTarget AUCTarget ASRTarget TPR@1%FPRMatched AUCMatched ASRMatched TPR@1%FPR
CIFAR-100, 2K → 4K96.9391.4263.1697.2292.1163.25
CIFAR-100, 4K → 2K84.3577.1914.9285.6778.3916.88
CIFAR-10 → Tiny ImageNet, both 4K78.4572.0110.7378.4971.9911.21
Tiny ImageNet → CIFAR-10, both 4K96.5291.1265.1296.6591.3462.90

Appendix E.2 (page 22) also reports center sensitivity on CIFAR-100. AUC remains above 94% for centers from 0.50 to 1.75, while TPR@1%FPR ranges from 49.16% to 63.25%. Overall membership separation therefore remains strong across this range, with greater variation visible at the strict low-FPR operating point. TPR@1%FPR measures detection at one operating point, whereas AUC summarizes the full ROC curve. The transfer experiments above show that shadow-selected centers remain effective across the tested datasets and training stages.

We separately evaluate decision-threshold transfer while fixing λ⋆=0.75\lambda^\star=0.75. All shadows and targets are trained for 4K epochs, and all metrics below are percentages.

First, we select the ASR-maximizing threshold on the shadow and apply it unchanged to the target. Across all tested cross-dataset pairs, the resulting ASR is within 1.20 percentage points of the target-optimal ASR. Representative examples are:

Shadow → TargetASR: shadow-selected thresholdASR: target-selected threshold (oracle)
CIFAR-10 → Tiny ImageNet71.1972.18
Tiny ImageNet → CIFAR-1089.5490.15

The target-selected threshold uses target membership labels and serves only as an oracle reference.

Second, we calibrate a threshold to 1% FPR using only shadow non-members and apply it unchanged to the target. We compare cross-dataset transfer with the usual matched-dataset shadow calibration:

Shadow → TargetTransferred TPRTarget FPRMatched-shadow TPRMatched-shadow FPR
CIFAR-10 → Tiny ImageNet18.651.6812.871.06
Tiny ImageNet → CIFAR-1053.580.6362.301.00

The matched-shadow reference uses an independently trained shadow on the target's dataset. Both calibration procedures use shadow data only, and the actual target FPR is measured rather than fixed.

These results support effective threshold transfer in the tested settings. ASR remains close to the target-selected oracle reference, while thresholds calibrated to 1% shadow FPR produce target FPRs between 0.59% and 1.68% across the full transfer grid.

AuthorsOpenReview ↗

Response to Reviewer KboM, part 3/3

W5: Query-budget comparisons.

TSC requires a minimum of three queries, one at each evaluated noise level. The default 15-query configuration averages five raw losses per level before taking logarithms, reducing sampling variability in the temporal contrast. We have added a low-query comparison across eight DDPM checkpoints covering four datasets at two training stages (Appendix E.1, Table 14, page 21).

Table 14: Query efficiency across eight DDPM checkpoints. Results are equal-weight averages over four datasets at 2K and 4K epochs. All metrics are percentages.

MethodQueriesAUCASRTPR@1%FPR
Naive loss168.1463.714.00
SimA172.9668.095.29
PIA273.79‾\underline{73.79}68.65‾\underline{68.65}9.01
PIAN271.4167.4710.93‾\underline{10.93}
TSC375.9870.5014.95
TSC1580.5474.7923.28

Three-query TSC achieves 14.95% mean TPR@1%FPR, compared with 9.01% for PIA and 10.93% for PIAN, each using two queries. TSC also achieves higher mean AUC and ASR. Averaging five draws per level further raises its mean TPR to 23.28%, showing that its advantage is already present without multi-draw averaging.

PIA and PIAN obtain an image-specific noise estimate from the model and use it to construct the subsequent noisy input. With the image representation, conditioning, and attack timestep fixed, repeating their original two-query procedure produces the same score and provides no additional averaging benefit. Exploiting further queries would require extending their scoring procedure, for example by aggregating across timesteps or augmentations.

  We appreciate the care you have taken in reviewing our paper. Your feedback and requested experiments have helped us improve both the presentation and evaluation, and we hope the revisions provide the clarification and evidence you were looking for.

Reviewer_MedHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates the importance of difficulty calibration for MI in diffusion models. It shows huge benefits in MIAs for finetuning, when we subtract loss from the base model when evaluating finetune-based membership. The authors also propose to use special calibration for diffusion models during inference phase - by finding the region of maximum separation between memebers and nonmembers and calculate distinctive value in the neighbourhood of the center of that region.

Critical strengths
  1. Showing the importance of difficulty calibration for MIA both using training checkpoints and generation steps in diffusion.
Critical weaknesses

W1 From what I understand RMC is a direct application of previous idea, e. g.proposed earlier here: On the Importance of Difficulty Calibration in Membership Inference Attacks, ICLR 2022 (not even cited)

W2 TSC problems: 1) is not general MIA, it can be applied only for diffusion 2) compared with older methods at most from 2024, related work is at most citing works from 2024 3) is not better than CLiD 4) is not explained well enough to reproduce the method - see Q1 5) experiments are not extensive.

Questions for authors

Most important: see weaknesses. Additionally: Q1. How lambda* is selected? Q2. How is Tab1 CLiD baseline calculated? Is it benefiting from recalibration? Q3. In tab 1 we distinguish finetuned-members/ non-members, not real ones?

4 comments
AuthorsOpenReview ↗

Response to Reviewer MedH, part 1/2

Thank you for the time and effort you have put into reviewing our paper! We appreciate your recognition of the importance of difficulty calibration across training checkpoints and diffusion timesteps. Below, we clarify the relationship to prior work, the intended scope, and the evaluation protocol.

W1: Relationship with previous difficulty-calibration methods.

We thank the reviewer for pointing out Watson et al. (ICLR 2022). We have added this reference and discussed its relationship to score calibration in the Related Work section 2, page 3. Difficulty calibration and reference-score comparisons are established ideas. As acknowledged in our introduction, RMC was inspired by reference-based approaches to auditing LLM fine-tuning data. Our contribution is to systematically evaluate prior-checkpoint calibration for membership inference in diffusion-model fine-tuning. RMC uses the available pretrained checkpoint to calibrate denoising losses, requiring no additional training to construct the calibration reference, and demonstrates strong effectiveness without captions.

W2.1: Scope of TSC.

TSC is designed for diffusion and flow-matching models, whose noise-dependent loss trajectories provide the reference used by the attack. Like diffusion-model attacks such as SecMI and PIA, it exploits the structure of the denoising process. Applicability to arbitrary model families is outside the paper’s stated scope. Within this setting, our evaluations cover pixel-space and latent-space models, conditional and unconditional models, convolutional and transformer architectures, and different prediction/loss parameterizations.

W2.2 and W2.5: Recent baselines and experimental coverage.

We have added SimA MC, from the more recent work (Rao et al., 2025), to Tables 2–4 using 15 queries (pages 8-9), matching TSC’s default budget. We have also added its description to Related Work (2 page 3), and the baseline description (4.1 page 7).

Table 2:

Method (queries)CIFAR-10 AUCASRTPRTiny ImageNet AUCASRTPRSTL-10-U AUCASRTPR
Naive (20)85.3978.217.3564.3760.522.9364.1560.502.51
SecMI (12)88.2480.9110.2066.3761.882.9168.1263.333.03
PIA (2)91.90‾\underline{91.90}84.88‾\underline{84.88}25.8068.99‾\underline{68.99}64.25‾\underline{64.25}4.4369.46‾\underline{69.46}64.51‾\underline{64.51}4.08
PIAN (2)89.2683.3230.90‾\underline{30.90}67.0463.446.10‾\underline{6.10}66.7263.364.32‾\underline{4.32}
SimA MC (15)90.2683.0613.2267.0162.542.9667.2962.872.92
TSC (15)95.9190.1562.3178.6372.1812.2777.8471.299.57

Table 3:

Method (queries)Pokémon AUCASRTPRMS-COCO AUCASRTPRFlickr AUCASRTPR
Naive (10)84.8477.4328.3776.6169.927.1677.9671.6610.08
SecMI (12)83.5976.4715.1478.3371.466.8481.1474.247.40
PIA, t=500 (2)69.8666.1319.7154.0853.962.7654.7353.241.96
PIA, optimal t (2)94.5087.8861.3075.6170.9017.36‾\underline{17.36}74.1869.5817.12‾\underline{17.12}
PIAN (2)53.3454.151.9269.2064.0813.8869.1565.1012.40
SimA MC (15)78.7071.5531.7380.55‾\underline{80.55}72.96‾\underline{72.96}11.2082.86‾\underline{82.86}76.40‾\underline{76.40}13.04
TSC (15)91.37‾\underline{91.37}84.03‾\underline{84.03}55.77‾\underline{55.77}85.5578.4229.1684.6077.9830.28
While SimA might improve AUC and ASR at some experiments, our method TSC remains highly effective for TPR at low FPR region.

To evaluate a different architecture, we added an unconditional CIFAR-10 DiT-S/2 experiment under the DDPM training setup: DiT-S/2 on CIFAR-10 at 2K epochs. All metrics are percentages. Parentheses give query counts.

Method (queries)AUCASRTPR@1%FPR
Naive (20)70.5665.412.84
SecMI (12)77.3271.113.10
PIA (2)78.18‾\underline{78.18}71.477.53
PIAN (2)77.6171.77‾\underline{71.77}11.21‾\underline{11.21}
SimA MC (15)74.1068.233.60
TSC (15)86.0378.6427.35

TSC achieves 27.35% TPR@1%FPR, compared with 11.21% for the strongest evaluated baseline, PIAN (Table 5). Training details appear in Appendix B (page 15).

We have also added low-query evaluations (Appendix E.1, page 21), together with analyses of cross-dataset and cross-training-stage center transfer, center sensitivity, and threshold transfer (Appendix E.2, pages 22-24). These additions complement the existing evaluations across datasets, training stages, augmentation settings, conditioning, and prediction/loss combinations.

AuthorsOpenReview ↗

Response to Reviewer MedH, part 2/2

W2.3: Comparison with CLiD. CLiD is a strong caption-conditioned attack, and we do not claim that TSC universally outperforms it. TSC supports auditing without captions or an auxiliary caption-generation model and also applies to unconditional models. We acknowledge that CLiD’s generated-caption experiments demonstrate that exact training captions are not always necessary. We additionally evaluated CLiD with BLIP-2-generated captions in the Pokémon overfit setting. With oracle-tuned α, its results, alongside caption-free TSC, are:

MethodAUCASRTPR@1%FPR
CLiD (BLIP-2 captions)68.0268.0265.8265.8220.1920.19
TSC (no captions)91.3791.3784.0384.0355.7755.77
This provides a concrete example of the usefulness of caption-free auditing alongside CLiD’s generated-caption approach. When a pretrained checkpoint is available, Table 1 (page 5) separately shows that 2-query RMC outperforms the 15-query, caption-conditioned CLiD baseline on all three evaluated fine-tuning datasets.

W2.4 and Q1: Reproducibility and selection of λ⋆\lambda^\star.

The center is a setting-level hyperparameter, fixed across candidate images. For the DDPM and Stable Diffusion benchmarks, we select it by maximizing AUC on a disjoint shadow data split, then fix it before target evaluation. Target membership labels are not used for this selection. We use λ⋆=0.75\lambda^\star = 0.75 log-SNR for DDPM and 0.250.25 for Stable Diffusion, with Δ = 2.5. Timestep-dependent baselines are also tuned on shadow data, while SecMI retains its fixed protocol.

We have expanded the implementation details (4.1, page 7) to specify the repeated-query computation explicitly. At each of the three noise levels, we average five raw losses, take the logarithm of each average, and compute the temporal contrast. This requires 15 queries. One draw per level gives the three-query variant, evaluated in Appendix E.1, Table 14 (page 21). For the prediction/loss experiments, the shared centers are selected from first-half shadow-model AUC curves and reported in Table 4 and Appendix E.6 (page 26). Appendix E.2 (pages 22-24) additionally examines center sensitivity and transfer across datasets and training stages.

Q2: Construction of the CLiD baseline in Table 1. We calculate the baseline using the authors’ threshold-based CLiD implementation, including its native normalization and published 15-query timestep configuration. We tune its mixing coefficient α to maximize AUC on the evaluated model’s labeled split and compute the reported metrics using that selected score. Table 1 therefore reports an oracle-tuned CLiD baseline using its original scoring procedure.

Q3: Meaning of membership in Table 1. Table 1 evaluates membership in the audited model’s fine-tuning dataset. Members are images included in that fine-tuning stage, while non-members are held-out images from the same data distribution. These labels do not establish whether either group appeared in the base model’s pretraining corpus.

  Thank you again for your time and valuable feedback. We appreciate the questions and we hope our responses and the added results address your concerns.

Thanks for the answers and additional effort during the rebuttal. In general, it only partially addressed my concerns.

I think that this submission should not be accepted in the current form. It may look from the rebuttal that the experiments are extensive (when described by the authors), but from what I remember reading this work, they were rather not, and on small scale, which additionally raises questions about the generality and usefulness of the method. Additionally, I think its originality is still in question and also with the narrowed scope the work/presented results do not look significant enough in my opinion.

Additional comment for BLIP-2-generated captions in the Pokémon overfit setting: Given my direct experience with this dataset, I think general captions provided would not help here due to the specifics of this dataset, and could rather misdirect the model, so this is not a fair example in my opinion (see what captions are produced and what is on the image - generated general captions are not helpful at all).

Thank you for following up. We appreciate your point about the Pokémon captions. We would also like to clarify where our contribution builds on prior work and which experiments were part of the original submission.

Originality and relation to Watson et al.

RMC shares the principle of calibrating a candidate’s score against a reference score. Watson et al. construct that reference by training a classifier on auxiliary data. In their white-box forgetting variant, the auditor starts from the trained target and continues training it on separate data. The change in score helps identify examples from the target’s original training set.

RMC asks whether an image was used in a particular fine-tuning run. When the starting checkpoint is available, the auditor can measure how well the model denoised that same image before fine-tuning and compare it with the loss after fine-tuning. Both checkpoints already exist. As acknowledged in our introduction, this choice of reference was inspired by work on auditing LLM fine-tunes. Our contribution here is also empirical: across the three Stable Diffusion fine-tunes in Table 1, one query to each checkpoint detects 90.63%–96.86% of fine-tuning members at 1% FPR, without captions, showing how extremely effective this method is for auditing finetuned diffusion models.

TSC makes a separate methodological contribution. It uses the image’s losses at two surrounding noise levels as a reference for its loss at the center, so the auditor can exploit the shape of the loss trajectory using a single checkpoint. If the remaining originality concern also applies to TSC, we would appreciate knowing which prior method the reviewer considers closest to this temporal comparison

Scope and experimental evidence.

The original submission evaluated DDPMs on CIFAR-10, CIFAR-100, STL-10-U, and Tiny ImageNet, as well as Stable Diffusion v1.4 fine-tunes on Pokémon, MS-COCO, and Flickr. It also evaluated all nine prediction and training-loss combinations for CIFAR-100 flow-matching models, with and without augmentation. That controlled experiment let us test whether the informative noise region shifts with the training objective, a question that directly affects how an auditor selects the evaluation timestep.

The revision extends this evidence in the directions raised during review. We added the recent SimA baseline at the same 15-query budget as TSC (Tables 2–4), tested transfer across datasets and training stages (Appendix E.2), and evaluated a transformer backbone (Table 5). This extends the architecture evidence beyond U-Nets.

Generated captions on Pokémon.

We agree that a general-purpose captioner may produce unhelpful captions for Pokémon images. This is a practical difficulty an auditor can face when checking whether a character-generation model was trained or fine-tuned on copyrighted artwork. A rights holder may have both the images and the released model, yet have no way to know how those images were described during training. The training caption might name a character that the captioner describes only in generic terms. Equally, the captioner might correctly identify the character even though the training caption described only its appearance. Even an accurate description of the image can therefore differ in ways that matter for a caption-conditioned audit.

This issue can arise across specialized collections, including comics, animation, and game artwork. CLiD shows that generated captions work well on MS-COCO. Our Pokémon experiment illustrates why an image-only approach can also be useful.

We appreciate the continued discussion and hope the reviewer will consider both the original experiments and the evidence added in the revision when making their final assessment.

Reviewer_tmhhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the confounding effect of sample difficulty on membership inference attacks (MIAs) against diffusion models. It proposes two calibration methods—Reference Model Calibration (RMC) and Temporal Self-Calibration (TSC)—that draw on the training process and the diffusion process, respectively, to mitigate the influence of intrinsic sample difficulty on membership inference. The effectiveness of RMC is evaluated on Stable Diffusion models fine-tuned on three datasets, while TSC is evaluated across six datasets and four baseline methods.

Critical strengths

1.The paper addresses an important problem and proposes an intuitive and straightforward solution. The RMC approach draws inspiration from prior work on large models, which is an interesting direction.

2.The observation that intrinsic image difficulty persists across different noise levels during denoising, whereas memorization manifests primarily at particular timesteps, is interesting and offers useful insights.

Critical weaknesses

1.The baseline comparisons raise concerns regarding fairness. In Tables 2–4, TSC uses 15 queries, whereas PIA and PIAN use only two. Although Figure 5 provides a comparison under equal query budgets, it includes only the Naive baseline and omits the strongest baselines, PIA and PIAN. The authors should provide query budget–performance curves that include appropriately extended versions of these methods. If PIA and PIAN cannot effectively exploit additional queries, the specific reasons should be explained.

2.The evidence supporting the key observation could be strengthened. Although Figure 3 visually illustrates the effectiveness of TSC, the provenance of the results is unclear. For example, the authors should specify which dataset was used and explain precisely how the curves were computed. Without these details, the current evidence is insufficiently convincing.

Questions for authors

See “Critical Weaknesses”.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tmhh

Thank you for the time and effort you have put into reviewing our paper! We appreciate your recognition of the problem’s importance, the simplicity of our calibration approaches, and the observation underlying TSC.

Weakness 1: Query budgets and comparison with PIA/PIAN.

PIA and PIAN first query the model on the clean image to obtain an image-specific noise estimate, which PIAN additionally normalizes. They then construct a noisy image using this estimate and perform a second query to compute the prediction error. With the image representation, conditioning, and attack timestep fixed, this procedure is deterministic. Repeating these evaluations therefore produces the same score, providing no additional averaging benefit. Exploiting further queries would require extending the original scoring procedure, for example by aggregating scores across multiple timesteps or image augmentations. TSC requires only three queries, one at each evaluated noise level. Our default 15-query configuration averages five noise realizations per level to reduce sampling variability in the temporal contrast. To clarify the contribution of this averaging, we added a low-query comparison across eight DDPM checkpoints, covering four datasets at two training stages (Table 14 - Appendix E.1, page 21). Table 14: Query efficiency across eight DDPM checkpoints. Results are equal-weight averages over four datasets at 2K and 4K epochs. All metrics are percentages.

MethodQueriesAUCASRTPR@1%FPR
Naive loss168.1463.714.00
SimA172.9668.095.29
PIA273.79‾\underline{73.79}68.65‾\underline{68.65}9.01
PIAN271.4167.4710.93‾\underline{10.93}
TSC375.9870.5014.95
TSC1580.5474.7923.28

With only 3 queries, TSC achieves a mean TPR@1%FPR of 14.95%, compared with 9.01% for PIA and 10.93% for PIAN, each using two queries. TSC also exceeds both methods in mean AUC and ASR. Averaging five draws per level further raises TSC’s mean TPR to 23.28%. Thus, its advantage is already present at the minimum 3-query setting, with additional averaging providing a further improvement.

Weakness 2: Provenance and computation of Figure 3.

We agree that the provenance of these curves should be explicit. Figure 3 shows two selected member/non-member pairs from a CIFAR-10 DDPM checkpoint trained for 4K epochs. For each image, we evaluate reconstruction loss on a grid with nominal log-SNR spacing of 0.25. Each plotted value is the logarithm of the mean loss over 5 Gaussian noise draws. We have expanded the caption to include these details and clarify that the figure presents illustrative individual examples. The four corresponding candidate images are now provided in Appendix B, Figure 7, page 15 with colors matching their loss trajectories.

We have also added Figure 5 in 3.3.2, page 7 over CIFAR-10. Panel (a) compares reconstruction log-loss with the TSC score, illustrating how TSC recovers high-loss members that an absolute-loss threshold misses. Panel (b) evaluates detection across difficulty quintiles, using average reconstruction loss across noise levels as a proxy for model-relative image difficulty. For each method, a single global threshold targeting 1% FPR is applied unchanged across all quintiles. Baseline TPR decreases as difficulty increases, reaching at most 5.57% in the hardest quintile, whereas TSC’s TPR increases from 52.5% in the easiest quintile to 68.0% in the hardest. These results support TSC’s ability to detect members whose high reconstruction loss obscures their membership signal.

  Once again, thank you for the time and effort you have devoted to reviewing our paper. Your questions have helped us improve the work, and we hope the additional results and clarifications address your concerns.

Thank you for the additional experiments and clarifications. The new results, particularly the difficulty-stratified analysis in Figure 5, address my concerns and provide convincing evidence for TSC’s behavior on harder-to-reconstruct members. I will maintain my original score. I hope future work will explore black-box settings and develop more practical RMC approaches.

Comments on the paper

1 comment

Dear Reviewers,

Thanks again for your time and feedback. The revised manuscript is now available and includes the additional experiments and clarifications discussed in our rebuttal.

When you have had a chance to review the changes, we would appreciate a brief follow-up letting us know whether they address your concerns. If any questions remain, we are more than happy to answer them while the discussion is open.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.