- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LoFIT, a membership inference attack for T2I diffusion models that exploits differences in conditional denoising between training members and non-members. The method measures the signed alignment between the denoising residual under the original caption and the prediction shift induced by alternative captions. The authors hypothesize that member samples exhibit less local denoising improvement than non-members. LoFIT aggregates these alignment scores across multiple text conditions and diffusion states without needing model gradients.
Critical strengths
-
LoFIT is a straightforward interpretation of membership inference through directional changes in denoising loss. It requires only forward denoiser evaluations, without gradients or additional classifier training.
-
LoFIT achieves good improvements over existing approaches: CLiD, SD-MIA, and PFAMI, particularly in low-FPR membership detection on fine-tuned diffusion models. The results are consistent across multiple datasets and Stable Diffusion backbones. The paper also includes controlled comparisons using identical reference predictions and diffusion states.
Critical weaknesses
-
The observation that membership information can be extracted from differences in conditional denoising behavior has already been explored by CLiD and, through generation-based sensitivity in SD-MIA. LoFIT is the normalized first-order component of CLiD's conditional loss discrepancy, as dervied in Section 5.3. The main contribution is therefore a modification of an existing membership statistic rather than a new mechanism. The novelty appears relatively incremental.
-
All evaluated models are in the Stable Diffusion family and use UNet-based denoisers with epsilon-prediction objectives. Modern T2I models use DiTs and flow-matching models, such as SD3 and FLUX. LoFIT's effectiveness across different architectures and prediction objectives is not explored in the paper. This is needed because the proposed membership signal is based on an empirical hypothesis about local conditional denoising fit rather than a theoretically property of diffusion/flow models.
-
LoFIT achieves amazing membership discrimination on fine-tuned models (Table 1), but performance drops considerably on pretrained SD v1.5 (Table 2), with TPR@5%FPR below SD-MIA. Moreover, the fine-tuning settings are designed such that it exposes training examples multiple times during training. This raises questions about whether the strong membership signal is due to extensive fine-tuning and overfitting rather than LoFIT's discrimination. The relatively weak performance on pre-trained models limits the parctical utility of LoFIT.
Questions for authors
-
Can the authors clarify whether the LoFIT hypothesis is expected to hold for DiT-based and flow-matching models?
-
How sensitive is LoFIT to the number of training epochs and the amount of fine-tuning? Section 5.1 shows different training checkpoints, but it would be useful to see whether the strong results survive under fewer repeated exposures of each sample.
-
The experiments assume access to the original training captions, and Appendix C.3 shows a significant performance drop when using generated captions. Could the authors discuss how this requirement affects LoFIT's real-world MIA utility?
5 comments
W2 and Q1. DiT architectures, flow matching, and video generation
TL;DR: LoFIT improves detection on transformer-based CogVideoX and flow-matching Wan, extending beyond UNet denoisers.
The same first-order score applies to noise and velocity prediction. We evaluate CogVideoX-2b [2] and Wan2.1 [3], both transformer models. For Wan, we replace the noise target with the flow-matching velocity target and use the same normalized residual interaction.
Each video's training caption and empty reference share noise at 12 fixed levels, costing 24 condition evaluations. CLiD_th uses .
| Model | Dataset | Training updates | Members | Non-members | CLiD_th AUROC | CLiD_th TPR@5%FPR | LoFIT AUROC | LoFIT TPR@5%FPR |
|---|---|---|---|---|---|---|---|---|
| CogVideoX-2b | Anime-Shooter | 200 | 500 | 4,000 | 0.870 | 0.530 | 0.948 | 0.692 |
| CogVideoX-2b | OpenVid | 4,000 | 1,000 | 4,000 | 0.594 | 0.114 | 0.656 | 0.116 |
| Wan2.1 | Anime-Shooter | 200 | 500 | 3,998 | 0.564 | 0.074 | 0.610 | 0.112 |
| Wan2.1 | OpenVid | 7,000 | 1,000 | 4,000 | 0.547 | 0.072 | 0.604 | 0.105 |
LoFIT improves both metrics over CLiD_th across these settings. We will add these results to the main experimental summary and document the noise/velocity formulations and full training settings in the revised appendix.
W3. Extensive fine-tuning and weaker pretrained discrimination
TL;DR: LoFIT's AUROC advantage appears before extensive fine-tuning, and additional sampling improves pretrained-model detection without further training.
The advantage is present within the first few training epochs. The three-seed COCO/Pokemon experiments in Q2 show earlier AUROC separation for LoFIT than CLiD_th under identical predictions. Strong low-FPR detection develops with greater exposure.
The pretrained result improves by increasing attack sampling alone. On the unchanged pretrained model, AUROC/TPR@5%FPR rises from 0.602/0.101 to 0.635/0.114, versus SD-MIA's 0.571/0.112. The new run uses condition evaluations, compared with 4,800 for SD-MIA. We will report these results separately from the fine-tuning study in the revision.
Q2. Sensitivity to training exposure
TL;DR: Figure 3 shows LoFIT improving earlier than the baselines, and new 1–10-epoch experiments establish an early AUROC advantage.
LoFIT reaches high discrimination earliest among the baselines in Figure 3. On Flickr30k, it reaches AUROC 0.994 at 40k updates, while official CLiD_th continues improving through 80k.
We additionally evaluate much lower exposure, starting at one epoch. Both attacks share eleven references (five combined edits, five random captions, and empty) and 48 timestep/noise states, costing 576 condition evaluations. CLiD_th's alpha is calibrated separately at each checkpoint. COCO/Pokemon use training seeds 1/2/3 and test sets of 2,000/2,000 and 333/334; Flickr30k uses one run and 8,000/8,000. AUROC is mean ± SD where multiple seeds are available; TPR is the mean.
| Dataset | Epochs (updates) | LoFIT AUROC | CLiD_th AUROC | LoFIT TPR@1%FPR | CLiD_th TPR@1%FPR |
|---|---|---|---|---|---|
| COCO | 1 (10) | 0.511 ± 0.001 | 0.497 ± 0.000 | 0.010 | 0.011 |
| COCO | 2 (20) | 0.522 ± 0.001 | 0.497 ± 0.000 | 0.011 | 0.011 |
| COCO | 5 (50) | 0.550 ± 0.001 | 0.510 ± 0.021 | 0.015 | 0.010 |
| COCO | 10 (100) | 0.590 ± 0.001 | 0.552 ± 0.002 | 0.018 | 0.009 |
| Pokemon | 1 (2) | 0.496 ± 0.003 | 0.452 ± 0.002 | 0.012 | 0.010 |
| Pokemon | 2 (4) | 0.517 ± 0.002 | 0.459 ± 0.002 | 0.011 | 0.015 |
| Pokemon | 5 (10) | 0.549 ± 0.004 | 0.490 ± 0.001 | 0.004 | 0.014 |
| Pokemon | 10 (20) | 0.575 ± 0.003 | 0.515 ± 0.004 | 0.009 | 0.012 |
| Flickr30k | 1 (625) | 0.524 | 0.516 | 0.016 | 0.012 |
| Flickr30k | 2 (1,250) | 0.550 | 0.531 | 0.017 | 0.012 |
| Flickr30k | 5 (3,125) | 0.614 | 0.566 | 0.028 | 0.015 |
The early AUROC advantage precedes strong low-FPR detection. At 100 epochs, LoFIT's TPR@1%FPR reaches 0.604/0.307 on COCO/Pokemon, versus 0.087/0.074 for CLiD_th. We will add the full exposure study, including empty-reference controls and training settings, to the revision.
Q3. Practical effect of missing original captions
TL;DR: Generated captions enable image-only LoFIT audits, and LoFIT further improves MoFit using the same optimized conditions.
LoFIT retains an advantage without the original training caption. In the manuscript's BLIP-2 comparison, AUROC is 0.641/0.669/0.621 on Flickr30k/COCO/Pokemon, versus 0.603/0.650/0.576 for official CLiD_th with oracle alpha. Gaussian embedding probes raise it to 0.842/0.904/0.802 on these fine-tuned models; on pretrained LAION-MI, caption references give 0.546 versus 0.532 for Gaussian probes.
LoFIT also improves MoFit, an existing caption-free attack. We keep MoFit's optimized conditions fixed and replace only its loss-gap terms [4]. Both methods fit their scaler and mixing weight on the same calibration split. Scoring costs 15 forwards after MoFit's 1,300/1,200 gradient-based optimization passes per COCO/Pokemon image.
| Method | COCO AUROC [95% CI] | COCO TPR@1%FPR [95% CI] | Pokemon AUROC [95% CI] | Pokemon TPR@1%FPR [95% CI] |
|---|---|---|---|---|
| CLiD_th, BLIP-2, published protocol | 0.648 [0.631, 0.665] | 0.041 [0.028, 0.060] | 0.571 [0.527, 0.612] | 0.045 [0.015, 0.075] |
| LoFIT, BLIP-2, no MoFit optimization | 0.757 [0.741, 0.772] | 0.098 [0.066, 0.125] | 0.594 [0.549, 0.635] | 0.033 [0.003, 0.093] |
| MoFit | 0.677 [0.660, 0.692] | 0.062 [0.047, 0.089] | 0.599 [0.556, 0.641] | 0.036 [0.015, 0.075] |
| MoFit + LoFIT | 0.815 [0.802, 0.827] | 0.140 [0.104, 0.177] | 0.692 [0.653, 0.731] | 0.057 [0.009, 0.102] |
The improvement on unchanged conditions shows that LoFIT complements caption-free condition construction. We will distinguish supplied-caption and image-only audits in the revised abstract and introduction and add the full MoFit comparison and settings.
References
[1] Antoni Kowalczuk et al. Privacy Attacks on Image AutoRegressive Models. ICML, 2025.
[2] Zhuoyi Yang et al. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. ICLR, 2025.
[3] Team Wan et al. Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314, 2025.
[4] Joonsung Jeon et al. No Caption, No Problem: Caption-Free Membership Inference via Model-Fitted Embeddings. ICLR, 2026.
[5] Shengfang Zhai et al. Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy. NeurIPS, 2024.
[6] Tao Qi et al. Black-box Membership Inference Attacks on the Pre-training Data of Image-generation Models. CVPR, 2026.
We thank the reviewer for your detailed feedback. We clarify the method’s novelty and provide new evidence across model architectures, training exposures, and caption-free settings.
W1. Novelty relative to conditional-loss and sensitivity attacks
The current manuscript does not sufficiently explain the motivation and significance of our method. We will add the following explanation to the revision.
TL;DR: CLiD's gap measures a loss change, the normalized gap an average rate of change, and LoFIT an instantaneous rate. This local-fit formulation improves detection and extends to autoregressive cross-entropy.
Local conditional optimality motivates examining the loss response to conditioning changes. Let L(c)=ℓ(f(c),y), where f(c) is the prediction under continuous condition representation c and y is the fixed target.
At a smooth interior conditional optimum, ∇cL(c0)=0. We hypothesize that training pairs exhibit better local fit and less potential for loss reduction through conditional adjustments. However, a small condition derivative can also arise because the prediction barely changes. We separate these effects by normalizing the directional loss response by the corresponding prediction change:
∣∇cf(c0)Δc∣2∇cL(c0)⊤Δc=∣∇cf(c0)Δc∣2∇fℓ(f0,y)⊤∇cf(c0)Δc≈G.
where Δc=cj−c0, and ∇cf denotes the prediction Jacobian with respect to conditioning. This approximation follows from the chain rule.
With f0=f(c0) and nonzero reference shift δ=f(cj)−f0≈∇cf(c0)Δc, the normalized response is
G=⟨∇fℓ(f0,y),∣δ∣2δ⟩=dsdℓ!(f0+s∣δ∣2δ,y)s=0.
where s is output distance. For finite references, G remains the exact instantaneous loss rate along the observed prediction shift.
LoFIT separates the instantaneous response from the finite loss changes measured by CLiD. CLiD's discrepancy term measures the endpoint loss change ΔL [5]. Dividing by ∣δ∣2 gives the average rate; LoFIT measures the instantaneous rate at the original prediction. For denoising residual r=f0−y,
∣δ∣2ΔL=G+∣δ∣2,G=2v,v=∣δ∣2⟨r,δ⟩.
The average rate includes the effect of finite displacement, so an endpoint loss increase can coexist with an initial decrease. SD-MIA examines generation stability under textual perturbations [6]; LoFIT directly measures whether the induced direction improves the queried sample's training loss.
On identical Flickr30k predictions, unnormalized first-order scoring reaches AUROC 0.909, compared with 0.906 for normalization alone and 0.813 for the raw gap; combining both reaches 0.935. Over the normalized gap, LoFIT improves TPR@1%FPR by 12.0/9.8/14.5 percentage points on Flickr30k/COCO/Pokemon without extra queries.
The same formulation extends to autoregressive cross-entropy. The cross-entropy gradient p−ey replaces the denoising-loss gradient in G, where p is the original token distribution and ey the observed token's one-hot vector. We keep the image-token prefix and target fixed and change the caption. On LlamaGen GPT-XL trained for 10 epochs on 2,500 COCO images, we test 2,000 members and 2,000 non-members at the same twelve-forward budget:
Kowalczuk's statistic [1] is selected on calibration data and fixed across checkpoints; the CLiD_th mixture is also calibrated. The gains show that the scoring contribution extends beyond denoising. We will add the full Jacobian connection, cross-entropy decomposition, AR implementation, and complete comparisons to the revised appendix.