- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
L017: By testing mainstream sampling acceleration methods including SDE and ODE solvers → maybe its better to name them directly here, as all the methods including the non-accelerated techniques are kind of a SDE/ODE sovlers.
The paper’s text and flow looks to me like a memo, the simplest example is authors try to randomly redefine the acronyms, e.g., DM all over the paper.
Figure.1 maybe the trained-DM ? as the pre-trained makes the authors biased that it have been trained on something else that celeb or ffhq?
Contribution.1 Isn’t just similar to use the some pretrained model like dinov2 or something to measure the similarity? which is already quite established. Have the authors tried to test some generators that were trained on multiple domains? e.g., a sreamlined generator that is trained on multiple domains, now try to extract its training data? I think this is key important experiment that authors need to add, otherwise its too narrow the scope, which makes it a workshop paper might be a better fit.
Its good that authors considered both latent and pixel space diffusion models
Is FID a good metric for the reconstruction quality here that authors used for the quantifying the quality? maybe if you are using the a insightface model its better to use that to quantify?
Table.11, Tabl.12 and Table.3 so comparing the first line and the last line for the all the stages of your method and without the authors methods we gain ~15 minute saving on 8 hour scale? how siginificant is this results? it is acknowledgeable that the authors put this all the timing of the different stages.
Also, although there is a discussion about the applicability of the conditional generator in the appendix but maybe more experiments are required for this, especially by noting my earlier comment about more streamlined generator consideration.
I consider increasing my score if the additional experiments are provided, especially the multi domain one.
Critical strengths
Please see the summary
Critical weaknesses
Please see the summary
Questions for authors
Please see the summary
2 comments
Rebuttal (2/2)
Pixel-space Diffusion Models
Its good that authors considered both latent and pixel space diffusion models
Thank you for recognizing this aspect of the evaluation. Table 2 includes both latent-space diffusion models and a pixel-space DDPM. We will retain these comparisons and state their data and model scope explicitly.
Image Quality Metric
Is FID a good metric for the reconstruction quality here that authors used for the quantifying the quality? maybe if you are using the a insightface model its better to use that to quantify?
The FID here is not used to evaluate the reconstruction fidelity, but to evaluate the distributional discrepancy between generated and reference image sets. For the reconstruction fidelity, InsightFace similarity already measures the correspondence between each generated face and its matched training image.
Efficiency Analysis
Table.11, Tabl.12 and Table.3 so comparing the first line and the last line for the all the stages of your method and without the authors methods we gain ~15 minute saving on 8 hour scale? how siginificant is this results? it is acknowledgeable that the authors put this all the timing of the different stages.
The main reported benefit is the number of distinct matched training entries obtained within approximately the same runtime. We take Table. 3 for example here.
| Method | Distinct matched training entries | Reported total time | Matches/hour |
|---|---|---|---|
| 60-step SDE-DPM-Solver | 128 | 8 h 28 min | 15.12 |
| GFR, combined | 236 | 8 h 12 min | 28.78 |
These reported figures correspond to 16 minutes less runtime, 84.4% more distinct matched entries, and a 1.90× throughput ratio.
[R3] Detecting, Explaining, and Mitigating Memorization in Diffusion Models. ICLR 2024.
Rebuttal (1/2)
Thank you for your comments.
Paper Writing
We will reconsider these naming problem and avoid the ambiguity.
Definition of Extraction
Using pretrained features is a natural approach, and the choice of representation determines which visual content a match preserves. For models trained on faces, our objective is to assess the recovery of facial content while allowing changes in background and illumination. This motivates a representation specifically sensitive to the face.
Our choice followed preliminary comparisons of alternative metrics. When we initially used pixel-space MSE, we observed retrieved pairs with similar black backgrounds and skin tones but visibly different faces. With CLIP and DINO features, we also observed pairs whose similarity was strongly influenced by background and illumination. These observations exposed a mismatch between overall image similarity and the facial content relevant to our task. We additionally observed generated faces resembling training faces under different backgrounds or lighting, which motivates accommodating such variations in the evaluation.
Face-recognition representations provide a well-developed way to emphasize facial identity while reducing sensitivity to these nuisance factors. We therefore use InsightFace features to operationalize extraction in this face-focused setting.
In all, we do not claim the paradigm of using some pretrained model to measure the similarity as one of our contributions. Instead, we provide a more precise way to measure the privacy risk in the context of human faces and we focus on the efficiency problem of extraction attacks, which has been rarely studied.
Experiments on Multiple Domains and conditional generators
The existing non-face experiment in Appendix B evaluates LSUN-Churches using SSCD similarity. Table 6 reports 69 matched training examples in 5.70 hours for SDE-DPM-Solver++ and 195 in 4.97 hours for GFR, corresponding to 12.11 and 39.24 matches/hour. This provides evidence in an additional scene domain. We agree that a generator jointly trained on diverse content would test a broader setting.
We also validate the effectiveness of our method in a conditional diffusion models trained on ImageNet, a multi-domain dataset. Specifically, we use the conditional LDM pretrained on ImageNet as the victim model and evalute the effciency of extraction attack on 10 classes, including insects, birds, flowers and mammals. Note that the extraction is defined by SSCD similarity here following [R3]. We compare GFR with 60-step DPM-Solver and 60-step SDE-DPM-Solver.
GFR extracts about 13.5 more images than SDE-DPM-Sovler, which is also more efficient than its ODE counterpart. Moreover, we find that classes behave quite differently in extraction attacks. For the great grey owl, all three methods are able to extract around 100 images under the 4000 generation budget, while that number for the koala is 0 for all of them. We believe that intra-class diverity is a important factor for memorization and we leave this problem for future research.