- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies memorization in multi-modal diffusion models that use both reference images and text prompts. The authors propose a memorization score based on object masking and reconstruction similarity and find that multi-modal encoders contribute more to memorization than diffusion backbones. They also investigate privacy leakage through image reconstruction and compare mitigation strategies applied to the encoder and diffusion backbone.
Critical strengths
The paper addresses an important problem in multi-modal diffusion models. Existing memorization studies mainly focus on text-conditioned generation, while this work considers the additional role of reference images.
The experiments cover two different architectures, BLIP-Diffusion and ELITE. The authors compare training and non-training samples, validate their metric against a leave-one-out approach, and reverse the training datasets to examine potential distribution effects. These experiments provide useful evidence for the main observations.
Critical weaknesses
The main weakness of the paper is that the proposed memorization metric is not fully validated. Object reconstruction may reflect general predictability rather than memorization. Although Table 1 reports over 91% overlap with the LOO metric, Kendall's Tau is relatively low (0.186–0.266). Considering that the LOO metric measures representation stability rather than image reconstruction, the claim of equivalence seems too strong.
Also, the contribution of the encoder is not clear. Table 2 shows that encoder fine-tuning reduces memorization scores on both datasets. Random initialization also removes useful visual information, not just memorization. These results show that the encoder is important for reconstruction, but do not clearly demonstrate that it is the main source of memorization.
Finally, since the encoder-level and diffusion-level mitigation methods use different techniques, it is difficult to determine whether the improvement comes from the intervention location or the mitigation method itself. Without comparisons under the same conditions, it is unclear whether encoder-level mitigation is actually more effective.
Questions for authors
Q1. Could the authors conduct more controlled ablation studies to isolate the encoder's contribution to memorization? For example, selectively modifying the encoder and diffusion backbone while controlling their general reconstruction capabilities would provide stronger evidence for the main claim.
Q2. Could the authors compare encoder-level and diffusion-level mitigation under more comparable conditions?
Q3. Could the authors provide additional modality ablations for Table 3, including removing target caption information from both the encoder input and diffusion prompt, and measuring similarity between retrieved reference images and target images?
Q4. Could the authors provide a direct computational cost comparison between the proposed metric and the LOO baseline under the same evaluation setting?
2 comments
Image-Retrieved Modality Analysis and Computational Cost Comparison with LOO
Q3: Could the authors provide additional modality ablations for Table 3, including removing target caption information from both the encoder input and diffusion prompt, and measuring similarity between retrieved reference images and target images?
We add extra experiments that retrieve the most related image according to the target image (instead of caption) and do the reconstruction based on these images and blank captions. To be noted that this experiment has nothing to do with the original privacy leakage risk but only used to show the modality importance during reconstruction. The results are shown in table below:
| Setting | BLIP-Diffusion SSIM ↑ | BLIP-Diffusion LPIPS ↓ | BLIP-Diffusion SSCD ↑ | ELITE SSIM ↑ | ELITE LPIPS ↓ | ELITE SSCD ↑ |
|---|---|---|---|---|---|---|
| Most memorized samples (image retrieved by target image + blank caption) | 0.166 ± 0.014 | 0.779 ± 0.028 | 0.099 ± 0.010 | 0.155 ± 0.013 | 0.791 ± 0.025 | 0.091 ± 0.008 |
The results achieve even better results than the original results for Retrieve Image + Blank Caption in table 3. This proves that Visual modality plays a more important role in this reconstruction.
Q4: Could the authors provide a direct computational cost comparison between the proposed metric and the LOO baseline under the same evaluation setting?
We report the evaluation for 1M samples and training cost for these 2 methods (on BLIP-Diffusion) in table below:
| Extra Training Time | Evaluation Time | Total overhead | |
|---|---|---|---|
| Our method | 0 | 154.1 h | 154.1 h |
| LOO | 118.6 h | 147.9 h | 266.5 h |
The training is done on 4 H100 GPU and evaluation is done on 4 A100 GPU. Therefore, the actual additional overhead incurred by LOO is far greater than that of our proposed method.
LOO Consistency, Encoder Contribution, and Mitigation Comparison
We agree that “equivalent” overstates what Table 1 shows. We revise it to “consistent with the LOO metric, especially on the most memorized samples.” We clarify two points. (1) Overlap should be read against chance: with ∣DC∣ = 100K, random top-K overlap is 0.1% / 1% / 5% for K = 100 / 1000 / 5000, versus 91.0–94.2% observed. (2) We now additionally report Kendall’s test over all Candidate sets with τ = 0.684 / p = 2.21e-188.
The main idea of table 2 is not that the memorization level has decreased, but rather that the gap in memorization levels between CC3M and COCO-stuff has been narrowed. We observed that fine-tuning the encoder using the training samples from the diffusion backbone (COCO-Stuff) significantly narrowed the memory gap between CC3M and COCO-Stuff (second experiments in table 2). This experiment acts as a more extreme test based on that finding: we discovered that even when the diffusion backbone has memorization of the COCO-Stuff training samples, the memorization levels for CC3M and COCO-Stuff become the same once the encoder’s memory is completely erased. Therefore, we conclude that the memorization within the encoder plays the primary role.
We agree that currently there is currently no direct way to quantitatively compare their mitigation strengths because these two approaches operate on different model architectures and pre-training paradigms. To make this comparison as fair as possible, we have applied the most effective mitigation method described in the original paper for each approach.
If the encoder is further fine-tuned, to preserve the same generalization ability of model (performance), the diffusion-backbone need to be fine-tuned as well. This is because the multi-modal encoder and the diffusion backbone within the multi-modal diffusion model are trained separately, in two distinct stages. Therefore, to evaluate the whole model’s memorization level, it’s nearly impossible to isolate any components without changing model itselves generalization ability (performance).
Since these two approaches operate on different model architectures and pre-training paradigms, there is currently no direct way to quantitatively compare their respective strengths. To make this comparison as fair as possible, we have applied the most effective mitigation method described in the original paper for each approach.