- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors analyze memorization in LoRA finetuning of pretrained diffusion models and provide an SVD-based mitigation method.
Specifically, they apply SVD to the LoRA updates, and find that the singular vectors that do not align with the top singular subspaces of the pretrained weights are related to memorization. So they propose to prune or reweight such components and find that doing so reduces the SSCD similarity between generated images and the training images. To validate their method, the authors conduct experiments on different architectures (SD1.5, SDXL, and FLUX1.) and different datasets (PokenMon, CelebA, and LAION-ART).
Critical strengths
- The authors study a practical and meaningful setting, that is the memorization in the LoRA finetuning of pretrained diffusion models, where the downstream data can be scarce and memorization can be severe.
- The method is stated clearly and compared against a reasonably wide range of other methods.
Critical weaknesses
- The method and observations lack justification, and I do not intuitively get why alignment with the top singular subspaces with the pretrained weights matters for memorization.
- The theory is largely decorative and not very relevant, and the core ideas that need to be justified (alignment generalization) are directly used as an assumption.
- The evaluation is not entirely reasonable, where the major improvement is reducing the SSCD (while a reasonably high SSCD is partly expected for getting closer to the finetuning distribution). The other two metrics (CLIP and aesthetic score) do not directly reflect the effectiveness of finetuning, which is aimed at generating a specific distribution of downstream images rather than improving generic CLIP/aesthetic scores. Also, CLIP score is quite old, and the authors may test their method on more modern metrics like GenEval.
Questions for authors
- For Fig.2, it would be better to include some intuitive examples as in Fig.1.
- From Fig.3, it seems that real-world LoRA without mitigation is not memorizing? And also BAF seems not substantially better than other methods like RTA.