- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper centers around memorization in diffusion models, but rather than the traditional lens of viewing it w.r.t its notion and ability for extraction of training data, the paper focuses on a different setting. This setting examines generated images from a model that evolves, via fine-tuning of the text condition, throughout this process. The paper discusses the different stages it defines during that process, where the memory of the model passes through the ‘abstraction’ regime, until reaching the ‘locking’ phase which generates the input image verbatim.
Critical strengths
- The overall concept of the paper is easy to understand and follow.
- The idea of changing the paradigm of memorization from observation at the static model to some process of optimization on its condition is novel and interesting.
- The notion of semantic convergence before the locking stage is demonstrated effectively as can be seen in Fig. 1b.
Critical weaknesses
- The authors discuss memorization and claim to reframe it differently. This new framing is not well justified. It is done using an optimized condition and examines the output along this process, but it does not establish how this reflects memorization, rather than a characteristic of the condition optimization. While the paper demonstrates the results of abstractions along this process, I believe this is a known phenomenon, both from the fact that gradient descent fits low frequency and high energy first [1], along with methods that showed that for personalization methods this transition happens explicitly [2].
- The process itself (Eq. 2) resembles the personalization methods as mentioned in the paper (e.g. Textual Inversion [3]), and it is not grounded why this process with certain 'memories' is better than performing the same with such an approach.
- The structure of the paper is not well written, I think the lack of related works in the main paper affects readability, along some preliminary background on diffusion model before the definitions of the training objective (Eq.4).
- Mathematical framework: section 4 provides a mathematical model that is supposed to ground the idea of the abstraction regime emergence. However, these dynamics leave us with the residual and susceptibility (L216-221), and both are general and have an extra degree of freedom which does not help to establish the notion of the abstraction regime. The paper doesn’t specify or measure the covariance to support this. So, for example, even if hypothetically the abstraction regime would not exist, we would still have this separation of features with larger and smaller eigenvalues, and this mathematical setting can be used to explain the exact opposite ordering (surface before semantics).
[1] On the Spectral Bias of Neural Networks, Rahaman Et. Al., ICML 2019
[2] Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics, Voronov Et. Al., Neurips 2023
[3] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, Gal Et. Al., ICLR 2023
Questions for authors
- I think the core claim of the paper is confusing, as formulating memorizing in a different way is important, but the actual current formulation lacks a lot of information that would be important for such a formulation. For a memorization analysis, I would expect an analysis that can differentiate different populations: what happens when the analysis is done on something that is actually out of distribution (not just images that model did not generate as the authors name it, but actual images the model finds hard to produce)? Within the generated images, are there different behaviors according to different prompts? What about training data vs. non training data (the “traditional” memorization)?
- As this sits closely to the personalization methods, can this analysis help improve personalization methods, e.g. when the desired concept is more/less abstract?
- Regarding W2: What is the difference of using the learnable vectors rather than using known methods like textual inversion? Does the regime of abstraction emerge also in this scenario? If this is the case, what is the contribution of this method using the learnable vectors?
4 comments
Rebuttal (2/2)
Answers to questions
Q1: We partially address this question in the opening remark of our rebuttal (1/2). On the experiments on strongly out-of-distribution targets: we are running new experiments with purely geometrical targets, pixelated targets, and blurry real images, instances which we don’t expect SD2.1 to be good at reproducing. It needs to be noted that the SD2.1 VAE can become a bottleneck in very out-of-distribution images, since its encoding/decoding may become lossy. Our preliminary results show that the abstraction window can shrink and even disappear if the internal dictionary of features of SD2.1 does not cover the necessary features for quickly acquiring the target. This gives further confirmation that abstraction needs the target to excite highly susceptible, semantic channels.
Q2: We thank the reviewer for the interesting suggestion. Except for some preliminary results (see also response to reviewer 668R), we have not yet devoted our study to finding direct applications of this analysis. It is plausible that the learned conditioning vector can be manipulated to deploy the wanted concept/subject in different settings, similarly to what is done in Textual Inversion. Then, the shift of the conditioning’s nature from semantic to positional can explain some of the performance deterioration in late training, as observed in [2].
Q3: We chose our conditioning scheme to be as general as possible: our setup could be extended to any model that can be conditioned through attention, via direct learning in KV space (e.g. in LLMs). In fact, we tested multiple conditioning methods: training in token space (as in Textual Inversion), training part of the empty caption conditioning, training a LoRA for the weights of the network (Fig.7 in E.3). Each one of those variations still produces the abstraction phase, which is the real object of our study.
[1] On the Spectral Bias of Neural Networks, Rahaman Et. Al., ICML 2019
[2] Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics, Voronov Et. Al., Neurips 2023
[3] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion, Gal Et. Al., ICLR 2023
We hope these clarifications address the reviewer's questions, and we would be glad to provide any further detail during the discussion.
Thank you for your elaborated response.
Your clarification regarding your connection to memorization is important, and I think it helped me understand that in the abstract, the text should say so explicitly. Inverting the ‘traditional’ memorization as a failure mode and asking how a model recalls a single data point plants the idea and expectation that the reader will receive information on how that specific point is memorized. However, this is not the focus of the paper, and I think this is very important for the clarity of the paper’s main idea.
Regarding the gradient descent bias, the low frequency is not in the pixel space. It is in the dominant directions of the loss being optimized, and if the authors would analyze the low frequency of the cost I believe they will experience the same phenomena. I still believe that this abstraction for fine-tuning a specific image is a result that alone, without any valid takeaways from it, is not surprising. I understand the scientific goal, but I believe its execution needs further analysis on how this can give us new information, whether for specific data memorization, “hardness” of wiping data out as done in concept erasure, or other surprising usages.
Regarding the mathematical framework, I still don’t see how the framework explains the abstraction regime emergence. Any ordering is possible depending on the prior, so the framework does not predict semantic-first behavior; this is just observed in SD2.1. The presented theory does not distinguish the high variance features, and the fact that this aligns with SD2.1 does not mean the framework is true. It needs further analysis to support some separation.
Overall, I think the idea of the paper is interesting, but it needs additional supporting analysis, more than what a revision should contain. Therefore, I’m keeping my score.
Response
We thank the reviewer for their continued engagement. However, we believe that some of the remaining criticisms rest on an interpretation of our contribution that differs from what we actually claim.
On memorization, our goal is precisely to study how a single new datum is acquired, following which properties become reproducible along the optimization of its learned conditioning signal. We agree that the first two sentences in the abstract lead to a possible misunderstanding; we will change the opening to: "Modern generative models are typically studied for their ability to generalize, while memorization is treated as a failure mode. We invert this perspective and ask how a generative model can be made to recall a single new datum."
On spectral bias, we agree that preferential learning directions are a generic feature of gradient-based optimization, shaped by the loss. In our setting, since there is a single datum, the central question is how the structure already encoded by the prior determines which properties are acquired first. This internal structure is what shapes the fast learning directions and creates a correspondence with semantic modes. Generic optimization bias alone does not account for the additional generative and functional signatures we observe.
On the theory, we explicitly present an interpretative framework, not a derivation of SD2.1's internal dynamics. The theoretical goal is a conditional explanation: identifying properties of a prior under which semantic-first learning can emerge, and deriving consequences that can be investigated experimentally. The fact that different priors can produce different orderings is precisely part of our perspective, rather than a contradiction of it. We acknowledge that direct verification of the proposed susceptibility mechanism remains open, and it will inspire future work.
We understand the reviewer's interest in a more complete mechanistic explanation and concrete applications, but these are extensions beyond the claims of the present paper. We believe the empirical findings and their characterization constitute a scientific contribution in their own right.
Rebuttal (1/2)
We thank the reviewer for their interesting comments. We are glad they found our work easy to read and understand, and the memorization paradigm novel and interesting. The result of a semantic-first ordering is central in our manuscript, and we are happy that the reviewer found it demonstrated in an effective way.
Before addressing all the points, we would like to stress an important dichotomy. Standard memorization in ML: a failure mode where training data are internalized and reproduced by the generative model. Our connotation of memorization: the process of learning a conditioning vector (the memory) that forces a well-trained generative model to reproduce a single new datum (e.g., think of a human adult memorizing a poem). In particular, we pay specific attention to the dynamics of this memorization process, and we show how it unveils the internal relevance structure of the generative prior. Our framework is not meant to be informative about the standard memorisation of pre-training data.
Weaknesses
Relation to previous work and novelty
Gradient descent bias for low frequency and high energy [1]: The “semantic” directions preferred during abstraction do not correspond to low-frequency/high-energy details in pixel space: Fig. 1 explicitly shows that fine, high frequency details like the eyes’ shape are fixed faster than the person’s position, their silhouette and background, which are of arguably lower frequency and higher contrast. Abstraction emerges as a byproduct of an optimization process that explicitly requires faithful reproduction of the target image.
Personalization methods [2]: We thank the reviewer for pointing out missing references (added to the revised paper). We agree with the reviewer, multiple personalization methods are based on conditioning, similarly to what is discussed in this paper. Some hints of the phenomenology that we center our work around are indeed present. The main difference lies in the scientific goal: we are not presenting a way of improving personalization in diffusion models or making it more efficient. Our framework builds on similar methods, but is aimed at probing the hidden relevance structure of the prior, how it emerges dynamically during memorization of a single image, and how different stages of this memorization process are compatible with function, i.e., how useful the memory can be beyond the reproduction task. The question is thus more general, and extends beyond SD2.1 and diffusion models, and even to different data modalities. While the late stage of training in [2] shows an undesired degradation of personalization performance, in our framework verbatim reconstruction is the only explicit training goal. Yet, the dynamics naturally pass through abstraction, and we shift the perspective to noticing how this is non-trivial and may tell us something about human memory. Moreover, this research direction is closely aligned with current results in computational neuroscience, focused on the hippocampus-neocortex interaction and on episodic memory (see Extended Related Works).
Improving clarity
We agree with the reviewer that some background knowledge is not fully covered in the current version; we will use the additional page to add more references in the introduction and to better introduce the diffusion framework.
Mathematical framework
We thank the reviewer for pointing to this limitation, already partially acknowledged in the current Limitation (iii). We agree we are not able to map back the ingredients of our framework to the experimental setup with SD2.1, given that we have no direct access to the internal features of this model. However, the simplicity of the framework allows the identification of a clear condition for the emergence of semantic-first abstraction: high susceptibility of the corresponding semantic modes. This condition is not necessarily realised in all models. In fact, it is completely possible that a different model, trained on different data and with a very different prior, will not exhibit the same semantic abstraction. For example, if one considers a model that was never trained on human faces, it should be impossible for it to enter an “identity”-preserving abstraction phase, since it lacks the necessary feature modes (see also reply to Q1). However, in our experiments, we do see a semantic-first phase with SD2.1-compatible data, and we also find a correspondence with two qualitative predictions obtained from the mathematical framework (see Sec. 5).