- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes an inference-time method for controlling memorization in flow-based generative models using the Moreau envelope of a nearest-neighbor memorization score. The resulting gradient field is added to the model’s velocity field, with positive strength pushing samples away from memorized training points and negative strength attracting them toward selected anchors. The authors provide an optimal-transport interpretation through Wasserstein gradient flows and JKO steps, and evaluate the method on a memorizing RetinaMNIST flow model, a synthetic 2D setting, and a flow-based language model. The method reduces measured memorization on RetinaMNIST and can also steer language generation toward chosen word anchors without retraining.
Critical strengths
-
The paper presents a mathematically principled inference-time intervention that does not require retraining the generative model.
-
The paper is a joy to read. It distinguishes itself from many contemporary works (that unfortunately tend to rely more heavily on AI tools) through clarity, an enjoyable edginess, and rigor.
-
The connection between the Moreau envelope, proximal mappings, optimal transport, and Wasserstein gradient flows is elegant and gives the perturbation a clear theoretical interpretation rather than treating it as an ad hoc steering heuristic.
-
The method provides a single signed control parameter that can both mitigate memorization and enforce attraction toward selected anchors.
-
The authors include meaningful ablations against raw ∇S and nearest-neighbor repulsion.
-
The evaluation considers memorization in multiple spaces, including latent and raw pixel space, and supplements these with perceptual diagnostics, which helps guard against improvements that merely optimize the intervention space.
Critical weaknesses
-
The empirical evaluation is quite limited in scale. The main image experiment uses only 1,000 low-resolution RetinaMNIST images and a deliberately memorizing model, with no evaluation on a realistic, larger-scale generative model or dataset.
-
The practical scalability of the method is unclear. Computing the perturbation requires access to the anchor/training set during inference, nearest-neighbor queries, and repeated inner iterations of the proximal solver. The paper does not report wall-clock overhead or analyze how this would scale to (potentially) millions of training examples.
-
The memorization metric is sensitive to the representation space. Latent-space and pixel-space memorization rates differ substantially, and Euclidean distance in either space is only an indirect proxy for perceptual copying. It would be interesting to see how humans perceive the degree of memorization.
-
The choice to perform the intervention in a 200-dimensional PCA subspace of the 256-dimensional VAE latent space is insufficiently justified.
-
The language experiment demonstrates attraction toward selected anchor words rather than mitigation or enforcement of actual training-data memorization. It therefore supports the generality of the geometric steering mechanism, but provides limited evidence for the broader memorization claims across modalities.
-
There also appears to be a sign inconsistency in the theoretical exposition: the text states that descending the Moreau functional moves away from the memorized distribution, whereas the equations imply that gradient descent moves toward the low-S, more memorized proximal state.
Questions for authors
-
How does the method scale to realistic generative-model settings with substantially more training examples? In particular, what is the wall-clock overhead of nearest-neighbor queries and the repeated inner proximal iterations during sampling?
-
On the same tangent of scale / realistic deployment, the language experiment appears to demonstrate generic attraction toward selected anchor words. However, it would be interesting to see the effect of the steering on larger language models. Could the method actually steer a model away from a memorized sentence? Can the method be evaluated on a more realistic and diverse image-generation setting, such as COCO or ImageNet-scale data, rather than only the deliberately memorizing 1,000-image RetinaMNIST setup? As this might be out of scope for a rebuttal, a discussion of this is fully sufficient (any extra experiments would still be appreciated).
-
Why is the perturbation applied in the first 200 PCA directions of the 256-dimensional VAE latent space?
Final comment: I like the paper, the way it is written, and the ideas it presents. I consider it valuable to the community. My main concern is its practical relevance. Memorization appears to be more of an issue in large-scale deployed systems (e.g., copyright infringement). While the theory underlying the presented approach appears solid, it is not uncommon that scaling such methods up leads to different computational dynamics that, in turn, cause unexpected output behavior. If there is anything the authors can present or argue to mitigate this concern, I am very much leaning towards clear acceptance.
3 comments
Nice work.
I appreciate the responses and the effort. My concerns have been eased, and I will raise my score to 4.
Thank you for your quick response and for raising your score, we appreciate your constructive feedback.
Response to Reviewer L4CU
We thank the reviewer for the very positive assessment and for the precise concerns. We start with the theoretical inconsistency, then address practical relevance, which we understand to be the main concern.
Sign inconsistency (W6). The reviewer is right. Since MλS is small near memorized points, the field +γ∇MλS with γ>0 ascends Eλ: mitigation is ascent and enforcement is descent. The sentence after Prop. 3 is corrected, with a line relating it to the sign convention of Eq. 2.
Practical relevance and scaling (Final comment, W1, W2, Q1). Memorization in large deployed models is concentrated on a small subset of the training data, typically heavily duplicated images (Somepalli et al., 2023; Carlini et al., 2023). This is exactly where our construction applies, because the anchor set D need not be the training set: it can contain only the memorized, copyrighted or sensitive samples, or a neighborhood of them. Appendix E demonstrates this on a 32-point zone: repulsion lowers in-zone memorization to 1% while leaving samples elsewhere essentially undisplaced. The method thus has two regimes: in low-data settings (our RetinaMNIST study), memorization is widespread and D is the whole small dataset; at scale, memorization is localized and D is the small set at risk. In both, the cost depends on ∣D∣, not on the size of the training set.
On cost, General response §2 gives measured wall-clock times and shows that the per-step cost reduces to one kNN query over D.
Steering away from memorized sentences (W5, Q2). We agree: the anchors in Sec. 4.2 are not training data, so the experiment demonstrates the steering mechanism in a language model rather than mitigation of training-data memorization. Flow-based language models appeared this year, and their memorization has not been characterized yet. Identifying memorized outputs, e.g. through an extraction study against the training corpus, is a prerequisite for a mitigation experiment and is beyond the scope of this work.
Larger and more diverse image datasets (Q2). Computation is not the limiting factor. With the candidate cache (General response §2), each Euler step needs one kNN query over D:
The open question is rather which representation makes S meaningful for diverse, multi-class data: Stable Diffusion latents with PCA as here, or features such as DINOv2 or CLIP. This calls for a dedicated study, which we mention as future work in the new limitations paragraph; the scaling discussion is in Appendix D.2.
Sensitivity to the representation space (W3). Absolute memorization rates depend on the space, because S depends on its geometry (93.0% latent vs. 45.3% pixel at baseline). This is why we report both, together with a DINOv2 copy-detection score (Fig. 8, center); all three decrease together as γ grows. We agree a human perceptual study would be valuable; it is beyond our scope, and we mention it in the limitations.
Why 200 PCA directions (W4, Q3). The leading principal components carry the coarse, visually dominant structure of retinal images (overall color, shape, major features). Perturbing only a few of them alters images drastically; we increased the number of directions until changes became perceptually fine-grained. The 200 directions explain 99.8% of the latent variance, and memorization measured in the PCA subspace tracks the full latent space closely (Fig. 8, left), so the projection does not distort the effect.