acceptodds
Under review as a conference paper at ICLR 2027

Mastering memorization in flow-based generative models via the Moreau Envelope

Abstract

Flow-based models have shown unprecedented generation capabilities across a wide range of modalities, but they suffer from the not fully understood phenomenon of memorization - the verbatim reproduction of data samples despite valid model convergence. This degrades their creativity and applicability in low data regimes as well as their reliability in secure settings. On the other hand, memorization could be the key to a new form of controlled generation, by memorizing features corresponding to specific states to observe, but this has not yet been studied for this purpose. We thus introduce a novel framework to achieve controlled memorization, with a dial allowing model optimization to decide to what extent it should be mitigated or enforced. This is made possible by turning a memorization metric into a Kantorovich potential via the Moreau Envelope, leveraging the proximal of the envelope to build an attractive or repulsive memorization field. We further provide an Optimal Transport interpretation of the proposed method, and derive a proximal gradient descent algorithm for the computation of the proximal during model inference. Moreover, we propose a number of illustrative use cases across various modalities; these range from preventing memorization in low data regime image generation and leverage the perturbation in text domain to steer flow-based language models token generation towards using specific words, showcasing the versatility of the proposed method.

Primary area:
generative models
Keywords:
Generative Modeling, Memorization, Flow-Matching, Flow-based models, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_L4CUOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an inference-time method for controlling memorization in flow-based generative models using the Moreau envelope of a nearest-neighbor memorization score. The resulting gradient field is added to the model’s velocity field, with positive strength pushing samples away from memorized training points and negative strength attracting them toward selected anchors. The authors provide an optimal-transport interpretation through Wasserstein gradient flows and JKO steps, and evaluate the method on a memorizing RetinaMNIST flow model, a synthetic 2D setting, and a flow-based language model. The method reduces measured memorization on RetinaMNIST and can also steer language generation toward chosen word anchors without retraining.

Critical strengths
  • The paper presents a mathematically principled inference-time intervention that does not require retraining the generative model.

  • The paper is a joy to read. It distinguishes itself from many contemporary works (that unfortunately tend to rely more heavily on AI tools) through clarity, an enjoyable edginess, and rigor.

  • The connection between the Moreau envelope, proximal mappings, optimal transport, and Wasserstein gradient flows is elegant and gives the perturbation a clear theoretical interpretation rather than treating it as an ad hoc steering heuristic.

  • The method provides a single signed control parameter that can both mitigate memorization and enforce attraction toward selected anchors.

  • The authors include meaningful ablations against raw ∇S and nearest-neighbor repulsion.

  • The evaluation considers memorization in multiple spaces, including latent and raw pixel space, and supplements these with perceptual diagnostics, which helps guard against improvements that merely optimize the intervention space.

Critical weaknesses
  • The empirical evaluation is quite limited in scale. The main image experiment uses only 1,000 low-resolution RetinaMNIST images and a deliberately memorizing model, with no evaluation on a realistic, larger-scale generative model or dataset.

  • The practical scalability of the method is unclear. Computing the perturbation requires access to the anchor/training set during inference, nearest-neighbor queries, and repeated inner iterations of the proximal solver. The paper does not report wall-clock overhead or analyze how this would scale to (potentially) millions of training examples.

  • The memorization metric is sensitive to the representation space. Latent-space and pixel-space memorization rates differ substantially, and Euclidean distance in either space is only an indirect proxy for perceptual copying. It would be interesting to see how humans perceive the degree of memorization.

  • The choice to perform the intervention in a 200-dimensional PCA subspace of the 256-dimensional VAE latent space is insufficiently justified.

  • The language experiment demonstrates attraction toward selected anchor words rather than mitigation or enforcement of actual training-data memorization. It therefore supports the generality of the geometric steering mechanism, but provides limited evidence for the broader memorization claims across modalities.

  • There also appears to be a sign inconsistency in the theoretical exposition: the text states that descending the Moreau functional moves away from the memorized distribution, whereas the equations imply that gradient descent moves toward the low-S, more memorized proximal state.

Questions for authors
  1. How does the method scale to realistic generative-model settings with substantially more training examples? In particular, what is the wall-clock overhead of nearest-neighbor queries and the repeated inner proximal iterations during sampling?

  2. On the same tangent of scale / realistic deployment, the language experiment appears to demonstrate generic attraction toward selected anchor words. However, it would be interesting to see the effect of the steering on larger language models. Could the method actually steer a model away from a memorized sentence? Can the method be evaluated on a more realistic and diverse image-generation setting, such as COCO or ImageNet-scale data, rather than only the deliberately memorizing 1,000-image RetinaMNIST setup? As this might be out of scope for a rebuttal, a discussion of this is fully sufficient (any extra experiments would still be appreciated).

  3. Why is the perturbation applied in the first 200 PCA directions of the 256-dimensional VAE latent space?


Final comment: I like the paper, the way it is written, and the ideas it presents. I consider it valuable to the community. My main concern is its practical relevance. Memorization appears to be more of an issue in large-scale deployed systems (e.g., copyright infringement). While the theory underlying the presented approach appears solid, it is not uncommon that scaling such methods up leads to different computational dynamics that, in turn, cause unexpected output behavior. If there is anything the authors can present or argue to mitigate this concern, I am very much leaning towards clear acceptance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer L4CU

We thank the reviewer for the very positive assessment and for the precise concerns. We start with the theoretical inconsistency, then address practical relevance, which we understand to be the main concern.

Sign inconsistency (W6). The reviewer is right. Since MλSM_\lambda S is small near memorized points, the field +γ∇MλS+\gamma \nabla M_\lambda S with γ>0\gamma > 0 ascends EλE_\lambda: mitigation is ascent and enforcement is descent. The sentence after Prop. 3 is corrected, with a line relating it to the sign convention of Eq. 2.

Practical relevance and scaling (Final comment, W1, W2, Q1). Memorization in large deployed models is concentrated on a small subset of the training data, typically heavily duplicated images (Somepalli et al., 2023; Carlini et al., 2023). This is exactly where our construction applies, because the anchor set DD need not be the training set: it can contain only the memorized, copyrighted or sensitive samples, or a neighborhood of them. Appendix E demonstrates this on a 32-point zone: repulsion lowers in-zone memorization to 1% while leaving samples elsewhere essentially undisplaced. The method thus has two regimes: in low-data settings (our RetinaMNIST study), memorization is widespread and DD is the whole small dataset; at scale, memorization is localized and DD is the small set at risk. In both, the cost depends on ∣D∣|D|, not on the size of the training set.

On cost, General response §2 gives measured wall-clock times and shows that the per-step cost reduces to one kNN query over DD.

Steering away from memorized sentences (W5, Q2). We agree: the anchors in Sec. 4.2 are not training data, so the experiment demonstrates the steering mechanism in a language model rather than mitigation of training-data memorization. Flow-based language models appeared this year, and their memorization has not been characterized yet. Identifying memorized outputs, e.g. through an extraction study against the training corpus, is a prerequisite for a mitigation experiment and is beyond the scope of this work.

Larger and more diverse image datasets (Q2). Computation is not the limiting factor. With the candidate cache (General response §2), each Euler step needs one kNN query over DD:

  • Memory. 100k anchors at d=4,096d = 4{,}096 (Stable Diffusion latents of 256×256256\times256 images, as for ImageNet-256) take 1.6 GB in fp32, and at d=16,384d = 16{,}384 (512×512512\times512 images, as for COCO) 6.6 GB, which fits on a single GPU.
  • Compute. In brute force, the query for a batch of 64 is one matrix product of 64×105×d64 \times 10^5 \times d multiply-adds, about 2×10112\times10^{11} FLOPs at d=16,384d = 16{,}384. Beyond that, partitioned and quantized indices (FAISS, Johnson et al., 2021; ScaNN with SOAR, Guo et al., 2020; Sun et al., 2023) handle millions to billions of vectors with sublinear queries, and a localized DD keeps the anchor set small in the first place.

The open question is rather which representation makes SS meaningful for diverse, multi-class data: Stable Diffusion latents with PCA as here, or features such as DINOv2 or CLIP. This calls for a dedicated study, which we mention as future work in the new limitations paragraph; the scaling discussion is in Appendix D.2.

Sensitivity to the representation space (W3). Absolute memorization rates depend on the space, because SS depends on its geometry (93.0% latent vs. 45.3% pixel at baseline). This is why we report both, together with a DINOv2 copy-detection score (Fig. 8, center); all three decrease together as γ\gamma grows. We agree a human perceptual study would be valuable; it is beyond our scope, and we mention it in the limitations.

Why 200 PCA directions (W4, Q3). The leading principal components carry the coarse, visually dominant structure of retinal images (overall color, shape, major features). Perturbing only a few of them alters images drastically; we increased the number of directions until changes became perceptually fine-grained. The 200 directions explain 99.8% of the latent variance, and memorization measured in the PCA subspace tracks the full latent space closely (Fig. 8, left), so the projection does not distort the effect.

Reviewer_L4CUOpenReview ↗

Nice work.

I appreciate the responses and the effort. My concerns have been eased, and I will raise my score to 4.

Thank you for your quick response and for raising your score, we appreciate your constructive feedback.

Reviewer_3FmKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduced an inference-time perturbation framework that can mitigate memorization in flow-based generative models. The key idea is to add the gradient of the Moreau envelope of a memorization metric (Eq. 6) to the trained velocity. The proximal operator can be approximated by using a Nesterov-accelerated proximal gradient method. The experiments show that the proposed method is effective in mitigating memorization in low-data regimes.

Critical strengths
  1. The idea of the paper is easy to understand. To mitigate memorization in the low-data regime, the paper proposed adding the gradient of a memorization metric to the trained velocity.
  2. The experimental results demonstrate the ability of the proposed method to mitigate memorization.
Critical weaknesses
  1. I am curious about the additional computational cost of the sampling process with Algorithm 1, compared to the original sampling process with the trained velocity.
  2. There is no discussion or explanation of Figure 1.
Questions for authors
  1. Could the authors present some results on the computational costs of the inference process using the approximation in Algorithm 1?
  2. Could the authors give some intuitive explanation or discussion of Figure 1?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3FmK

We thank the reviewer for the positive evaluation.

Q1 — Computational cost (W1). In our RetinaMNIST experiment, unperturbed sampling takes 1.26 s (128 samples, 100 Euler steps, small UNet), and the perturbation adds 0.67 s with N=10N = 10 inner iterations, the setting used in the paper (1.82 s with N=30N = 30). This added time depends on the anchor set and on NN, not on the generative model: the corresponding ratio (1.5×) is specific to our small UNet and shrinks for larger models. The per-step cost also reduces to a single nearest-neighbor query (General response §2). Convergence against the inner budget is reported in Appendix D.1, and costs in Appendix D.2.

Q2 — Explanation of Figure 1 (W2). Figure 1 illustrates Algorithm 1 in two dimensions. Lines are the Voronoi cells of the dataset anchors; each trace starts at a sample xx and follows the APGM iterates to the proximal point, the most memorized point within a radius set by λ\lambda. The iterates move toward the cell's anchor and stay within its cell. The field ∇MλS(x)=(x−proxλS(x))/λ\nabla M_\lambda S(x) = (x - \mathrm{prox}_{\lambda S}(x))/\lambda points from that proximal point back to xx: following it (γ>0\gamma > 0) moves the sample away from its most memorized neighbor, and following its opposite (γ<0\gamma < 0) moves it toward it. We added this explanation to the caption and to Sec. 3.4.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 3FmK, as the discussion period ends soon, we wanted to check whether our response addressed your questions on computational cost and on Figure 1. We would be happy to clarify anything further.

Thank you for your responses and additional discussion, which address my main concerns. I will maintain my positive evaluation.

Reviewer_YcLzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses memorization control in flow-based generative models, supporting both mitigation and enforcement. It turns a memorization metric into a Kantorovich potential via the Moreau envelope, uses the proximal of the envelope to build an attractive or repulsive memorization field, and derives a proximal gradient (APGM) algorithm to compute the proximal at inference time. The method requires no retraining of pretrained models and is given an Optimal Transport interpretation. The authors illustrate it on low-data-regime image generation and on steering token generation toward specific words in the text domain.

Critical strengths
  1. Meaningful and well-motivated problem. Memorization control for flow-based models is practically important, and the framing that memorization can be either mitigated or deliberately enforced is a useful reframing that supports controlled generation.
  2. Compact, practical, training-free method. Building an inference-time perturbation field via the Moreau envelope is conceptually simple and requires no retraining, which makes it easy to apply to existing pretrained models.
Critical weaknesses
  1. The evaluation score S is not robust. When the dataset contains many duplicate or near-duplicate samples, a generated point can lie close to both nn1 and nn2 and still yield a large S value—memorization persists but is not captured by S. The paper neither acknowledges this failure mode nor mitigates it, which undermines the reliability of the reported memorization measurements and the conclusions drawn from them.
  2. Experimental validation is too limited to support the claimed versatility. Real-world task evaluation covers only a small-scale RetinaMNIST image experiment and a small-scale text setup, and the compared baselines are mostly internal ablations derived from the same S-based objective rather than independent existing intervention strategies. This makes it hard to judge whether the method is competitive beyond the authors' own objective.
Questions for authors
  1. How does S behave when the dataset contains duplicate or near-duplicate samples? Can you provide a failure case and a mitigation, or an alternative metric that remains valid in that regime?
  2. Does the fixed inner-iteration budget of APGM achieve satisfactory convergence in the non-convex regime? Can you quantify the resulting approximation error, e.g., by varying the budget?
  3. Can intermediate APGM iterates cross Voronoi boundaries and cause oscillatory behavior? How is this handled?
  4. Can you compare against independent existing intervention strategies (not ablations of your own objective) on the same benchmarks?
  5. Can you extend the real-world evaluation beyond RetinaMNIST and the small-scale text setup, or explicitly scope the claims accordingly?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YcLz (1/2)

We thank the reviewer for the precise questions, which led us to find a better solver setting and to strengthen our results.

Q1 — Robustness of SS to duplicates and near-duplicates (W1). Our reported scores are not affected, which we verified directly.

  • Exact duplicates. Our training set contains none, in latent, PCA or pixel space. The failure mode cannot arise from them in our experiments.
  • Near-duplicates. We define them with the paper's own criterion rather than a new threshold: training image jj is a near-duplicate of ii if S(i)<τS(i) < \tau when ii is scored against the rest of the training set, i.e. jj would count as a copy of ii. Only 3.5% of training images have one, always as isolated pairs. A near-duplicate pair can only hide a copy whose distance to its nearest training image is comparable to the gap within the pair. We tested this by recomputing memorization with the second neighbor taken outside the near-duplicate group of the first (256 samples per γ\gamma):
γ\gamma00.10.20.3
Latent, standard → merged (%)93.0 → 94.176.2 → 78.138.7 → 40.22.7 → 3.9
Pixel, standard → merged (%)45.3 → 45.726.2 → 26.210.5 → 10.50.4 → 0.4

Rates change by at most 2 points (latent) and 0.4 points (pixel), and the trend is unchanged. The share of samples whose status changes stays flat across γ\gamma (1.2–2.0% latent, ≤\leq 0.4% pixel), so the perturbation does not lower SS by placing samples between near-duplicates (Appendix F.3).

  • As the driving potential, SS is built on an anchor set DD that need not be the raw training set, so DD can be deduplicated or clustered beforehand without changing the construction (Sec. 3.2).
  • Independently of SS, DINOv2 top-1 similarity to reconstructed training images, a standard copy-detection statistic measured in a space the perturbation does not act on, falls from 0.953 to 0.880 across the γ\gamma sweep (Fig. 8, center).

Q2 — Convergence of the inner budget. Yes, provided the inner step is well chosen. We compared the computed field with a 20,000-iteration reference along sampling trajectories (64 samples). Its fixed point does not depend on λˉ\bar\lambda (General response §1), and references computed with steps 0.05 and 0.5 agree (median cosine ≥0.999\geq 0.999).

Direction (median cosine similarity with the reference; 1 = same direction):

Settingt=0.1t = 0.1t=0.3t = 0.3t=0.6t = 0.6t=0.9t = 0.9
λˉ=0.05\bar\lambda = 0.05, N=30N = 30 (submission)0.900.890.930.97
λˉ=1\bar\lambda = 1, N=10N = 100.960.960.981.00
λˉ=1\bar\lambda = 1, N=30N = 300.9990.9980.9981.000

Magnitude (median norm ratio to the reference; 1 = exact):

Settingt=0.1t = 0.1t=0.3t = 0.3t=0.6t = 0.6t=0.9t = 0.9
λˉ=0.05\bar\lambda = 0.05, N=30N = 30 (submission)0.150.230.390.71
λˉ=1\bar\lambda = 1, N=10N = 100.390.530.761.02
λˉ=1\bar\lambda = 1, N=30N = 301.051.061.011.00

The submitted setting already pointed in nearly the correct direction but delivered only part of the magnitude, mostly at small tt. There, the schedule λ(t)=λ0/t\lambda(t) = \lambda_0/t keeps the field weak by design, while samples are still close to noise, and also makes the proximal search radius largest, so a fixed number of small steps covers less of it. With λˉ=1\bar\lambda = 1, 30 iterations reach the reference within 9%, and N=10N = 10 keeps the direction (cosine ≥0.96\geq 0.96) with a smaller magnitude early on. We use N=10N = 10 in the paper: it takes 1.5× instead of 2.4× the unperturbed sampling time, and gives a slightly better memorization–fidelity trade-off (at held-out FD ≈0.65\approx 0.65: 11.7% latent memorization vs. 18.8% for N=30N = 30). Appendix D.1 reports both.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer YcLz (2/2)

Q3 — Can APGM iterates cross Voronoi boundaries? Geometrically, S=d12/d22≤1S = d_1^2/d_2^2 \leq 1, with equality exactly on cell boundaries, so boundaries are maxima of SS. The proximal step minimizes SS plus a quadratic, which pulls iterates toward the cell's anchor and away from its boundary. Empirically, the fraction of solves whose proximal point lies in a different cell than the starting point is 0% for λˉ≤1\bar\lambda \leq 1 and N∈{10,30}N \in \{10, 30\}, at every time tested. Crossings appear only with much larger steps (up to 11% at λˉ=20\bar\lambda = 20), when Nesterov extrapolation overshoots. Our implementation recomputes the nearest neighbor at each iterate, so a 0% crossing rate also means it coincides with the fixed-anchor definition of Eq. 8.

Q4 — Comparison with independent methods (W2). As detailed in General response §3, the dissimilarity component of AMG (Chen et al., 2024) is the only part of AMG that applies to our unconditional setting, and our ∇S\nabla S baseline is its closest velocity-space analogue. The closest competitor, the Riemannian Laplace approach of Gegenfurtner et al. (2026), also perturbs the velocity but through a posterior over model parameters, and has no public code. Nearest-neighbor repulsion is a second independent heuristic. At γ=0.2\gamma = 0.2 (256 samples):

FieldS<τS<\tau latent (%)S<τS<\tau pixel (%)FD (held-out)NFE (dopri5)
Unperturbed93.045.30.563602
AMG-style ∇S\nabla S, norm-matched46.514.10.604860
NN repulsion, norm-matched38.312.90.652602
∇MλS\nabla M_\lambda S (ours)38.710.50.627602

At their natural scale, both baselines leave memorization at the unperturbed level (92.6% and 92.2%). We therefore also give them the envelope's own magnitude at every tt (norm-matched). Against nearest-neighbor repulsion, our field reaches the same latent memorization with lower pixel memorization and better fidelity. The norm-matched ∇S\nabla S removes less memorization at this scale, with a slightly better fidelity for the amount it removes; it works only with the magnitude supplied by the envelope, and its irregularity costs the adaptive solver 43% more evaluations (860 vs. 602), which is the smoothing effect of the envelope. Results at N=30N = 30 lead to the same conclusions (Appendix D.1).

Q5 — Scope of the evaluation. We respectfully maintain the versatility claim, which concerns the mechanism rather than the scale of the evaluation: the same training-free construction mitigates or enforces memorization with a single signed parameter, applies to any representation space and any anchor set, and is demonstrated in three settings of different nature (2D mixtures, latent image generation, a flow language model). We agree that our experiments are small-scale, and the new limitations paragraph states this explicitly. Scaling is discussed in General response §2 and in our answer to L4CU.

We thank the reviewer for considering our response and for raising their score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank all three reviewers for their careful reading and constructive questions. Several questions are shared, notably computational cost (all reviewers), convergence of the inner solver (YcLz, 3FmK) and baselines (YcLz). We answer them once here and refer to this comment from the individual responses. A revised PDF is uploaded, with all changes in blue.

1. Inner solver convergence: a better step, stronger results

Following YcLz's question, we measured the convergence of Algorithm 1 against a 20,000-iteration reference. The fixed point of Eq. 17 does not depend on the inner step λˉ\bar\lambda: setting xk+1=xk=x⋆x^{k+1} = x^k = x^\star gives x⋆=y−λ∇S(x⋆)x^\star = y - \lambda \nabla S(x^\star). References computed with two different steps agree accordingly (median cosine ≥0.999\geq 0.999). The step only controls how fast the solver gets there.

Our default λˉ=0.05\bar\lambda = 0.05 was conservative: the field pointed in nearly the same direction as the converged one (median cosine 0.89–0.97) but reached only part of its magnitude. With λˉ=1.0\bar\lambda = 1.0, the solver gets much closer to the proximal point at the same cost per iteration (details in our answer to YcLz, Q2), and memorization control becomes markedly stronger: at γ=0.3\gamma = 0.3, latent memorization drops from 42.2% to 2.7% (pixel: from 10.9% to 0.4%). All RetinaMNIST results are regenerated with this step, and the γ\gamma sweep now covers [0,0.3][0, 0.3] since the effect saturates sooner (Appendix D.1).

2. Computational cost and scalability

Measured for our RetinaMNIST experiment: small UNet, 1,000 anchors in a 200-dimensional PCA space, 128 samples, 100 Euler steps, on one NVIDIA L4 GPU (Appendix D.2).

SettingTime (s)Added time (s)Relative
Unperturbed1.26—1.0×
Moreau, λˉ=1\bar\lambda = 1, N=10N = 10 (used in the paper)1.93+0.671.5×
Moreau, λˉ=1\bar\lambda = 1, N=30N = 303.08+1.822.4×
  • The added time depends on the anchor set and the inner budget, not on the generative model. The ratios above are specific to our small UNet; the same added time is a smaller fraction of sampling with a larger model.
  • The cost reduces to one kNN query per step. As written, each inner iteration searches all nn anchors, i.e. O(N n k)O(N\,n\,k) per sample and step. Since the proximal search is local (its radius is set by λ\lambda), it suffices to search once for the KK nearest anchors of the starting point and to iterate over those only. With K=64K = 64, this reproduces the exact field (median relative error 10−610^{-6}, maximum 0.02%) with identical memorization, and already reduces sampling time by 8% (N=10N = 10) and 14% (N=30N = 30) at n=1000n = 1000. The cost becomes one kNN query plus O(NKk)O(N K k); that single query is what GPU and approximate indices (FAISS, Johnson et al., 2021; ScaNN, Guo et al., 2020) serve routinely at the scale of millions to billions of vectors.
  • nn is the size of the anchor set, not of the training set (see our answer to L4CU on practical relevance).

(continued in the next comment)

AuthorsOpenReview ↗

General response to all reviewers (2/2)

3. Baselines

The closest competitor is the Riemannian Laplace approach of Gegenfurtner et al. (2026), which also perturbs the velocity of a flow model, but through a Laplace posterior over model parameters, and has no public code. Apart from it, we know of no method acting on the velocity of a flow model at inference time. The closest training-free method for diffusion models is Anti-Memorization Guidance (AMG; Chen et al., CVPR 2024), designed for ϵ\epsilon-prediction diffusion models. AMG combines three components: despecification and caption deduplication guidance, which act on the text conditioning, and dissimilarity guidance. Only the latter applies to unconditional generation. It adds c31−αˉt ∇xtσtc_3\sqrt{1-\bar\alpha_t}\,\nabla_{x_t}\sigma_t to ϵ^\hat\epsilon, where σt\sigma_t is a normalized nearest-neighbor distance computed on the predicted clean image x^0\hat x_0 and the guidance is switched on by a dynamic threshold. There is no exact correspondence between ϵ\epsilon-guidance and velocity guidance for our models, which use minibatch OT couplings (RetinaMNIST) and a Gaussian-mixture source (2D). We therefore use the closest analogue, our ∇S\nabla S baseline: the gradient of a nearest-neighbor similarity, evaluated at xtx_t and added to the velocity. We present and cite it as an AMG-style baseline and detail these adaptations in Appendix F.2 (results in our answer to YcLz, Q4). The text-specific components of AMG have no counterpart in our experiments, which cover unconditional generation; conditional and prompted generation, for instance with classifier-free guidance, is untested, although nothing in the construction is specific to the unconditional case. We state this in the new limitations paragraph.

4. Summary of revisions (in blue in the revised PDF)

  • Sec. 3.2: the anchor set need not be the raw training set (deduplicated or restricted).
  • Sec. 3.3: sign error corrected (mitigation is ascent of EλE_\lambda), thanks to L4CU.
  • Sec. 3.4 and Fig. 1 caption: discussion of Figure 1, thanks to 3FmK; step independence of the fixed point.
  • Sec. 4.1, Table 1, Figs. 2, 3, 4b, 7 and 8, Appendix F: RetinaMNIST results regenerated with λˉ=1\bar\lambda = 1, N=10N = 10; AMG-style baseline.
  • Related work: AMG discussed.
  • Conclusion: new limitations paragraph.
  • Appendix D.1: convergence study and Voronoi cell crossing. Appendix D.2: wall-clock times, complexity, candidate-cached solver, scaling.
  • Appendix F.2: AMG-style baseline. Appendix F.3: duplicates and near-duplicates.

References: Chen, Liu & Xu, Towards Memorization-Free Diffusion Models, CVPR 2024. Gegenfurtner et al., Reducing Memorisation in Generative Models via Riemannian Bayesian Inference, 2026. Johnson, Douze & Jégou, Billion-scale similarity search with GPUs, IEEE Trans. Big Data. Guo et al., Accelerating Large-Scale Inference with Anisotropic Vector Quantization, ICML 2020. Sun, Guo & Kumar, SOAR: Improved Indexing for Approximate Nearest Neighbor Search, NeurIPS 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.