acceptodds
Under review as a conference paper at ICLR 2027

Toward Generalization from Memorized Diffusion Models with Housekeeping-Current Injection

Abstract

We propose a sampling protocol that reduces copying in memorized diffusion models while retaining the large-scale structure of the data distribution. We describe memorization with a relative free-energy model, in which narrow local minima around individual training examples lie within broader energy wells. Using the Entropic Bound (EB) together with the continuity equation, we distinguish density-changing transport from housekeeping currents that preserve a prescribed reference density. This motivates housekeeping-current injection (HCI), which adds such a current during sampling to steer trajectories away from memorized paths. In the energy picture, HCI helps trajectories leave the narrow wells around training examples; the learned landscape itself is not changed, and the diffusion model is not retrained. We also add a simple transport correction to alleviate the concentration that can occur with HCI alone. A 2D ring illustrates the landscape and HCI sampling; experiments on an 8D Gaussian mixture model, CelebA-32 and CelebA-64 evaluate trained models that show strong memorization. Compared with recent sampling-based methods, HCI substantially reduces the memorization fraction at a smaller loss of sample quality.

Primary area:
generative models
Keywords:
memorization, generalization, entropy production, housekeeping current, entropic bound

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Vp8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes housekeeping-current injection (HCI), a sampling intervention for memorized diffusion models. It adds a drift that preserves a fitted reference distribution, together with a transport correction when needed. Experiments on a ring, a Gaussian mixture, and CelebA show reduced near-copy rates without retraining the diffusion model.

Critical strengths

The distinction between preserving a reference distribution and changing the memorized model’s distribution is interesting. The reduction in near-copy rates is substantial, and the comparisons at similar FID, brightness-normalized measurements, and replication runs make the empirical result more convincing.

Critical weaknesses
  1. I am not yet convinced that the image results demonstrate generalization beyond deforming memorized examples. Section 5.3 explicitly describes the outputs this way. Brightness normalization rules out a simple intensity shift, but pixel-distance thresholds can still miss perceptually similar copies after geometric changes. The reported NN ratio of 0.54, versus 1.00 for held-out faces, also suggests substantial remaining proximity to training images. A perceptual evaluation would help establish what has actually been recovered.

  2. The theory explains reference preservation, but not when HCI should reduce copying. Proposition 1 makes HCI alone KL-neutral, while Proposition 2 shows a density change without determining whether mass moves away from training examples. The paper acknowledges that the relaxation in Section 4.2 is not the learned sampler, and the practical method adds a heuristic transport correction. I am missing a clearer connection between the theoretical construction and the observed benefit.

Questions for authors
  1. Does the reduction in copying persist with perceptual nearest-neighbor distances or after aligning generated and training images? This would help distinguish new samples from transformed copies.

  2. What determines whether a reference-preserving current reduces rather than increases copying? Could you test sensitivity to the sign and orientation of the skew matrix while keeping injection strength and transport correction fixed?

  3. How does HCI compare with applying a matched-strength transformation in the same PCA subspace only to the final samples? This would help isolate the benefit of intervening during sampling.

7 comments
AuthorsOpenReview ↗

Weakness 1

I am not yet convinced that the image results demonstrate generalization beyond deforming memorized examples. Section 5.3 explicitly describes the outputs this way. Brightness normalization rules out a simple intensity shift, but pixel-distance thresholds can still miss perceptually similar copies after geometric changes. The reported NN ratio of 0.54, versus 1.00 for held-out faces, also suggests substantial remaining proximity to training images. A perceptual evaluation would help establish what has actually been recovered.

We agree that the pixel criterion alone cannot establish this. The revised paper adds a perceptual evaluation and a test against deformation, and it reports the remaining proximity to the training images.

  • Perceptual evaluation. On CelebA-64 in color, a copy is now decided by SSCD with a threshold that flags 1% of held-out images. The SSCD score decreases from 86.6% for the baseline to 37.2% at η=1.0\eta=1.0, while FID increases from 19.8 to 24.5 (Section 5.3, Figure 4, SI Table 5). On CelebA-32, we report the SSCD score in addition to fmemf_{\rm mem}. It decreases from 93.3% to 56.4% in the main setting (SI I.2, SI Table 6).
  • Test against deformation. We changed only the final baseline sample by a rotation, a reversed rotation, a random shift and a shift matched to HCI, within the same principal components that KK uses and with the displacement matched to HCI (Section 5.4, Figure 6b, SI Figure 12b, SI I.4). These changes raise rr, but the images remain the same person, while the HCI samples of these rows show a different face.
  • Geometric changes. The examples of the sample figures are selected with tests that are stricter than the pixel ratio. They include the distance ratio after alignment, the SSCD and DINOv2 similarity, and tests for blended and composited training images (SI H.3).
  • Remaining proximity. We agree that substantial proximity remains, and the revised paper reports it in place of the NN ratio.
    • On CelebA-32, 75.4% of the HCI samples have rr between 1/3 and 0.78, where 99.0% of held-out images have rr above 0.78 (Figure 7a, SI Table 8). We state in Section 5.4 that most samples stay closer to the region where the memorized model pulls, and that early stopping keeps more diversity than HCI.
    • On CelebA-64, the nearest training image is the same as for the baseline sample in 79.1% of all samples. In 4.9%, HCI gives a face with a different nearest training image that passes all our copy tests (Figure 5, SI Figure 11, SI H.3).
  • What is recovered. HCI removes most near-exact copies and gives fewer defective images than early stopping (Figure 7b, SI Table 9). It produces new faces samples, and many samples remain close to a training image under the perceptual measures.
AuthorsOpenReview ↗

Weakness 2

Intuition. The memorized model keeps the broad shape of the reference and adds a narrow well at each training point (Figure 1a). Along a level set of the reference, the reference density is constant, so a flow along the level set does not change the reference (Proposition 3). The learned density is not constant along the level set, because it has a peak at every training point. So the same flow carries probability out of these peaks. This is the source term of Proposition 2 (Eq. 11), which is nonzero where the density ratio ωt\omega_t changes along the flow. At every sampling step there is a competition. HCI pushes the sample along the level set, and the learned score pulls it back to the training point. Memorization decreases when the push that remains after all later pulls moves the sample out of the copy region. The calculation below makes this statement precise near one training point.

HCI reduces memorization when some of its displacement remains at the end of sampling. We show this condition below with a local calculation which includes the learned score.

For a fixed reference, HCI alone does not change KL (Proposition 1). But the copy fraction can change because it depends only on the probability near training examples. On a ring with a uniform reference, if we rotate the density peaks from the training points into the gaps, KL stays the same, but this probability becomes smaller.

We calculate the decrease near one training point using the learned score. Let yy be the position along a reference level set, with the training point at y=0y=0. Each GMM step applies HCI first and then the learned-score update (SI F.5). Near the training point, HCI moves the sample by ηδk\eta\delta_k, and the learned score pulls it back. We approximate this pull as linear,

y_{k+1}\simeq a_k(y_k+\eta\delta_k)+\xi_k,\quad 0<a_k<1. \tag{R1}

Here aka_k is the fraction of the distance left after the pull, and ξk\xi_k is the noise. In this approximation, aka_k and δk\delta_k depend only on the step. We compare paths with and without HCI for the same start and the same noise. Their difference follows dk+1=ak(dk+ηδk)d_{k+1}=a_k(d_k+\eta\delta_k), with d0=0d_0=0, so

d_N=\eta\sum_{k=0}^{N-1}\delta_k\prod_{j=k}^{N-1}a_j. \tag{R2}

Each displacement is reduced by the pull at that step and by all later pulls. An early displacement goes through more pulls, so both the strength and the timing of HCI matter.

Now assume that the final density without HCI, qN0q_N^0, is symmetric and decreases strictly with distance from the training point. In the linear approximation, HCI shifts this peak by dNd_N. For the copy region ∣y∣<ϵ|y|<\epsilon,

\Pr_{\rm HCI}(|y_N|<\epsilon) =\int_{-\epsilon}^{\epsilon}q_N^0(y-d_N)\\,dy \leq\int_{-\epsilon}^{\epsilon}q_N^0(y)\\,dy =\Pr_0(|y_N|<\epsilon). \tag{R3}

The copy interval contains the most probability when the peak is at its center. So copying near this training point decreases whenever dN≠0d_N\ne 0. If we use −K-K, only the sign of dNd_N changes, and the decrease is the same. This agrees with the experiment, where reversing the sign of KK gives similar values (SI I.2, Table 7). This calculation is local. A sample that moves to another training point can still count as a copy there.

Our HCI preserves the reference and moves samples along its level sets (Proposition 3). On the ring, samples move into the gaps and stay on the ring. At similar copy fractions, the midpoint occupancy is about 42% for HCI, 12% for churn, and 5% for KTS (Section 5.1, Figure 2).

SI Figure 8 illustrates how two reverse processes that start from the same density reach different final distributions when their velocities differ. Between training points in the GMM, the learned score pulls samples inward much more strongly than the reference score, so the distribution contracts with HCI alone (SI G.3, Table 4). HCI moves samples along the level sets and cannot correct this inward pull. The normal correction in Eq. (99) replaces only the normal component of the learned score with the reference value.

SI Table 3 (two-component reference) shows the two roles. The normal correction alone leaves a copy fraction of 0.664. Adding HCI lowers it to 0.004, with a KS distance of 0.033. HCI alone gives a copy fraction of 0.013, but the KS distance is 0.836.

The theory therefore gives two statements. The reference is preserved for any η\eta (Theorem 1), and a current that preserved the learned density itself could not change the copy fraction (SI C.6), so the reference must differ from the memorized model. The theory does not determine where a sample lands after it leaves the copy region. This depends on the learned score, and the transport correction is a heuristic for this part (Section 4.3).

AuthorsOpenReview ↗

Question 1

Does the reduction in copying persist with perceptual nearest-neighbor distances or after aligning generated and training images? This would help distinguish new samples from transformed copies.

Yes. The reduction persists with a perceptual measure and after alignment, and it is smaller than under the pixel criterion.

  • Perceptual measure. The revised paper reports the SSCD score, which is the fraction of samples above the threshold that flags 1% of held-out images.
    • On CelebA-64, the SSCD score decreases from 86.6% for the baseline to 37.2% at η=1.0\eta=1.0 (Section 5.3, Figure 4, SI Table 5).
    • On CelebA-32, it decreases from 93.3% to 56.4% in the main setting (SI I.2), and SI Table 6 gives its dependence on η\eta.
  • After alignment. For each generated image, we searched over shifts, rotations up to 10 degrees, scales from 0.9 to 1.1 and flips of the training images, and we took the smallest distance. The values below are for CelebA-32 in the setting of SI Table 6 and are reported in this response.
    • With the same rule d1/d2<1/3d_1/d_2<1/3 after alignment, the copy fraction decreases from 92.0% for the memorized model to 0.9% for HCI with the normal correction.
    • With a threshold that flags 1% of held-out images, it decreases from 96.9% to 63.5%.
  • So the HCI samples are rarely near-exact copies, also after alignment. But most of them are still closer to a training image than held-out images are, which agrees with the distribution of rr in Figure 7(a) and SI Table 8.
  • The aligned distance ratio is also one of the tests with which we select the examples of the sample figures (SI H.3).
Reviewer_ZEmuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Overparametrized diffusion models can memorize their training set if the sample complexity is small or if they are trained for too long (Gu et al. 2023, Baptista et al. 2025, Bonnaire et al. 2025). This paper proposes a method to mitigate this memorization without compromising the quality of generated samples too much. The proposed method is training-free, i.e., one does not need to retrain a new model but can reuse a pretrained score model and it only changes the sampling algorithm.

The authors propose the Housekeeping-Current Injection (HCI) method, which relies on the fact that adding a divergence-free probability current to the Fokker-Planck equation does not change the marginal distribution if the true score is used. The authors thus propose a specific current that steers the trajectories away from memorized training samples.

They support their claims with experiments on several datasets (2D ring, 8D Gaussian mixture model and CelebA at several resolutions), comparing their performance to other training-free methods such as Churn and KTS. The authors claim that their method outperforms the other methods in the literature in terms of memorization and FID (for the same FID the proposed method has less memorization than previous methods). Moreover, for simple toy models (2D ring and Gaussian mixture), they show that while Churn and KTS generate samples outside of the data manifold or concentrated around the training data, HCI mitigates these issues.

Critical strengths
  • The authors propose a training-free method to mitigate memorization in diffusion models by modifying the sampling dynamics. According to the authors' experiments, their method outperforms the previous training-free methods it is compared to (Churn, KTS, CPSample). Since memorization is a common issue in score-based generative models, having a training-free method to mitigate it without compromising sample quality would be valuable to the community.
Critical weaknesses
  • Section 5.4, on CelebA-64 in color, presents experiments whose details are explained neither in the main text nor in the appendix. Moreover, some generated samples labelled as non-memorized according to the Euclidean distance criterion seem to be memorized to the naked eye (M3 with PCA HCI). I think that, if it worked, it would be the most compelling experiment of the article, so it is a real pity that there are not enough details and analysis. If this section were complete, I could increase my rating.
  • The first part, on entropic bounds, seems a bit unrelated to the experimental part, and its results are never used.
Questions for authors
  • The authors say that one of their main contributions is the free energy picture of memorization. I believe that this picture is close to Biroli et al. (2024, Nature Communications). How do the two pictures differ?
  • There are a lot of parameters to be fitted (η,K\eta, K, the fitted Gaussian reference). How long does it take to fit them, and how sensitive are the results to these parameters? For the experiments, the authors make a specific choice of KK, the skew-symmetric matrix, but a priori this is a degree of freedom one can tune. Did the authors try to see the effect of the choice of KK?
  • How is this method better than just doing early stopping as in Bonnaire et al. 2025 and Favero et al. 2025?
  • In Fig. 4, the generated images seem to be slight variations of images present in the training set. Can we not say that the generated samples are correlated with the training set? The same holds for Fig. 5: some images labelled as non-memorized by the L2 distance ratio criterion still seem to be memorized to the naked eye, e.g. M3 with PCA HCI. Is the Euclidean distance criterion the right one to measure memorization?
  • In Section 5.4, what are M1, M2 and M3?
8 comments
AuthorsOpenReview ↗

Weakness 1

Section 5.4, on CelebA-64 in color, presents experiments whose details are explained neither in the main text nor in the appendix. Moreover, some generated samples labelled as non-memorized according to the Euclidean distance criterion seem to be memorized to the naked eye (M3 with PCA HCI). I think that, if it worked, it would be the most compelling experiment of the article, so it is a real pity that there are not enough details and analysis. If this section were complete, I could increase my rating.

We revise the experiment section of the paper. The CelebA-64 experiment is now Section 5.3, and its details are in SI H.

  • Models. M1, M2 and M3 were three fine-tuned models. We fine-tuned the public DDIM model pretrained on CelebA at 64×6464\times64 in color on 128 training images for 8,000 updates. We repeat the fine-tuning three times with different random seeds and report the mean and standard deviation over the three models (Section 5.3, SI H.1).
  • Sampling. We draw 10,240 samples per model with deterministic DDIM over 100 steps, from the same initial noise for the baseline and for HCI. The reference is a Gaussian fitted to 32,768 samples of the pretrained model, KK rotates within the 8 planes of the 16 leading principal components, HCI is active during the final 25% of the schedule, and the normal correction is applied at every step (Section 5.3, SI H.2).
  • Metric. We agree that the Euclidean distance criterion is not the right one for these images. We replaced it with SSCD, which is a perceptual copy detection metric. A sample is counted as a copy when its SSCD similarity to the nearest training image is at least 0.510, which is the 99th percentile for held-out images, and quality is measured by FID against 3,962 held-out images (Section 5.3, SI H.1).
  • Results. The SSCD score decreases from 86.6% for the baseline to 37.2% at η=1.0\eta=1.0, while FID increases from 19.8 to 24.5. Figure 4 shows four values of η\eta, and SI Table 5 lists the values with the standard deviation over the three models. The normal correction alone does not change the two measures, and HCI alone gives a similar SSCD score to HCI with the normal correction (SI H.2).
  • Samples that look memorized. The reviewer noted that some samples labelled as non-memorized looked memorized to the naked eye. The revised experiment treats this point in three ways.
    • A copy is now decided by SSCD and not by the Euclidean distance.
    • We report the copies that remain. At η=1.0\eta=1.0 the SSCD score is 37.2%, so more than one third of the samples are still counted as copies.
    • We show and count the samples that are below the threshold but still close to a training image. Figure 5 shows 24 initial noises with the baseline sample, the HCI sample and the nearest training image of each, and SI Figure 11 extends it to 48 initial noises. The bottom row shows samples whose nearest training image is the same as for the baseline sample. These are 79.1% of all samples, where 43.1% have SSCD below the threshold and 36.0% at or above it (SI H.3). The top row shows faces with a different nearest training image that pass all our copy tests, and these are 4.9% of all samples.

So HCI lowers the SSCD score substantially, but many samples keep the same nearest training image.

AuthorsOpenReview ↗

Weakness 2

The first part, on entropic bounds, seems a bit unrelated to the experimental part, and its results are never used.

The entropic bound is the theoretical introduction of the method, and we made this connection explicit in the revised paper (abstract, Sections 3 and 4).

  • The EB with the decomposition of the probability current gives one bound for the transport current and one for the housekeeping current (Section 3, Eq. 7). Transport makes the density evolve, and housekeeping changes the sample paths without changing the density.
  • This decomposition leads to the method. Theorem 1 shows that the housekeeping currents are the only currents that can be added to the reverse process without changing the reference density (Section 4.1, Eq. 9). The paragraph "From the split to an intervention" at the end of Section 3 states this step, and HCI adds such a current during sampling.
  • The constructed field u=−Ksrefu=-Ks_{\rm ref} is a housekeeping current of the reference (Proposition 3). The transport (normal) correction acts on the transport sector. It removes the normal part of the score gap TDerrT\mathcal D_{\rm err} and thereby limits the transport bias ΔW∥\Delta W_\parallel of Eq. 7 (SI D.5, Proposition 4 in SI F.1).
  • The experiments test the two parts separately. On the GMM, HCI alone removes the copies but contracts the samples, and the transport correction restores the distribution (Section 5.2, SI Table 3). SI G.3 and Table 4 measure the transport error that causes the contraction.
  • The endpoint branch explains the comparison with early stopping. When the learned velocity differs from the exact one, the Wasserstein error is controlled by the velocity error (SI B.2, Eqs. 32 and 33), and SI E.3 compares early stopping and HCI on the ring.
  • We added SI Figure 8, which illustrates the two branches of the bound.
AuthorsOpenReview ↗

Question 1

The authors say that one of their main contributions is the free energy picture of memorization. I believe that this picture is close to Biroli et al. (2024, Nature Communications). How do the two pictures differ?

The two pictures are consistent, and they are used for different purposes.

  • Biroli et al. (2024) analyze the reverse process with the exact empirical score in the limit of high dimension and many data points. They identify the speciation time, which follows from a change of curvature of the effective potential, and the collapse time, at which the trajectories are attracted to one training point. The collapse is a condensation as in a random energy model and is characterized by an excess entropy. Their result is the time at which each regime starts, as a function of the dimension and the number of data.
  • Our Section 2 writes the learned density relative to a reference, p^t=pt⋆ωt\hat p_t=p_t^\star\omega_t, and uses the relative free energy Ft[q]=DKL(q∥pt⋆)\mathcal F_t[q]=D_{\rm KL}(q\|p_t^\star) (Eqs. 1 and 2). Memorization appears as narrow minima of ΔEt=−log⁡ωt\Delta E_t=-\log\omega_t at the training points inside the broad basin of the reference (Figure 1). We use this description to ask which change of the sampling dynamics is allowed. A housekeeping current of the reference does not change the relative free energy of any density (Proposition 1), but it changes a learned density that deviates from the reference (Proposition 2).
  • So the difference is the following. Biroli et al. describe when and why the collapse onto the training points happens, and they do not modify the sampler. Their paper does not discuss probability currents, the entropy production or the housekeeping current. Our paper starts from a model that has already collapsed and uses the current decomposition to move the samples out of the narrow minima during sampling.
  • The two pictures agree on the time scale. In their analysis the collapse happens late in the reverse process, and HCI acts in the low-noise window, which is the final 25% of the schedule in the image experiments (Section 4.4).
  • In the revised paper, the free-energy picture is no longer listed as a contribution. The contributions are the housekeeping currents for diffusion models from the EB, their properties, and the HCI sampler (Section 1).

Reference

Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Marc Mézard. Dynamical regimes of diffusion models. Nature Communications, 15:9957, 2024.

Reviewer_8R4wOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors consider diffusion models which memorize their data. To reduce memorization, they inject a new current to the reverse diffusion process. In several benchmarks, the authors show that this current reduced memorization and allows the model to sample from data manifold.

Critical strengths

The theory part looks solid.

Critical weaknesses

I guess a key question is what is a practical setting in which this method is useful. Why start with an overtrained model that memorizes data in the first place? Are there "memorized" diffusion models out there that need to be rescued?

Writing gets obscure every now and then.

Questions for authors

In, paired CelebA, generated faces with HCI (and other methods as well) don't look very good. Can it be that these methods are just sampling off the data manifold?

How much of the work is done by HCI and how much is by transport?

What is "normal correction"?

4 comments
AuthorsOpenReview ↗

Critical Weakness 1

I guess a key question is what is a practical setting in which this method is useful. Why start with an overtrained model that memorizes data in the first place? Are there "memorized" diffusion models out there that need to be rescued?

Yes. Public models such as Stable Diffusion already copy some of their training images. Carlini et al. (2023) extracted more than one hundred training images from Stable Diffusion and Imagen. The developers of Stable Diffusion v1 also wrote in the model card that the model memorized some images which were duplicated in the training data. So memorization is a problem of real models that people use.

Companies already spend much effort on this problem. When OpenAI developed DALL·E 2, some internal models reproduced training images, so the team removed duplicate images and trained the model again (Nichol, 2022). Stability AI removed near-duplicate images with the copy detection model SSCD before training Stable Diffusion 3 (Esser et al., 2024), which adds a clustering step over the whole training set. Google reports deduplication and lower weights for similar images in Imagen 3, and deduplication in Imagen 4. HiDream.ai also reports deduplication to reduce the memorization risk (Cai et al., 2026).

But copying can remain after the data is cleaned. Hong et al. (2025) found memorization in Stable Diffusion 2 although the training data was deduplicated by URL, and they suggested that repeated image layouts are a possible cause. So a model can be useful for image generation and still copy some training examples.

This case is the practical setting of our work. A team already has a useful model, the model copies some training images, and retraining is expensive. Our method keeps the trained model and reduces copying only during sampling. HCI changes only the score that the sampler uses, and the trained network stays fixed (Section 4.4, Eq. 15).

The experiments of the revised paper follow this setting. In Section 5.3, we fine-tune a public pretrained model on 128 images, and after 8,000 updates the model reproduces its training images with an SSCD score of 86.6% (setup in SI H.1). HCI is applied to this frozen model and lowers the SSCD score to 37.2%, while FID increases from 19.8 to 24.5 (Figures 4 and 5).

Regarding the alternative of not overtraining, Section 5.4 compares HCI with early stopping of the same training run on CelebA-32 (Figure 7, values in SI I.4). The early-stopped checkpoints have FID 41.6 and 52.2 with 10.4% and 0.2% copies, and HCI on the memorized model has FID 30.6 with 5.6% copies. Among the non-copies, 41.0% and 60.9% of the early-stopped samples show a defect, against 21.3% for HCI. We also state there that early stopping keeps more diversity than HCI. Early stopping requires that the stopping point is chosen during training, whereas HCI can be applied to a model that is already trained.

References

  • Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. 32nd USENIX Security Symposium, pp. 5253–5270, 2023.
  • CompVis. Stable Diffusion v1 model card. 2022.
  • Alex Nichol. DALL·E 2 pre-training mitigations. OpenAI, 2022.
  • Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12606–12633, 2024.
  • Imagen 3 Team, Google. Imagen 3. arXiv:2408.07009, 2024.
  • Google. Imagen 4 model card. May 20, 2025.
  • Qi Cai et al. HiDream-O1-Image: A natively unified image generative foundation model with pixel-level unified transformer. arXiv:2605.11061, 2026.
  • Chunsan Hong, Tae-Hyun Oh, and Minhyuk Sung. MemBench: Memorized image trigger prompt dataset for diffusion models. Transactions on Machine Learning Research, 2025.
AuthorsOpenReview ↗

Question 1

In, paired CelebA, generated faces with HCI (and other methods as well) don't look very good. Can it be that these methods are just sampling off the data manifold?

We address this question by measuring image quality and memorization together, and by checking the data directly where the target is known. These results are in the revised paper, and we point to them below.

  • Quality and memorization on CelebA-32. The paired CelebA experiment uses 32 × 32 grayscale faces. Please see Section 5.4 with Figure 7(a) and SI Table 8, and the paired samples of all methods in Figure 6 and SI Figure 12. HCI reduces fmemf_{\rm mem} from 93.2% to 5.6%, while FID changes from 18.6 to 30.6. There is a quality cost compared with the memorized model. However, HCI still has better FID than early stopping at 220k updates, CPSample, and KTS, which also leave more pixel-level copies.
  • Quality of the individual images. We inspected every image that is not a copy for blur, low contrast and grain, compared with held-out images. Please see Figure 7(b) and SI Table 9, with the definitions in SI I.4. The share of defective images of HCI is close to the level of held-out images, and it is about two to three times higher for early stopping.
  • Perceptual measure. We added SSCD because pixel distance can miss visually similar copies. For CelebA-32, the SSCD score is in SI I.2 and SI Table 6. Memorization decreases under this measure too, but more than half of the HCI samples are still flagged.
  • CelebA-64. The CelebA-64 results give stronger evidence that a large reduction in memorization can be obtained while keeping image quality. Please see Section 5.3 with Figures 4 and 5, and SI H with SI Table 5. HCI with the normal correction lowers the SSCD score from 86.6% to 37.2%, while FID changes from 19.8 to 24.5.
  • GMM, where the target density is known. We can check the distribution exactly. Please see Section 5.2 with Figure 3, and SI Table 3. HCI with the normal correction removes the copies, and the distribution of the exact log-density of its samples stays at the sampling floor, while isotropic noise at a similar copy fraction moves the samples away from the target.
  • Ring. We can also check the structure directly. Figure 2 in Section 5.1 shows that HCI keeps the samples on the circle and moves them into the gaps between training points, while churn spreads them off the circle. SI E.3 with Figure 10 and Table 2 compares HCI with early stopping at almost equal copy fractions, where HCI keeps the narrow ring and the early model spreads samples inside and outside the ring.

For faces, FID and perceptual tests do not prove that every generated image lies on the data manifold. The image comparisons support a reduction in memorization while retaining useful image quality, and the GMM and ring experiments directly show how HCI can reduce memorization without moving samples away from the data structure.

AuthorsOpenReview ↗

Question 2

How much of the work is done by HCI and how much is by transport?

The reduction of memorization is done by HCI. The transport correction does not reduce memorization by itself. It keeps the distribution on the reference and improves the quality of the samples.

Where to see this in the revised paper

  • GMM (Section 5.2, Figure 3, SI Table 3, SI G.3). The normal correction alone lowers fmemf_{\rm mem} only from 0.99 to 0.66, and HCI alone lowers it to 0.013. But HCI alone contracts the samples, and adding the normal correction brings the distribution back to the sampling floor while fmemf_{\rm mem} is 0.004. SI G.3 with Table 4 measures the reason, which is the inward pull of the memorized score between training points.
  • CelebA-64 (SI H.2). The normal correction alone changes neither the SSCD score nor FID. HCI alone and HCI with the normal correction give almost the same SSCD score at η=0.7\eta=0.7 (56.7% and 56.4%).
  • CelebA-32 (SI I.2). Here the normal correction acts on the quality. With the reference of the main comparison, it removes the grain of the samples, and FID is 30.6.

Why the method has two parts (Section 4.3)

  • Proposition 3 gives the property of the constructed housekeeping current, and the paragraph "Adding transport" introduces the transport correction, whose form is given in Proposition 4 (SI F.1).
  • We are introducing the normal transport correction because we used the constructed skew matrix KK for the housekeeping current.
  • To be specific, the general condition for the housekeeping current is ∇⋅(prefu)=0\nabla\cdot(p_{\rm ref}u)=0, but we are taking a specific choice u=−Ksrefu=-Ks_{\rm ref} that satisfies ∇⋅u=0\nabla\cdot u=0 and u⋅sref=0u\cdot s_{\rm ref}=0 separately (Proposition 3). As a consequence, the HCI by KK cannot cross the level sets, and the motion across them is determined by the normal component of the score network.
  • The normal correction replaces this normal component by that of the reference. So HCI moves the samples away from the training points along the level sets, and the correction keeps them at the right place across the level sets.
AuthorsOpenReview ↗

Question 3

What is "normal correction"?

The normal correction is the transport correction of Section 4.3 (paragraph "Adding transport"). Its definition is Proposition 4 in SI F.1 (Eqs. 98 and 99), and the reverse process with HCI and the correction is Eq. 97. We explain it below with one sampling step.

Let's consider a single Euler–Maruyama step of the reverse variance-preserving SDE from step kk to k−1k-1 with step size Δt\Delta t.

  • xk−1=xk+βkΔt[12xk+s^(xk)]+βkΔt,zkx_{k-1}=x_k+\beta_k\Delta t\Bigl[\tfrac12x_k+\hat s(x_k)\Bigr]+\sqrt{\beta_k\Delta t}\\,z_k, where zk∼N(0,I)z_k\sim\mathcal N(0,I), s^=−ε/σk\hat s=-\varepsilon/\sigma_k is the score of the trained network, and β\beta is the pre-defined schedule.

  • HCI introduces changes in the drift, ∇⋅(ptkrefuk)=0\nabla\cdot\bigl(p^{\rm ref}_{t_k}u_k\bigr)=0. Writing the probability flux as a rotated density gradient, ptkrefuk=−K∇ptkrefp^{\rm ref}_{t_k}u_k=-K\nabla p^{\rm ref}_{t_k} with a constant skew-symmetric KK, and dividing by ptkrefp^{\rm ref}_{t_k} gives the field in terms of the reference score skref=∇log⁡ptkrefs^{\rm ref}_k=\nabla\log p^{\rm ref}_{t_k} uk=−K∇log⁡ptkref=−Kskref.u_k=-K\nabla\log p^{\rm ref}_{t_k}=-Ks^{\rm ref}_k.

  • s⊤Ks=0s^\top Ks=0, uku_k is orthogonal to the score (tangent to the level sets of the reference), and uku_k moves samples along the surfaces of constant reference density (Proposition 3). Here, the choice of KK itself is free. Note that this is a heuristic choice made in the current paper that satisfies ∇⋅(ptkrefuk)=0\nabla\cdot\bigl(p^{\rm ref}_{t_k}u_k\bigr)=0.

  • The normal correction is a separate modification. Because uku_k is tangent to the level sets, HCI from KK cannot move probability across them. Across level sets, the distribution is set by the component of the network score along the normal nk=skref/∥skref∥n_k=s^{\rm ref}_k/\lVert s^{\rm ref}_k\rVert. The smallest change of s^\hat s that replaces this normal component by that of the reference is s^,c=arg⁡min⁡s∥s−s^∥2\hat s^{\\,c}=\arg\min_s\lVert s-\hat s\rVert^2 subject to nk⊤s=nk⊤skrefn_k^\top s=n_k^\top s^{\rm ref}_k, i.e. s^,c=s^+nknk⊤(skref−s^)\hat s^{\\,c}=\hat s+n_kn_k^\top\bigl(s^{\rm ref}_k-\hat s\bigr). It has no free parameters, and in the image experiments it is applied at every step.

Then, replacing s^\hat s by s^,c+η,wk,ck,uk\hat s^{\\,c}+\eta\\,w_k\\,c_k\\,u_k, the reverse step used in the paper is

  • xk−1=xk+βkΔt[12xk+s^(xk)+nknk⊤(skref(xk)−s^(xk))−η,wk,ck,Kskref(xk)]+βkΔt,zk,x_{k-1}=x_k+\beta_k\Delta t\Bigl[\tfrac12x_k+\hat s(x_k)+n_kn_k^\top\bigl(s^{\rm ref}_k(x_k)-\hat s(x_k)\bigr)-\eta\\,w_k\\,c_k\\,K s^{\rm ref}_k(x_k)\Bigr]+\sqrt{\beta_k\Delta t}\\,z_k,

    • wk=max⁡(0,,1−tk/0.25T)w_k=\max(0,\\,1-t_k/0.25T) is the time window,
    • ckc_k is a scalar per step, which is calibrated once and then kept fixed (SI F.6).
  • We can use Figure 1 to explain what the normal correction is.

  • The reference prefp_{\rm ref} is the broad basin in Fig. 1 (a), and it is the ring in Fig. 1 (b) and (c). The memorized model has narrow wells at the training points, so the samples are moving toward each training point.

  • HCI drifts samples to move away from these wells following the tangential direction of the ring since it rotates the reference score. If the strength of this drift is large enough (η\eta is large) then a sample can escape from the well. This reduces memorization, but the quality of the sample depends on where it landed.

  • The normal correction guides the sample to stay in the reference. In the ring example, it will move samples that are off from the ring so that the samples can stay on the circle and between data points.

  • The GMM experiment shows this effect with numbers. HCI alone moves samples between training points, where the memorized score pulls inward much more strongly than the reference, and the samples contract. The normal correction removes this contraction (SI G.3, Tables 3 and 4).

Comments on the paper

1 comment

We thank all reviewers for the careful reading of our paper and for the comments and questions. We revised the paper to answer the questions and address the concerns in the revised version of the submitted paper. The changes are marked in red. In the responses to each reviewer, we point to the section, figure, or table of the revised paper that contains the answer.

The main changes in the revised paper address the clarifications and the experiment results, summarized as follows.

  • Sampling with HCI and the normal correction (Sections 4.3 and 4.4, SI F): We write the update of the sampler and the definition of the normal correction explicitly.
  • CelebA-64 (Section 5.3, SI H): We rewrote this experiment with three fine-tuned models, the perceptual measure SSCD, and an analysis of the generated samples.
  • CelebA-32 (Section 5.4, SI I): We added the comparison with early stopping, the distribution of the distance ratio, the quality of the individual images, the SSCD score, the variants of K, and the changes applied to the final samples.
  • Ring and GMM (SI E.3, SI G): We added the comparison of early stopping and HCI on the ring, and we rewrote the GMM section so that the roles of HCI and the normal correction can be read from one table.

We hope that the revised paper and the explanations below resolve the concerns of the reviewers. We are glad to answer any further question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.