- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes housekeeping-current injection (HCI), a sampling intervention for memorized diffusion models. It adds a drift that preserves a fitted reference distribution, together with a transport correction when needed. Experiments on a ring, a Gaussian mixture, and CelebA show reduced near-copy rates without retraining the diffusion model.
Critical strengths
The distinction between preserving a reference distribution and changing the memorized model’s distribution is interesting. The reduction in near-copy rates is substantial, and the comparisons at similar FID, brightness-normalized measurements, and replication runs make the empirical result more convincing.
Critical weaknesses
-
I am not yet convinced that the image results demonstrate generalization beyond deforming memorized examples. Section 5.3 explicitly describes the outputs this way. Brightness normalization rules out a simple intensity shift, but pixel-distance thresholds can still miss perceptually similar copies after geometric changes. The reported NN ratio of 0.54, versus 1.00 for held-out faces, also suggests substantial remaining proximity to training images. A perceptual evaluation would help establish what has actually been recovered.
-
The theory explains reference preservation, but not when HCI should reduce copying. Proposition 1 makes HCI alone KL-neutral, while Proposition 2 shows a density change without determining whether mass moves away from training examples. The paper acknowledges that the relaxation in Section 4.2 is not the learned sampler, and the practical method adds a heuristic transport correction. I am missing a clearer connection between the theoretical construction and the observed benefit.
Questions for authors
-
Does the reduction in copying persist with perceptual nearest-neighbor distances or after aligning generated and training images? This would help distinguish new samples from transformed copies.
-
What determines whether a reference-preserving current reduces rather than increases copying? Could you test sensitivity to the sign and orientation of the skew matrix while keeping injection strength and transport correction fixed?
-
How does HCI compare with applying a matched-strength transformation in the same PCA subspace only to the final samples? This would help isolate the benefit of intervening during sampling.
7 comments
Weakness 2
Intuition. The memorized model keeps the broad shape of the reference and adds a narrow well at each training point (Figure 1a). Along a level set of the reference, the reference density is constant, so a flow along the level set does not change the reference (Proposition 3). The learned density is not constant along the level set, because it has a peak at every training point. So the same flow carries probability out of these peaks. This is the source term of Proposition 2 (Eq. 11), which is nonzero where the density ratio changes along the flow. At every sampling step there is a competition. HCI pushes the sample along the level set, and the learned score pulls it back to the training point. Memorization decreases when the push that remains after all later pulls moves the sample out of the copy region. The calculation below makes this statement precise near one training point.
HCI reduces memorization when some of its displacement remains at the end of sampling. We show this condition below with a local calculation which includes the learned score.
For a fixed reference, HCI alone does not change KL (Proposition 1). But the copy fraction can change because it depends only on the probability near training examples. On a ring with a uniform reference, if we rotate the density peaks from the training points into the gaps, KL stays the same, but this probability becomes smaller.
We calculate the decrease near one training point using the learned score. Let be the position along a reference level set, with the training point at . Each GMM step applies HCI first and then the learned-score update (SI F.5). Near the training point, HCI moves the sample by , and the learned score pulls it back. We approximate this pull as linear,
y_{k+1}\simeq a_k(y_k+\eta\delta_k)+\xi_k,\quad 0<a_k<1. \tag{R1}
Here is the fraction of the distance left after the pull, and is the noise. In this approximation, and depend only on the step. We compare paths with and without HCI for the same start and the same noise. Their difference follows , with , so
d_N=\eta\sum_{k=0}^{N-1}\delta_k\prod_{j=k}^{N-1}a_j. \tag{R2}
Each displacement is reduced by the pull at that step and by all later pulls. An early displacement goes through more pulls, so both the strength and the timing of HCI matter.
Now assume that the final density without HCI, , is symmetric and decreases strictly with distance from the training point. In the linear approximation, HCI shifts this peak by . For the copy region ,
\Pr_{\rm HCI}(|y_N|<\epsilon) =\int_{-\epsilon}^{\epsilon}q_N^0(y-d_N)\\,dy \leq\int_{-\epsilon}^{\epsilon}q_N^0(y)\\,dy =\Pr_0(|y_N|<\epsilon). \tag{R3}
The copy interval contains the most probability when the peak is at its center. So copying near this training point decreases whenever . If we use , only the sign of changes, and the decrease is the same. This agrees with the experiment, where reversing the sign of gives similar values (SI I.2, Table 7). This calculation is local. A sample that moves to another training point can still count as a copy there.
Our HCI preserves the reference and moves samples along its level sets (Proposition 3). On the ring, samples move into the gaps and stay on the ring. At similar copy fractions, the midpoint occupancy is about 42% for HCI, 12% for churn, and 5% for KTS (Section 5.1, Figure 2).
SI Figure 8 illustrates how two reverse processes that start from the same density reach different final distributions when their velocities differ. Between training points in the GMM, the learned score pulls samples inward much more strongly than the reference score, so the distribution contracts with HCI alone (SI G.3, Table 4). HCI moves samples along the level sets and cannot correct this inward pull. The normal correction in Eq. (99) replaces only the normal component of the learned score with the reference value.
SI Table 3 (two-component reference) shows the two roles. The normal correction alone leaves a copy fraction of 0.664. Adding HCI lowers it to 0.004, with a KS distance of 0.033. HCI alone gives a copy fraction of 0.013, but the KS distance is 0.836.
The theory therefore gives two statements. The reference is preserved for any (Theorem 1), and a current that preserved the learned density itself could not change the copy fraction (SI C.6), so the reference must differ from the memorized model. The theory does not determine where a sample lands after it leaves the copy region. This depends on the learned score, and the transport correction is a heuristic for this part (Section 4.3).
Question 1
Does the reduction in copying persist with perceptual nearest-neighbor distances or after aligning generated and training images? This would help distinguish new samples from transformed copies.
Yes. The reduction persists with a perceptual measure and after alignment, and it is smaller than under the pixel criterion.
- Perceptual measure. The revised paper reports the SSCD score, which is the fraction of samples above the threshold that flags 1% of held-out images.
- On CelebA-64, the SSCD score decreases from 86.6% for the baseline to 37.2% at (Section 5.3, Figure 4, SI Table 5).
- On CelebA-32, it decreases from 93.3% to 56.4% in the main setting (SI I.2), and SI Table 6 gives its dependence on .
- After alignment. For each generated image, we searched over shifts, rotations up to 10 degrees, scales from 0.9 to 1.1 and flips of the training images, and we took the smallest distance. The values below are for CelebA-32 in the setting of SI Table 6 and are reported in this response.
- With the same rule after alignment, the copy fraction decreases from 92.0% for the memorized model to 0.9% for HCI with the normal correction.
- With a threshold that flags 1% of held-out images, it decreases from 96.9% to 63.5%.
- So the HCI samples are rarely near-exact copies, also after alignment. But most of them are still closer to a training image than held-out images are, which agrees with the distribution of in Figure 7(a) and SI Table 8.
- The aligned distance ratio is also one of the tests with which we select the examples of the sample figures (SI H.3).
Weakness 1
We agree that the pixel criterion alone cannot establish this. The revised paper adds a perceptual evaluation and a test against deformation, and it reports the remaining proximity to the training images.