- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why memorization varies across regions within the same diffusion model. It develops a point-dependent analysis of kernel density estimation in the high-dimensional regime (P=\exp(\alpha N)). For hard-sphere kernels, it derives a transition governed by local probability mass, and a leave-one-out argument connects low coverage to self-kernel dominance at training points. Experiments with pixel-space U-Nets on CIFAR-10, grayscale CelebA, and a MNIST/CIFAR-10 mixture report increasing nearest-neighbor-ratio memorization with local sparsity. Additional experiments examine class differences, training dynamics, and model capacity. The paper interprets these observations through a finite effective kernel width.
Critical strengths
1.The paper identifies local effective sample size, rather than dataset size alone, as a relevant quantity for understanding which examples are vulnerable to memorization. The hard-sphere analysis makes this intuition explicit through the competition between sample complexity and local coverage. This provides an interpretable framework for discussing the coexistence of copying and novel generation.
2.The experiments examine dataset size, local sparsity, class composition, training duration, and model capacity. The held-out-data dominance baseline is a valuable control for geometric correlations. The training-dynamics observations are particularly interesting and suggest a connection between data isolation and the onset of memorization.
Critical weaknesses
1.The main qualitative conclusion overlaps substantially with prior work. Zhou et al. [1] discuss single-sample dominance, interpolation among nearby samples, and memorization of isolated samples. Chen [2] studies how score smoothing produces interpolation. Zhang et al. [3], already cited in the submission, connect locally sparse data to memorization and locally abundant data to learning shared statistics. These works already provide much of the intuition behind the claim that sparse regions favor memorization while dense regions support interpolation. The local coverage criterion is a useful addition, but the paper should explain more clearly what new predictions it provides beyond these results. In particular, the explicit threshold is derived for hard-sphere KDE, while its connection to Gaussian kernels and trained diffusion models remains largely qualitative. Section 4.1 reports correlations with nearest-neighbor sparsity without directly testing the predicted threshold, and the effective kernel width proposed in Section 4.3 is not directly estimated. This makes the additional predictive value of the theory difficult to assess.
2.The memorization measures are affected by the same distances used to describe sparsity. Sparsity is measured using distances between neighboring training samples, while both local dominance and binary memorization depend on the ratio (d_1/d_2). Even if the distance to the nearest training sample, (d_1), stays fixed, a larger (d_2) increases dominance and makes (d_1/d_2<\tau) easier to satisfy. Thus, isolated samples may receive higher memorization scores without being reproduced more accurately in absolute distance. The authors acknowledge the possible correlation and provide a useful test-data baseline. However, this comparison does not fully separate differences in copying accuracy from differences in training-sample spacing. Moreover, dominance and binary memorization are computed from the same ratio, so their agreement does not provide independent confirmation. Reporting absolute nearest-neighbor distances and an additional copying measure would help establish whether the observed trend reflects more accurate reproduction of isolated training samples.
[1] Smoothing the Score Function to Enhance Generalization in Diffusion Models
[2] On the Interpolation Effect of Score Smoothing in Diffusion Models
[3] Generalization of Diffusion Models Arises with a Balanced Representation Space
Questions for authors
1.What predictions does the local coverage criterion provide beyond the related work above? Can these predictions be tested in a setting where local coverage is known?
2.How does the hard-sphere result extend to Gaussian KDE and the outputs of trained diffusion models? What evidence supports describing training and capacity effects through an effective kernel width?
3.Does the relationship between sparsity and memorization remain when copying is measured using absolute distance or perceptual similarity?
5 comments
Thank you for the response and the additional experiments. The rebuttal clarifies the mathematical generality of Theorem 1 and the proposed local coverage criterion. However, I'm not convinced that these results provide sufficiently distinct, testable predictions beyond the established relationship between local sparsity, sample dominance, and memorization. In particular, the explicit transition threshold is derived for hard-sphere kernels, while the additional experiments mainly demonstrate qualitative correlations rather than quantitatively validate the predicted transition in trained diffusion models.
Although I think the paper is well-developed and the additional experiments are useful, I still find the incremental novelty and additional insights beyond prior work somewhat limited. Therefore, I will maintain my original score.
Thank you for your follow-up. We are glad that our response clarified the local coverage criterion, and we appreciate your positive assessment of the paper’s development and the additional experiments. We understand that your remaining reservation concerns the significance of the theoretical advance beyond the established qualitative picture and its testable predictions.
We appreciate your acknowledgment of the mathematical generality of Theorem 1. As our paper is primarily a theoretical contribution, we believe that its novelty should be assessed through the mathematical results it establishes relative to prior work. Theorem 1 provides a general local characterization of KDE statistics for arbitrary kernels and data distributions, including their dependence on sample complexity. Its scope extends substantially beyond the specific settings analyzed in the cited works: separated clusters and Gaussian mixtures in Zhang et al., and uniformly spaced training samples on a one-dimensional subspace in Chen (2026). We ask that the paper be judged primarily on the novelty and significance of its contribution to theoretical understanding.
Connecting the theory to trained diffusion models involves several approximations: describing the learned models by an effective KDE, using hard-sphere kernels to obtain an explicit threshold, and taking the high-dimensional limit (N\to\infty) with (P=e^{\alpha N}). The hard-sphere kernel serves as an idealization illustrating the general result, while Gaussian kernels are covered by the framework of Theorem 1. Quantitatively comparing the explicit threshold’s numerical location with trained models would additionally require accounting for these approximations.
Despite these approximations, our experiments support the relevance of the theoretical results. We recognize that qualitative links between sparsity and memorization have been observed previously. However, our experiments also examine a more specific prediction of Theorem 1 than the sparsity-memorization correlation alone: as dataset size increases, memorization recedes first in well-covered regions and persists longest in poorly covered regions.
We thank the reviewer for their detailed comments and helpful suggestions.
Critical point 1: Relation to prior work and novelty
Thank you for suggesting these relevant works. Our main theoretical contribution is a general characterization of local KDE statistics for arbitrary kernels and data distributions (Theorem 1). This framework describes how individual samples locally compete with the combined contribution of nearby training points as sample size increases. We then test empirically how local geometry predicts differences in memorization risk across training samples. We agree that the qualitative sparsity-memorization connection also appears in Zhang et al. and other cited works. Our contribution makes the joint dependence on local geometry and sample complexity in KDE quantitative; neither Zhou et al. nor Chen provides the local KDE characterization of Theorem 1.
Chen studies how score smoothing promotes interpolation, with the analytical results focusing on uniformly spaced samples on a one-dimensional manifold. Zhou et al. analyze Gaussian-component dominance and develop smoothing methods to reduce memorization. We therefore view these works as complementary: they substantiate the relevance of understanding KDE, while our analysis advances that understanding locally, and now discuss these references in our manuscript. In particular, building on Zhou et al.'s suggestion of geometry-dependent smoothing, we also discuss using our local coverage criterion to guide smoothing as a future direction.
Memorization measure
Thank you for highlighting the possible confound between our memorization metric and local sparsity. We have added experiments using SSCD features (Pizzi et al., 2022), which compare images in the feature space of a pretrained copy-detection model rather than pixel space. We report both raw cosine similarities, which do not involve the second-neighbor distance, and local dominance. Both measures support the hypothesis on CIFAR-10 and CelebA (Section C.5). Together with the held-out-data baselines, these checks provide evidence that the relationship is not simply an artifact of the shared pixel-distance statistics. We did not include raw pixel distances to the nearest neighbor on their own, as these have been found not to be good predictors of memorization on CIFAR-10 before (see e.g. Carlini et al. 2023, Section 5).
Effective kernel width
We would first like to clarify that within our KDE description, a finite effective kernel width is a modeling assumption needed to describe novel generation: as the width approaches zero, the KDE collapses onto the training data (Section 4.3)
However, we noticed several trends in the empirical results in in section 4.3 and section C.4 of the appendix that relate to this effective description.
First, alignment between the trained denoiser and the corresponding KDE estimator increases during training, with agreement strongest at high noise levels and improving subsequently at lower noise levels (Figures 3c and 8). Thus, as training progresses, the model approximates KDE estimators with progressively smaller kernel widths. We evaluate this alignment on both noised training and held-out data.
Second, memorization initially occurs predominantly in sparse regions and later extends to denser regions (Figures 3b and 7). This is consistent with a decreasing effective kernel width, which moves the threshold for single-sample dominance toward regions of higher coverage.
Varying model capacity produces corresponding trends (Figures 8–9): smaller U-Nets retain strong alignment with KDE at high noise levels but show larger discrepancies at low noise levels, while larger models exhibit stronger memorization. Together, these observations support interpreting increased training time and capacity as allowing the model to approximate KDE at progressively smaller effective kernel widths.
Further questions, addressed together
Can these predictions be tested in a setting where local coverage is known? How does the hard-sphere result extend to Gaussian KDE and the outputs of trained diffusion models?
Our main theoretical result, Theorem 1, applies to general kernel functions, including Gaussian kernels, under the stated convergence assumptions. Obtaining a corresponding explicit threshold for Gaussian kernels requires evaluating the kernel statistic appearing in Theorem 1 for that case.
We have also added experiments measuring memorization directly against local coverage (Section C.6). We observe the expected decrease in memorization as coverage increases. Here, coverage is estimated from the available data rather than known analytically. These results provide a more direct empirical check of the coverage dependence predicted by the hard-sphere analysis in trained diffusion models.
We hope these clarifications and additional experiments address your concerns. If so, we would be grateful if you would consider raising your score.