- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether widening a latent representation delays memorization in diffusion model training. With a fixed signal distribution and fixed hidden width, wider synthetic latents reduce copying and delay empirical-score fitting. Vector-latent image experiments support this trend more strongly on CelebA than CIFAR-10. A frozen random-feature model suggests an explanation through changes in activation input scale and feature learning rates, but the connection to memorization in trained networks remains conditional. A spatial-DiT does not reproduce the trend.
Critical strengths
S1. The synthetic experiments address a real measurement problem. Adding noise coordinates can hide copies from the full-space nearest-neighbor test. After projection onto the signal subspace, copying still falls from 29.9% to 1.2%, against a 0.4% fresh-sample baseline. Calibrated signal copies and empirical-score alignment strengthen this finding (Tables 1-2).
S2. The spectral intervention is informative. Holding the activation input scale fixed reduces the endpoint growth of the signal-to-sample edge ratio from ~3.17 to ~1.09. The exact dynamics also clarify that feature directions learn in parallel.
S3. The experimental limitations are reported clearly. Reconstruction-identity checks, five-seed results, and the negative spatial-DiT pilot help establish where the observed effect holds.
Critical weaknesses
W1. The mechanism is not yet connected to memorization in the trained model. The surrogate uses frozen tanh features at one diffusion time, whereas the main network learns GELU features across diffusion times. At latent widths 5 and 40, first-layer pre-activation second moments evaluated at are nearly equal at the first saved checkpoint (50k steps; Appendix C). These uncentered moments include bias and time embeddings; their later separation does not establish the proposed causal sequence. The fixed-scale intervention changes surrogate spectra and noise covariance, without testing its effect on copying in the trained MLP. The authors shall clarify what connects slower feature learning to copying, including whether analysis across diffusion times along the reverse sampling path is needed.
W2. Related work and novelty need a more precise comparison. Ye et al. [1] report less copying at higher dimensions with a fixed 12M-parameter model and an equivalent nearest-neighbor criterion. George and Macris [2] study asymptotic error and sample complexity for optimal random-feature scores; Achilli et al. [3] and Ross et al. [4] study geometric memorization and manifold dimensions. These provide relevant background, but do not reproduce the present training-delay intervention. George et al. [5] show that orthogonal contributions can dominate total test error at small diffusion times, obscuring the transition between memorization and generalization along the data subspace. This motivates directional diagnostics but differs from reduced NN-detector sensitivity.
W3. Transfer from synthetic nuisance coordinates to real latents remains unresolved. Synthetic excess coordinates are independent Gaussian noise; wider real latents may encode redundant functions of the same factors. Repeating signal coordinates can preserve average energy, while zero padding reduces clean-data energy per coordinate without adding clean-data nuisance noise. The effect therefore depends on scaling and covariance as well as width. Separately trained VAEs change both representation and decoder. The widest CelebA latents have worse FID than intermediate widths, measured against the diffusion training subset, leaving the delay at comparable quality on an independent reference set unclear (Appendix D).
W4. The timing predictor has limited independent validation. The main pressure predictor is fitted using the observed hitting times. The held-out-width evaluation concerns a different one-constant clock, which is worse than a dimension-only baseline on CelebA (log-time error 0.407 versus 0.125) and modestly better on CIFAR-10 (0.151 versus 0.198). Both evaluations include reconstruction-limited widths (Table 5). These results support an exploratory diagnostic, but do not independently validate the main predictor.
Questions for authors
Q1. Can an input-scale intervention in the trained MLP change signal-projected copying and empirical-score alignment while preserving the data construction?
Q2. What would establish the link between the spectral groups and generated-sample memorization? Does this require analysis across diffusion times along the reverse sampling path?
Q3. How do the statistics in Appendix G.6 support the proposed mechanism in the trained MLP? Would redundant coordinates produce the same delay, and does the image effect persist at matched quality against an independent reference set?
Q4. As a secondary robustness check, does the delay persist for signal dimensions greater than five under signal-projected detection and empirical-score alignment, beyond the RFNN spectral sweeps?
Q5. Can the main pressure predictor be evaluated on held-out widths above the reconstruction-identity threshold, against the dimension-only baseline under the same protocol?
References
[1] Zeqi Ye, Qijie Zhu, Molei Tao, and Minshuo Chen. Provable Separations between Memorization and Generalization in Diffusion Models. ICLR, 2026.
[2] Anand Jerry George and Nicolas Macris. Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data. arXiv:2603.22962, 2026.
[3] Beatrice Achilli, Enrico Ventura, Gianluigi Silvestri, Bao Pham, Gabriel Raya, Dmitry Krotov, Carlo Lucibello, and Luca Ambrogioni. Losing dimensions: Geometric memorization in generative diffusion. arXiv:2410.08727, 2024.
[4] Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse C. Cresswell, and Gabriel Loaiza-Ganem. A Geometric Framework for Understanding Memorization in Generative Models. ICLR, 2025.
[5] Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Denoising Score Matching with Random Features: Insights on Diffusion Models From Precise Learning Curves. AISTATS, PMLR 300:1405-1413, 2026.
3 comments
Response to Reviewer tiyw (2/3): mechanism controls, real latents, architecture
W3 / Q3. Redundant coordinates, and the mechanism in the trained MLP. We ran both constructions you describe at d = 40: "tile" (eight copies of the signal, per-coordinate energy equal to d = 5) and "zero pad" (35 exact zeros). Neither adds nuisance noise.
| construction, 1M steps | alpha | projected copying |
|---|---|---|
| d = 5 baseline | 0.669 | 16.1% |
| d = 40, noise (paper) | 0.243 | 0.7% |
| d = 40, tile | 0.130 | 4.5% |
| d = 40, zero pad | 0.102 | 0.4% (chance) |
Both delay copying, so the delay does not come from nuisance noise or per-coordinate energy. The frozen-feature model treats a tiled input as identical to d = 5, so it predicts no delay there. Redundant coordinates do delay copying, though not identically: by copying, tile sits between the two baselines.
Four further controls isolate the cause:
| condition, 1M steps | alpha | projected copying |
|---|---|---|
| d = 5 + 35 noise input channels (not diffused) | 0.68 | 17.0% |
| d = 5 + 35 unpredictable output targets | 0.666 | 16.3% |
| zero pad + exact scores of the zero coordinates supplied by a fixed term | 0.690 | 19.8% |
| zero pad | 0.102 | 0.4% |
Extra inputs alone or extra targets alone do nothing. Removing the work of denoising the extra coordinates removes the delay. The mechanism in the trained fixed-width MLP is therefore the load of denoising the extra coordinates on the shared hidden layers. Two properties:
- Selective. The zero-padded network learns the population score in the signal subspace as fast as d = 5 (per-coordinate error 0.89 vs 1.09 at the first evaluation).
- Capacity-dependent. Hidden width 768 or 2048 instead of 256 shrinks the d = 8 vs d = 40 alignment gap at 1M steps from 0.565 to 0.112 and 0.088. An onset delay remains at 2048 (threshold crossed at 0.15–0.20M steps at d = 40 vs 0.05M at d = 8).
This answers the first half of W1: the link to copying in the trained network is competition for a fixed-width trunk.
Real latents and architecture. On CelebA VAEs retrained from scratch (d = 80–200; five seeds, 2M steps):
| denoiser | time to 5% memorization, d = 80 / 120 / 160 / 200 | ratio 200/80 | memorized at 2M, d = 80 / 200 |
|---|---|---|---|
| paper's MLP (SGD) | 0.30 / 0.57 / 0.84 / 1.16M | 3.87 | 0.670 / 0.182 |
| same MLP, Adam | 0.20 / 0.48 / 0.68 / 1.27M | 6.35 | 0.882 / 0.242 |
| small transformer, Adam | within 50k steps in 19 of 20 runs | 1.49 (35k vs 52k) | 0.955 / 0.981 |
The MLP result replicates on independent VAEs and under Adam. The transformer shows a small onset shift and no plateau decline. On the synthetic zero-pad construction it has about a tenth of the MLP's alignment gap (0.065 vs 0.565), with or without adaLN or attention, while an MLP with a residual stream or LayerNorm keeps the delay (gaps 0.30–0.50). We do not attribute the difference to one component. The claim is scoped to fixed-width MLP denoisers in the revised title claim, abstract and conclusion.
Matched quality. You are right. Training-subset FID rewards copies, so it cannot settle this, and we did not run an independent-reference comparison. The revision states it as untested. App. G.6 is a surrogate diagnostic and does not support a mechanism in the trained MLP; its heading will say so.
Response to Reviewer tiyw (3/3): intrinsic dimension, held-out evaluation, related work
Q4. Signal dimension above five: the delay persists. Same MLP, 10-dimensional signal, two seeds, 1.5M steps, detection projected onto the 10 signal coordinates.
| d (intrinsic 10) | alpha | projected copying | full-space copying |
|---|---|---|---|
| 10 | 0.43 | 21.9% | 21.9% |
| 20 | 0.39 | 5.3% | 3.2% |
| 40 | 0.32 | 0.3% | 0.1% |
| 80 | 0.09 | 0.0% | 0.0% |
Both readouts fall monotonically, by far more than the seed spread (below 0.01).
Q5 / W4. Held-out evaluation above the identity threshold.
| held-out log error, 5% threshold | spectral clock | dimension-only clock |
|---|---|---|
| CelebA, d >= 80 | 0.25 | 0.13 |
| CIFAR-10, d >= 160 | 0.14 | 0.13 |
The spectral clock does not beat the dimension-only baseline on either dataset. We have not yet evaluated the main pressure predictor under this protocol; the final version will report it, and we do not expect it to do better. The revision demotes the predictor to a diagnostic in the contribution list and the abstract.
W2. Related work. The revision adds a paragraph on each reference. Ye et al. [1] find less memorization at higher data dimension at the end of training with a fixed parameter count; they vary the data dimension itself and do not record onset time. The direction agrees with ours. George and Macris [2] and George et al. [5] study ridge-optimal random-feature scores, where memorization is governed by the feature-to-sample ratio and not by training time. We agree with your reading of [5]: our signal-projected test and alpha are directional diagnostics in that sense, and detector-sensitivity loss is a separate effect that the signal-copy control in Table 1 isolates. Achilli et al. [3] and Ross et al. [4] are static analyses that do not vary ambient dimension at fixed intrinsic dimension or study training time.
Summary. The empirical finding and its detector control stand, and now replicate on independent VAEs, under a second optimizer, and at intrinsic dimension 10. The frozen-feature interpretation does not transfer to the trained network. The interventions you asked for identify a different, tested mechanism in the MLP, and the claim is scoped to MLP denoisers. We would be grateful if you would reconsider the soundness and informativeness scores in that light.
Response to Reviewer tiyw (1/3): input-scale intervention and diffusion times
Thank you for questions we could act on. We ran the interventions behind Q1, Q3, Q4 and Q5, with success criteria fixed before each run. This response is in three parts.
Q1 / W1. Input-scale intervention in the trained MLP: no effect. We multiplied the network input by a fixed gain g, leaving data, loss, sampler and metrics unchanged. g = 0.45 gives d = 5 the d = 40 input variance; g = 2.2 does the reverse. Two seeds, 1M steps, against five-seed baselines.
The gain closes 6% of the width gap in alpha at d = 5 and moves the wrong way at d = 40. Our criterion required movement beyond the seed spread in the predicted direction at both widths, and it fails. Input scale does not set memorization timing in the trained MLP, so the operating-point account describes the frozen-feature spectra but does not transfer. The revision says so in the abstract, Sec. 3 and Sec. 5. Two caveats: the gain also scales the diffusion noise, and under Adam it changes the first layer's effective step size.
W1 / Q2. Linking feature learning to copying; diffusion times. In the frozen model each readout mode relaxes at its eigenvalue, but reading the sample group as memorization requires the alignment weights of Prop. G.2, which we have not measured. In the trained network the measurable link is alpha at small t, which tracks projected copying across widths and over training. Across diffusion times (five seeds, 1M steps):
The width ordering holds at every t. For t >= 0.3 the empirical and population scores nearly coincide in the signal coordinates, so alpha is uninformative there. We have not integrated alpha along the sampling path, and the final version will state the single-time analysis as a restriction.
On the early statistics: the App. C quantity was uncentered and included bias and time embedding. The centered variance from the latent input is not lower at d = 40 at initialization (0.081 vs 0.058 at d = 5) but is 2.6x lower by 50k steps. Alpha separates in the same window, so this does not establish causal order.