acceptodds
Under review as a conference paper at ICLR 2027

How Excess Latent Dimensionality Delays Memorization in Diffusion Models

Abstract

Latent diffusion models operate on representations whose dimension can exceed the data's intrinsic dimension. Because diffusion models can reproduce individual training samples, it is important to understand whether this excess width affects memorization. We find that widening the latent space at fixed hidden width delays memorization in the tested vector-latent multilayer perceptron (MLP) score models. In controlled synthetic experiments, the memorized fraction after 5M training steps falls from to as the latent dimension grows from 5 to 40. This decline persists when memorization is measured only in the known signal subspace, showing that declining detector sensitivity alone cannot explain the result. On CelebA, above the width at which the variational autoencoder (VAE) preserves sample identity, the mean time to reach memorization grows by ; CIFAR-10 provides supporting evidence. A frozen-feature model gives exact learning dynamics for individual feature directions. Its spectra and a controlled intervention support an explanation based on the variance entering the nonlinearity, although connecting this mechanism to memorization requires additional alignment assumptions. Spectral timing estimates calibrated on trained runs show dataset-dependent accuracy. These results identify latent width as a factor in memorization timing for vector-latent MLPs and motivate treating latent dimensionality as a variable when evaluating memorization and privacy risk in diffusion models. The mechanism in trained networks and transfer to other architectures remain unresolved.

Primary area:
generative models
Keywords:
diffusion models, memorization, generalization, latent diffusion, random features, random matrix theory, score matching, intrinsic dimension

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tiywOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether widening a latent representation delays memorization in diffusion model training. With a fixed signal distribution and fixed hidden width, wider synthetic latents reduce copying and delay empirical-score fitting. Vector-latent image experiments support this trend more strongly on CelebA than CIFAR-10. A frozen random-feature model suggests an explanation through changes in activation input scale and feature learning rates, but the connection to memorization in trained networks remains conditional. A spatial-DiT does not reproduce the trend.

Critical strengths

S1. The synthetic experiments address a real measurement problem. Adding noise coordinates can hide copies from the full-space nearest-neighbor test. After projection onto the signal subspace, copying still falls from 29.9% to 1.2%, against a 0.4% fresh-sample baseline. Calibrated signal copies and empirical-score alignment strengthen this finding (Tables 1-2).

S2. The spectral intervention is informative. Holding the activation input scale fixed reduces the endpoint growth of the signal-to-sample edge ratio from ~3.17 to ~1.09. The exact dynamics also clarify that feature directions learn in parallel.

S3. The experimental limitations are reported clearly. Reconstruction-identity checks, five-seed results, and the negative spatial-DiT pilot help establish where the observed effect holds.

Critical weaknesses

W1. The mechanism is not yet connected to memorization in the trained model. The surrogate uses frozen tanh features at one diffusion time, whereas the main network learns GELU features across diffusion times. At latent widths 5 and 40, first-layer pre-activation second moments evaluated at t=0.1t=0.1 are nearly equal at the first saved checkpoint (50k steps; Appendix C). These uncentered moments include bias and time embeddings; their later separation does not establish the proposed causal sequence. The fixed-scale intervention changes surrogate spectra and noise covariance, without testing its effect on copying in the trained MLP. The authors shall clarify what connects slower feature learning to copying, including whether analysis across diffusion times along the reverse sampling path is needed.

W2. Related work and novelty need a more precise comparison. Ye et al. [1] report less copying at higher dimensions with a fixed 12M-parameter model and an equivalent nearest-neighbor criterion. George and Macris [2] study asymptotic error and sample complexity for optimal random-feature scores; Achilli et al. [3] and Ross et al. [4] study geometric memorization and manifold dimensions. These provide relevant background, but do not reproduce the present training-delay intervention. George et al. [5] show that orthogonal contributions can dominate total test error at small diffusion times, obscuring the transition between memorization and generalization along the data subspace. This motivates directional diagnostics but differs from reduced NN-detector sensitivity.

W3. Transfer from synthetic nuisance coordinates to real latents remains unresolved. Synthetic excess coordinates are independent Gaussian noise; wider real latents may encode redundant functions of the same factors. Repeating signal coordinates can preserve average energy, while zero padding reduces clean-data energy per coordinate without adding clean-data nuisance noise. The effect therefore depends on scaling and covariance as well as width. Separately trained VAEs change both representation and decoder. The widest CelebA latents have worse FID than intermediate widths, measured against the diffusion training subset, leaving the delay at comparable quality on an independent reference set unclear (Appendix D).

W4. The timing predictor has limited independent validation. The main pressure predictor is fitted using the observed hitting times. The held-out-width evaluation concerns a different one-constant clock, which is worse than a dimension-only baseline on CelebA (log-time error 0.407 versus 0.125) and modestly better on CIFAR-10 (0.151 versus 0.198). Both evaluations include reconstruction-limited widths (Table 5). These results support an exploratory diagnostic, but do not independently validate the main predictor.

Questions for authors

Q1. Can an input-scale intervention in the trained MLP change signal-projected copying and empirical-score alignment while preserving the data construction?

Q2. What would establish the link between the spectral groups and generated-sample memorization? Does this require analysis across diffusion times along the reverse sampling path?

Q3. How do the statistics in Appendix G.6 support the proposed mechanism in the trained MLP? Would redundant coordinates produce the same delay, and does the image effect persist at matched quality against an independent reference set?

Q4. As a secondary robustness check, does the delay persist for signal dimensions greater than five under signal-projected detection and empirical-score alignment, beyond the RFNN spectral sweeps?

Q5. Can the main pressure predictor be evaluated on held-out widths above the reconstruction-identity threshold, against the dimension-only baseline under the same protocol?

References

[1] Zeqi Ye, Qijie Zhu, Molei Tao, and Minshuo Chen. Provable Separations between Memorization and Generalization in Diffusion Models. ICLR, 2026.

[2] Anand Jerry George and Nicolas Macris. Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data. arXiv:2603.22962, 2026.

[3] Beatrice Achilli, Enrico Ventura, Gianluigi Silvestri, Bao Pham, Gabriel Raya, Dmitry Krotov, Carlo Lucibello, and Luca Ambrogioni. Losing dimensions: Geometric memorization in generative diffusion. arXiv:2410.08727, 2024.

[4] Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse C. Cresswell, and Gabriel Loaiza-Ganem. A Geometric Framework for Understanding Memorization in Generative Models. ICLR, 2025.

[5] Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Denoising Score Matching with Random Features: Insights on Diffusion Models From Precise Learning Curves. AISTATS, PMLR 300:1405-1413, 2026.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tiyw (1/3): input-scale intervention and diffusion times

Thank you for questions we could act on. We ran the interventions behind Q1, Q3, Q4 and Q5, with success criteria fixed before each run. This response is in three parts.

Q1 / W1. Input-scale intervention in the trained MLP: no effect. We multiplied the network input by a fixed gain g, leaving data, loss, sampler and metrics unchanged. g = 0.45 gives d = 5 the d = 40 input variance; g = 2.2 does the reverse. Two seeds, 1M steps, against five-seed baselines.

conditionalpha at 1Mprojected copying at 1M
d = 5, g = 0.450.64515.4%
d = 5 baseline0.669 [0.649, 0.685]16.1% [15.2, 17.2]
d = 5, g = 2.20.69618.4%
d = 40, g = 0.450.2310.8%
d = 40 baseline0.243 [0.237, 0.249]0.7% [0.6, 0.7]
d = 40, g = 2.20.2010.5%

The gain closes 6% of the width gap in alpha at d = 5 and moves the wrong way at d = 40. Our criterion required movement beyond the seed spread in the predicted direction at both widths, and it fails. Input scale does not set memorization timing in the trained MLP, so the operating-point account describes the frozen-feature spectra but does not transfer. The revision says so in the abstract, Sec. 3 and Sec. 5. Two caveats: the gain also scales the diffusion noise, and under Adam it changes the first layer's effective step size.

W1 / Q2. Linking feature learning to copying; diffusion times. In the frozen model each readout mode relaxes at its eigenvalue, but reading the sample group as memorization requires the alignment weights of Prop. G.2, which we have not measured. In the trained network the measurable link is alpha at small t, which tracks projected copying across widths and over training. Across diffusion times (five seeds, 1M steps):

t0.010.030.10.31.0
alpha, d = 50.160.400.670.911.03
alpha, d = 400.050.130.240.280.54

The width ordering holds at every t. For t >= 0.3 the empirical and population scores nearly coincide in the signal coordinates, so alpha is uninformative there. We have not integrated alpha along the sampling path, and the final version will state the single-time analysis as a restriction.

On the early statistics: the App. C quantity was uncentered and included bias and time embedding. The centered variance from the latent input is not lower at d = 40 at initialization (0.081 vs 0.058 at d = 5) but is 2.6x lower by 50k steps. Alpha separates in the same window, so this does not establish causal order.

AuthorsOpenReview ↗

Response to Reviewer tiyw (2/3): mechanism controls, real latents, architecture

W3 / Q3. Redundant coordinates, and the mechanism in the trained MLP. We ran both constructions you describe at d = 40: "tile" (eight copies of the signal, per-coordinate energy equal to d = 5) and "zero pad" (35 exact zeros). Neither adds nuisance noise.

construction, 1M stepsalphaprojected copying
d = 5 baseline0.66916.1%
d = 40, noise (paper)0.2430.7%
d = 40, tile0.1304.5%
d = 40, zero pad0.1020.4% (chance)

Both delay copying, so the delay does not come from nuisance noise or per-coordinate energy. The frozen-feature model treats a tiled input as identical to d = 5, so it predicts no delay there. Redundant coordinates do delay copying, though not identically: by copying, tile sits between the two baselines.

Four further controls isolate the cause:

condition, 1M stepsalphaprojected copying
d = 5 + 35 noise input channels (not diffused)0.6817.0%
d = 5 + 35 unpredictable output targets0.66616.3%
zero pad + exact scores of the zero coordinates supplied by a fixed term0.69019.8%
zero pad0.1020.4%

Extra inputs alone or extra targets alone do nothing. Removing the work of denoising the extra coordinates removes the delay. The mechanism in the trained fixed-width MLP is therefore the load of denoising the extra coordinates on the shared hidden layers. Two properties:

  • Selective. The zero-padded network learns the population score in the signal subspace as fast as d = 5 (per-coordinate error 0.89 vs 1.09 at the first evaluation).
  • Capacity-dependent. Hidden width 768 or 2048 instead of 256 shrinks the d = 8 vs d = 40 alignment gap at 1M steps from 0.565 to 0.112 and 0.088. An onset delay remains at 2048 (threshold crossed at 0.15–0.20M steps at d = 40 vs 0.05M at d = 8).

This answers the first half of W1: the link to copying in the trained network is competition for a fixed-width trunk.

Real latents and architecture. On CelebA VAEs retrained from scratch (d = 80–200; five seeds, 2M steps):

denoisertime to 5% memorization, d = 80 / 120 / 160 / 200ratio 200/80memorized at 2M, d = 80 / 200
paper's MLP (SGD)0.30 / 0.57 / 0.84 / 1.16M3.870.670 / 0.182
same MLP, Adam0.20 / 0.48 / 0.68 / 1.27M6.350.882 / 0.242
small transformer, Adamwithin 50k steps in 19 of 20 runs1.49 (35k vs 52k)0.955 / 0.981

The MLP result replicates on independent VAEs and under Adam. The transformer shows a small onset shift and no plateau decline. On the synthetic zero-pad construction it has about a tenth of the MLP's alignment gap (0.065 vs 0.565), with or without adaLN or attention, while an MLP with a residual stream or LayerNorm keeps the delay (gaps 0.30–0.50). We do not attribute the difference to one component. The claim is scoped to fixed-width MLP denoisers in the revised title claim, abstract and conclusion.

Matched quality. You are right. Training-subset FID rewards copies, so it cannot settle this, and we did not run an independent-reference comparison. The revision states it as untested. App. G.6 is a surrogate diagnostic and does not support a mechanism in the trained MLP; its heading will say so.

AuthorsOpenReview ↗

Response to Reviewer tiyw (3/3): intrinsic dimension, held-out evaluation, related work

Q4. Signal dimension above five: the delay persists. Same MLP, 10-dimensional signal, two seeds, 1.5M steps, detection projected onto the 10 signal coordinates.

d (intrinsic 10)alphaprojected copyingfull-space copying
100.4321.9%21.9%
200.395.3%3.2%
400.320.3%0.1%
800.090.0%0.0%

Both readouts fall monotonically, by far more than the seed spread (below 0.01).

Q5 / W4. Held-out evaluation above the identity threshold.

held-out log error, 5% thresholdspectral clockdimension-only clock
CelebA, d >= 800.250.13
CIFAR-10, d >= 1600.140.13

The spectral clock does not beat the dimension-only baseline on either dataset. We have not yet evaluated the main pressure predictor under this protocol; the final version will report it, and we do not expect it to do better. The revision demotes the predictor to a diagnostic in the contribution list and the abstract.

W2. Related work. The revision adds a paragraph on each reference. Ye et al. [1] find less memorization at higher data dimension at the end of training with a fixed parameter count; they vary the data dimension itself and do not record onset time. The direction agrees with ours. George and Macris [2] and George et al. [5] study ridge-optimal random-feature scores, where memorization is governed by the feature-to-sample ratio and not by training time. We agree with your reading of [5]: our signal-projected test and alpha are directional diagnostics in that sense, and detector-sensitivity loss is a separate effect that the signal-copy control in Table 1 isolates. Achilli et al. [3] and Ross et al. [4] are static analyses that do not vary ambient dimension at fixed intrinsic dimension or study training time.

Summary. The empirical finding and its detector control stand, and now replicate on independent VAEs, under a second optimizer, and at intrinsic dimension 10. The frozen-feature interpretation does not transfer to the trained network. The interventions you asked for identify a different, tested mechanism in the MLP, and the claim is scoped to MLP denoisers. We would be grateful if you would reconsider the soundness and informativeness scores in that light.

Reviewer_w69UOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

On the one hand, diffusion models are trained on data in a latent space of dimension dlatentd_{\mathrm{latent}}, which is typically larger than the intrinsic dimension dintrinsicd_{\mathrm{intrinsic}} of the data distribution. On the other hand, diffusion models are known to memorize a fraction of their dataset (Somepalli et al. 2023) and its onset happens only after a large number of epochs (Bonnaire et al. 2025). The aim of this paper is to understand the effect of dlatentd_{\mathrm{latent}} on the onset of memorization during training.

In order to study this effect, the authors consider three different settings:

  • a synthetic dataset made of a Gaussian Mixture Model with 10 modes of dimension dintrinsic=5d_{\mathrm{intrinsic}}=5. They then add extra components with white noise until they reach dlatentd_{\mathrm{latent}}. The extended vector is then rotated by a random rotation matrix. They train an MLP at fixed hidden width on the score matching task for 5M steps and measure the fraction of memorized datapoints as well as the distance to the population score. They observe that increasing the latent dimension decreases the fraction of memorized datapoints.
  • The authors then focus on a tractable model: the random feature neural network as in Bonnaire et al. (2025). They study the spectrum of the empirical feature correlation matrix, which indicates the modes that are learned first. The authors observe empirically that the spectrum is composed of 4 parts: signal, noise-dimension, sample and tail modes. The authors argue that increasing dlatentd_{\mathrm{latent}} slows down the sample modes associated with memorization more than the signal modes associated with generalization.
  • The authors also do experiments on two real data sets (CelebA and CIFAR-10). To control the latent space, the authors train a VAE for each latent dimension and train an MLP to learn the score for 5M steps and report the fraction of memorized data points and FID of generated samples. They observe that increasing the latent dimension delays the onset of memorization.

Overall, this paper claims that increasing the latent dimension of the training samples delays the onset of memorization during training time.

Critical strengths

None

Critical weaknesses
  • The authors choose a fixed number of training steps (5M) both on synthetic data and on real images, for all values of the latent dimension. This seems arbitrary, and the number of steps needed to reach comparable performance might change across values of dlatentd_{\mathrm{latent}}. If the timescale needed to generalize also increases with dlatentd_{\mathrm{latent}}, then the proposed explanation is irrelevant. Likewise, the intrinsic dimension of the synthetic dataset is kept fixed to 5. Studying the effect of the intrinsic dimension would improve the message of this work. The only architecture studied is an MLP in the main text. In appendix E there are experiments on SOTA architectures which are not really discussed in the main text and which do not reproduce the behavior of the MLP.

  • The paper is extremely hard to read and understand. For instance, the authors write numerical values, sometimes to two decimal places, in every paragraph, which do not necessarily help the reader understand and make it hard to grasp the main message.

Questions for authors
  • l.38-39: "collapse during reverse sampling (Biroli et al., 2024) is distinct from the late-training memorization observed in some settings (Bonnaire et al., 2025)". What do the authors mean? In what sense is it different?
  • In the experiments, why stop at 5M steps? It would also be interesting to study the effect of the latent dimension on the timescales of generalization and memorization (cf. Bonnaire et al. 2025).
1 comment
AuthorsOpenReview ↗

Response to Reviewer w69U

Thank you for the review. We answer your two questions first.

Q1. Collapse during sampling vs. late-training memorization. These are two different time axes. Biroli et al. (2024) study diffusion time: when the score is the exact empirical score, reverse trajectories collapse onto a training point at a certain time. Bonnaire et al. (2025) study training time: a network first approximates the population score and approaches the empirical score only later. Our paper is about the second: how long training takes to reach the regime where sampling copies. The revision rewrites the sentence.

Q2. Why 5M steps, and how does d change the two timescales? 5M is where the narrowest latent reaches 30% memorization; its curve is still rising slowly (26.5% at 3M), which is why our claim is about timing and not final levels. We computed both clocks for every width. The generalization clock is the step at which the population score error (known exactly here) is lowest.

dstep of min. score errortime to 1% memorizedtime to 5% memorizedmemorized at 5M
50.05M (first evaluation)0.14M0.35M30.0%
100.05M0.30M0.72M17.1%
200.05M1.10Mnot by 5M3.7%
400.05Mnot by 5Mnot by 5M0.1%
800.15Mnot by 5Mnot by 5M0.0%
240still falling at 5Mnot by 5Mnot by 5M0.0%

In the main-text range the generalization clock does not move while the memorization clock grows by more than an order of magnitude, so the delay is not a uniform slowdown. Beyond d of about 80 both clocks lengthen, and the revision qualifies the "25x across the sweep" sentence, which mixed the two.

A new control gives a more direct answer. Padding the latent with 35 zero coordinates keeps copying at chance through 1M steps, yet the network learns the population score in the signal subspace as fast as at d = 5 (per-coordinate error 0.89 vs 1.09 at the first evaluation). Only the approach to the training set is delayed.

On images we cannot separate the clocks as cleanly, because training-subset FID rewards copies. The revision states this limitation.

Intrinsic dimension fixed at 5. We trained the same MLP on a 10-dimensional signal at d = 10, 20, 40, 80 (two seeds, 1.5M steps). Signal-projected copying falls from 21.9% to 5.3%, 0.3% and 0.0%, and alpha from 0.43 to 0.09, both monotonically. The delay persists.

Only an MLP; the DiT pilot does not reproduce the behavior. We agree, and we ran a cleaner test. On CelebA VAEs retrained from scratch (d = 80–200), five seeds, 2M steps:

denoisertime to 5% memorization, d = 80 / 200memorized at 2M, d = 80 / 200
paper's MLP (SGD)0.30M / 1.16M0.670 / 0.182
same MLP, Adam0.20M / 1.27M0.882 / 0.242
small transformer, Adam35k / 52k0.955 / 0.981

The MLP result replicates on new VAEs and under a second optimizer. The transformer memorizes almost immediately at every width. The delay is a property of fixed-width MLP denoisers, and the revision scopes the title claim, abstract and conclusion accordingly, with these results in the main text.

Relevance of the explanation. New interventions (details in our response to reviewer tiyw) show that the frozen-feature explanation does not hold in the trained MLP: rescaling the input has no effect. What does hold: the delay comes from having to denoise the extra coordinates with a fixed-width network. Supplying their exact scores through a fixed term restores narrow-latent timing, and widening the hidden layer from 256 to 2048 shrinks the gap from 0.565 to 0.088. The revision presents the frozen-feature analysis as a description of that model only.

Readability. We agree. The revision removes inline numbers from Secs. 2–4 except the headline results, moves the rest to tables, and uses one name per eigenvalue group.

Reviewer_Lq1MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether giving diffusion models extra latent dimensions makes them copy training examples less quickly. With hidden network width fixed, the authors find delayed copying in synthetic data and small image experiments. A signal-subspace control shows that the decline survives corrections for reduced detection sensitivity. The proposed explanation uses a random-feature model: extra dimensions change the scale entering the activation function, weakening nonlinear features and slowing their learning. A fixed-variance experiment partly supports this explanation. They also explore spectral estimates of copying times. The main contribution is a controlled empirical finding and a possible explanation, but currently limited to vector-latent MLP (the DiT experiments do not reproduce the trend).

Critical strengths

The raised phenomenon is interesting

Critical weaknesses
  1. The spatial-latent DiT experiments do not reproduce the trend. Positive results remain concentrated in small-data, vector-latent MLPs with fixed hidden width. Although the authors acknowledge this boundary, it substantially limits the finding’s usefulness for widely used diffusion models

  2. Limited theoretical novelty. [1] already includes variance-dependent nonlinear coefficients and spectral timescales. [2] establishes spectral learning dynamics. The added dimensionality intervention offers a relatively narrow theoretical advance.

  3. The mechanism remains unverified in trained networks. The fixed-variance intervention only tests a frozen random-feature model. In actual MLPs, early activation statistics are nearly identical across dimensions and separate later, leaving cause and effect unclear. Without directly intervening on these networks and measuring both spectra and copying, the proposed explanation remains speculative

[1] Why Diffusion Models Don’t Memorize: The Role of Implicit Dynamical Regularization in Training.

[2] An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Models

Questions for authors

none

1 comment
AuthorsOpenReview ↗

Response to Reviewer Lq1M

Thank you for the review. We answer your two questions first.

Q1. Collapse during sampling vs. late-training memorization. These are two different time axes. Biroli et al. (2024) study diffusion time: when the score is the exact empirical score, reverse trajectories collapse onto a training point at a certain time. Bonnaire et al. (2025) study training time: a network first approximates the population score and approaches the empirical score only later. Our paper is about the second: how long training takes to reach the regime where sampling copies. The revision rewrites the sentence.

Q2. Why 5M steps, and how does d change the two timescales? 5M is where the narrowest latent reaches 30% memorization; its curve is still rising slowly (26.5% at 3M), which is why our claim is about timing and not final levels. We computed both clocks for every width. The generalization clock is the step at which the population score error (known exactly here) is lowest.

dstep of min. score errortime to 1% memorizedtime to 5% memorizedmemorized at 5M
50.05M (first evaluation)0.14M0.35M30.0%
100.05M0.30M0.72M17.1%
200.05M1.10Mnot by 5M3.7%
400.05Mnot by 5Mnot by 5M0.1%
800.15Mnot by 5Mnot by 5M0.0%
240still falling at 5Mnot by 5Mnot by 5M0.0%

In the main-text range the generalization clock does not move while the memorization clock grows by more than an order of magnitude, so the delay is not a uniform slowdown. Beyond d of about 80 both clocks lengthen, and the revision qualifies the "25x across the sweep" sentence, which mixed the two.

A new control gives a more direct answer. Padding the latent with 35 zero coordinates keeps copying at chance through 1M steps, yet the network learns the population score in the signal subspace as fast as at d = 5 (per-coordinate error 0.89 vs 1.09 at the first evaluation). Only the approach to the training set is delayed.

On images we cannot separate the clocks as cleanly, because training-subset FID rewards copies. The revision states this limitation.

Intrinsic dimension fixed at 5. We trained the same MLP on a 10-dimensional signal at d = 10, 20, 40, 80 (two seeds, 1.5M steps). Signal-projected copying falls from 21.9% to 5.3%, 0.3% and 0.0%, and alpha from 0.43 to 0.09, both monotonically. The delay persists.

Only an MLP; the DiT pilot does not reproduce the behavior. We agree, and we ran a cleaner test. On CelebA VAEs retrained from scratch (d = 80–200), five seeds, 2M steps:

denoisertime to 5% memorization, d = 80 / 200memorized at 2M, d = 80 / 200
paper's MLP (SGD)0.30M / 1.16M0.670 / 0.182
same MLP, Adam0.20M / 1.27M0.882 / 0.242
small transformer, Adam35k / 52k0.955 / 0.981

The MLP result replicates on new VAEs and under a second optimizer. The transformer memorizes almost immediately at every width. The delay is a property of fixed-width MLP denoisers, and the revision scopes the title claim, abstract and conclusion accordingly, with these results in the main text.

Relevance of the explanation. New interventions (details in our response to reviewer tiyw) show that the frozen-feature explanation does not hold in the trained MLP: rescaling the input has no effect. What does hold: the delay comes from having to denoise the extra coordinates with a fixed-width network. Supplying their exact scores through a fixed term restores narrow-latent timing, and widening the hidden layer from 256 to 2048 shrinks the gap from 0.565 to 0.088. The revision presents the frozen-feature analysis as a description of that model only.

Readability. We agree. The revision removes inline numbers from Secs. 2–4 except the headline results, moves the rest to tables, and uses one name per eigenvalue group.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.