acceptodds
Under review as a conference paper at ICLR 2027

Orthogonality of Generalization and Memorization Trajectories in Diffusion Models

Abstract

Training over-parameterized diffusion models with limited data typically exhibits a two-phase phenomenon in which the model generalizes early in training before progressively memorizing the training data. Although this qualitative behavior has been observed and studied in many recent works, a precise quantitative characterization of the geometry and dynamics underlying this transition is still lacking. In this paper, we characterize this transition through the geometry of the denoiser outputs. We show that the training trajectory can be decomposed into two directions associated with the population and empirical distributions, which are orthogonal in expectation. Across synthetic and real-world datasets, this decomposition reveals a quantitative phase-transition phenomenon with the denoising trajectory: the trajectory first moves predominantly toward the population Bayes-optimal denoiser, then it makes a sharp, nearly orthogonal turn toward the empirical Bayes-optimal denoiser as memorization emerges. We further study this behavior through the spectrum of the denoiser output-space training kernel, observing that the population direction is learned through substantially faster kernel modes than the empirical direction. Motivated by this separation, we propose a Direction-Aware Orthogonal (DAO) regularization, which selectively penalizes denoising motion along an estimated empirical direction. Experiments show that DAO is highly effective for mitigating memorization while further improving generation quality.

Primary area:
generative models
Keywords:
diffusion models, memorization, generalization, training dynamics

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_MvC3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyze the two-phase behavior of over-parameterized diffusion models trained on small datasets (early generalization, late memorization). The authors look at the direction of the denoiser output and decompose it into two components dpopd_{pop} and dempd_{emp} (and d_res). d_pop is the gap between ϕpop\phi_{pop} and ϕθ0\phi_{\theta_0}, while ϕpop\phi_{pop} is a posteria mean which is the model supposed to learn. Since the actual training target is minimizing loss over data points, at the beginning. ϕθK\phi_{\theta_K} goes toward ϕpop\phi_{pop} and, as the training progresses, it goes to ϕemp\phi_{emp}.

The authors project \dθk\d_{\theta_k} onto the plane spanned by dpopd_{pop} and dempd_{emp}. So, the projection of the \dθk\d_{\theta_k} is represented ak×dpop+bk×dempa_k\times d_{pop} + b_k \times d_{emp}. For dpopd_{pop}, the authors use analytical solution for the synthetic toy model, and surrogate methods for real data.

The authors empirically show that during generalization phase, aka_k grows from 0 to almost 1 while bkb_k stays close to 0. And at some point, as phase 2 kicks in aka_k stays as a constant and bkb_k grows to 1.

The authors attempt to explain why the sudden change happens by analyzing the learning dynamics.

The novelty of the paper is a cleaner quantitative picture of a known phenomenon, plus a regularizer built on it, with the explanation of the phenomenon left roughly where prior work had it.

Critical strengths
  1. A simple, well-defined geometric coordinate system that reveals a consistent, sharp phenomenon. The decomposition is cheap to compute, and the near-right-angle turn appears in four architectures (MLP, DDPM U-Net, EDM, JiT) on six datasets, across a wide range of evaluation noise levels (Fig. 7), with the residual quickly becoming negligible so that the two coordinates capture nearly all of the motion. The large-N control (Fig. 8) shows that the turn disappears with N = 50,000 (the right sanity check) and makes the finite-data interpretation convincing.

  2. DAO is a direct consequence of the geometry, has two untuned hyperparameters, and produces large effects. Memorization ratio drops from 16–82% to 0.04–4.1% while best FID improves in every setting, and Fig. 4b shows a_k is essentially unchanged under DAO, which is direct evidence that the penalty is selective as claimed rather than slowing all learning.

Critical weaknesses
  1. The theoretical section mostly restates the empirical observation rather than explaining it. The authors try to explain the phase changes with partial success. Lemma 1 is the Pythagorean identity for nested conditional expectations (E[x0∣xt,DN]−E[x0∣xt]E[x0 | x_t, D_N] − E[x0 | x_t] is a mean-zero innovation given xtx_t) and holds for any Bayes estimator, not specifically for diffusion models. Theorem 1 follows by integrating the exact coordinate ODEs in Lemma 3 under Assumption 1, but Assumption 1 directly posits what should be explained: ρ>0\rho > 0 is by definition a˙>0\dot{a} > 0, and Assumption 1(ii) literally contains both conclusions of part (ii) ($|1 − a_\tau| \le \epsilon $ and the sign condition for b˙\dot{b}). The Kpop,Kemp,β,ρK_{pop}, K_{emp}, \beta, \rho are measured along the trajectory, not derived from any property of the data or architecture, so the result does not say why Kpop≫KempK_{pop} \gg K_{emp}.

  2. The DAO evaluation has no baselines beyond vanilla EDM and no ablations, and the analysis is global while the method is per-sample. Not compared: explicit early stopping (which is effectively the surrogate itself), score regularization (Baptista et al.), noisy-data training (Shah et al.), limited-data augmentation methods (Zhang et al.), standard regularizers, or the most informative control: an isotropic proximity penalty ∣ϕθ−ϕ^pop∣2|\phi_{\theta} - \hat{\phi}_{pop}|^2 (self-distillation to the early checkpoint) that would test whether direction-awareness matters at all.

Questions for authors

Clarifications.

  1. Please report FID and memorization ratio at the same checkpoint, both at the best-FID checkpoint and at the final one. Does DAO’s FID at 30M images still beat the baseline’s best FID?

  2. What reference statistics does FID use, both for Table 1 and for selecting the surrogate? If the full dataset is used, can you show DAO with a selection rule available in a true limited-data setting (held-out denoising loss on a small split, or a fixed fraction of the budget)? How sensitive is DAO to the surrogate choice of the selected iteration?

  3. Ablations: (a) the isotropic penalty; (b) the simpler form without the η\eta term, with numbers; (c) sensitivity to cpenc_{pen} and η0\eta_0. What fraction of the gain survives in (a)? This is the key test of whether orthogonality is doing the work.

  4. Per-input analysis. Please report the bank size M, the null cosine scale (or a shuffled-pairing baseline), the distribution of per-input cosines cos⁡(dpop(j),demp(j))\cos(d_{pop}^{(j)}, d_{emp}^{(j)}), and a 2D histogram of per-input (ak(j),bk(j)a_k^{(j)}, b_k^{(j)}) at a few checkpoints. Do all inputs turn together, or do some (outliers, low-density samples) memorize first?

3 comments
AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer MvC3,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

I appreciate the all efforts the authors put in the rebuttal. I like the new theoretical framework (ne Theorem 1). Also, the extended evaluations also help me to convince the validity of the paper. Therefore, I raise my rating.

We are glad to hear that our rebuttal helps to address the reviewer's concerns. We would like to sincerely thank the reviewer once again for the response and thoughtful review, which has motivated us to think more comprehensively about our work and further strengthen both its theoretical and empirical aspects.

Reviewer_wLgLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the transition from generalization to memorization in diffusion models by decomposing denoiser-output training trajectories into population and empirical directions. The authors establish orthogonality in expectation, analyze the dynamics using an output-space training kernel, and propose DAO regularization to mitigate memorization. Experiments across multiple architectures and datasets support the geometric characterization, while DAO improves generation quality and substantially reduces memorization in limited-data settings.

Critical strengths

I found that this paper is interesting and was well written. Two points are consider as critical strengths:

  • The geometric characterization offers an interesting perspective on diffusion training dynamics. The nearly orthogonal trajectories are observed across different architectures, datasets, and noise levels, with additional synthetic experiments supporting the theoretical assumptions.
  • DAO achieves substantial reductions in memorization while consistently improving best FID across all four evaluated settings. The qualitative comparisons further support its effectiveness
Critical weaknesses
  • My first concern is theoretical explanation limitation in theorem 1. Although the proofs appear sound under the stated assumptions, Theorem 1 relies on conditions that already impose key aspects of the two-stage behavior. Appendix C validates these conditions in a synthetic setting, but the theorem remains a conditional characterization rather than a general explanation of why the trajectory emerges.
  • I think that there is an insufficient ablation of DAO. Appendix D.1 acknowledges that a simpler directional penalty also improves generation quality, however, there is no quantitative comparison is provided. In such the case, without comparisons against this simpler variant and basic early-reference regularization, the specific benefit of DAO's proposed design remains unclear.
Questions for authors

I have following questions: (1) Could the authors clarify what Theorem 1 establishes beyond Assumption 1, and whether its assumptions can be justified for practical diffusion architectures? (2) Could the authors provide quantitative comparisons between full DAO, the simpler directional penalty, and a basic early-reference regularizer? (3) Table 1 reports best FID and final memorization ratio at potentially different checkpoints. Could the authors report both metrics at the same checkpoints, particularly those achieving the best FID?

1 comment
AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer wLgL,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

Reviewer_Hc93OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the transition from generalization to memorization in diffusion-model training through the geometry of denoiser outputs. The authors decompose the learned denoiser displacement into a population direction, pointing toward the population Bayes-optimal denoiser, an empirical direction, pointing from the population to the empirical Bayes-optimal denoiser, and an orthogonal residual. They show that the population and empirical directions are orthogonal in expectation and empirically observe a characteristic two-stage trajectory: training first progresses predominantly toward the population solution and subsequently makes a sharp turn toward the empirical solution as memorization develops. The paper further analyzes this behavior through directional spectral properties of the denoiser output-space training kernel and proposes Direction-Aware Orthogonal (DAO) regularization, which selectively penalizes motion toward the empirical solution. Experiments across synthetic data and several image datasets and architectures show consistent trajectory geometry, while DAO substantially reduces measured memorization and improves FID in the studied limited-data regimes.

Critical strengths
  1. The empirical training-trajectory phenomenon is interesting and remarkably consistent. The strongest contribution, in my view, is the observation that diffusion-model optimization organizes around the population and empirical directions in a clean two-stage manner. Across several datasets and architectures, the residual rapidly becomes small, the model first approaches the population solution, and only later progresses substantially toward the empirical solution. This provides an intuitive geometric characterization of the generalization-to-memorization transition.

  2. The paper connects the geometric observation to a practical intervention. DAO follows naturally from the proposed decomposition rather than being an unrelated regularization technique, and infact is a good way to probe the hypothesis. The reported reductions in memorization are substantial, while generation quality improves rather than simply trading off against memorization. I also appreciate the breadth of the empirical evaluation and the attempt to connect the observed trajectory to directional spectral rates in the output-space kernel.

Critical weaknesses
  1. The orthogonality result itself appears to be a classical conditional-expectation/Hilbert-space fact, and its novelty should be positioned more carefully. The empirical and population Bayes-optimal denoisers are nested conditional expectations, ϕemp=E[x0∣xt,DN]\phi_{\mathrm{emp}}=\mathbb{E}[x_0\mid x_t,D_N] and ϕpop=E[x0∣xt]\phi_{\mathrm{pop}}=\mathbb{E}[x_0\mid x_t]. The fact that the additional conditional-expectation innovation is orthogonal in L2L^2 to quantities measurable with respect to the coarser sigma-algebra is a standard projection property of conditional expectation. Thus, I do not view the orthogonality in Lemma 1 by itself as a diffusion-specific theoretical discovery. The more interesting contribution is that actual diffusion-model optimization follows these population/empirical projection directions in a pronounced two-stage trajectory. I think the paper should distinguish these aspects more clearly.

  2. The dependence on finite training-set size NN is central to the interpretation but is not sufficiently characterized. Since the empirical distribution converges toward the population distribution as NN grows, one expects the empirical Bayes denoiser to approach the population Bayes denoiser under suitable conditions, making demp=ϕemp−ϕpopd_{\mathrm{emp}}=\phi_{\mathrm{emp}}-\phi_{\mathrm{pop}} small. Orthogonality can formally persist while this vector vanishes, but then the normalized angle alone becomes less informative. The appendix provides an encouraging synthetic comparison between N=256N=256 and N=50,000N=50{,}000, where the empirical-turning phase largely disappears, but a two-point comparison does not characterize the dependence on NN. A systematic empirical study or theoretical characterization of ∥demp∥\|d_{\mathrm{emp}}\|, the magnitude of displacement along this direction would substantially strengthen the interpretation. I would also suggest looking into the onset of the empirical-turning phase as a function of NN .

Questions for authors

See weaknesses. Upon addressing point 2 I would positively consider raising the score. Disproving point 2 it may also work.

4 comments
AuthorsOpenReview ↗

Rebuttal [1/2]

We thank the reviewer for the clear summary of our work and the positive assessment of our geometric findings and proposed method. We appreciate the constructive feedback on the scope of Lemma 1 and the suggested analysis of the dependence on training-set size, both of which have helped us strengthen the paper. We address these points below.

The orthogonality result itself appears to be a classical conditional-expectation/Hilbert-space fact, and its novelty should be positioned more carefully. The empirical and population Bayes-optimal denoisers are nested conditional expectations, ϕemp=E[x0∣xt,DN]\phi_{\mathrm{emp}} = \mathbb{E}[x_0 \mid x_t, D_N] and ϕpop=E[x0∣xt]\phi_{\mathrm{pop}} = \mathbb{E}[x_0 \mid x_t]. The fact that the additional conditional-expectation innovation is orthogonal in L2L^2 to quantities measurable with respect to the coarser sigma-algebra is a standard projection property of conditional expectation. Thus, I do not view the orthogonality in Lemma 1 by itself as a diffusion-specific theoretical discovery. The more interesting contribution is that actual diffusion-model optimization follows these population/empirical projection directions in a pronounced two-stage trajectory. I think the paper should distinguish these aspects more clearly.

Answer: We agree with the reviewer. The orthogonality in Lemma 1 follows from a standard property of conditional expectation, and we do not claim the identity itself as a theoretical contribution. As also pointed out by the reviewer, our contribution is to use the resulting population–empirical coordinate system to study actual diffusion-model optimization, reveal its pronounced two-stage trajectory, and use this geometry to motivate DAO. We have revised the paper to make this positioning explicit (see page 2, line 80 and page 4, line 200-202).

AuthorsOpenReview ↗

Rebuttal Follow-up

Dear Reviewer Hc93,

We would like to once again sincerely thank you for your time and thoughtful feedback on our work.

As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.

Thank you again for your time and consideration.

Thank you for the detailed response as well as extension of Appendix B.5. I will raise my score accordingly

We appreciate the reviewer's response and would like to thank them once again for their valuable feedback, which has encouraged us to think more deeply about our work and helped us improve the clarity and completeness of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.