- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyze the two-phase behavior of over-parameterized diffusion models trained on small datasets (early generalization, late memorization). The authors look at the direction of the denoiser output and decompose it into two components and (and d_res). d_pop is the gap between and , while is a posteria mean which is the model supposed to learn. Since the actual training target is minimizing loss over data points, at the beginning. goes toward and, as the training progresses, it goes to .
The authors project onto the plane spanned by and . So, the projection of the is represented . For , the authors use analytical solution for the synthetic toy model, and surrogate methods for real data.
The authors empirically show that during generalization phase, grows from 0 to almost 1 while stays close to 0. And at some point, as phase 2 kicks in stays as a constant and grows to 1.
The authors attempt to explain why the sudden change happens by analyzing the learning dynamics.
The novelty of the paper is a cleaner quantitative picture of a known phenomenon, plus a regularizer built on it, with the explanation of the phenomenon left roughly where prior work had it.
Critical strengths
-
A simple, well-defined geometric coordinate system that reveals a consistent, sharp phenomenon. The decomposition is cheap to compute, and the near-right-angle turn appears in four architectures (MLP, DDPM U-Net, EDM, JiT) on six datasets, across a wide range of evaluation noise levels (Fig. 7), with the residual quickly becoming negligible so that the two coordinates capture nearly all of the motion. The large-N control (Fig. 8) shows that the turn disappears with N = 50,000 (the right sanity check) and makes the finite-data interpretation convincing.
-
DAO is a direct consequence of the geometry, has two untuned hyperparameters, and produces large effects. Memorization ratio drops from 16–82% to 0.04–4.1% while best FID improves in every setting, and Fig. 4b shows a_k is essentially unchanged under DAO, which is direct evidence that the penalty is selective as claimed rather than slowing all learning.
Critical weaknesses
-
The theoretical section mostly restates the empirical observation rather than explaining it. The authors try to explain the phase changes with partial success. Lemma 1 is the Pythagorean identity for nested conditional expectations ( is a mean-zero innovation given ) and holds for any Bayes estimator, not specifically for diffusion models. Theorem 1 follows by integrating the exact coordinate ODEs in Lemma 3 under Assumption 1, but Assumption 1 directly posits what should be explained: is by definition , and Assumption 1(ii) literally contains both conclusions of part (ii) ($|1 − a_\tau| \le \epsilon $ and the sign condition for ). The are measured along the trajectory, not derived from any property of the data or architecture, so the result does not say why .
-
The DAO evaluation has no baselines beyond vanilla EDM and no ablations, and the analysis is global while the method is per-sample. Not compared: explicit early stopping (which is effectively the surrogate itself), score regularization (Baptista et al.), noisy-data training (Shah et al.), limited-data augmentation methods (Zhang et al.), standard regularizers, or the most informative control: an isotropic proximity penalty (self-distillation to the early checkpoint) that would test whether direction-awareness matters at all.
Questions for authors
Clarifications.
-
Please report FID and memorization ratio at the same checkpoint, both at the best-FID checkpoint and at the final one. Does DAO’s FID at 30M images still beat the baseline’s best FID?
-
What reference statistics does FID use, both for Table 1 and for selecting the surrogate? If the full dataset is used, can you show DAO with a selection rule available in a true limited-data setting (held-out denoising loss on a small split, or a fixed fraction of the budget)? How sensitive is DAO to the surrogate choice of the selected iteration?
-
Ablations: (a) the isotropic penalty; (b) the simpler form without the term, with numbers; (c) sensitivity to and . What fraction of the gain survives in (a)? This is the key test of whether orthogonality is doing the work.
-
Per-input analysis. Please report the bank size M, the null cosine scale (or a shuffled-pairing baseline), the distribution of per-input cosines , and a 2D histogram of per-input () at a few checkpoints. Do all inputs turn together, or do some (outliers, low-density samples) memorize first?
3 comments
I appreciate the all efforts the authors put in the rebuttal. I like the new theoretical framework (ne Theorem 1). Also, the extended evaluations also help me to convince the validity of the paper. Therefore, I raise my rating.
We are glad to hear that our rebuttal helps to address the reviewer's concerns. We would like to sincerely thank the reviewer once again for the response and thoughtful review, which has motivated us to think more comprehensively about our work and further strengthen both its theoretical and empirical aspects.
Rebuttal Follow-up
Dear Reviewer MvC3,
We would like to once again sincerely thank you for your time and thoughtful feedback on our work.
As we approach the end of the rebuttal discussion period, we would appreciate hearing whether our response has addressed your concerns. We would also be happy to address any further questions or concerns whenever your schedule permits.
Thank you again for your time and consideration.