- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper try to explain why equivariant diffusion might generalize better by looking at training time. It sets up two random-feature denoisers with the same frozen first layer, one with plain features and one with features averaged over a symmetry group, and trains only the readout on the same projected target. The main claim is selective delay. Averaging slows down the low-signal block of features a lot more than the high-signal block (S), so the ratio τ_mem/τ_gen goes up. I think it is a qualified paper overall.
Critical strengths
The author acknowledges some limitations of the methods, which is Sincerely. Additionally, asking how symmetry changes when things get learned, instead of how many samples you need, is a nice angle, and the matched Base/Avg setup makes it easy to see what averaging alone does. The numbers also provide support for this story. For C2, S only slows by 6.2 to 10.4% while N takes twice as long, so the ratio grows by 1.81 to 1.88, and all 90 matched controls show no gain at all.
Critical weaknesses
-
The main exact results is proved on the population loss, where there is no training set to memorize. So the "memorization time" is really just when the weaker part of the true score gets learned. The conclusion does say the link to memorization is open, but the main text still calls these the start of generalization and memorization. Apart from the QM9 run in the appendix, is there any setting where the wider window shows up as better held-out error or later copying?
-
The theory need a nonzero target in the fixed subspace of the group. For rotations of 3D coordinates that subspace is just zero, so models like EDM or EGNN, which is what most people mean by equivariant diffusion, aren't covered. Can the result be extended to equivariant outputs, or should the title be narrowed?
-
The experiments mostly check the theorems on hand-built synthetic examples. The training set size is never varied, even though that's how Bonnaire et al. tie the late phase to memorization. The only real-data check is QM9 in the Appendix, where the generalization time isn't really resolved. The GNN-Dist and EGNN-NoCoord controls help, but they don't pass the quality check either, so the symmetry effect is still hard to separate. I think a cleaner real-data experiment belongs in the main text.
-
How invariance thins out the kernel spectrum and what that buys in sample complexity has already been worked out, for example by Tahmasebi & Jegelka, and slower learning along small-eigenvalue directions is the usual spectral bias story (Bordelon et al., Wang & Pehlevan). In the cyclic case the slow block is put in the group-moved directions by design, so the result follows pretty directly. It would help to spell out what is new beyond putting these together.
References
[1] T. Bonnaire, R. Urfin, G. Biroli, M. Mézard. Why diffusion models don't memorize: The role of implicit dynamical regularization in training. NeurIPS 2025.
[2] B. Tahmasebi, S. Jegelka. The exact sample complexity gain from invariances for kernel regression. NeurIPS 2023.
[3] B. Bordelon, A. Canatar, C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. ICML 2020.
[4] E. Hoogeboom, V. G. Satorras, C. Vignac, M. Welling. Equivariant diffusion for molecule generation in 3D. ICML 2022.
[5] V. G. Satorras, E. Hoogeboom, M. Welling. E(n) equivariant graph neural networks. ICML 2021.
[6] B. Wang, C. Pehlevan. An analytical theory of spectral bias in the learning dynamics of diffusion models. NeurIPS 2025.
Questions for authors
Please see the weaknesses.
Response to Reviewer M1no
We thank the reviewer for the careful reading and constructive comments.
W1. Is the analysis based on population loss, and is there evidence of better held-out error or later copying beyond QM9?
Response. Our theory analyzes training loss on a fixed dataset, averaging only over diffusion noise. The joint limit n, p, d → ∞ gives a tractable spectral approximation of the training Gram matrix. If only n → ∞ with p, d fixed, training moments converge to population moments. With fixed aspect ratios in the joint limit, however, a deterministic difference between the approximate training and test spectra remains (Bonnaire et al., 2025, Section 3).
Our "gen" and "mem" labels follow Bonnaire et al.'s interpretation of the two separated spectral components (Theorem 3.2). The "mem" time measures when the corresponding score component is learned. Precise evidence linking these fitting times to statistical generalization and training-sample memorization is still lacking.
Beyond QM9, we will add more extensive experiments in the final version.
W2. Can the result cover EDM/EGNN vector scores, or should its scope be narrowed?
Response. Under standard 3D rotations, the fixed-vector subspace is zero, so the projected-target theorem does not cover EDM/EGNN vector-score outputs. We will state this restriction in the title and scope. Possible extensions include vector-valued equivariant features and equivariant averaging:
fG(x)=∫Gg−1f(gx)dμ(g),
where μ is the normalized Haar measure. These allow nonzero equivariant outputs; extending the timing result requires a new analysis of their Gram operators and target contributions, which we leave to future work.
W3. Has training-set size been varied, do the controls pass the quality check, and can real-data results be included in the main text?
Response. Section 6.3 reports the QM9 comparison of EGNN and a raw-coordinate GNN, trained on 4,096 molecules for 500k steps with three paired seeds. We evaluate held-out noise-prediction loss, generation quality, and copying of training molecules.
QM9 summary across three paired seeds
Here k denotes 1,000 training steps; >500k is right-censored. Generation crossings occur at the first evaluation, leaving earlier onset unresolved. Paired ratios compare EGNN with the raw GNN; lower quality ratios are better.
EGNN shows later copying in all three pairs while passing the best-quality criterion. The scale study covers 24/48 planned runs: at N = 8192, two of three pairs favor EGNN, but the quality ratio is 1.333 and fails the criterion. The EGNN/GNN-Dist and EGNN-NoCoord/raw-GNN controls at N = 4096 also fail (1.199 and 1.444).
W4. What is new beyond invariant-kernel theory and spectral bias, given that the cyclic example is designed to delay the weak block?
Response. We share the spectral viewpoint of prior work: invariance modifies the kernel spectrum, and smaller eigenvalues imply slower fitting. Our contribution is to jointly analyze how nonlinear feature averaging changes the training operator and target coupling, while matching learning directions across models.
First, we derive the operator change from the group action. A group-averaged Hermite–Mehler expansion computes covariances between transformed nonlinear features; the orbit-variance identity identifies the variation removed by averaging. For finite data, a conditional approximation preserves the sample-dependent zeroth and first noise-Hermite terms exactly and controls higher-order fluctuations.
Second, we match target-relevant subspaces across operators. Averaging can rotate eigenspaces, so eigenvalues alone cannot identify corresponding learning directions. We combine standard eigenvalue and projector perturbation bounds with control of target contributions and overlap with common reference subspaces, preserving block interpretation under approximation error. The cyclic example fixes alignment for an explicit covariance calculation; the general analysis also handles eigenspace rotation and finite-sample approximation error.
We will add these discussions in the final version.
Reference
T. Bonnaire, R. Urfin, G. Biroli, and M. Mézard. (2025). Why diffusion models don't memorize: The role of implicit dynamical regularization in training. NeurIPS 2025. arXiv:2505.17638.