acceptodds
Under review as a conference paper at ICLR 2027

Equivariant Diffusion can have a relatively larger generalization regime than non-equivariant one

Abstract

Equivariant diffusion models use known symmetries to guide denoising, but how symmetry shapes the course of learning remains unclear. Understanding this effect is important for explaining when training favors useful structure over less relevant variation. This paper identifies selective delay as a mechanism by which symmetry separates these stages of learning.Averaging over symmetry-related inputs can slow the learning of less relevant variation more strongly than the learning of the target's main structure. This increases the relative separation between the two stages,even when both become slower. We establish this mechanism in a tractable denoising model, characterize when it produces a wider learning regime under discrete and continuous symmetries, and verify its predictions numerically. These results connect symmetry to the timing of learning and provide a principle for choosing symmetries that favor task-relevant structure early in training.

Primary area:
learning theory
Keywords:
Diffusion Models, Equivariant Representation Learning, Group Averaging, Random Features, Learning Dynamics, Spectral Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M1noOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper try to explain why equivariant diffusion might generalize better by looking at training time. It sets up two random-feature denoisers with the same frozen first layer, one with plain features and one with features averaged over a symmetry group, and trains only the readout on the same projected target. The main claim is selective delay. Averaging slows down the low-signal block of features a lot more than the high-signal block (S), so the ratio τ_mem/τ_gen goes up. I think it is a qualified paper overall.

Critical strengths

The author acknowledges some limitations of the methods, which is Sincerely. Additionally, asking how symmetry changes when things get learned, instead of how many samples you need, is a nice angle, and the matched Base/Avg setup makes it easy to see what averaging alone does. The numbers also provide support for this story. For C2, S only slows by 6.2 to 10.4% while N takes twice as long, so the ratio grows by 1.81 to 1.88, and all 90 matched controls show no gain at all.

Critical weaknesses
  1. The main exact results is proved on the population loss, where there is no training set to memorize. So the "memorization time" is really just when the weaker part of the true score gets learned. The conclusion does say the link to memorization is open, but the main text still calls these the start of generalization and memorization. Apart from the QM9 run in the appendix, is there any setting where the wider window shows up as better held-out error or later copying?

  2. The theory need a nonzero target in the fixed subspace of the group. For rotations of 3D coordinates that subspace is just zero, so models like EDM or EGNN, which is what most people mean by equivariant diffusion, aren't covered. Can the result be extended to equivariant outputs, or should the title be narrowed?

  3. The experiments mostly check the theorems on hand-built synthetic examples. The training set size is never varied, even though that's how Bonnaire et al. tie the late phase to memorization. The only real-data check is QM9 in the Appendix, where the generalization time isn't really resolved. The GNN-Dist and EGNN-NoCoord controls help, but they don't pass the quality check either, so the symmetry effect is still hard to separate. I think a cleaner real-data experiment belongs in the main text.

  4. How invariance thins out the kernel spectrum and what that buys in sample complexity has already been worked out, for example by Tahmasebi & Jegelka, and slower learning along small-eigenvalue directions is the usual spectral bias story (Bordelon et al., Wang & Pehlevan). In the cyclic case the slow block is put in the group-moved directions by design, so the result follows pretty directly. It would help to spell out what is new beyond putting these together.

References

[1] T. Bonnaire, R. Urfin, G. Biroli, M. Mézard. Why diffusion models don't memorize: The role of implicit dynamical regularization in training. NeurIPS 2025.

[2] B. Tahmasebi, S. Jegelka. The exact sample complexity gain from invariances for kernel regression. NeurIPS 2023.

[3] B. Bordelon, A. Canatar, C. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. ICML 2020.

[4] E. Hoogeboom, V. G. Satorras, C. Vignac, M. Welling. Equivariant diffusion for molecule generation in 3D. ICML 2022.

[5] V. G. Satorras, E. Hoogeboom, M. Welling. E(n) equivariant graph neural networks. ICML 2021.

[6] B. Wang, C. Pehlevan. An analytical theory of spectral bias in the learning dynamics of diffusion models. NeurIPS 2025.

Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer M1no

We thank the reviewer for the careful reading and constructive comments.

W1. Is the analysis based on population loss, and is there evidence of better held-out error or later copying beyond QM9?

Response. Our theory analyzes training loss on a fixed dataset, averaging only over diffusion noise. The joint limit n, p, d → ∞ gives a tractable spectral approximation of the training Gram matrix. If only n → ∞ with p, d fixed, training moments converge to population moments. With fixed aspect ratios in the joint limit, however, a deterministic difference between the approximate training and test spectra remains (Bonnaire et al., 2025, Section 3).

Our "gen" and "mem" labels follow Bonnaire et al.'s interpretation of the two separated spectral components (Theorem 3.2). The "mem" time measures when the corresponding score component is learned. Precise evidence linking these fitting times to statistical generalization and training-sample memorization is still lacking.

Beyond QM9, we will add more extensive experiments in the final version.

W2. Can the result cover EDM/EGNN vector scores, or should its scope be narrowed?

Response. Under standard 3D rotations, the fixed-vector subspace is zero, so the projected-target theorem does not cover EDM/EGNN vector-score outputs. We will state this restriction in the title and scope. Possible extensions include vector-valued equivariant features and equivariant averaging:

fG(x)=∫Gg−1f(gx) dμ(g),f_G(x) = \int_G g^{-1} f(gx)\,d\mu(g),

where μ is the normalized Haar measure. These allow nonzero equivariant outputs; extending the timing result requires a new analysis of their Gram operators and target contributions, which we leave to future work.

W3. Has training-set size been varied, do the controls pass the quality check, and can real-data results be included in the main text?

Response. Section 6.3 reports the QM9 comparison of EGNN and a raw-coordinate GNN, trained on 4,096 molecules for 500k steps with three paired seeds. We evaluate held-out noise-prediction loss, generation quality, and copying of training molecules.

QM9 summary across three paired seeds

MeasureEGNNRaw-coordinate GNN
Mean test loss at 500k steps0.2290.303
Generation time10k for all seeds10k for all seeds
Copying time (seeds 0/1/2)420k / 440k / >500k400k / 360k / 150k
Median paired copying delay80k steps; median window ratio 1.222—
Mean paired best-quality ratio0.797 (criterion ≤ 1.1)—

Here k denotes 1,000 training steps; >500k is right-censored. Generation crossings occur at the first evaluation, leaving earlier onset unresolved. Paired ratios compare EGNN with the raw GNN; lower quality ratios are better.

EGNN shows later copying in all three pairs while passing the best-quality criterion. The scale study covers 24/48 planned runs: at N = 8192, two of three pairs favor EGNN, but the quality ratio is 1.333 and fails the criterion. The EGNN/GNN-Dist and EGNN-NoCoord/raw-GNN controls at N = 4096 also fail (1.199 and 1.444).

W4. What is new beyond invariant-kernel theory and spectral bias, given that the cyclic example is designed to delay the weak block?

Response. We share the spectral viewpoint of prior work: invariance modifies the kernel spectrum, and smaller eigenvalues imply slower fitting. Our contribution is to jointly analyze how nonlinear feature averaging changes the training operator and target coupling, while matching learning directions across models.

First, we derive the operator change from the group action. A group-averaged Hermite–Mehler expansion computes covariances between transformed nonlinear features; the orbit-variance identity identifies the variation removed by averaging. For finite data, a conditional approximation preserves the sample-dependent zeroth and first noise-Hermite terms exactly and controls higher-order fluctuations.

Second, we match target-relevant subspaces across operators. Averaging can rotate eigenspaces, so eigenvalues alone cannot identify corresponding learning directions. We combine standard eigenvalue and projector perturbation bounds with control of target contributions and overlap with common reference subspaces, preserving block interpretation under approximation error. The cyclic example fixes alignment for an explicit covariance calculation; the general analysis also handles eigenspace rotation and finite-sample approximation error.

We will add these discussions in the final version.

Reference

T. Bonnaire, R. Urfin, G. Biroli, and M. Mézard. (2025). Why diffusion models don't memorize: The role of implicit dynamical regularization in training. NeurIPS 2025. arXiv:2505.17638.

Reviewer_z3fxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how symmetry affects the learning dynamics of diffusion denoising models, focusing on a mechanism termed selective delay. The authors propose that symmetry averaging can delay the learning of less relevant variants more than highly contributed ones, thereby increasing the relative separation between their learning stages.

To analyze this phenomenon, the authors introduce two random feature models (RFM), Base-RFM with basic architecture and Avg-RFM with group averaging. They also define a generalization regime through the convergence times of two feature subspaces, S and N. They provide an explicit construction for finite cyclic groups, derive sufficient conditions for general finite groups, and extend the analysis to continuous symmetry averaging on a torus. Experiments on synthetic settings support the predicted selected delays.

Critical strengths
  1. Presentation is clear and well organized. The paper is generally well written with a clear progression from the problem formulation to the theoretical results. Appropriate explanations with mathematical definitions make the main ideas relatively easy to follow.
  2. The theoretical setup is clear and controlled. The comparison between Base-RFM and Avg-RFM is well designed to isolate the effect of orbit averaging. Also, theoretical analysis are well structured. The paper provides an exact construction for cyclic groups and extends the analysis through sufficient spectral conditions for general finite groups and continuous torus averaging. The results offer a mathematically grounded explanation of when selective delay can occur.
Critical weaknesses
  1. The selection of subspaces S and N is not sufficiently specified. The paper distinguishes S and N using their initial target contributions and additional spectral conditions. However, these conditions do not uniquely determine the subspaces, and different admissible choices may yield different learning time separations. It would helpful to clarify whether there is a principled procedure for identifying S and N in a given learning problem, and how sensitive the results are to their selection.
  2. The connection between the defined generalization regime and actual generalization remains unclear. The paper defines the generalization regime through the relative parameter convergence times of S and N. However, these subspaces are distinguished by their initial target contributions rather than their actual roles in generalization and memorization. A feature direction receiving a small initial gradient does not necessarily correspond to memorization or task irrelevant variation. Therefore, while the paper establishes a separation in learning times, it remains unclear whether this separation corresponds to improved generalization or delayed memorization in the conventional sense.
Questions for authors

Is orbit averaging essential to the selective-delay mechanism? The proposed mechanism relies on the reduction of feature variance through orbit averaging. Could a similar separation of learning times arise from symmetry based data augmentation or other forms of symmetry regularization, without explicitly averaging the features?

In addition, the experiments are limited to carefully constructed synthetic settings. The numerical experiments mainly validate the predictions of the theoretical results under controlled constructions. Although this supports the proposed mechanism, it provides limited evidence of its applicability to more general learning problems. Would the framework also works for experiments involving real world data and a natural symmetry group, such as image rotations or reflections?

2 comments
AuthorsOpenReview ↗

Response to Reviewer z3fx

We thank the reviewer for the careful reading and constructive comments.

W1. How are the subspaces SS and NN selected, and how sensitive is the time separation to that choice?

Response. We select shared guidelines for dominant target-coupled directions (SS) and contrasting directions affected by the group action (NN), fixing both before measuring learning times. For example, in the finite-cyclic experiments (Appendix F.2), QS\mathbf Q_S is the leading rank-qq eigenspace of the average of the two models' trace-normalized target-contribution matrices, with qq fixed by design. To construct QN\mathbf Q_N, we map independent Gaussian probes of the non-fixed input directions through the shared weights, remove their QS\mathbf Q_S components, and take the leading rank-qq left-singular subspace. These guidelines are not unique and need not be Gram spectral subspaces. The theorem requires matched active spectral blocks satisfying the target-contribution, overlap, and spectral-margin conditions in Section 5.3. Admissible choices may change the quantitative separation, and sensitivity to alternative choices has not been systematically tested, left as future work.

W2. Does the SS–NN fitting-time separation imply conventional generalization or delayed memorization?

Response. Yes, what you said is right. However, to the best of our knowledge, this problem is largely open, supported by related work[1] (NeurIPS 2025 best paper), and we also admit this limitation in the conclusion.

[1] T. Bonnaire, R. Urfin, G. Biroli, M. Mézard. Why diffusion models don't memorize: The role of implicit dynamical regularization in training. NeurIPS 2025.

Q1. Is explicit orbit averaging necessary, or can augmentation or other symmetry regularization produce a similar separation?

Response. Explicit orbit averaging is not necessary, and augmentation or regularization could produce a similar separation if they induce suitable relative changes in blockwise learning rates. Recall our proof covers orbit averaging. For frozen features and the projected target, Appendix B.4 gives

=\mathbf U_{\mathrm{avg}}^{(n)}+\mathbf V_G^{(n)}, \qquad \mathbf V_G^{(n)}\succeq0,$$ where $\mathbf U_{\mathrm{aug}}^{(n)}$ is the Gram matrix under uniform group augmentation and $\mathbf V_G^{(n)}$ is the noise-averaged feature orbit covariance. The target cross-moment is unchanged because the projected target is fixed by the group, so augmentation adds a nonnegative quadratic loss term. Positive-semidefinite ordering alone does not determine relative block delays. Extending the result requires spectral, target-coupling, and subspace-matching checks; learned features require further analysis. A matched empirical comparison remains future work. **Q2. Does the framework also work for experiments with real data and natural symmetries?** *Response.* **Yes, it works. Finite rotation and reflection groups fit the framework when the projected target is nonzero, the data distribution is invariant, and suitable spectral blocks exist.** Standard 3D rotations have no nonzero fixed vector, so molecular equivariant vector scores require a vector-valued extension of the theorem. **QM9 molecular generation provides empirical evidence to support the theory** (see Section 6.3). As shown in the table below, EGNN has a larger generalization window than GNN. **QM9 ($N=4096$; seeds 0/1/2)** | Measure (training steps) | Raw-coordinate GNN | EGNN | | --- | --- | --- | | Generation start (all seeds) | 10k | 10k | | Memorizationor Copying onset | 400k / 360k / 150k | 420k / 440k / $>500$k | | Generalization window | 390k / 350k / 140k | 410k / 430k / $>490$k |

Thank you for the response. However, my main concerns remain unresolved. To my understanding, there is still no principled procedure for selecting S and N. The response only describes heuristic guidelines used in a specific experiment and the sensitivity of the results to this choice remains untested. Also for generalization and memorization regimes, I still cannot see how S-N separation connects to actual generalization or memorization performance. Therefore, I will maintain my score.

Reviewer_Hn3fOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper discusses training trajectories of toy equivariant diffusion models in a simplified setup. Under very restricted assumptions, for example, a Gaussian data distribution and random-feature models to be trained, the authors show that, the learning speed of less relevant features will be slower than that of the target main structures.

Critical strengths

-- It's interesting to see the connections between the group properties and the learning speed of the features.

Critical weaknesses
  1. The paper does not cover any potential empirical consequences of their work. In particular, what can practitioners do by learning the results of their theoretical work? I think this is very important for any theoretical work even if they did experiments and analysis in a toy setup.

  2. Section 5 is a bit overly technical and dense, as is the main text. To be more efficient, to show the general message of their theoretical work, and to save space for more empirical analysis.

  3. The empirical study in Section 6 is only for numerical verification of their theoretical claims without any neural network training. It does not add any additional insights. It would be great if the authors could discuss a bit about how their theoretical work aligns with real-world diffusion model training. And whether we can see a similar phenomenon in a real practical case

Questions for authors

I have mentioned several serious problems in the critical weakness section. I am particularly interested in points 1 and 3 which I would just repeat here:

  1. What can a practitioner learn from this empirical result, and what do the results suggest for future empirical and theoretical study?

  2. Can you show some experiments involving neural network training that can support your theoretical claims (in the setting assumed by the theory); in addition, can you show that the described phenomena happened in real-world diffusion training processes?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Hn3f

We thank the reviewer for the constructive comments. We will revise in the final version.

W1 & Q1: (1) Practical implications: what can practitioners learn from this work? (2) Suggestions for future empirical and theoretical study?

R1: (1) Practical implications: For tasks with a known symmetry compatible with the target, our theory implies practitioners to use symmetry-compatible models better with fewer training epochs than non-symmetry ones in practice. This is because our theory implies that symmetry models can reach useful generation quality earlier and delay overfitting or memorization, potentially requiring fewer training epochs when getting comparable performance (see Figure 1 and 4 in the manuscript). (2) Future Suggestions: For theory study, our analysis can be extended to more practical network architectures (e.g., EGNN), continuous groups (e.g., SO(3)), optimization algorithms (e.g., Adam), and generative models (e.g., equivariant flow matching). For empirical study, practitioners can consider using fewer training epochs with equivariant models to do large-scale experiments, thus reducing computational costs.

W2: Presentation: Can Section 5 and the main text be made less dense while foregrounding the main message?

R2: Thank you for your recommendation. We will reconstruct Section 5 to convey the core insights of their theoretical framework while reserving space for a more extensive empirical analysis. Specifically, we will present Section 5 in three steps: target coupling identifies the compared directions, the Gram spectrum sets their fitting rates, and the rate comparison explains the timing separation.** We will motivate definitions, retain essential assumptions, and move repeated derivations and approximation bounds to the appendix.

W3&Q2: Do the experiments provide evidence beyond numerical checks, including evidence from trainable (in the setting assumed by the theory) or real-world diffusion models?

R3: Yes, we did these experiments, which support our theory claims. (1) Experiments with neural network training in the setting assumed by the theory. Figure 3(d) (in the manuscript) shows that Avg-RFM has a larger generalization window (about 5$\times$) than Base-RFM, supporting our theory claims. (2) Experiments about real-word diffusion models. We move Section 6.3 from the appendix to report end-to-end training of EGNN and a raw-coordinate GNN on the QM9 dataset for the molecular generalization task with three paired seeds. As shown in the table below, EGNN has a larger generalization window with later memorization (or copying) time and lower mean test loss in all three pairs, while meeting the prespecified best-quality criterion, supporting our theory claim. (See Figure 4 in the manuscript for details).

QM9 summary across three paired seeds

MeasureEGNNGNN
Mean test loss at 500k steps0.2290.303
Generation time τgen\tau_{\rm gen}10k for all seeds10k for all seeds
Meomorization or Copying time τmem\tau_{\rm mem} (seeds 0/1/2)420k / 440k / >500>500k400k / 360k / 150k
Median paired copying delay80k steps; median copying-window ratio 1.222-
Mean paired best-quality ratio0.797 (lower is better; criterion ≤1.1\leq 1.1)-

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.