acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Improving the Training Dynamics of EDM2

Abstract

Diffusion models have achieved remarkable success in image generation, yet their training dynamics can strongly affect both convergence speed and the persistence of high-quality generation. In this work, we investigate the optimization dynamics of EDM2 on small-scale image generation benchmarks and identify factors that influence its generation quality and training stability. We introduce several improvements, including learning-rate scheduling, activation statistics regulation, adaptive reconstruction objectives, and modifications to EDM2-specific components, which improve generation quality and extend the persistence of high-performance states. Furthermore, we analyze the optimization geometry induced by the EDM2 parameterization and find that, compared with an EDM-style control model, it exhibits lower sensitivity to relative parameter perturbations during early training. These results highlight the importance of considering optimization dynamics beyond final performance when improving diffusion models.

Primary area:
generative models
Keywords:
Diffusion Models, EDM2, Training Dynamics, Optimization, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NNVzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the training behavior of EDM2 on small image datasets. The authors find that their EDM2 baseline performs worse than EDM and can lose generation quality during later training. They introduce several changes, including data augmentation, learning-rate scheduling, weight centralization, and a frequency-aware loss. They also study how EDM2 components affect late-stage performance and use Hessian analysis to examine its optimization behavior.

Critical strengths
  1. The paper shows several techniques that improve EDM2 on small datasets. The best reported configurations outperform EDM in their settings.

  2. The paper provides explanations for the proposed changes and ablations to examine their effects. It also studies the training trajectories beyond the best FID reached during training.

Critical weaknesses
  1. The motivation is not clear enough. EDM2 already outperforms EDM in the original paper, including on ImageNet-64 at a similar model size and with fewer sampling evaluations. Here, the authors build their own EDM2 configuration for smaller datasets, where the weaker performance may come from hyperparameters that are not well suited to these settings. The paper needs to separate this possibility from a problem specific to EDM2. The original EDM2 paper also discusses late-stage overfitting and the benefit of dropout. Although this paper already uses dropout, it remains unclear how the behavior studied here differs from that observation.

  2. The contribution beyond tuning the training recipe seems limited. Data augmentation and learning-rate scheduling are useful, but they are standard tools and account for most of the FID improvement in Table 1. Weight centralization and the frequency-aware loss provide much smaller additional gains, and their motivations need stronger support. For weight centralization, zero-mean inputs and targets do not imply that intermediate features should also have zero mean. Normalization methods such as RMSNorm work without explicitly centering features. The relation between RMS, mean, and variance does not by itself show that the observed mean offsets harm generation quality. It is unclear whether the small improvement comes from reducing these offsets or from other effects of the changed weight parameterization. For the frequency-aware loss, the model already works well without it. Adding this loss changes FID from 1.70 to 1.70 on CIFAR-10, 1.98 to 1.93 on AFHQv2, and 2.02 to 1.99 on FFHQ. Why is middle-frequency reconstruction a problem that needs to be addressed here? The paper does not show that it is a bottleneck in EDM2 training. The connection between this loss and the training problems studied in the paper is therefore unclear.

  3. I would like to see a stronger, well-tuned EDM2 baseline and clearer evidence of what these changes add beyond standard tuning. The training-dynamics analysis should also explain what new understanding it provides beyond the original EDM2 paper.

  4. The scope is quite limited. The paper proposes several improvements, but only tests them on EDM2 and small datasets. Whether these changes also help on larger datasets, where EDM2 already works well, is still unknown. Their usefulness for other architectures is also unclear. If these techniques only help EDM2 under specific small-dataset settings, I see limited contribution beyond adapting its training recipe to those settings.

Questions for authors
  1. How were the baseline learning rate, decay schedule, and dropout rates selected? Does the late-stage degradation remain after tuning these settings under a comparable training budget?

  2. What evidence shows that activation mean offsets and middle-frequency reconstruction errors are responsible for the observed training problems? Can the authors separate these explanations from other effects of the proposed changes?

  3. Do the proposed changes improve EDM2 on ImageNet-64, where the original paper already provides a strong configuration? This would help clarify whether the findings extend beyond the small-dataset settings studied here.

8 comments

Response to Reviewer NNVz (Part 1: Questions 1)

We sincerely thank the reviewer for the thoughtful and constructive comments. We appreciate the reviewer’s recognition of our analysis of EDM2 training dynamics. We agree that the current manuscript would benefit from additional explanations, more comprehensive statistical evidence, and further disclosure of experimental details. In the revised manuscript, we will address these concerns through additional analyses, new experiments, visualizations, and implementation details. We respond to the reviewer’s comments point by point below.


Question 1: How were the learning rate, learning rate decay strategy, and Dropout ratio of the baseline model selected? After tuning these settings under a comparable training budget, does the late-stage performance degradation still persist?

1.1 How were the learning rate, learning rate decay strategy, and Dropout ratio of the baseline model selected?

Response: We thank the reviewer for asking us to describe the hyperparameter selection procedure more systematically. We agree that, in addition to reporting the final hyperparameter values, disclosing the search strategy, number of experiments, and corresponding computational cost is important for improving reproducibility.

We adopted hyperparameter search strategies tailored to each dataset, using the EDM2 configuration on ImageNet-64 and the publicly available EDM configuration for the corresponding dataset as initial references. Specifically, the initial learning rate and its decay schedule were primarily referenced from the EDM2 configuration on ImageNet-64. Most other hyperparameters (e.g., Dropout) were initially referenced from the EDM configuration for the corresponding dataset, since the EDM2 configuration on ImageNet-64 itself inherits many design choices from EDM.

(1) Initial pilot experiment Before conducting the formal hyperparameter search, we first performed a pilot experiment using the above initial configuration and extended training to 200M images to observe the FID trajectory and roughly identify the training stage at which the best FID occurred. This pilot experiment showed that the best FID was reached before the end of training. Therefore, our subsequent hyperparameter search primarily focused on this stage to avoid unnecessarily long training for all candidate configurations. We emphasize that this initial 200M-image training run was conducted to explore training dynamics and was independent of the subsequent formal hyperparameter grid search.

(2) Learning rate and decay duration search

  • CIFAR-10: We tested 11 initial learning rates ranging from 0.0050.005 to 0.0250.025, with an interval of 0.0020.002, together with seven learning rate decay durations: {50k,35k,25k,20k,15k,10k,7k}\{50\mathrm{k}, 35\mathrm{k}, 25\mathrm{k}, 20\mathrm{k}, 15\mathrm{k}, 10\mathrm{k}, 7\mathrm{k}\}. This resulted in 77 combinations. The experimental results show that when the decay duration was 50k50\mathrm{k} or 35k35\mathrm{k}, lower initial learning rates (approximately 0.0100.010–0.0150.015) performed relatively better, but the FID remained above 2.152.15, with the best results occurring at around 75M images. When the decay duration was 25k25\mathrm{k}, 20k20\mathrm{k}, or 15k15\mathrm{k}, an initial learning rate of approximately 0.0170.017 achieved better results, with FID values around 2.052.05–2.112.11 and the best results occurring at around 55M images. In contrast, when the decay duration was further reduced to 10k10\mathrm{k} or 7k7\mathrm{k}, the generation quality deteriorated substantially, with the FID exceeding 2.52.5 for most combinations. Based on these results, we selected an initial learning rate of 0.0170.017 and a decay duration of 20k20\mathrm{k} for the CIFAR-10 baseline.
  • AFHQv2-64 and FFHQ-64: We narrowed the search ranges based on the preliminary experimental results. For each dataset, we tested six initial learning rates ranging from 0.0130.013 to 0.0230.023, with an interval of 0.0020.002, together with four decay durations: {15k,20k,25k,30k}\{15\mathrm{k}, 20\mathrm{k}, 25\mathrm{k}, 30\mathrm{k}\}. This resulted in 24 combinations for each dataset.

Therefore, the baseline learning rate and decay duration searches across the three datasets comprised 125 experiments in total. All these formal search experiments were trained for 80M images.

(3) Dropout and other hyperparameters We also conducted independent Dropout searches based on the EDM reference configuration for each dataset:

  • CIFAR-10: We tested 0.09,0.11,0.130.09, 0.11, 0.13, and 0.150.15, and finally selected 0.130.13.
  • AFHQv2-64: We tested 0.23,0.25,0.270.23, 0.25, 0.27, and 0.290.29, and finally selected 0.250.25.
  • FFHQ-64: We tested 0.03,0.05,0.070.03, 0.05, 0.07, and 0.090.09, and finally selected 0.050.05.

The candidate values for AFHQv2-64 and FFHQ-64 were selected around the corresponding EDM reference configurations, and the final values were determined based on the respective experimental results.

Response to Reviewer NNVz (Part 2: Questions 1)

(4) Configurations for subsequent experiments and search for the annealing starting point

For the subsequent experiment groups (Configurations B–F), we did not repeat the full grid search. Instead, we conducted smaller local searches around the learning rate and decay duration selected for the baseline.

Taking Configuration C on CIFAR-10 as an example, we tested the initial learning rates {0.015,0.017,0.019}\{0.015, 0.017, 0.019\} and decay durations {15k,20k,25k}\{15\mathrm{k}, 20\mathrm{k}, 25\mathrm{k}\}. The results showed that the better hyperparameter combinations identified by this local search were very close to those used in the baseline. Therefore, to ensure fair comparisons among subsequent modifications, we retained the baseline learning-rate and decay settings for the subsequent experiment groups. We followed the same principle for AFHQv2-64 and FFHQ-64, conducting limited local searches around their respective baseline configurations and ultimately adopting the same learning-rate and decay settings as the baselines. Dropout was also selected following the same general approach.

After introducing learning-rate annealing, we adjusted the total number of training steps for some datasets according to the changes in the training stage at which the best FID occurred.

The annealing starting point was selected as follows. First, we identified the training time corresponding to the best FID, denoted by TbestT_{\mathrm{best}}. We then selected a checkpoint close to 0.5Tbest0.5T_{\mathrm{best}} and saved the complete training state. Starting from this checkpoint, we resumed training and tested learning-rate turning points at the following candidate times:

0.5Tbest,0.6Tbest,0.7Tbest,0.8Tbest.0.5T_{\mathrm{best}},\quad 0.6T_{\mathrm{best}},\quad 0.7T_{\mathrm{best}},\quad 0.8T_{\mathrm{best}}.

For each resumed run, we restored the model parameters, EMA parameters, optimizer state, and training progress, rather than loading only the model weights. Consequently, all candidate annealing schedules shared exactly the same training trajectory before branching, and differed only in the learning-rate schedule adopted from the specified turning point onward. For subsequent searches over later turning points, we also reused checkpoints from the preceding turning-point experiments to reduce redundant training.


(5) Computational cost

All 125 formal baseline experiments for the learning-rate and decay-duration search were trained for 80M images. Each experiment used four NVIDIA A30 GPUs, and all experiments used the same global training random seed. The following table reports the wall-clock time per experiment and the total GPU hours for each dataset, where total GPU hours are calculated as Wall-clock time per run×Number of runs×4\text{Wall-clock time per run} \times \text{Number of runs} \times 4:

DatasetNumber of runsWall-clock time per runTotal GPU hours
CIFAR-107717 h 32 min 20 s~5,402
AFHQv2-64241 d 8 h 18 min~3,101
FFHQ-64241 d 9 h 12 min~3,187
Total125—~11,690

These figures estimate the computational resources required for the formal baseline hyperparameter search. The initial 200M-image pilot experiment, subsequent local searches, dropout searches, and annealing starting-point searches are not included in the above total of 125 experiments.

Response to Reviewer NNVz (Part 3: Questions 1)

(6) General hyperparameter tuning procedure

Based on these experiments, we summarize a practical tuning procedure for adapting EDM2 to a new dataset:

  1. Select an initial reference configuration. Start from a well-established training configuration for a publicly available dataset of a comparable scale. Conduct a pilot experiment to observe the FID trajectory, identify the approximate location of the best performance, and examine subsequent performance trends.
  2. Jointly search the learning rate and decay duration. Conduct a joint search over the initial learning rate and decay duration around the reference configuration. Since these two hyperparameters are coupled, joint tuning provides more useful information than adjusting either parameter independently. The search range for the decay duration can be initially determined based on the dataset scale and pilot results.
  3. Determine the annealing starting point through checkpoint branching. First identify the training time corresponding to the best FID, TbestT_{\mathrm{best}}. Then save a complete checkpoint near 0.5Tbest0.5T_{\mathrm{best}}, resume training from it, and test different learning-rate turning points. This avoids restarting training for every candidate and therefore reduces computational cost. Subsequent searches over later turning points can also reuse checkpoints from earlier turning-point experiments.
  4. Further tune other hyperparameters. After determining a suitable learning-rate schedule, adjust other hyperparameters, such as dropout and loss-related parameters, including the frequency-loss weight and frequency-scale boundaries.
  5. Evaluate both peak performance and training stability. A configuration should not be selected solely based on its best FID during early training. Promising candidates should be trained further to examine their behavior after reaching their best performance, so that we can assess whether they can maintain generation quality over time.

We will include the above search details and computational costs in the appendix of the revised manuscript to improve the transparency and reproducibility of our experiments.


1.2 After tuning these settings under a comparable training budget, does the late-stage performance degradation still persist?

Response: Yes, it still persists. Through progressive refinement of hyperparameter configurations and training strategies, we found that late-stage performance degradation cannot be simply attributed to improper selection of the initial learning rate or decay duration.

  • In our initial pilot experiment, we directly adopted EDM2's initial learning rate and decay settings from ImageNet-64 together with EDM's Dropout setting on CIFAR-10, training the model up to 200M images. We observed that the model reached a relatively good FID around 40M images (though still approximately 4.14.1), and as training continued, the FID deteriorated further above 66 near 200M images.
  • Considering that EDM2 uses markedly different learning rate and decay settings on ImageNet-64 and ImageNet-512, we hypothesized that these hyperparameters are strongly dataset-dependent. Thus, we first systematically searched the learning rate and decay duration, obtaining the tuned Configuration A. However, as shown in Figure 1(a), late-stage performance degradation remained prominent even after this tuning.
  • Subsequently, considering the small scale of CIFAR-10 and the potential risk of severe overfitting, we introduced the exact same data augmentation strategy as EDM (differing only in the conditioning injection mechanism; see Appendix B.2), yielding Configuration B. This modification substantially improved generation quality, bringing the FID close to EDM levels.
  • However, Configuration B did not eliminate the late-stage issue. Compared with Configuration A, the late-stage FID of Configuration B did not exhibit simple monotonic deterioration, but instead showed severe oscillations, failing to stably maintain generation quality. We hypothesized that the learning rate near the optimum was too large and therefore applied learning rate annealing. Although annealing eliminated the FID oscillations and improved peak generation quality, late-stage FID degradation reappeared.

These results demonstrate that merely tuning the learning rate, decay duration, and Dropout—or introducing data augmentation and learning rate annealing—is insufficient to fully resolve the late-stage training degradation of EDM2 under the current small-scale dataset settings. Therefore, we further investigated other factors affecting training dynamics and model stability, as discussed in Section 2.5 of the manuscript.

Reviewer_S6bQOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the training dynamics of EDM2 on small-scale image generation benchmarks. Starting from a baseline that combines the EDM2 network with the EDM training scheme, the authors cumulatively add non-leaky data augmentation, a piecewise late-stage learning-rate annealing, weight centralization to reduce activation mean offsets, and an adaptive frequency-aware reconstruction loss. This loss puts more weight on middle-frequency errors depending on the noise level and training progress. The authors further observe that EDM2 reaches a strong FID quickly but does not stay there. They propose a stability-oriented configuration that removes EDM2’s uncertainty-based loss weighting and modifies its MP components, which extends the high-performance regime on CIFAR-10. Finally, a Hessian-based analysis with a scale-corrected sensitivity measure indicates that EDM2 is less sensitive to relative parameter perturbations than an EDM-style control model during early-to-middle training, which the authors relate to its tolerance of larger learning rates and faster convergence.

Critical strengths

1. Analysis of how long good performance persists, not only the best FID.

Rather than reporting only the best FID, the paper presents full FID trajectories over training. It separates two aspects of training dynamics: how quickly a model reaches a high-performance regime, and how long it stays there. The original EDM2 paper discusses FID degradation mainly as overfitting of its larger models trained without dropout on ImageNet-512. It does not study the pattern observed here on small datasets, where FID reaches an early optimum and then degrades or fluctuates. This observation and problem setting are therefore meaningful.

2. Detailed and transparent reporting.

The appendix documents the hyperparameters of every configuration, the full piecewise learning-rate schedules, and the search ranges of the new hyperparameters. Configurations C–E are evaluated with five sampling seeds. On CIFAR-10, Configuration E is additionally checked with five training seeds and a sweep over sampling steps. The paper also reports unfavorable findings openly: weight centralization does not change the persistence behavior, the frequency-aware loss gives limited gains on CIFAR-10, and Configuration F is not a universal replacement for Configuration E.

Critical weaknesses

1. Limited scope and insufficient separation of architectural effects from training settings.

  • Experiments are limited to CIFAR-10, AFHQv2-64, and FFHQ-64. Validation on ImageNet-64 with official EDM2 configurations would help establish whether the observations extend beyond the authors’ adapted setup.

  • Configuration A uses an initial learning rate of 0.017, above the official ImageNet-64 presets of 0.007–0.012. This is not necessarily inappropriate, but its selection is insufficiently justified. Appendix B.3 states that lower learning rates improve stability at the cost of convergence speed, without reporting the tested values or quantitative results.

  • Dataset-specific learning-rate schedules complicate the persistence analysis. After 62M images, AFHQv2 and FFHQ use approximately 1/142 of the initial maximum or less, while CIFAR-10 retains approximately 1/9–1/16. Although the paper discusses learning-rate effects and reports C/D trajectories in Appendix C, controlled schedule comparisons are needed to distinguish these effects from architectural and dataset differences.

  • Each checkpoint is evaluated over 39 post-hoc EMA lengths, with the best retained. The checkpoint-selection procedure and its comparability with the EDM baseline need clarification, particularly given the small reported gains.

2. Insufficient justification and validation of the proposed components.

  • At fixed RMS, a larger activation mean reduces variance, but the paper does not establish that this harms useful representations or generation quality. Weight centralization yields only 0.01–0.04 improvements in mean FID and does not substantially improve persistence. Matched training-seed comparisons would help establish whether these gains are reproducible.

  • The choice to emphasize middle frequencies lacks direct validation through per-frequency error analysis or comparisons with higher-frequency emphasis and Kim et al. (2025). The theoretical analysis preserves the optimal denoiser but does not explain the practical benefit. Configuration E improves main-seed FID over D by only 0.03–0.05 on AFHQv2 and FFHQ, with no gain on CIFAR-10; the fixed frequency-aware objective is worse than D on all three datasets.

  • Most of the minimum-FID improvement comes from augmentation and learning-rate scheduling. Although the persistence and optimization analyses offer separate contributions, the practical benefits of weight centralization and the adaptive frequency-aware loss remain insufficiently established.

Questions for authors
  1. How was the initial learning rate of 0.017 selected? Appendix B.3 discusses lower learning rates qualitatively; could you provide the tested values and quantitative trajectories, and clarify whether the short high-performance regime also occurs with official EDM2 configurations on ImageNet-64?

  2. How much of the difference in persistence across datasets is attributable to their different learning-rate schedules? Could controlled comparisons within each dataset separate schedule effects from the effects of the proposed interventions?

  3. How were the checkpoints in Table 1 selected, and how would the comparison with EDM change under a matched checkpoint- and EMA-selection protocol?

  4. What evidence supports the premises behind weight centralization and middle-frequency emphasis? In particular, can you show that activation mean offsets harm generation and provide per-frequency error analysis or comparisons with higher-frequency emphasis, including the searched (0.4, 1.2) setting?

10 comments

Response to Reviewer s6bq (Part 1: Questions 1)

We sincerely thank the reviewer for the thoughtful and constructive comments. We appreciate the reviewer’s recognition of our analysis of EDM2 training dynamics, particularly that we do not merely report the best FID but also present the FID trajectory throughout the entire training process. We agree that the current manuscript would benefit from additional explanations, more comprehensive statistical evidence, and further disclosure of experimental details.

In the revised manuscript, we will address these concerns through additional analyses, new experiments, visualizations, and implementation details. We respond to the reviewer’s comments point by point below.


Question 1: How was the initial learning rate of 0.017 selected? Does a similar phenomenon also occur on ImageNet-64?

1.1 How was the initial learning rate of 0.017 selected?

Response:

We thank the reviewer for asking us to describe the hyperparameter selection procedure more systematically. We agree that, in addition to reporting the final hyperparameter values, disclosing the search strategy, number of experiments, and corresponding computational cost is important for improving reproducibility.

We adopted hyperparameter search strategies tailored to each dataset, using the EDM2 configuration on ImageNet-64 and the publicly available EDM configuration for the corresponding dataset as initial references. Specifically, the initial learning rate and its decay schedule were primarily referenced from the EDM2 configuration on ImageNet-64. Most other hyperparameters (e.g., Dropout) were initially referenced from the EDM configuration for the corresponding dataset, since the EDM2 configuration on ImageNet-64 itself inherits many design choices from EDM.

or all candidate configurations.

We emphasize that this initial 200M-image experiment was conducted separately to explore the training dynamics and was not included in the subsequent formal grid search.

(2) Search over the initial learning rate and decay duration

For CIFAR-10, we tested 11 initial learning rates ranging from 0.005 to 0.025, with a step size of 0.002, combined with seven learning-rate decay durations: {50k, 35k, 25k, 20k, 15k, 10k, 7k}. This resulted in 77 combinations.

The results show that when the decay duration was 50k or 35k, relatively lower initial learning rates (approximately 0.010–0.015) performed better. However, their FID values remained above 2.15, with the best results occurring at approximately 75M training images. When the decay duration was 25k, 20k, or 15k, an initial learning rate of approximately 0.017 achieved better results, with FID values around 2.05–2.11 and the best results occurring at approximately 55M training images. In contrast, further reducing the decay duration to 10k or 7k substantially degraded generation quality, with most configurations producing FID values above 2.5. Based on these results, we selected an initial learning rate of 0.017 and a decay duration of 20k for the CIFAR-10 baseline.

For AFHQv2-64 and FFHQ-64, we narrowed the search ranges based on the preliminary results. For each dataset, we tested six initial learning rates ranging from 0.013 to 0.023, with a step size of 0.002, combined with four decay durations: {15k, 20k, 25k, 30k}. This resulted in 24 combinations for each dataset.

Therefore, the baseline search over the initial learning rate and decay duration involved 125 experiments across the three datasets. All these formal search experiments were trained for 80M images.

(3) Dropout and other hyperparameters

We also conducted a separate search over dropout values, using the EDM configuration for each dataset as the reference:

  • CIFAR-10: We tested 0.09, 0.11, 0.13, and 0.15, and selected 0.13.
  • AFHQv2-64: We tested 0.23, 0.25, 0.27, and 0.29, and selected 0.25.
  • FFHQ-64: We tested 0.03, 0.05, 0.07, and 0.09, and selected 0.05.

The candidate values for AFHQv2-64 and FFHQ-64 were selected around the corresponding EDM reference configurations, and the final values were determined based on the experimental results for each dataset.

Response to Reviewer s6bq (Part 2: Questions 1)

(4) Configurations for subsequent experiments and search for the annealing starting point

For the subsequent experiment groups (Configurations B–F), we did not repeat the full grid search. Instead, we conducted smaller local searches around the learning rate and decay duration selected for the baseline.

Taking Configuration C on CIFAR-10 as an example, we tested the initial learning rates {0.015,0.017,0.019}\{0.015, 0.017, 0.019\} and decay durations {15k,20k,25k}\{15\mathrm{k}, 20\mathrm{k}, 25\mathrm{k}\}. The results showed that the better hyperparameter combinations identified by this local search were very close to those used in the baseline. Therefore, to ensure fair comparisons among subsequent modifications, we retained the baseline learning-rate and decay settings for the subsequent experiment groups. We followed the same principle for AFHQv2-64 and FFHQ-64, conducting limited local searches around their respective baseline configurations and ultimately adopting the same learning-rate and decay settings as the baselines. Dropout was also selected following the same general approach.

After introducing learning-rate annealing, we adjusted the total number of training steps for some datasets according to the changes in the training stage at which the best FID occurred.

The annealing starting point was selected as follows. First, we identified the training time corresponding to the best FID, denoted by TbestT_{\mathrm{best}}. We then selected a checkpoint close to 0.5Tbest0.5T_{\mathrm{best}} and saved the complete training state. Starting from this checkpoint, we resumed training and tested learning-rate turning points at the following candidate times:

0.5Tbest,0.6Tbest,0.7Tbest,0.8Tbest.0.5T_{\mathrm{best}},\quad 0.6T_{\mathrm{best}},\quad 0.7T_{\mathrm{best}},\quad 0.8T_{\mathrm{best}}.

For each resumed run, we restored the model parameters, EMA parameters, optimizer state, and training progress, rather than loading only the model weights. Consequently, all candidate annealing schedules shared exactly the same training trajectory before branching, and differed only in the learning-rate schedule adopted from the specified turning point onward. For subsequent searches over later turning points, we also reused checkpoints from the preceding turning-point experiments to reduce redundant training.


(5) Computational cost

All 125 formal baseline experiments for the learning-rate and decay-duration search were trained for 80M images. Each experiment used four NVIDIA A30 GPUs, and all experiments used the same global training random seed. The following table reports the wall-clock time per experiment and the total GPU hours for each dataset, where total GPU hours are calculated as Wall-clock time per run×Number of runs×4\text{Wall-clock time per run} \times \text{Number of runs} \times 4:

DatasetNumber of runsWall-clock time per runTotal GPU hours
CIFAR-107717 h 32 min 20 s~5,402
AFHQv2-64241 d 8 h 18 min~3,101
FFHQ-64241 d 9 h 12 min~3,187
Total125—~11,690

These figures estimate the computational resources required for the formal baseline hyperparameter search. The initial 200M-image pilot experiment, subsequent local searches, dropout searches, and annealing starting-point searches are not included in the above total of 125 experiments.

Response to Reviewer s6bq (Part 3: Questions 1)

(6) General hyperparameter tuning procedure

Based on these experiments, we summarize a practical tuning procedure for adapting EDM2 to a new dataset:

  1. Select an initial reference configuration. Start from a well-established training configuration for a publicly available dataset of a comparable scale. Conduct a pilot experiment to observe the FID trajectory, identify the approximate location of the best performance, and examine subsequent performance trends.
  2. Jointly search the learning rate and decay duration. Conduct a joint search over the initial learning rate and decay duration around the reference configuration. Since these two hyperparameters are coupled, joint tuning provides more useful information than adjusting either parameter independently. The search range for the decay duration can be initially determined based on the dataset scale and pilot results.
  3. Determine the annealing starting point through checkpoint branching. First identify the training time corresponding to the best FID, TbestT_{\mathrm{best}}. Then save a complete checkpoint near 0.5Tbest0.5T_{\mathrm{best}}, resume training from it, and test different learning-rate turning points. This avoids restarting training for every candidate and therefore reduces computational cost. Subsequent searches over later turning points can also reuse checkpoints from earlier turning-point experiments.
  4. Further tune other hyperparameters. After determining a suitable learning-rate schedule, adjust other hyperparameters, such as dropout and loss-related parameters, including the frequency-loss weight and frequency-scale boundaries.
  5. Evaluate both peak performance and training stability. A configuration should not be selected solely based on its best FID during early training. Promising candidates should be trained further to examine their behavior after reaching their best performance, so that we can assess whether they can maintain generation quality over time.

We will include the above search details and computational costs in the appendix of the revised manuscript to improve the transparency and reproducibility of our experiments.


1.2 Does a similarly short high-performance period also occur on ImageNet-64?

Response:

We thank the reviewer for this suggestion. We agree that investigating whether a similarly short high-performance period also occurs on ImageNet-64 is important, as it would help determine whether our findings extend beyond the small-scale datasets considered in this study.

However, conducting complete training experiments on ImageNet-64 requires substantial computational resources. Due to our current computational constraints, we are unable to complete the corresponding controlled experiments on this dataset at present. Therefore, we cannot yet provide direct experimental conclusions regarding whether this phenomenon also occurs on ImageNet-64.

We will clarify the scope of our experiments in the revised manuscript and restrict our conclusions to the datasets and training settings investigated in this work, rather than directly generalizing them to larger-scale datasets such as ImageNet-64. We believe that systematically validating the effectiveness of these modifications on larger-scale datasets is a valuable direction for future research.

Reviewer_ysBtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the training dynamics of an EDM2 adaptation on datasets CIFAR-10, AFHQv2-64, and FFHQ-64, distinguishing rapid convergence to good generation quality from the persistence of that quality during continued training. The authors progressively introduce modifications to improve training dynamics, including non-leaky augmentation, learning-rate annealing, weight centralization, and a frequency-aware reconstruction objective whose strength depends on noise level and training progress. They report improved best-checkpoint FID over their adapted baseline. To address late-stage FID deterioration, they additionally remove uncertainty-based loss weighting and modify magnitude-preserving branch merging, which they find results in a tradeoff between convergence speed and stability/persistence (found to vary across datasets). Separately, they compare local Hessian geometry in a shared convolutional parameter subspace of an EDM2-style model and an EDM-style control. They report lower local gradient sensitivity for the EDM2-style model during early-to-middle training under their chosen parameter-relative perturbation measure. The empirical study is limited to small-scale image generation; broader architectural and dataset generalization remains uncertain.

Critical strengths
  1. The paper is well written, easy to follow, and includes ample appendix material containing further details that are pertinent.
  2. The authors carry out a systematic investigation of training dynamics, examining activation statistics, learned loss weights, and training trajectories, with targeted ablations and robustness checks to explore possible explanations.
  3. The proposed changes are clearly motivated, mathematically formulated, and evaluated through progressive ablations. This makes the reasoning and experimental approach easy to follow and assess.
  4. The plots are informative and clearly show the differences between convergence speed, best FID, and persistence of good performance, including the tradeoffs and differences across datasets.
Critical weaknesses
  1. The statistical support for the incremental improvements is limited. The values in parentheses in Table 1 reflect variation across sampling seeds for a single selected checkpoint, not across training runs. The training-seed robustness check (Figure 9b) suggests that the best FID of Configuration E on CIFAR-10 varies by roughly 0.05 across five training seeds, which is comparable to or larger than the C →\to D (1.72→1.701.72 \to 1.70), D →\to E (1.70→1.701.70 \to 1.70), and E →\to F (1.70→1.661.70 \to 1.66) differences, while AFHQv2-64 and FFHQ-64 appear to use a single training seed. In addition, each reported FID is the best over many checkpoints and 39 post-hoc EMA lengths at a fixed sampling seed, which biases the reported values downward. The EDM row appears to use published numbers rather than a rerun under the same selection protocol (the EDM trajectory in Figure 3a reaches roughly 1.83). Table 2 illustrates the concern: the training-progress weighting λt\lambda_t (Eq. 5, with Tmax⁡=7×107T_{\max} = 7 \times 10^7) is already ≈0.77\approx 0.77 after 1M training images and ≈0.96\approx 0.96 by 35.7M, so it is nearly constant over most of training and scales a component with λ=0.005\lambda = 0.005 on FFHQ-64, yet it is credited with improving FID from 2.19 to 1.99. Together with the ≈0.01\approx 0.01 FID variation across λ\lambda values on CIFAR-10 (Appendix B.5.3), this makes it difficult to determine which of the proposed components (weight centralization, the frequency-aware objective, and the late-stage modifications) yield reliable improvements beyond the non-leaky augmentation and learning-rate annealing of Configurations B and C.

  2. The recipes introduce many hyperparameters, such as dataset-specific learning-rate schedules and frequency-loss choices, but the paper gives limited guidance on how someone would tune these for their own application. The search ranges are provided, but the number and duration of trials, total tuning cost, and reliability of cheaper selection methods are unclear. Reporting these details and identifying transferable defaults would make the findings easier to use and build on, including allowing the community to investigate whether they extend to other architectures and diffusion formulations. A complete public code release after publication, with exact experiment configurations, pretrained and intermediate checkpoints, and the full Hessian comparison pipeline, would help facilitate this research. The supplementary code is a useful start, but currently lacks checkpoints and the complete EDM-style control and parameter-scale/ratio analysis needed to reconstruct Figure 4.

  3. The Hessian analysis does not seem to establish the proposed explanation of faster convergence. The sensitivity measure remains dependent on parameter units, and the compared models have different initialization and learning-rate histories, making it difficult to isolate the effect of the architecture. The authors acknowledge these limitations, but the connection between the measured local geometry and the observed convergence behavior remains unresolved. Moreover, because forced weight normalization makes the EDM2 backbone weights exactly scale-invariant, the stored weight radius (d\sqrt{d}) is a convention: rescaling it changes S\mathcal{S} by the inverse factor (Eq. 84) without changing the network function, and a correspondingly rescaled Adam learning rate leaves training unchanged (up to Adam's ϵ\epsilon). The cross-model ratio ρ\rho, including where it crosses one, therefore depends on this convention rather than solely on the local geometry of the two parameterizations. Finally, the FID trajectory of the EDM-style control (final FID 1.77±0.021.77 \pm 0.02) is not shown, so it is not established that EDM2 converges faster than this specific control.

  4. While authors discuss possible limitations of their EDM2 ported baseline, and show improvements from augmentation and scheduling, it is not clearly established why the adapted EDM2 baseline underperforms EDM, nor if it was tuned (at all, or how extensively), prior to introducing the proposed changes. Would grid searching hyper parameters for this baseline, like done so extensively for the improved recipes, have resulted in large reductions in FID?

Questions for authors
  1. In Section 2.4 (lines 223-225), you state that "low-frequency structures are typically learned earlier during optimization, whereas high-frequency details are acquired later and are more difficult to optimize." Could you provide a reference for this frequency-principle claim? The nearby citations in lines 226-227 accompany the separate statement about the importance of high-frequency features in diffusion models.

  2. How should a practitioner select the learning-rate breakpoints/rates and frequency-loss strength/band for a new dataset and training budget (+ GPU hours, etc.)? Could you please provide some search procedure, number and duration of trials, and total tuning compute, including whether short runs or shared checkpoints reliably select settings with good final FID and persistence? Were the baselines given comparable tuning budgets? This guidance would help the community apply the findings and investigate their transfer to other architectures and diffusion formulations.

  3. Could you please include Configuration E alongside the two ablation curves in Figure 3 b, so that the mentioned delay in reaching the optimum (and the improved stability of high-performance regime) can be assessed directly? Also please clarify which configuration 'later' refers to in lines 369-371 -- the orange curve appears to reach its minimum earlier than the blue curve. (E is shown in Figs 1 and 10, but including all three curves together would make the progressive comparison much easier to follow)

  4. Have you measured the Hessian-based local gradient sensitivity of Configurations E and F across training, particularly around the onset of E's FID deterioration? Does E show an increase in sensitivity while F exhibits a different trajectory? If this comparison has not been performed, could you add it to help connect the geometry analysis to the stability interventions? Evaluating both configurations would likely have to occur under a common probe objective to help distinguish geometry changes from the impact of uncertainty-based loss weighting removal.

  5. Noise-dependent weights amplify individual samples' gradient contributions, but larger gradients do not generally imply larger parameter updates under Adam, which (approximately) cancels constant global gradient scaling through its moment normalization. Since your weighting varies across noise levels, it may also change gradient directions and variability. Has actual parameter-update magnitudes or relative update sizes been measured to support the proposed step-size explanation in Section 2.5 (lines 361-365)?

  6. After publication, could you release the complete code and exact configurations for the reported experiments, together with pretrained models and intermediate training/EMA checkpoints? For Figure 4, could you also include the EDM-style control implementation and configuration, parameter-scale measurements, and code/data for the sensitivity ratios? These artifacts would make it easier to reproduce the analysis and explore whether the findings transfer to other settings without repeating all training from scratch.

  7. Could you please clarify why the adapted EDM2 baseline underperforms EDM on all three datasets, and what tuning was performed before settling on Configuration A? In particular, how were its augmentation and learning-rate settings selected, and were the EDM and EDM2 comparisons given comparable tuning budgets?

  8. Could you report the best FID (mean ±\pm standard deviation) across at least three training seeds for Configurations C-F on all three datasets, using a fixed checkpoint and EMA selection protocol, and rerun EDM under the same protocol? This would help establish which components provide improvements beyond training-run variability, including the λt\lambda_t ablation in Table 2.

  9. Could you show the FID trajectory of the EDM-style control model used in Section 3 and Appendix C.1, so that its convergence speed can be compared directly with the EDM2-style model?

  10. Could you please include qualitative samples for the baseline and final configurations on each dataset, ideally including checkpoints before and after the late-stage FID degradation of Configuration E? The paper currently contains no generated samples, which makes it difficult to assess what the reported FID changes correspond to visually.

Some minor points:

  • Figures 5 and 7 list a model capacity of 5.4 Mparams, while Appendix C.1 states approximately 54M parameters in the complete network. Please clarify.
  • "Duration (Mimg)" in Tables 3, 5, 6, and 8 appears to denote the selected checkpoint rather than the total training length (e.g., Configuration A on CIFAR-10 is listed as 50 Mimg, while Figure 1 shows its trajectory beyond 150 Mimg). Please clarify.

Multi-seed evidence for the incremental improvements, clarifying the baseline selection/tuning budgets, and also providing evidence relating the Hessian geometry analysis to the observed training behavior (Configs E, F in particular), would strengthen my rating of the paper.

8 comments

Response to Reviewer ysBt (Part 1: Questions 1-2)

We sincerely thank the reviewer for the thoughtful and constructive comments. We appreciate the recognition of our analysis of EDM2 training dynamics, particularly the distinction between rapidly achieving a good FID and stably maintaining that performance throughout training. We agree that the current manuscript would benefit from additional explanations, more comprehensive statistical evidence, and further disclosure of experimental details.

In the revised manuscript, we will address these concerns through additional analyses, experiments, visualizations, and implementation details. We respond to each question below.


Question 1: References Related to the Frequency Principle

Response:

Thank you for pointing out the need for clearer theoretical references. We will incorporate prior work on the spectral bias of neural networks, particularly Rahaman et al., On the Spectral Bias of Neural Networks (ICML 2019), and Xu et al., Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks (2019), to provide a more solid theoretical context for our discussion of the Frequency Principle.

These works provide theoretical and empirical evidence that neural networks tend to learn lower-frequency components of a target function before progressively learning higher-frequency components. This finding provides relevant theoretical support for our frequency-oriented analysis. We will also clarify in the manuscript that the Frequency Principle serves as a motivation for our investigation, rather than a standalone explanation for all the behaviors observed in diffusion models.

Question 2: Hyperparameter Search Strategy

Response:

Thank you for asking us to describe the hyperparameter selection process more systematically. We agree that, in addition to reporting the final hyperparameters, disclosing the search strategy, the number of experiments, and the corresponding computational cost is important for improving reproducibility.

We adopted dataset-specific hyperparameter search strategies, using the EDM2 configuration on ImageNet-64 and the publicly available EDM configurations for the corresponding datasets as initial references. Specifically, the initial learning rate and its decay strategy were primarily informed by the EDM2 setup on ImageNet-64. Most other hyperparameters, such as dropout, were initialized based on the EDM configurations for the corresponding datasets, since the ImageNet-64 EDM2 configuration itself also inherits many design choices from EDM.

(1) Initial pilot experiment

Before conducting the formal hyperparameter search, we first performed a pilot experiment using the aforementioned reference configuration and extended training to 200M images. The purpose was to observe how FID evolved throughout training and to approximately identify the training stage at which the best FID occurred. This pilot experiment showed that the best FID was reached before the end of training. Therefore, we focused our subsequent hyperparameter search primarily on this stage, avoiding unnecessarily long training runs for all candidate configurations.

We emphasize that this initial 200M-image experiment was conducted separately to explore the training dynamics and was not included in the subsequent formal grid search.

(2) Search over the initial learning rate and decay duration

For CIFAR-10, we tested 11 initial learning rates ranging from 0.005 to 0.025, with a step size of 0.002, combined with seven learning-rate decay durations: {50k, 35k, 25k, 20k, 15k, 10k, 7k}. This resulted in 77 combinations.

The results show that when the decay duration was 50k or 35k, relatively lower initial learning rates (approximately 0.010–0.015) performed better. However, their FID values remained above 2.15, with the best results occurring at approximately 75M training images. When the decay duration was 25k, 20k, or 15k, an initial learning rate of approximately 0.017 achieved better results, with FID values around 2.05–2.11 and the best results occurring at approximately 55M training images. In contrast, further reducing the decay duration to 10k or 7k substantially degraded generation quality, with most configurations producing FID values above 2.5. Based on these results, we selected an initial learning rate of 0.017 and a decay duration of 20k for the CIFAR-10 baseline.

For AFHQv2-64 and FFHQ-64, we narrowed the search ranges based on the preliminary results. For each dataset, we tested six initial learning rates ranging from 0.013 to 0.023, with a step size of 0.002, combined with four decay durations: {15k, 20k, 25k, 30k}. This resulted in 24 combinations for each dataset.

Therefore, the baseline search over the initial learning rate and decay duration involved 125 experiments across the three datasets. All these formal search experiments were trained for 80M images.

Response to Reviewer ysBt (Part 2: Question 2)

(3) Dropout and other hyperparameters

We also conducted a separate search over dropout values, using the EDM configuration for each dataset as the reference:

  • CIFAR-10: We tested 0.09, 0.11, 0.13, and 0.15, and selected 0.13.
  • AFHQv2-64: We tested 0.23, 0.25, 0.27, and 0.29, and selected 0.25.
  • FFHQ-64: We tested 0.03, 0.05, 0.07, and 0.09, and selected 0.05.

The candidate values for AFHQv2-64 and FFHQ-64 were selected around the corresponding EDM reference configurations, and the final values were determined based on the experimental results for each dataset.


(4) Configurations for subsequent experiments and search for the annealing starting point

For the subsequent experiment groups (Configurations B–F), we did not repeat the full grid search. Instead, we conducted smaller local searches around the learning rate and decay duration selected for the baseline.

Taking Configuration C on CIFAR-10 as an example, we tested the initial learning rates {0.015,0.017,0.019}\{0.015, 0.017, 0.019\} and decay durations {15k,20k,25k}\{15\mathrm{k}, 20\mathrm{k}, 25\mathrm{k}\}. The results showed that the better hyperparameter combinations identified by this local search were very close to those used in the baseline. Therefore, to ensure fair comparisons among subsequent modifications, we retained the baseline learning-rate and decay settings for the subsequent experiment groups. We followed the same principle for AFHQv2-64 and FFHQ-64, conducting limited local searches around their respective baseline configurations and ultimately adopting the same learning-rate and decay settings as the baselines. Dropout was also selected following the same general approach.

After introducing learning-rate annealing, we adjusted the total number of training steps for some datasets according to the changes in the training stage at which the best FID occurred.

The annealing starting point was selected as follows. First, we identified the training time corresponding to the best FID, denoted by TbestT_{\mathrm{best}}. We then selected a checkpoint close to 0.5Tbest0.5T_{\mathrm{best}} and saved the complete training state. Starting from this checkpoint, we resumed training and tested learning-rate turning points at the following candidate times:

0.5Tbest,0.6Tbest,0.7Tbest,0.8Tbest.0.5T_{\mathrm{best}},\quad 0.6T_{\mathrm{best}},\quad 0.7T_{\mathrm{best}},\quad 0.8T_{\mathrm{best}}.

For each resumed run, we restored the model parameters, EMA parameters, optimizer state, and training progress, rather than loading only the model weights. Consequently, all candidate annealing schedules shared exactly the same training trajectory before branching, and differed only in the learning-rate schedule adopted from the specified turning point onward. For subsequent searches over later turning points, we also reused checkpoints from the preceding turning-point experiments to reduce redundant training.


(5) Computational cost

All 125 formal baseline experiments for the learning-rate and decay-duration search were trained for 80M images. Each experiment used four NVIDIA A30 GPUs, and all experiments used the same global training random seed. The following table reports the wall-clock time per experiment and the total GPU hours for each dataset, where total GPU hours are calculated as Wall-clock time per run×Number of runs×4\text{Wall-clock time per run} \times \text{Number of runs} \times 4:

DatasetNumber of runsWall-clock time per runTotal GPU hours
CIFAR-107717 h 32 min 20 s~5,402
AFHQv2-64241 d 8 h 18 min~3,101
FFHQ-64241 d 9 h 12 min~3,187
Total125—~11,690

These figures estimate the computational resources required for the formal baseline hyperparameter search. The initial 200M-image pilot experiment, subsequent local searches, dropout searches, and annealing starting-point searches are not included in the above total of 125 experiments.

Response to Reviewer ysBt (Part 3: Question 2)

(6) General hyperparameter tuning procedure

Based on these experiments, we summarize a practical tuning procedure for adapting EDM2 to a new dataset:

  1. Select an initial reference configuration. Start from a well-established training configuration for a publicly available dataset of a comparable scale. Conduct a pilot experiment to observe the FID trajectory, identify the approximate location of the best performance, and examine subsequent performance trends.
  2. Jointly search the learning rate and decay duration. Conduct a joint search over the initial learning rate and decay duration around the reference configuration. Since these two hyperparameters are coupled, joint tuning provides more useful information than adjusting either parameter independently. The search range for the decay duration can be initially determined based on the dataset scale and pilot results.
  3. Determine the annealing starting point through checkpoint branching. First identify the training time corresponding to the best FID, TbestT_{\mathrm{best}}. Then save a complete checkpoint near 0.5Tbest0.5T_{\mathrm{best}}, resume training from it, and test different learning-rate turning points. This avoids restarting training for every candidate and therefore reduces computational cost. Subsequent searches over later turning points can also reuse checkpoints from earlier turning-point experiments.
  4. Further tune other hyperparameters. After determining a suitable learning-rate schedule, adjust other hyperparameters, such as dropout and loss-related parameters, including the frequency-loss weight and frequency-scale boundaries.
  5. Evaluate both peak performance and training stability. A configuration should not be selected solely based on its best FID during early training. Promising candidates should be trained further to examine their behavior after reaching their best performance, so that we can assess whether they can maintain generation quality over time.

We will include the above search details and computational costs in the appendix of the revised manuscript to improve the transparency and reproducibility of our experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.