- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the training behavior of EDM2 on small image datasets. The authors find that their EDM2 baseline performs worse than EDM and can lose generation quality during later training. They introduce several changes, including data augmentation, learning-rate scheduling, weight centralization, and a frequency-aware loss. They also study how EDM2 components affect late-stage performance and use Hessian analysis to examine its optimization behavior.
Critical strengths
-
The paper shows several techniques that improve EDM2 on small datasets. The best reported configurations outperform EDM in their settings.
-
The paper provides explanations for the proposed changes and ablations to examine their effects. It also studies the training trajectories beyond the best FID reached during training.
Critical weaknesses
-
The motivation is not clear enough. EDM2 already outperforms EDM in the original paper, including on ImageNet-64 at a similar model size and with fewer sampling evaluations. Here, the authors build their own EDM2 configuration for smaller datasets, where the weaker performance may come from hyperparameters that are not well suited to these settings. The paper needs to separate this possibility from a problem specific to EDM2. The original EDM2 paper also discusses late-stage overfitting and the benefit of dropout. Although this paper already uses dropout, it remains unclear how the behavior studied here differs from that observation.
-
The contribution beyond tuning the training recipe seems limited. Data augmentation and learning-rate scheduling are useful, but they are standard tools and account for most of the FID improvement in Table 1. Weight centralization and the frequency-aware loss provide much smaller additional gains, and their motivations need stronger support. For weight centralization, zero-mean inputs and targets do not imply that intermediate features should also have zero mean. Normalization methods such as RMSNorm work without explicitly centering features. The relation between RMS, mean, and variance does not by itself show that the observed mean offsets harm generation quality. It is unclear whether the small improvement comes from reducing these offsets or from other effects of the changed weight parameterization. For the frequency-aware loss, the model already works well without it. Adding this loss changes FID from 1.70 to 1.70 on CIFAR-10, 1.98 to 1.93 on AFHQv2, and 2.02 to 1.99 on FFHQ. Why is middle-frequency reconstruction a problem that needs to be addressed here? The paper does not show that it is a bottleneck in EDM2 training. The connection between this loss and the training problems studied in the paper is therefore unclear.
-
I would like to see a stronger, well-tuned EDM2 baseline and clearer evidence of what these changes add beyond standard tuning. The training-dynamics analysis should also explain what new understanding it provides beyond the original EDM2 paper.
-
The scope is quite limited. The paper proposes several improvements, but only tests them on EDM2 and small datasets. Whether these changes also help on larger datasets, where EDM2 already works well, is still unknown. Their usefulness for other architectures is also unclear. If these techniques only help EDM2 under specific small-dataset settings, I see limited contribution beyond adapting its training recipe to those settings.
Questions for authors
-
How were the baseline learning rate, decay schedule, and dropout rates selected? Does the late-stage degradation remain after tuning these settings under a comparable training budget?
-
What evidence shows that activation mean offsets and middle-frequency reconstruction errors are responsible for the observed training problems? Can the authors separate these explanations from other effects of the proposed changes?
-
Do the proposed changes improve EDM2 on ImageNet-64, where the original paper already provides a strong configuration? This would help clarify whether the findings extend beyond the small-dataset settings studied here.
8 comments
Response to Reviewer NNVz (Part 2: Questions 1)
(4) Configurations for subsequent experiments and search for the annealing starting point
For the subsequent experiment groups (Configurations B–F), we did not repeat the full grid search. Instead, we conducted smaller local searches around the learning rate and decay duration selected for the baseline.
Taking Configuration C on CIFAR-10 as an example, we tested the initial learning rates and decay durations . The results showed that the better hyperparameter combinations identified by this local search were very close to those used in the baseline. Therefore, to ensure fair comparisons among subsequent modifications, we retained the baseline learning-rate and decay settings for the subsequent experiment groups. We followed the same principle for AFHQv2-64 and FFHQ-64, conducting limited local searches around their respective baseline configurations and ultimately adopting the same learning-rate and decay settings as the baselines. Dropout was also selected following the same general approach.
After introducing learning-rate annealing, we adjusted the total number of training steps for some datasets according to the changes in the training stage at which the best FID occurred.
The annealing starting point was selected as follows. First, we identified the training time corresponding to the best FID, denoted by . We then selected a checkpoint close to and saved the complete training state. Starting from this checkpoint, we resumed training and tested learning-rate turning points at the following candidate times:
For each resumed run, we restored the model parameters, EMA parameters, optimizer state, and training progress, rather than loading only the model weights. Consequently, all candidate annealing schedules shared exactly the same training trajectory before branching, and differed only in the learning-rate schedule adopted from the specified turning point onward. For subsequent searches over later turning points, we also reused checkpoints from the preceding turning-point experiments to reduce redundant training.
(5) Computational cost
All 125 formal baseline experiments for the learning-rate and decay-duration search were trained for 80M images. Each experiment used four NVIDIA A30 GPUs, and all experiments used the same global training random seed. The following table reports the wall-clock time per experiment and the total GPU hours for each dataset, where total GPU hours are calculated as :
| Dataset | Number of runs | Wall-clock time per run | Total GPU hours |
|---|---|---|---|
| CIFAR-10 | 77 | 17 h 32 min 20 s | ~5,402 |
| AFHQv2-64 | 24 | 1 d 8 h 18 min | ~3,101 |
| FFHQ-64 | 24 | 1 d 9 h 12 min | ~3,187 |
| Total | 125 | — | ~11,690 |
These figures estimate the computational resources required for the formal baseline hyperparameter search. The initial 200M-image pilot experiment, subsequent local searches, dropout searches, and annealing starting-point searches are not included in the above total of 125 experiments.
Response to Reviewer NNVz (Part 3: Questions 1)
(6) General hyperparameter tuning procedure
Based on these experiments, we summarize a practical tuning procedure for adapting EDM2 to a new dataset:
- Select an initial reference configuration. Start from a well-established training configuration for a publicly available dataset of a comparable scale. Conduct a pilot experiment to observe the FID trajectory, identify the approximate location of the best performance, and examine subsequent performance trends.
- Jointly search the learning rate and decay duration. Conduct a joint search over the initial learning rate and decay duration around the reference configuration. Since these two hyperparameters are coupled, joint tuning provides more useful information than adjusting either parameter independently. The search range for the decay duration can be initially determined based on the dataset scale and pilot results.
- Determine the annealing starting point through checkpoint branching. First identify the training time corresponding to the best FID, . Then save a complete checkpoint near , resume training from it, and test different learning-rate turning points. This avoids restarting training for every candidate and therefore reduces computational cost. Subsequent searches over later turning points can also reuse checkpoints from earlier turning-point experiments.
- Further tune other hyperparameters. After determining a suitable learning-rate schedule, adjust other hyperparameters, such as dropout and loss-related parameters, including the frequency-loss weight and frequency-scale boundaries.
- Evaluate both peak performance and training stability. A configuration should not be selected solely based on its best FID during early training. Promising candidates should be trained further to examine their behavior after reaching their best performance, so that we can assess whether they can maintain generation quality over time.
We will include the above search details and computational costs in the appendix of the revised manuscript to improve the transparency and reproducibility of our experiments.
1.2 After tuning these settings under a comparable training budget, does the late-stage performance degradation still persist?
Response: Yes, it still persists. Through progressive refinement of hyperparameter configurations and training strategies, we found that late-stage performance degradation cannot be simply attributed to improper selection of the initial learning rate or decay duration.
- In our initial pilot experiment, we directly adopted EDM2's initial learning rate and decay settings from ImageNet-64 together with EDM's Dropout setting on CIFAR-10, training the model up to 200M images. We observed that the model reached a relatively good FID around 40M images (though still approximately ), and as training continued, the FID deteriorated further above near 200M images.
- Considering that EDM2 uses markedly different learning rate and decay settings on ImageNet-64 and ImageNet-512, we hypothesized that these hyperparameters are strongly dataset-dependent. Thus, we first systematically searched the learning rate and decay duration, obtaining the tuned Configuration A. However, as shown in Figure 1(a), late-stage performance degradation remained prominent even after this tuning.
- Subsequently, considering the small scale of CIFAR-10 and the potential risk of severe overfitting, we introduced the exact same data augmentation strategy as EDM (differing only in the conditioning injection mechanism; see Appendix B.2), yielding Configuration B. This modification substantially improved generation quality, bringing the FID close to EDM levels.
- However, Configuration B did not eliminate the late-stage issue. Compared with Configuration A, the late-stage FID of Configuration B did not exhibit simple monotonic deterioration, but instead showed severe oscillations, failing to stably maintain generation quality. We hypothesized that the learning rate near the optimum was too large and therefore applied learning rate annealing. Although annealing eliminated the FID oscillations and improved peak generation quality, late-stage FID degradation reappeared.
These results demonstrate that merely tuning the learning rate, decay duration, and Dropout—or introducing data augmentation and learning rate annealing—is insufficient to fully resolve the late-stage training degradation of EDM2 under the current small-scale dataset settings. Therefore, we further investigated other factors affecting training dynamics and model stability, as discussed in Section 2.5 of the manuscript.
Response to Reviewer NNVz (Part 1: Questions 1)
We sincerely thank the reviewer for the thoughtful and constructive comments. We appreciate the reviewer’s recognition of our analysis of EDM2 training dynamics. We agree that the current manuscript would benefit from additional explanations, more comprehensive statistical evidence, and further disclosure of experimental details. In the revised manuscript, we will address these concerns through additional analyses, new experiments, visualizations, and implementation details. We respond to the reviewer’s comments point by point below.
Question 1: How were the learning rate, learning rate decay strategy, and Dropout ratio of the baseline model selected? After tuning these settings under a comparable training budget, does the late-stage performance degradation still persist?
1.1 How were the learning rate, learning rate decay strategy, and Dropout ratio of the baseline model selected?
Response: We thank the reviewer for asking us to describe the hyperparameter selection procedure more systematically. We agree that, in addition to reporting the final hyperparameter values, disclosing the search strategy, number of experiments, and corresponding computational cost is important for improving reproducibility.
We adopted hyperparameter search strategies tailored to each dataset, using the EDM2 configuration on ImageNet-64 and the publicly available EDM configuration for the corresponding dataset as initial references. Specifically, the initial learning rate and its decay schedule were primarily referenced from the EDM2 configuration on ImageNet-64. Most other hyperparameters (e.g., Dropout) were initially referenced from the EDM configuration for the corresponding dataset, since the EDM2 configuration on ImageNet-64 itself inherits many design choices from EDM.
(1) Initial pilot experiment Before conducting the formal hyperparameter search, we first performed a pilot experiment using the above initial configuration and extended training to 200M images to observe the FID trajectory and roughly identify the training stage at which the best FID occurred. This pilot experiment showed that the best FID was reached before the end of training. Therefore, our subsequent hyperparameter search primarily focused on this stage to avoid unnecessarily long training for all candidate configurations. We emphasize that this initial 200M-image training run was conducted to explore training dynamics and was independent of the subsequent formal hyperparameter grid search.
(2) Learning rate and decay duration search
Therefore, the baseline learning rate and decay duration searches across the three datasets comprised 125 experiments in total. All these formal search experiments were trained for 80M images.
(3) Dropout and other hyperparameters We also conducted independent Dropout searches based on the EDM reference configuration for each dataset:
The candidate values for AFHQv2-64 and FFHQ-64 were selected around the corresponding EDM reference configurations, and the final values were determined based on the respective experimental results.