- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares constant-LR AdamW + EMA, LR decay + EMA, and Schedule-Free AdamW for large diffusion model training. The main result is that Schedule-Free gives better FID and fixed-evaluation loss on a 675M SiT-XL/2 + REPA setup, while LR decay can also help but is sensitive to when decay starts. The paper explains these results through a trade-off between parameter stability and continued optimization.
Critical strengths
-
The paper studies a practical optimization question on a reasonably large and modern diffusion setup. The finding that LR decay and EMA are complementary is useful.
-
The analysis goes beyond final FID by also examining fixed-evaluation loss, parameter variability, and late-stage parameter movement. This gives a useful view of the trade-off between stability and continued optimization.
Critical weaknesses
-
The main empirical gains are not sufficiently validated across training randomness or hyperparameter choices. The reported FID values use multiple sampling seeds, but I could not find independent training-seed repetitions. This matters because the late-stage FID improvement of Schedule-Free over constant LR + EMA is fairly small. Also, Schedule-Free uses a larger LR, while the AdamW + EMA baseline does not appear to receive a comparable LR/EMA tuning budget.
-
The theoretical claim about vanishing stationarity for Schedule-Free should be stated more carefully. The general nonconvex bound in Theorem B.9 still contains a term proportional to the learning rate and gradient noise, so a fixed learning rate does not directly give a vanishing bound. The theorem also controls the gradient-evaluation sequence , rather than the evaluation weights used in practice.
Questions for authors
-
Can the authors provide independent training-seed repetitions for the main Schedule-Free vs. constant LR + EMA comparison, and a more matched hyperparameter sweep for AdamW + EMA?
-
Can the authors clarify the precise conditions under which the Schedule-Free stationarity guarantee vanishes, especially for fixed learning rate and for the evaluation sequence ?
2 comments
Response to Reviewer gcUZ (Part 2/2)
2. Regarding W2 & Q2: Scope of the theoretical guarantees
We agree that the scope of our original nonconvex guarantee requires further clarification. In the revised manuscript, the original Eq. (84), now renumbered as Eq. (87), retains a variance-dependent term proportional to under fixed and , so the bound does not vanish. This is a limitation of the worst-case guarantee, rather than proof that the actual stationarity error cannot vanish.
To clarify the scope of the existing guarantees, we have expanded Appendices B.4.2 and B.4.3. Appendix B.4.2 restates the existing convex objective-gap guarantee for the final averaged sequence x_T, following Defazio et al. (2024), while explicitly characterizing its residual term under a fixed step size. Appendix B.4.3 presents the corresponding nonconvex stationarity guarantee for the gradient-evaluation sequence y_t, clarifying that a vanishing bound requires a horizon-dependent base learning rate and does not establish last-iterate convergence of x_T.
More importantly, motivated by the residual variance term in Eq. (87) and the increasing-beta strategy explored in ScheduleFree+, we have added Appendix B.4.4 to extend the Lyapunov analysis to increasing interpolation coefficients under a fixed base learning rate and uniform averaging. Theorem B.11 establishes a schedule-dependent nonconvex stationarity bound, and Theorem B.12 specializes it to the log-linear annealing strategy used in ScheduleFree+.
For fixed endpoints beta_i < beta_f < 1 and an increasing annealing horizon, the leading variance-dependent contribution approaches gamma L sigma^2 [(1-beta_i)+(1-beta_f)]/2, compared with gamma L (1-beta_i) sigma^2 under constant beta_i. The remaining averaging term vanishes as O(log T/T).
These results show that increasing beta_t can reduce the variance-dependent bottleneck under a fixed base learning rate, although it does not eliminate the residual when the final interpolation coefficient remains below one. This provides theoretical support for the beta-annealing strategy used in ScheduleFree+ under the stated Schedule-Free SGD assumptions.
Response to Reviewer gcUZ (Part 1/2)
We thank Reviewer gcUZ for recognizing the practical relevance of our optimization study, the use of a modern large-scale diffusion setup, and the value of analyzing parameter variability and late-stage optimization dynamics beyond final FID. We also appreciate the reviewer’s recognition that LR decay and EMA can play complementary roles. We address the two concerns below.
1. Regarding W1 & Q1: Empirical robustness and hyperparameter tuning
To address the concern that our AdamW baseline might be suboptimally tuned, we have added a comprehensive learning-rate and EMA-decay ablation in Appendix A.5. We sweep four learning rates (5e-5, 1e-4, 2e-4, 3e-4) and three EMA coefficients (0.999, 0.9999, 0.99995). The results show that the official baseline configuration remains the strongest among the tested AdamW settings. At 400k steps, its FID-5K is 9.2139, compared with 9.0504 for our main Schedule-Free run.
We also report a learning-rate stability ablation in Appendix A.4. Standard AdamW with a learning rate of 4e-4, matching the Schedule-Free learning-rate scale, develops sustained NaNs after approximately 218.7k steps, whereas Schedule-Free remains stable under the same learning rate and warmup.
Furthermore, we have added a multi-seed robustness analysis in Appendix A.6, comparing two independent AdamW training seeds (0 and 88) under the baseline configuration. Their 400k FID-5K results are 9.2139 and 9.2230, respectively, supporting the reproducibility of the baseline.
We acknowledge that we do not yet have an independent Schedule-Free training-seed repetition. Due to limited computational resources, we could not complete such a run within the rebuttal period. Importantly, we did not conduct a multi-seed sweep to select the best Schedule-Free result; the reported result comes from our main training run. Thus, these additional experiments strengthen the validation of the AdamW baseline but do not quantify Schedule-Free's training-seed variability.