acceptodds
Under review as a conference paper at ICLR 2027

Large Diffusion Models Need More Than EMA

Abstract

Large-scale diffusion models are commonly trained with a constant learning rate (LR) and exponential moving average (EMA), despite the strong performance of LR decay and Schedule-Free optimization in training large language models. We study whether EMA is uniquely suited to diffusion training, or whether stronger alternatives have been underexplored. Through the lens of variance reduction, we theoretically show that averaging a constant-step trajectory has an inherent optimization limitation, whereas LR decay and Schedule-Free optimization admit vanishing stationarity guarantees under standard nonconvex assumptions. Empirically, LR decay and EMA provide complementary rather than substitutable benefits: decay alone does not outperform EMA with a constant LR, while combining decay with EMA further improves this standard baseline at most tested starting points, although the gain remains sensitive to when decay begins. Schedule-Free avoids this timing requirement and consistently outperforms EMA with a constant LR across training stages, achieving both lower fixed-evaluation objectives and better generation quality. Our analysis further shows that all three strategies substantially reduce parameter variability, yet stronger stabilization is not always better: EMA with a constant LR retains comparatively higher parameter variability, while aggressive decay combined with EMA achieves the lowest checkpoint variability but also weakens continued optimization. Schedule-Free substantially reduces parameter variability while preserving more late-stage progress, providing a better balance between parameter stability and continued optimization. Finally, the benefits of Schedule-Free extend beyond pretraining to ReFL and generator-side DMD, demonstrating its transferability across distinct diffusion optimization settings. Overall, our results suggest that large diffusion models need more than parameter averaging alone: effective optimization requires balancing variance reduction with continued optimization.

Primary area:
generative models
Keywords:
Optimization, Diffusion model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gcUZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares constant-LR AdamW + EMA, LR decay + EMA, and Schedule-Free AdamW for large diffusion model training. The main result is that Schedule-Free gives better FID and fixed-evaluation loss on a 675M SiT-XL/2 + REPA setup, while LR decay can also help but is sensitive to when decay starts. The paper explains these results through a trade-off between parameter stability and continued optimization.

Critical strengths
  • The paper studies a practical optimization question on a reasonably large and modern diffusion setup. The finding that LR decay and EMA are complementary is useful.

  • The analysis goes beyond final FID by also examining fixed-evaluation loss, parameter variability, and late-stage parameter movement. This gives a useful view of the trade-off between stability and continued optimization.

Critical weaknesses
  • The main empirical gains are not sufficiently validated across training randomness or hyperparameter choices. The reported FID values use multiple sampling seeds, but I could not find independent training-seed repetitions. This matters because the late-stage FID improvement of Schedule-Free over constant LR + EMA is fairly small. Also, Schedule-Free uses a larger LR, while the AdamW + EMA baseline does not appear to receive a comparable LR/EMA tuning budget.

  • The theoretical claim about vanishing stationarity for Schedule-Free should be stated more carefully. The general nonconvex bound in Theorem B.9 still contains a term proportional to the learning rate and gradient noise, so a fixed learning rate does not directly give a vanishing bound. The theorem also controls the gradient-evaluation sequence yty_t, rather than the evaluation weights xTx_T used in practice.

Questions for authors
  1. Can the authors provide independent training-seed repetitions for the main Schedule-Free vs. constant LR + EMA comparison, and a more matched hyperparameter sweep for AdamW + EMA?

  2. Can the authors clarify the precise conditions under which the Schedule-Free stationarity guarantee vanishes, especially for fixed learning rate and for the evaluation sequence xTx_T?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gcUZ (Part 1/2)

We thank Reviewer gcUZ for recognizing the practical relevance of our optimization study, the use of a modern large-scale diffusion setup, and the value of analyzing parameter variability and late-stage optimization dynamics beyond final FID. We also appreciate the reviewer’s recognition that LR decay and EMA can play complementary roles. We address the two concerns below.

1. Regarding W1 & Q1: Empirical robustness and hyperparameter tuning

To address the concern that our AdamW baseline might be suboptimally tuned, we have added a comprehensive learning-rate and EMA-decay ablation in Appendix A.5. We sweep four learning rates (5e-5, 1e-4, 2e-4, 3e-4) and three EMA coefficients (0.999, 0.9999, 0.99995). The results show that the official baseline configuration remains the strongest among the tested AdamW settings. At 400k steps, its FID-5K is 9.2139, compared with 9.0504 for our main Schedule-Free run.

We also report a learning-rate stability ablation in Appendix A.4. Standard AdamW with a learning rate of 4e-4, matching the Schedule-Free learning-rate scale, develops sustained NaNs after approximately 218.7k steps, whereas Schedule-Free remains stable under the same learning rate and warmup.

Furthermore, we have added a multi-seed robustness analysis in Appendix A.6, comparing two independent AdamW training seeds (0 and 88) under the baseline configuration. Their 400k FID-5K results are 9.2139 and 9.2230, respectively, supporting the reproducibility of the baseline.

We acknowledge that we do not yet have an independent Schedule-Free training-seed repetition. Due to limited computational resources, we could not complete such a run within the rebuttal period. Importantly, we did not conduct a multi-seed sweep to select the best Schedule-Free result; the reported result comes from our main training run. Thus, these additional experiments strengthen the validation of the AdamW baseline but do not quantify Schedule-Free's training-seed variability.

AuthorsOpenReview ↗

Response to Reviewer gcUZ (Part 2/2)

2. Regarding W2 & Q2: Scope of the theoretical guarantees

We agree that the scope of our original nonconvex guarantee requires further clarification. In the revised manuscript, the original Eq. (84), now renumbered as Eq. (87), retains a variance-dependent term proportional to γL(1−β)σ2\gamma L(1-\beta)\sigma^2 under fixed γ>0\gamma>0 and β<1\beta<1, so the bound does not vanish. This is a limitation of the worst-case guarantee, rather than proof that the actual stationarity error cannot vanish.

To clarify the scope of the existing guarantees, we have expanded Appendices B.4.2 and B.4.3. Appendix B.4.2 restates the existing convex objective-gap guarantee for the final averaged sequence x_T, following Defazio et al. (2024), while explicitly characterizing its residual term under a fixed step size. Appendix B.4.3 presents the corresponding nonconvex stationarity guarantee for the gradient-evaluation sequence y_t, clarifying that a vanishing bound requires a horizon-dependent base learning rate and does not establish last-iterate convergence of x_T.

More importantly, motivated by the residual variance term in Eq. (87) and the increasing-beta strategy explored in ScheduleFree+, we have added Appendix B.4.4 to extend the Lyapunov analysis to increasing interpolation coefficients under a fixed base learning rate and uniform averaging. Theorem B.11 establishes a schedule-dependent nonconvex stationarity bound, and Theorem B.12 specializes it to the log-linear annealing strategy used in ScheduleFree+.

For fixed endpoints beta_i < beta_f < 1 and an increasing annealing horizon, the leading variance-dependent contribution approaches gamma L sigma^2 [(1-beta_i)+(1-beta_f)]/2, compared with gamma L (1-beta_i) sigma^2 under constant beta_i. The remaining averaging term vanishes as O(log T/T).

These results show that increasing beta_t can reduce the variance-dependent bottleneck under a fixed base learning rate, although it does not eliminate the residual when the final interpolation coefficient remains below one. This provides theoretical support for the beta-annealing strategy used in ScheduleFree+ under the stated Schedule-Free SGD assumptions.

Reviewer_8jNqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares constant learning rate with EMA, late learning-rate decay, and Schedule-Free optimization for diffusion training. On SiT-XL/2 with REPA, Schedule-Free improves generation quality, with additional tests on reward fine-tuning and distillation.

Critical strengths
  • The paper studies a practical training choice that can improve diffusion models without changing their architecture.
  • The matched decay branches and fixed-data loss measurements make the comparisons useful.
  • The additional ReFL and DMD tests help show where the optimizer works and where it does not.
Critical weaknesses
  • The optimizer comparison needs a stronger tuning control. An unstable AdamW run at 4e-4 does not rule out better intermediate learning rates, warmup settings, or EMA choices.
  • Most checkpoint gains are small FID-5K differences. Three sampling seeds are used, but their spread and variation across independent training runs are not reported.
  • The nonconvex guarantee needs qualification. Equation (84) retains a noise term at fixed learning rate and momentum, and concerns gradient-evaluation points rather than the final averaged model.
  • The main pretraining evidence comes from one SiT+REPA setting. Most of the total-loss gain comes from the projection term, so the result without REPA remains unclear.
Questions for authors
  • How does Schedule-Free compare with the best retuned AdamW+EMA and decay+EMA settings under the same compute budget?
  • Can you report uncertainty for the FID differences and a matched FID-50K comparison that also includes decay+EMA?
  • Which parameter choices make the bound in Equation (84) vanish, and how do they relate to the fixed-parameter AdamW experiments?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8jNq (Part 1/3)

We thank Reviewer 8jNq for recognizing the practical value of our optimization study, the usefulness of our matched training comparisons and fixed-data loss measurements, and the additional insights from ReFL and DMD. We appreciate the reviewer’s constructive feedback and address the theoretical concerns below.

Regarding W3 & Q3: Scope and interpretation of the nonconvex guarantee

We agree that our original theoretical statement requires qualification. In the revised manuscript, Appendix B.4.3 clarifies that Eq. (87) retains a variance-dependent residual under fixed γ>0\gamma>0 and β<1\beta<1. With a suitable horizon-dependent base learning rate, the bound yields O(T−1/2)O(T^{-1/2}) best-iterate stationarity for the gradient-evaluation sequence yty_t, rather than a convergence guarantee for the final averaged weights xTx_T. Ordinary SGD also admits the same asymptotic order under comparable step-size choices.

Importantly, our theoretical analysis is not intended to rank optimization strategies by their worst-case convergence bounds, which generally cannot determine their relative performance in practical diffusion training. Instead, we use these bounds to characterize the strengths and limitations of different strategies, particularly how they balance variance reduction with continued optimization.

To provide a more complete theoretical understanding of Schedule-Free, we have made two further additions. First, Appendix B.4.2 restates the existing convex objective-gap guarantee for the final averaged weights xTx_T, providing a more general characterization beyond our scalar quadratic analysis.

More importantly, motivated by the variance-dependent residual in Eq. (87) and the increasing-β\beta strategy in ScheduleFree+, we added Appendix B.4.4 to extend the Lyapunov analysis to increasing interpolation coefficients. Theorem B.11 establishes a schedule-dependent nonconvex stationarity bound, while Theorem B.12 directly analyzes the log-linear annealing strategy used in ScheduleFree+.

For fixed endpoints βi<βf<1\beta_i<\beta_f<1, the leading variance-dependent contribution approaches

γLσ22[(1−βi)+(1−βf)]\frac{\gamma L\sigma^2}{2}\left[(1-\beta_i)+(1-\beta_f)\right]

as the annealing horizon increases, compared with γL(1−βi)σ2\gamma L(1-\beta_i)\sigma^2 under constant βi\beta_i. The remaining averaging term vanishes as O(log⁡T/T)O(\log T/T).

This provides theoretical support for how β\beta annealing can alleviate the late-stage variance bottleneck while keeping the base learning rate fixed, although the residual does not vanish.

These SGD results clarify the optimization trade-offs but do not directly establish convergence guarantees for the experimental SF-AdamW optimizer.

AuthorsOpenReview ↗

Response to Reviewer 8jNq (Part 2/3)

Regarding W4: Scope of the SiT+REPA experiments

Our study is motivated by a practical question: whether the widely adopted Constant LR + EMA recipe remains the best optimization choice for modern large-scale diffusion training. We therefore use SiT-XL/2 with REPA as a representative modern training setup. REPA substantially accelerates diffusion-transformer training, and removing it would require a considerably larger pretraining compute budget to reach comparable generation quality. Our goal is to evaluate optimization strategies in a strong, practical training setting, rather than establish optimizer superiority across all diffusion objectives or explain why individual components of the REPA objective respond differently to a particular optimizer.

We acknowledge that most of the fixed-evaluation total-loss improvement comes from the REPA projection term, as reported in Section 3.3. However, the denoising objective also improves: Appendix C.3 (Table 11) shows that Schedule-Free achieves lower denoising MSE at all eight evaluated fixed timesteps.

These results show that the observed objective improvement is not confined to the projection-loss term. However, they do not establish that the same gains would hold without REPA, which remains an open question.

AuthorsOpenReview ↗

Response to Reviewer 8jNq (Part 3/3)

1. Regarding W1 & Q1: Hyperparameter tuning and matched comparisons

To address the concern that the AdamW baseline may be insufficiently tuned, we have added a learning-rate and EMA ablation in Appendix A.5 (Table 5), covering four learning rates 5e-5, 1e-4, 2e-4, 3e-4 and three EMA coefficients {0.999,0.9999,0.99995}\{0.999,0.9999,0.99995\}.

At the matched 400k-step training budget, the official AdamW+EMA configuration remains the strongest among the tested AdamW settings, achieving FID-5K 9.2139 compared with 9.0504 for Schedule-Free. We also report a learning-rate stability control in Appendix A.4: standard AdamW at 4e-4, with the same 5k-step warmup as Schedule-Free, develops sustained NaNs after approximately 218.7k steps, whereas Schedule-Free remains stable.

These experiments strengthen our comparison against retuned AdamW+EMA configurations, although they do not exhaustively cover all learning-rate schedules or warmup settings.

2. Regarding W2 & Q2: FID uncertainty and training-seed robustness

We have expanded Appendix C.2 (Table 10) to report the mean and sample standard deviation over three sampling seeds for FID-5K, IS, and sFID. At 400k steps, the FID-5K results are:

MethodFID-5K
Schedule-Free9.0504 ± 0.0179
Constant LR + EMA9.2139 ± 0.1098
40k Decay + EMA9.1248 ± 0.2020

These standard deviations quantify sampling variability at fixed training checkpoints, rather than variability across independent training runs or statistical significance of the FID differences.

To further assess training-seed robustness, we have added Appendix A.6 (Table 6), comparing two independently trained AdamW baselines. Their 400k FID-5K results are 9.2139 and 9.2230, supporting the reproducibility of the baseline. Due to computational constraints, we could not complete an independent Schedule-Free training-seed repetition. Importantly, the reported Schedule-Free result comes from our main training run and was not selected from a multi-seed sweep.

Regarding the additional decay+EMA comparisons requested in Q1 and Q2: the relevant experiments are still being completed. We will report the matched comparisons, including FID-50K, in a subsequent response.

AuthorsOpenReview ↗

Additional response regarding LR decay (W1/Q1 and W2/Q2).

We have added matched long-horizon decay experiments in Section 3.2 (Figure 1(c), Appendix C.2, Table 12). At 800k steps, 40k-step cosine decay with EMA improves FID-50K from 1.9212 to 1.7319, approximately a 10% improvement over the corresponding Constant LR + EMA continuation. We observe a comparable improvement at 760k, further supporting the effectiveness of finishing decay at longer training horizons.

We have further reorganized Sections 2 and 3.4 around a training-stage-dependent stability–optimization trade-off. The unified nonconvex bounds motivate examining how diminishing transient contributions and potentially persistent variance-dependent terms affect the relative value of continued optimization and stronger stabilization. The revised analysis connects these theoretical insights to checkpoint variability, loss progress, and generation quality across training stages (Table 2, Figures 3–4).

Reviewer_y9SnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares constant-learning-rate AdamW with EMA, late-stage decay with EMA, and Schedule-Free AdamW for diffusion training. On a 675M-parameter SiT-XL/2 with REPA, Schedule-Free improves over constant LR + EMA. Additional experiments cover reward fine-tuning and distribution matching distillation (DMD). Simplified analyses interpret the results through parameter stability and continued optimization.

Critical strengths

S1. The main experiment addresses a useful practical question at substantial scale. At 800k steps, Schedule-Free improves FID-50K from 2.08 to 1.87 relative to constant LR + EMA, with improvements in sFID and Inception Score. This is the paper's strongest empirical result.

S2. The decay branches compare matched ending steps, and evaluating both online and EMA weights shows that decay and averaging can complement each other. The fixed-evaluation loss decomposition is also informative. The authors acknowledge mixed cross-reward results and worse DMD FID when Schedule-Free replaces the guidance optimizer.

Critical weaknesses

W1. The nonconvex guarantee does not support the stated contrast at fixed parameters. Eq. (84) retains the term γL(1−β)σ2/(1−pw)\gamma L(1-\beta)\sigma^2/(1-p_w), which does not vanish for fixed γ>0\gamma>0, β<1\beta<1, and nonzero gradient noise. Chae and Kim obtain a vanishing rate by choosing the base learning rate as a function of the training horizon (Corollary 4.6 [1]). Ordinary SGD admits a similar choice, as Theorem B.2 already notes. Lemma 2.4 and the surrounding discussion therefore need to specify this dependence and revise the comparison accordingly.

W2. The theoretical and conceptual novelty is limited. Most of Section 2 is standard or explicitly restated. The EMA reduction factor in Eq. (7) already appears in Zhang et al. [2, Theorem 2], and persistent bias under constant-step averaging has been analyzed more generally by Dieuleveut et al. [3]. Combining averaging with continued optimization is central to the original Schedule-Free work [5], while EDM2 studies decay jointly with averaging in diffusion training [4]. The explicit counterexample and quadratic Schedule-Free calculation are useful, but their additional theoretical insight needs clarification.

W3. The mechanism remains descriptive. The SGD analysis uses no diffusion-specific structure to predict the AdamW ranking or explain why most of the fixed-evaluation total-loss advantage comes from REPA's projection term. The general nonconvex bound controls stationarity at yty_t; convergence of the evaluated xtx_t is established only in the scalar quadratic analysis. Checkpoint spread includes systematic movement (Appendix A.3), and Figure 4 measures progress after decay has finished, with LR held at 10−610^{-6}. These diagnostics offer qualitative support, but do not establish that the proposed mechanism explains the observed optimizer ranking.

W4. The main comparison leaves tuning and averaging effects unresolved. Schedule-Free uses four times the baseline learning rate and adds warmup. The paper reports a single EMA coefficient of 9999/10000 for AdamW. AdamW's divergence at the larger rate is a useful control, but the reported evidence leaves its stable learning-rate range and alternative averaging timescales unresolved. Their interaction matters in diffusion training [4]. The intermediate FID gains are modest, and the stronger 800k result lacks a matched decay-plus-EMA comparison. Broader recommendations need clearer tuning evidence.

W5. The transfer experiments provide limited support for broader benefits. ReFL changes learning rate and warmup, and its cross-reward results are mixed (Appendix A.5). In DMD, AdamW is evaluated with online weights and generator-side Schedule-Free with averaged weights (Appendix A.6). The modest DMD FID gain therefore does not establish an advantage over AdamW with averaging. Generator-side Schedule-Free also worsens sFID at all three checkpoints (Table 6). The opposite FID effects of replacing the generator or guidance optimizer remain unexplained by the analysis.

Questions for authors

Q1. What parameter dependence makes Eq. (84) vanish, and what theoretical distinction from SGD remains under comparable choices?

Q2. What new insight does the analysis provide beyond the cited results? Which observation supports the proposed mechanism beyond trajectory smoothing and reduced progress after a finishing schedule?

Q3. What learning-rate, warmup, and EMA settings were explored for the main baseline? Are results available for alternative averaging timescales or decay plus EMA at 800k steps?

Q4. For ReFL and DMD, what tuning and averaging controls support attributing the gains to Schedule-Free? In particular, is an averaged AdamW DMD baseline available?

References

[1] Jiseok Chae and Donghwan Kim. Understanding Schedule-Free Methods in Nonconvex Optimization: Rate Guarantees and Escaping Saddles. 2026.

[2] Guodong Zhang et al. Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. NeurIPS, 2019.

[3] Aymeric Dieuleveut, Alain Durmus, and Francis Bach. Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains. arXiv:1707.06386, 2017.

[4] Tero Karras et al. Analyzing and Improving the Training Dynamics of Diffusion Models. CVPR, 2024.

[5] Aaron Defazio et al. The Road Less Scheduled. NeurIPS, 2024.

6 comments
AuthorsOpenReview ↗

Response to Reviewer y9Sn (Part 1/4)

We thank Reviewer y9Sn for recognizing the practical significance of our large-scale experiments, the complementary effects of LR decay and EMA, and the improvements in generation quality at 800k steps. We also appreciate the reviewer’s detailed theoretical feedback, which has helped us clarify and extend our analysis.

Regarding W1 & Q1: Nonconvex convergence guarantees and comparison with SGD

We agree that our original nonconvex statement requires qualification. We have revised Lemma 2.4 and Appendix B.4.3 to clarify that the original Eq. (84), now renumbered as Eq. (87), retains a variance-dependent residual under fixed γ>0\gamma>0, β<1\beta<1, and nonzero gradient noise.

With fixed β\beta and CC, the averaging rule in Eq. (86), and 1−pw1-p_w bounded away from zero, choosing a horizon-dependent base learning rate γT=min⁡{1L,1(1−β)T}\gamma_T = \min \lbrace \frac{1}{L}, \frac{1}{(1-\beta)\sqrt{T}} \rbrace

yields an O(T−1/2)O(T^{-1/2}) best-iterate stationarity bound for the gradient-evaluation sequence yty_t, following Chae and Kim (2026). Ordinary SGD achieves the same asymptotic order under comparable step-size choices; we do not claim a superior general nonconvex convergence rate.

Importantly, our theoretical analysis is not intended to rank optimization strategies by their worst-case convergence bounds, which generally cannot determine their relative performance in practical diffusion training. Instead, we use these bounds to characterize the strengths and limitations of different strategies, particularly how they balance variance reduction with continued optimization.

To provide a more complete theoretical understanding of Schedule-Free, we have made two further additions:

Convergence of the evaluation sequence xTx_T. Appendix B.4.2 (Theorem B.9) restates the existing convex objective-gap guarantee of Defazio et al. (2024). Under convexity and sample-wise Lipschitz assumptions, it provides an O(T−1/2)O(T^{-1/2}) bound for the final averaged weights xTx_T with a suitable horizon-dependent step size, complementing our scalar quadratic result in Appendix B.4.1.

Reducing the fixed-LR variance bottleneck. Motivated by the increasing-β\beta strategy in ScheduleFree+, we added Appendix B.4.4, extending the Lyapunov analysis to increasing interpolation coefficients. Theorem B.11 establishes a schedule-dependent nonconvex stationarity bound, while Theorem B.12 directly analyzes the log-linear annealing strategy used in ScheduleFree+. For fixed endpoints 12≤βi<βf<1\frac12\leq\beta_i<\beta_f<1, the leading variance-dependent contribution approaches

γLσ22[(1−βi)+(1−βf)]\frac{\gamma L\sigma^2}{2}\left[(1-\beta_i)+(1-\beta_f)\right]

as the annealing horizon increases, compared with γL(1−βi)σ2\gamma L(1-\beta_i)\sigma^2 under constant βi\beta_i. The remaining averaging term vanishes as O(log⁡T/T)O(\log T/T). This quantifies how β\beta annealing can alleviate the late-stage variance bottleneck while keeping the base learning rate fixed, although the residual does not vanish.

These results clarify the distinct guarantees for yty_t and xTx_T and extend our understanding of variance reduction in Schedule-Free SGD. They do not establish general nonconvex last-iterate convergence or guarantee the empirical ranking of SF-AdamW.

AuthorsOpenReview ↗

Response to Reviewer y9Sn (Part 2/4)

Regarding W2 & Q2: Theoretical novelty and additional insights

Our contribution is not a new general convergence-rate improvement, but a unified perspective connecting variance reduction, persistent optimization bias, and continued optimization in diffusion training.

We provide three complementary analytical results:

Persistent bias beyond quadratics (Lemma 2.2 / Theorem B.4). We construct a smooth, strongly convex nonquadratic objective whose constant-step SGD invariant mean satisfies mh=a(1−E[cos⁡U])>0=x⋆m_h=a(1-\mathbb E[\cos U])>0=x^\star. EMA preserves this biased mean, yielding a strictly positive expected objective-error lower bound. This explicitly distinguishes variance reduction from bias elimination.

Convergence of averaged evaluation weights (Appendix B.4.1 / Theorem B.8). For the stated quadratic model, Schedule-Free with a fixed stable base learning rate achieves O(T−1)O(T^{-1}) decay in the mean and variance of xTx_T, yielding an O(T−1)O(T^{-1}) expected objective gap.

New analysis of increasing interpolation coefficients (Appendix B.4.4). We extend the Lyapunov analysis of Schedule-Free SGD to increasing βt\beta_t. Theorem B.11 establishes a general schedule-dependent nonconvex stationarity bound, showing that the variance-dependent contribution can vanish under suitable annealing schedules even with a fixed base learning rate. Motivated by the log-linear β\beta annealing used in ScheduleFree+, Theorem B.12 further specializes this analysis to its practical schedule. For fixed endpoints 12≤βi<βf<1\frac12\leq\beta_i<\beta_f<1, the leading variance-dependent contribution approaches

γLσ22[(1−βi)+(1−βf)]\frac{\gamma L\sigma^2}{2}\left[(1-\beta_i)+(1-\beta_f)\right]

as the annealing horizon increases, compared with γL(1−βi)σ2\gamma L(1-\beta_i)\sigma^2 under constant βi\beta_i. Although this particular schedule retains a nonvanishing residual in the bound, the result theoretically supports the variance-reduction motivation behind the β\beta-annealing strategy explored in ScheduleFree+.

Our main contribution is a unified variance-reduction perspective. Rather than attempting to establish an optimizer ranking through multiple empirical diagnostics or worst-case convergence bounds, we analyze how EMA, LR decay, and Schedule-Free reduce stochastic variability through different mechanisms and the distinct optimization trade-offs they entail. This perspective clarifies their respective strengths and limitations, particularly the balance between parameter stability and continued optimization.

AuthorsOpenReview ↗

Response to Reviewer y9Sn (Part 3/4)

Regarding W3: Mechanism, REPA objectives, and diagnostic limitations

1. Why use a general stochastic optimization model? Our study is motivated by the contrast between the widespread use of LR scheduling in LLM training and the Constant LR + EMA recipe commonly used for large diffusion models. We investigate whether general principles of variance reduction and continued optimization can improve diffusion training, rather than derive an optimizer ranking from diffusion-specific assumptions. Our SiT+REPA experiments test these ideas under controlled architecture, data, and objective settings.

2. Why is the REPA projection-loss improvement larger? Our goal is to evaluate optimization strategies in a representative modern diffusion-training setup, rather than explain the different responses of individual loss components. As clarified in Section 3.3, most of the fixed-evaluation total-loss improvement comes from the REPA projection term. However, Appendix C.3 (Table 11) shows that Schedule-Free also achieves lower denoising MSE at all eight evaluated fixed timesteps. Thus, the improvement is not confined to the projection objective, although explaining the difference in magnitude remains beyond our scope.

3. What do the diagnostics establish? Appendix A.3.1 defines checkpoint variability as finite-window trace variance, which includes systematic parameter movement as well as stochastic fluctuations, rather than pure gradient-noise variance.

Figure 4 measures net parameter displacement and fixed-evaluation loss progress over 400k–405k, after decay has completed and the learning rate remains fixed at 10−610^{-6}. In this window, Decay+EMA exhibits the smallest parameter displacement but no further objective improvement, whereas Schedule-Free retains greater movement and continued loss reduction.

These diagnostics support the observed stability–progress trade-off, without establishing a causal explanation of the optimizer ranking.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Revisions and Responses

We thank the reviewers for their constructive feedback. The initial reviews raised important concerns regarding the theoretical contribution, the fairness and robustness of optimizer comparisons, and the explanatory depth of our empirical analysis. In response, we have substantially revised the manuscript, strengthened the theoretical analysis, and added extensive controlled experiments. The revised paper now provides a more unified and comprehensive account of variance reduction in diffusion optimization.

1. Strengthened theoretical analysis and clarified contributions (Section 2; Appendix B).

We have reorganized the theoretical analysis around a unified decomposition of nonconvex stationarity bounds into transient optimization and variance-dependent contributions (Section 2.4). Beyond clarifying the scope of existing results, we extend the nonconvex Lyapunov analysis of Schedule-Free SGD to increasing interpolation coefficients, establishing new schedule-dependent guarantees for β\beta annealing (Appendix B.4.4, Theorems B.11–B.12). We also clarify the distinction between gradient-evaluation and final averaged sequences (Appendix B.4.2–B.4.3). These revisions strengthen the theoretical foundation for understanding variance reduction and continued optimization.

2. Expanded optimizer controls and reproducibility experiments (Appendices A.4–A.6, A.8).

We have added systematic AdamW learning-rate and EMA-coefficient ablations (Appendix A.5, Table 5), confirming that the official REPA configuration remains the strongest among the tested AdamW settings. We further report independent baseline training runs (Appendix A.6, Tables 6–7), learning-rate stability controls (Appendix A.4), and a six-coefficient generator EMA ablation for DMD (Appendix A.8, Table 10). These experiments substantially strengthen the optimizer comparisons and address concerns about baseline tuning, training randomness, and averaging controls.

3. Extended long-horizon experiments and substantially revised the empirical analysis (Sections 3.2–3.4; Appendix A.3; Appendix C.2).

We have extended the finishing-decay experiments to 760k and 800k steps, using matched Constant LR + EMA continuations. The same 40k-step cosine decay improves FID-50K by approximately 10% at both endpoints (Figure 1(c), Table 12). We additionally analyze checkpoint variability, parameter displacement, and fixed-evaluation loss progress across training stages (Table 2, Figures 3–4). The results reveal a training-stage-dependent stability–optimization trade-off: Schedule-Free achieves a favorable balance between stabilization and continued optimization during mid-to-late training, whereas stronger finishing decay delivers substantial generation-quality improvements at longer horizons. The revised analysis directly connects these observations to the theoretical framework in Section 2.

4. Strengthened validation across diffusion training regimes (Section 3.5; Appendices A.7–A.8).

We have clarified the ReFL and DMD comparisons and expanded DMD evaluation with six generator-EMA controls. Generator-side Schedule-Free achieves better FID-5K than the best tested generator-EMA configuration at all three evaluated checkpoints (Appendix A.8, Table 10). We also substantially revised the analysis of the distinct generator and guidance optimization roles, explaining their different responses to Schedule-Free. Together with the ReFL results (Appendix A.7, Table 8), these findings strengthen the practical evidence beyond diffusion pretraining.

Overall, the revisions substantially address the central concerns raised during review. The revised manuscript presents stronger theoretical results, carefully controlled optimizer comparisons, additional long-horizon evidence, and a clearer account of when different variance-reduction strategies are effective. We believe these changes significantly strengthen both the technical rigor and the practical contribution of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.