acceptodds
Withdrawn from ICLR 2027

EMA Without the Lag: Bias-Corrected Iterate Averaging Schemes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Stochasticity in language model fine-tuning, often caused by the small batch sizes typically used in this regime, can destabilize training by introducing large oscillations in generation quality. A popular approach to mitigating this instability is to take an Exponential moving average (EMA) of weights throughout training. While EMA reduces stochasticity, thereby smoothing training, the introduction of bias from old iterates often creates a lag in optimization relative to vanilla training. In this work, we propose the Bias-Corrected Exponential Moving Average (BEMA), a simple and practical augmentation of EMA that retains variance-reduction benefits while eliminating bias. BEMA is motivated by a simple theoretical model wherein we demonstrate provable acceleration of BEMA over both a standard EMA and vanilla training. Through a suite of experiments on Language Models, we show that BEMA leads to improved convergence rates over both EMA and vanilla training in a variety of LM benchmarks, making BEMA a practical and theoretically motivated intervention for more stable and efficient fine-tuning.

Primary area:
learning theory
Keywords:
Optimization, Iterate Averaging, Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_B3UVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BEMA, parameter averaging method designed to reduce the optimization lag introduced by standard EMA while retaining its variance-reduction benefits. The authors derive a bias-corrected estimator under a stochastic quadratic optimization model and develop a practical implementation that adds a correction term to EMA. Experiments on LLM FT demonstrate improved convergence and generation performance compared with standard EMA and other stabilization methods.

Critical strengths
  • Overall, this paper is well written and easy to follow.
  • BEMA is simple to implement and introduces minimal modifications to standard EMA.
Critical weaknesses
  • There is a gap between the theoretical analysis and the practical implementation of BEMA. The theoretical guarantees are established for an MLE involving the inverse Hessian and uniform iterate averaging, whereas practical BEMA replaces these with a scalar correction and polynomial-decay EMA. It remains unclear to what extent the theoretical properties carry over to the implemented method.
  • The empirical evaluation of generation quality is relatively limited. Although improving generation stability is a central motivation, the experiments primarily use small subsets of BoolQ, GSM8K, and MMLU-HS, with only two training seeds. This makes it difficult to assess how consistently the reported improvements generalize across broader generation tasks and independent training runs.
Questions for authors
  • BEMA calculates bias correction based on the displacement relative to an initial checkpoint. Could the authors clarify whether this correction method remains reliable over the course of a long training process? Furthermore, would periodically updating the reference checkpoint help improve performance?
  • The paper notes that certain BEMA configurations lead to improved generation accuracy, even when the cross-entropy loss is relatively high. Could the authors elaborate on the reasons behind this phenomenon? Specifically, does BEMA primarily improve the optimization convergence process, or does it steer the model toward checkpoints that exhibit better generative performance?
Reviewer_Cm7LOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BEMA is a small change to EMA weight averaging for LM fine-tuning. EMA smooths checkpoint-to-checkpoint oscillation but lags, because it keeps weight on early iterates. BEMA adds a correction, μ^t=αt(θt−θ0)+μ^tEMA\hat\mu_t = \alpha_t(\theta_t - \theta_0) + \hat\mu^{\text{EMA}}_t with αt=(ρ+γt)−η\alpha_t = (\rho+\gamma t)^{-\eta}, on top of a power-law EMA with weight βt=(ρ+γt)−κ\beta_t = (\rho+\gamma t)^{-\kappa}. With η=∞\eta = \infty it is plain EMA.

The theory models constant-LR SGD on a quadratic as an OU process and treats a stabilizer as an estimator of the minimizer. It gives a minimax lower bound, shows the last iterate is inconsistent, shows the flat average has a bias term from θ0\theta_0 that dominates early, and shows the MLE μ^TMLE=A−1T(θT−θ0)+1T∫0Tθt dt\hat\mu^{\text{MLE}}_T = \frac{A^{-1}}{T}(\theta_T-\theta_0) + \frac1T\int_0^T\theta_t\,dt is unbiased and matches the lower bound. BEMA is this MLE with A−1/TA^{-1}/T replaced by a scalar schedule and the flat average replaced by a power-law EMA.

Experiments fine-tune Qwen2.5-1.5B on Tulu-3-SFT with Adam, constant LR, two seeds. Stabilizers are applied after training to checkpoints saved every 400 steps. Metrics are train/test cross-entropy and generation accuracy on small BoolQ, GSM8K and MMLU-HS subsets. BEMA is compared to no stabilization, EMA, OUEMA and DEMA, with sweeps over its hyperparameters, LR decay, batch size, and two other 1B models. Claims: BEMA converges faster than EMA, ends higher, and the best setup has no LR decay.

Critical strengths

S1. The estimation framing is clean and useful. Seeing EMA, last iterate and averaging variants as estimators of the OU mean gives a lower bound, explains EMA lag as a bias term, and derives the practical rule from the MLE instead of guessing it. The theory is standard but correct.

S2. Easy to adopt, and the sweeps are thorough. Two extra weight copies, two lines on top of EMA. The appendix reports full heatmaps for every hyperparameter rather than the best cell, and the limitations section is honest.

Critical weaknesses

W1. The gain over a tuned EMA is within noise, and the headline comparison is not fair. The paper's own heatmaps show EMA with a good κ\kappa gets very close to the best BEMA cell on all three tasks, closer than the seed error bars in Table 3. Without LR decay in Fig. 8, the best EMA and the best BEMA cells are 0.90 vs 0.93 on BoolQ, 0.42 vs 0.44 on GSM8K and 0.35 vs 0.36 on MMLU-HS, while the seed standard errors in Table 3 are about 0.04, 0.10 and at least 0.03. Fig. 1 compares at one κ\kappa where EMA is heavily over-averaged, and it uses a different κ\kappa for the loss panel and the accuracy panel. The heatmap numbers are also the best checkpoint over training, which inflates everything. The acceleration claim looks real. The "ends higher" claim is not supported with two seeds and 64 to 128 prompts.

W2. The EMA baseline is not standard EMA, and the obvious lag-free baselines are missing. All stabilizers are computed after training over checkpoints at a stride of 400 with a power-law weight. That is not the per-step EMA people actually run, and Fig. 11 shows the stride matters a lot. The simplest fix for lag is to drop early iterates: tail averaging or LAWA. LAWA is cited but not compared, tail averaging is not cited. The burn-in ablation is the closest thing and it hurts a lot, which goes against the paper's story that the lag is bias from θ0\theta_0. The stride also changes what the EMA is. In Algorithm 1, tt counts gradient steps while the update happens only every ϕ=400\phi = 400 steps, so βt=(10+t)−κ\beta_t = (10+t)^{-\kappa} is already small at the first update. At the default κ=0.5\kappa = 0.5, about 15 updates over 6000 steps leave roughly 70% of the weight on θ0\theta_0, the pretrained model, at the end of training. The flat average in the theory leaves 1/T1/T. So the EMA baseline is anchored to the starting point much more than a standard EMA, and much of the lag that BEMA removes may come from this choice.

W3. Smaller points. κ\kappa is picked by sweep and is problem dependent in the paper's own data (small for cross-entropy, large for accuracy, zero for Gemma), and η\eta sits next to a collapse point, so a user needs a 2D sweep per task. "No LR decay is best" only compares decay schedules at the same peak, not a smaller constant LR, which the theory itself points to. On Gemma and Llama the loss improves but downstream does not, and vanilla sometimes wins. Table 3 has two unexplained numbers per cell. One dataset, one main model, no code. The symbol η\eta means both LR scale and bias power. Novelty is modest: the MLE form is textbook, the authors say so; the contribution is the framing and the practical rule.

Questions for authors

Answers could change my score.

On W1. (1) Please compare EMA at its best κ\kappa against BEMA at its best (κ,η)(\kappa,\eta), at the final checkpoint, with error bars. Is the final-accuracy gain still there? (2) Why does Fig. 1 use κ=0.2\kappa = 0.2 for loss and κ=0.5\kappa = 0.5 for accuracy? (3) Can you add seeds or evaluation prompts? Two seeds is thin given Table 3. (4) What are the two numbers per cell in Table 3?

On W2. (5) Please add a standard per-step constant-β\beta EMA and tail averaging or LAWA as baselines. (6) Algorithm 1 suggests that tt in αt,βt\alpha_t, \beta_t counts gradient steps, not stabilizer updates. Please confirm and state the effective EMA window and the remaining weight on θ0\theta_0 at the end of training. (7) Does BEMA still beat EMA when the EMA keeps little weight on θ0\theta_0, for example with per-step updates? Why does burn-in hurt so much? (8) Is BEMA at κ=0\kappa = 0, which is just θ0+(1+αt)(θt−θ0)\theta_0 + (1+\alpha_t)(\theta_t - \theta_0), what actually helps on Gemma?

On W3. (9) Is there any rule for choosing κ\kappa and η\eta on a new task? (10) Does "no LR decay is best" hold against a smaller constant LR? (11) Will code be released?

Suggestions, not affecting the score: a second SFT dataset; a note on the relation to Adam's bias correction, which is the same idea on the gradient EMA.

Reviewer_gkHsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new stabilization method for deep learning optimizers. Building on top of exponential moving average (EMA), the paper introduces BEMA (Bias-corrected exponential moving average). The authors argue that EMA reduces stochasticity of model training, but introduces lag, slowing down the optimization process (particularly when small batches are updated like in large language models). BEMA is theoretically grounded and shown to be highly performant for LLM fine-tuning. The authors fine-tuned three LLMs of <1.5B scale on Tulu-3-SFT dataset and showed the performance on three benchmarks.

Critical strengths
  • Strong theoretical grounding - The authors provide rigorous theoretical justification behind the asymptotic behavior of BEMA by designing the optimization as a parameter estimation problem. Theoretical results are subsequently validated empirically establishing the authenticity.

  • Practical and easy to use - BEMA is a simple extension to EMA, therefore, making it compatible with existing optimizers very easily.

Critical weaknesses
  • Relatively weak experimental setup - All fine-tunings were conducted on small scale LLMs (<1.5B) and a single fine-tuning objective. The authors choose to evaluate the fine-tuned models only on three small scale benchmarks.

  • Early/quick convergence is not shown empirically - In both fig 1 and 5, it seems the training loss is not converged yet, therefore, the asymptotic behavior is not clearly visible.

  • Effectiveness over OUEMA is not very clear. Looks like BEMA performs very similar to OUEMA.

Questions for authors
  • Readers would be interested to see the performance of BEMA when a model is trained from scratch. Small-scale pre-training experiment would help assessing that capability.

  • See the first weakness: Add more fine-tuning objectives - instruction-tuning/language modeling/RL-based.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.