- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BEMA, parameter averaging method designed to reduce the optimization lag introduced by standard EMA while retaining its variance-reduction benefits. The authors derive a bias-corrected estimator under a stochastic quadratic optimization model and develop a practical implementation that adds a correction term to EMA. Experiments on LLM FT demonstrate improved convergence and generation performance compared with standard EMA and other stabilization methods.
Critical strengths
- Overall, this paper is well written and easy to follow.
- BEMA is simple to implement and introduces minimal modifications to standard EMA.
Critical weaknesses
- There is a gap between the theoretical analysis and the practical implementation of BEMA. The theoretical guarantees are established for an MLE involving the inverse Hessian and uniform iterate averaging, whereas practical BEMA replaces these with a scalar correction and polynomial-decay EMA. It remains unclear to what extent the theoretical properties carry over to the implemented method.
- The empirical evaluation of generation quality is relatively limited. Although improving generation stability is a central motivation, the experiments primarily use small subsets of BoolQ, GSM8K, and MMLU-HS, with only two training seeds. This makes it difficult to assess how consistently the reported improvements generalize across broader generation tasks and independent training runs.
Questions for authors
- BEMA calculates bias correction based on the displacement relative to an initial checkpoint. Could the authors clarify whether this correction method remains reliable over the course of a long training process? Furthermore, would periodically updating the reference checkpoint help improve performance?
- The paper notes that certain BEMA configurations lead to improved generation accuracy, even when the cross-entropy loss is relatively high. Could the authors elaborate on the reasons behind this phenomenon? Specifically, does BEMA primarily improve the optimization convergence process, or does it steer the model toward checkpoints that exhibit better generative performance?