- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how numerical precision affects the critical batch size in model training, using quantized stochastic gradient descent in linear regression. The authors develop theoretical risk bounds that predict lower precision favors larger batch sizes while leaving the scaling exponent with respect to training data unchanged. Experiments on synthetic data and language models show trends consistent with this prediction, although they do not conclusively establish the claimed relationship.
Critical strengths
- The paper has solid theoretical analysis. It extends existing work on low-precision training by considering more realistic coordinate-wise quantization errors.
- It tests the theory in real LLM pretraining.
Critical weaknesses
- bf16 and ufp4 runs use different learning rates, making it difficult to isolate the effect of precision.
Questions for authors
- Does the observed increase in critical batch size persist when bf16 and ufp4 use the same learning rate?
- real LLM pretraining does not use constant learning rate. They either use WSD or cosine annealing. Do the observations/conclusions still hold under those learning rate schedules?