acceptodds
Under review as a conference paper at ICLR 2027

On the Critical Batch Size for Low-Precision Training

Abstract

Large language models are increasingly trained with large batches and low-precision arithmetic to improve computational efficiency. The critical batch size (CBS) marks the onset of diminishing returns from larger batches, yet how this threshold changes with numerical precision remains poorly understood. We study the CBS of low-precision training through theoretical scaling laws for quantized stochastic gradient descent in sketched linear regression. Our analysis replaces the vector- and matrix-level error-alignment constraints of prior multiplicative quantization models with more practical bounds on the relative variance of independent, unbiased coordinate-wise rounding. Our theory predicts that lower precision increases the critical batch scale while preserving its scaling exponent with the data budget. The analysis explains this shift through a balance between quantization-induced noise and the cost of fewer optimization updates, accounting for parameter-rounding noise shared across a batch. We combine controlled regression experiments with matched-quality language-model comparisons to test the predicted scaling and its qualitative transfer to pretraining. This analysis motivates joint selection of numerical precision and batch size to exploit the useful range of data parallelism.

Primary area:
learning theory
Keywords:
critical batch size, low-precision training, scaling laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vzEyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how numerical precision affects the critical batch size in model training, using quantized stochastic gradient descent in linear regression. The authors develop theoretical risk bounds that predict lower precision favors larger batch sizes while leaving the scaling exponent with respect to training data unchanged. Experiments on synthetic data and language models show trends consistent with this prediction, although they do not conclusively establish the claimed relationship.

Critical strengths
  • The paper has solid theoretical analysis. It extends existing work on low-precision training by considering more realistic coordinate-wise quantization errors.
  • It tests the theory in real LLM pretraining.
Critical weaknesses
  • bf16 and ufp4 runs use different learning rates, making it difficult to isolate the effect of precision.
Questions for authors
  • Does the observed increase in critical batch size persist when bf16 and ufp4 use the same learning rate?
  • real LLM pretraining does not use constant learning rate. They either use WSD or cosine annealing. Do the observations/conclusions still hold under those learning rate schedules?
Reviewer_ZdtsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives risk bounds and a critical-batch-size law for quantized SGD in sketched linear regression under coordinate-wise unbiased rounding. Prediction: lower precision enlarges the critical batch size by a constant factor but leaves its data exponent unchanged. Validated on synthetic regression and one 302M language model trained in BF16 vs FP4 (UFP4).

Critical strengths
  • The analysis handles coordinate-wise rounding and batch-shared parameter noise within the existing framework, and tightens the bias bound in the smooth-target regime.
  • The prediction is simple and testable.
Critical weaknesses
  • The LM experiment does not support the claims: BF16 and FP4 use different learning rates although the theory says the critical batch scales with it, and the "identical exponent" is fit on five points over a 2.4× data range for one model with no seeds. The synthetic slopes also drift with ε at larger M, against the exponent-preservation claim.
  • The theoretical step beyond Zhang et al. (2026b) and Li et al. (2025) is incremental, and the practical conclusion (~10% larger CBS at FP4) is small relative to the uncontrolled factors.
Questions for authors
  1. Could you repeat BF16 vs FP4 at the same learning rate, with seeds?
  2. What if one extends the data range by at least 10× before claiming exponent preservation?
  3. Could you explain the slope change with ε in Fig. 1 at M=1536 and 2048?
  4. How do Adam and momentum, used in the experiments, change the prediction?
Reviewer_fmKXOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper provide theoretical justification as well as synthetic experiments that justifies larger critical batchsize to adopt for optimal loss in scaling laws under lower precision.

Critical strengths

clear motivation, clear conclusion on the role of precision errors to batchsize under sketching regresion.

Critical weaknesses

Bad presentation, complicated notation and theorem, which were not attempted to transform easy example and cases to explain the core intuition beyond the mere math result. Couple of missing labels in the figures, hard to exactly follow the point.

LLM pretraining under 4 bits is considered to be almost loseless

Questions for authors

"preserve the scaling exponent with respect to data budget D" is not clear, does it mean exponent of BF16 and FP4 is the same under what? First of all their assigned bit range are different, hard to believe they are the same unless presented. Or, if it means exponent bit is the same over the budget, why that is important? any empirical evidence? what is the role of block scaling factor like MX, NV, etc?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.