acceptodds
Under review as a conference paper at ICLR 2027

How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

Abstract

We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called *three-term law*). Fitting the proposed law on a large set of training runs, we find that it correctly recovers the scaling of the optimal batch size. Making use of training runs with suboptimal batch size, our proposed law can be robustly fit with a significantly smaller amount of training runs. We further show that the three-term law can be used to derive scaling laws for suboptimal batch sizes, and that it matches previous empirical findings related to the critical batch size.

Primary area:
optimization
Keywords:
scaling laws, LLM pretraining, hyperparameter scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4V9wOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper studies power laws associated with large scale LLM pretraining. In the famous chinchilla papers, the authors have devised a scaling law that determines the loss after pretraining as a function of the number of model parameters N and the training samples D, associated with corresponding scaling exponents. The paper expands this into a three term law that takes into account the batch size.
  • The authors use the data of Li at al and OpenEuroLLM to fit the coefficients. The procedure is performed using a Huber loss. The resulting fits are reported and analyzed.
  • For me, the main finding is that using the more complex parametric form is advantageous because the data can be fitted with runs at different batch sizes, which are usually collected anyways in the preparation of large runs.
  • Fitting literature curves with this three parameter model is the papers main contribution. In the considered cases this is beneficial because it works with less data points. However the dependence on other factors such as the learning rate is only weakly adressed.
Critical strengths
  • The authors address a critical challenge: Determine the optimal setup for pre-training a ML model with doing as little experimentation upfront, yet achieving optimal results for a given computational budget. I think the functional form of the scaling laws has so far not been given enough attention and including the batch size as explicit independent variable is a very useful step. Enabling determining the scaling laws from a wider range of runs, also with sub-optimal batch size is important.
  • The authors are very careful to show that the the most important literature results are reproduced by their fitted scaling laws. Most importantly, the optimal batch size and its dependence on the token budget are reproduced, a key finding of the recent years.
Critical weaknesses
  • I don't think we learn enough from the paper. Yes, we can add a dependency on the batch size to a power law. But the proposed functional form is not motivated (which leads to the observed E=0 case), it is merely adding a general dependency on batch size in the spirit of a Taylor Expansion. Furthermore, I don't think that cross-validation is a good metric for assessing the quality of a fit. Crucially, the power law models are used to do an informed decision about how to build a large-scale model. For this, any model is used in an extrapolative fashion. In my view, it would be critical to asses any model by what prediction it makes on the Scale of, let's say ten trillion tokens and 100 Billion parameters. Here, we would need a much more analysis of errors. Assessing these prediction with a careful analysis of the statistical errors would be absolutely required. Unfortunately, I see no single confidence interval, not even speaking of a chi-square analysis.
  • I am not sure the authors are aligned with the SOTA in the field. Most of todays approaches use the maximum update parameterization of hyperparameters to ensure optimal control of the scaling behavior of models. With this parametrization, the transfer of optimal batch sizes between model sizes is guaranteed, and recent works have studied for example studied in depth he relationship between the optimal learning rate and the batch size [1] and the dependency of the optimal batch size on the token budget [2].

[1] Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. Surge phenomenon in optimal learning rate and batch size scaling. arXiv preprint arXiv:2405.14578, 2024. [2] Filatov, Oleg, et al. "Time transfer: On optimal learning rate and batch size in the infinite data limit." arXiv preprint arXiv:2410.05838 (2024).

Questions for authors

None.

Reviewer_KxcHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an alternative scaling law form that factors the typical data term into two separate terms, one for the batch size and the other for the number of steps. This third term increases the number of parameters that need to be fit for the scaling law from 5 to 7. This form can be used to estimate the optimal batch size given a compute budget, and further means we can fit our scaling law with runs at suboptimal batch sizes potentially reducing the computational cost of fitting scaling laws in general.

Critical strengths

[S1] The three term law reduces to the established chinchilla scaling law.

[S2] Can estimate epsilon suboptimal batch sizes from 3TL.

Critical weaknesses

[W1] The optimal batch size estimates from the proposed 2TL and 3TL are quite similar to step law estimates, especially for 3TL. Even at 42% and 28% data the Step Law estimates seem as good as 3TL, which undermines the sample efficiency argument.

[W2] Some held-out evaluation to compare which methods extrapolate better would be valuable. That is fit the laws at small scale and then forecast configs at larger scale, train and evaluate them. It doesn't need to be a much larger scale but right now the validation points have nearly the same compute budget as the largest training point.

[W3] The consistency across datasets section hypothesizes the differences in fits has to do with the differences between the scaling ladders they are fit on. This is not something discovered in this work but a well established detail about scaling laws. The law fit is specific to the training setup the ladder was built on and will not necessarily generalize to other training configs.

Questions for authors

[Q1] Practically, is there a sell for why to use 3TL over Chinchilla with Step Law for hyperparameter scaling?

1 comment

[W1] Thank you for clarifying! The authors should consider revising the legends in figure two to more clearly state this. I would suggest replacing the explicit fit in the legend with something clear like “StepLaw”, and then change the black line to “StepLaw (Full Data)”.

I suspect the StepLaw “collapse” at 28% is at least in part due to how the grid is sampled, would be interesting to ablate this construction to see if a different sampling of the scaling ladder would perform better. My impression is that sampling two batch sizes per config is more harmful to the fit than if you were to more densely sample batch sizes at fewer configurations.

[W2] The authors have entirely overlooked my point. I am not asking for larger models necessarily. What this paper needs is evidence that the law extrapolates well or at least better than prior work. The whole reason we build scaling laws is so we can make informed decisions about how specific architectures and training configurations behave at much larger scales than where the law is fit. You have no evidence this law extrapolates. Some options for you:

  • Split the train and validation points by compute budget, tokens, and model size, so that you can demonstrate extrapolation along these three axes. Though it would be limited extrapolation, it would at least be a step in the right direction. To make this very concrete, you can drop the ~1B models from the scaling ladders, fit the laws on the remaining runs and measure how well you predict the ~1B models. This experiment does not require you to train any models, only fit scaling laws so its essentially free.
  • Run prescriptive experiments. Contrary to what the authors suggest, this does not require sweeping configurations at large scale. Pick a compute budget and for each scaling law you want to evaluate determine the compute optimal configuration, then train and evaluate that model. You can compare your law to the StepLaw in this way with only two runs. Using this protocol you can show that your reduced data fits produce nearly as good models to the full data fit.

[W3] Okay but this doesn’t explain the anomalies in your results. For example, you scaling fit on the Li scaling ladder sets E approximately 0.

[Q1] Great, thank you. This makes sense.

I appreciate the authors efforts during rebuttal. I will maintain my weak reject score, primarily because my central concern regarding extrapolation [W2] is unaddressed.

Reviewer_NYLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a scaling law that models loss using model size, batch size, and training steps separately. Under a fixed token budget, optimizing batch size recovers a Chinchilla-type loss model and an optimal batch-size scaling rule. The authors fit the law to two existing collections of AdamW training runs. On the Li dataset, retaining two batch sizes per configuration recovers similar scaling from 68 rather than 246 selected observations. The paper also studies near-optimal batch-size intervals and critical-batch-size behavior.

Critical strengths

The formulation connects batch-size scaling with loss prediction and makes useful use of suboptimal training configurations. Recovering a similar batch-size scaling rule from substantially fewer sampled settings is a promising practical result.

Critical weaknesses
  1. Figure 11 shows systematic errors near the token-budget boundaries. The near-optimal interval analysis consequently requires separate fits for each model size and token budget, limiting the usefulness of the global law.

  2. The 68/246 ratio counts observations after learning-rate selection. Learning-rate sweeps are still required, and training costs vary across configurations, so this ratio does not establish equivalent savings in FLOPs or GPU hours.

  3. The reduced-sweep experiment does not report variability across independently resampled batch-size selections. Five-fold cross-validation does not show whether a different pair of retained batch sizes would give equally reliable predictions.

  4. Section 4.3 derives the “at most 5%” compute penalty from the external EpochAI law without validating its loss-versus-token relationship on the evaluated training setups.

  5. Table 6 implies compute-optimal model-size exponents of approximately 0.374 and 0.669 for the three-term and directly fitted Chinchilla laws. This substantial disagreement is not resolved by validation on held-out model sizes.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.