- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper studies power laws associated with large scale LLM pretraining. In the famous chinchilla papers, the authors have devised a scaling law that determines the loss after pretraining as a function of the number of model parameters N and the training samples D, associated with corresponding scaling exponents. The paper expands this into a three term law that takes into account the batch size.
- The authors use the data of Li at al and OpenEuroLLM to fit the coefficients. The procedure is performed using a Huber loss. The resulting fits are reported and analyzed.
- For me, the main finding is that using the more complex parametric form is advantageous because the data can be fitted with runs at different batch sizes, which are usually collected anyways in the preparation of large runs.
- Fitting literature curves with this three parameter model is the papers main contribution. In the considered cases this is beneficial because it works with less data points. However the dependence on other factors such as the learning rate is only weakly adressed.
Critical strengths
- The authors address a critical challenge: Determine the optimal setup for pre-training a ML model with doing as little experimentation upfront, yet achieving optimal results for a given computational budget. I think the functional form of the scaling laws has so far not been given enough attention and including the batch size as explicit independent variable is a very useful step. Enabling determining the scaling laws from a wider range of runs, also with sub-optimal batch size is important.
- The authors are very careful to show that the the most important literature results are reproduced by their fitted scaling laws. Most importantly, the optimal batch size and its dependence on the token budget are reproduced, a key finding of the recent years.
Critical weaknesses
- I don't think we learn enough from the paper. Yes, we can add a dependency on the batch size to a power law. But the proposed functional form is not motivated (which leads to the observed E=0 case), it is merely adding a general dependency on batch size in the spirit of a Taylor Expansion. Furthermore, I don't think that cross-validation is a good metric for assessing the quality of a fit. Crucially, the power law models are used to do an informed decision about how to build a large-scale model. For this, any model is used in an extrapolative fashion. In my view, it would be critical to asses any model by what prediction it makes on the Scale of, let's say ten trillion tokens and 100 Billion parameters. Here, we would need a much more analysis of errors. Assessing these prediction with a careful analysis of the statistical errors would be absolutely required. Unfortunately, I see no single confidence interval, not even speaking of a chi-square analysis.
- I am not sure the authors are aligned with the SOTA in the field. Most of todays approaches use the maximum update parameterization of hyperparameters to ensure optimal control of the scaling behavior of models. With this parametrization, the transfer of optimal batch sizes between model sizes is guaranteed, and recent works have studied for example studied in depth he relationship between the optimal learning rate and the batch size [1] and the dependency of the optimal batch size on the token budget [2].
[1] Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. Surge phenomenon in optimal learning rate and batch size scaling. arXiv preprint arXiv:2405.14578, 2024. [2] Filatov, Oleg, et al. "Time transfer: On optimal learning rate and batch size in the infinite data limit." arXiv preprint arXiv:2410.05838 (2024).
Questions for authors
None.
[W1] Thank you for clarifying! The authors should consider revising the legends in figure two to more clearly state this. I would suggest replacing the explicit fit in the legend with something clear like “StepLaw”, and then change the black line to “StepLaw (Full Data)”.
I suspect the StepLaw “collapse” at 28% is at least in part due to how the grid is sampled, would be interesting to ablate this construction to see if a different sampling of the scaling ladder would perform better. My impression is that sampling two batch sizes per config is more harmful to the fit than if you were to more densely sample batch sizes at fewer configurations.
[W2] The authors have entirely overlooked my point. I am not asking for larger models necessarily. What this paper needs is evidence that the law extrapolates well or at least better than prior work. The whole reason we build scaling laws is so we can make informed decisions about how specific architectures and training configurations behave at much larger scales than where the law is fit. You have no evidence this law extrapolates. Some options for you:
[W3] Okay but this doesn’t explain the anomalies in your results. For example, you scaling fit on the Li scaling ladder sets E approximately 0.
[Q1] Great, thank you. This makes sense.
I appreciate the authors efforts during rebuttal. I will maintain my weak reject score, primarily because my central concern regarding extrapolation [W2] is unaddressed.