- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Under the increasingly common data-limited setting for LLM pretraining, compute-optimal scaling laws are required to help guide the development of production models. These scaling laws often prescribe the amount of data, the model size, the number of times a data point is revisited, and as has been recently investigated, the number of models to include in an ensemble. This paper operates at the edge of our current understanding of ensembling, producing scaling laws for ensembled language models, and investigates a hitherto unexamined axis: hyperparameter scaling. In prior work, hyperparameters were held constant across model size and data budgets. However, a recent work (the Lovelace paper) showed that weight decay could be increased for larger models to further reduce the loss, especially reducing overfitting. This paper seeks an improved scaling law that takes into account these hyperparameter settings and better models the effect of ensembling. However, to be very clear, their final scaling law only takes ensembling into account, it does not take hyperparameters into account. The authors instead propose scaling laws for the learning rate and weight decay, based on model size, data size, and revisitations of the data.
The authors also provide an important correction to prior work: scaling laws, representing a Pareto-optimal curve, should monotonically decrease. Thus, prior work with scaling laws whose loss increases with overfitting are invalid, since a Pareto-optimal agent, optimising over a whole slew of variables, could always choose to maintain the loss at a prior compute budget, rather than increasing the loss due to overfitting. I would posit that the authors are getting close to an unwanted generalisation of scaling laws, namely, that if such an agent existed that could sweep over hypothesis classes, optimisers, data filterings, associated hyperparameters, and so on, the scaling laws would become useless to the crowd training transformer-based LLMs. Thus, while the rejection of these prior scaling laws is well-motivated, perhaps it signals a bifurcation of the field: 1) compute-optimal scaling laws, 2) compute-actual scaling laws?
Note that in the above, I do not mean to invalidate the work done in this paper: I really do think that the authors' analysis of ensembling, learning rate, and weight decay scaling is interesting and important. I just want to highlight that being too strongly tied to theoretical correctness may lead to empirical uselessness.
Critical strengths
Firstly, the analysis of hyperparameter scaling is very interesting and is alone a clear criteria for acceptance. From my (admittedly) limited reading on the subject area, I believe this is an important factor for LLM training in production settings, and should be followed up with a more varied class of hyperparameters, and perhaps even their interplay. The authors did a fantastic job introducing, analysing, and validating their hypothesis.
Secondly, the analysis of ensembling as an extension of the Lovelace scaling law was well-motivated and especially the interplay with hyperparameter scaling made it evident that these were both useful elements of LLM production systems for attaining optimal losses.
Critical weaknesses
The only weakness I could readily identify in the entire setup and experiments was the transfer of these lower losses to improved downstream task performance, as was done in (some, not all) prior works.
Questions for authors
- From what I understood, for a given compute budget, the ensembling technique will result in smaller models. I am left wondering whether this adversely affects the performance achievable, since larger models have been shown to perform better in general? I understand that your scaling laws suggest that ensembling is more effective, but I somehow do not believe that this is always the case, nor that deploying M models will be possible / useful. For instance, an ML engineer having to serve the ensembled output of 4 models may have trouble loading them all in parallel (or not) and running them. How realistic is it to expect a production LLM to be deployed in such a manner?
- It seems that just sweeping the hyperparameters does not produce a monotonically decreasing curve. Why do you think that is? Comparing with the ensembled models, is it that the ensembled models somehow achieve such better learning performance that they do not reach the overfitting zone?
- When will it be necessary to specifically model overfitting in these scaling laws? I assume we are in agreement that with compute capacity outrunning data, at some point we will recover overfitting?
4 comments
Thank you for this detailed response! The table you present really solidified my confidence in and understanding of the work. I especially liked the nuance with respect to when ensembling is actually theoretically optimal. Very interesting!
Thank you!
Thank you! We put a lot of effort into these new experiments inspired by your thoughtful questions, and would really appreciate if you could consider raising your score in light of our response.
I unfortunately don't feel confident enough in my knowledge of the field to increase my score.
Rebuttal
Thank you for your thoughtful review and kind words! We are glad you found our work interesting and important.
We respond to your comments below. In particular, we add new results confirming that our lower losses translate to improved performance on downstream tasks. We also clarify our contributions around ensembling, monotonic gains with compute, and how our scaling law captures overfitting.
Below we provide new results confirming that our lower losses translate to improved downstream task performance.
We evaluate downstream performance for 12 configurations with varying U,N,R,M,C. For each configuration, we measure error rate on PIQA, SciQ, and ARC-Easy, following Kim et al. (2026), and BPB on the OLMES suite, following Lovelace et al. (2026).
Our validation loss is very strongly correlated with downstream task performance, with correlation coefficients of 0.96 (PIQA), 0.99 (SciQ), 0.98 (ARC-Easy), and over 0.99 (OLMES). These results confirm that our improved losses translate to improved downstream task performance. Additionally, our models compare favorably against the performance of models trained in Kim et al. (2026) and Lovelace et al. (2026) on comparable dataset sizes and compute budgets. We provide our full results table below.
You are correct that serving an ensemble carries engineering challenges. However, we think ensembling remains a relevant axis for scaling because:
It is not always the case that ensembling is compute-optimal. Ensembling is compute-optimal only for finite dataset sizes and sufficiently large compute budgets, as reflected by our scaling laws and empirical observations. Moreover, our new results on downstream tasks above confirm that any improvement in loss we achieve via ensembling also leads to stronger downstream performance.
Without ensembling, there is no guarantee that the predicted compute-optimal loss is monotonically decreasing with compute. Increasing compute implies we must increase epochs R and/or model size N, both of which exacerbate overfitting and therefore can result in worse performance. Scaling hyperparameters allows us to mitigate overfitting and observe gains for larger compute budgets, but for small dataset size U we still get non-monotonic gains as compute grows to infinity.
With ensembling, the predicted compute-optimal loss is guaranteed to monotonically decrease. As compute grows, we can always choose to increase the ensemble size M while holding N and R fixed. Since increasing M does not exacerbate overfitting, the optimal loss at a larger compute budget must be lower than the optimal loss at any smaller compute budgets.
Our scaling laws, as well as previous works on data-constrained scaling, already model “overfitting” in the sense that the validation loss will be higher than the (Chinchilla) train loss. This is exactly what is captured by the term P(R−1)δ(UγN)κ in Eq. (2). Indeed, for all our runs with R>1, we observe overfitting. If you are referring to non-monotonic loss as compute grows, please see our responses above.