acceptodds
Under review as a conference paper at ICLR 2027

Hyperparameter Transfer and Ensembling Improve Data-Constrained Scaling Recipes

Abstract

As growing compute budgets outpace the growth of high-quality training data, large language models must be trained with both compute and data constraints in mind. Unfortunately, existing prescriptions for data-constrained scaling suffer from two important limitations. First, whereas optimal hyperparameters vary wildly with model size and number of epochs, past works hold key hyperparameters such as weight decay fixed with scale. Second, past works neglect ensembling, a promising axis for scaling under data constraints. We address both limitations, leading to improved prescriptions for scaling data-constrained training pipelines. Our method prescribes how to scale key hyperparameters and allocate compute to scaling epochs, model parameters, or ensemble members. Controlling for training corpus size and compute budgets, our prescriptions achieve significantly lower validation loss than existing baselines, and this gap grows at larger scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, data-constrained training, ensembling, compute-optimal, hyperparameter transfer, weight decay

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_N4H3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Under the increasingly common data-limited setting for LLM pretraining, compute-optimal scaling laws are required to help guide the development of production models. These scaling laws often prescribe the amount of data, the model size, the number of times a data point is revisited, and as has been recently investigated, the number of models to include in an ensemble. This paper operates at the edge of our current understanding of ensembling, producing scaling laws for ensembled language models, and investigates a hitherto unexamined axis: hyperparameter scaling. In prior work, hyperparameters were held constant across model size and data budgets. However, a recent work (the Lovelace paper) showed that weight decay could be increased for larger models to further reduce the loss, especially reducing overfitting. This paper seeks an improved scaling law that takes into account these hyperparameter settings and better models the effect of ensembling. However, to be very clear, their final scaling law only takes ensembling into account, it does not take hyperparameters into account. The authors instead propose scaling laws for the learning rate and weight decay, based on model size, data size, and revisitations of the data.

The authors also provide an important correction to prior work: scaling laws, representing a Pareto-optimal curve, should monotonically decrease. Thus, prior work with scaling laws whose loss increases with overfitting are invalid, since a Pareto-optimal agent, optimising over a whole slew of variables, could always choose to maintain the loss at a prior compute budget, rather than increasing the loss due to overfitting. I would posit that the authors are getting close to an unwanted generalisation of scaling laws, namely, that if such an agent existed that could sweep over hypothesis classes, optimisers, data filterings, associated hyperparameters, and so on, the scaling laws would become useless to the crowd training transformer-based LLMs. Thus, while the rejection of these prior scaling laws is well-motivated, perhaps it signals a bifurcation of the field: 1) compute-optimal scaling laws, 2) compute-actual scaling laws?

Note that in the above, I do not mean to invalidate the work done in this paper: I really do think that the authors' analysis of ensembling, learning rate, and weight decay scaling is interesting and important. I just want to highlight that being too strongly tied to theoretical correctness may lead to empirical uselessness.

Critical strengths

Firstly, the analysis of hyperparameter scaling is very interesting and is alone a clear criteria for acceptance. From my (admittedly) limited reading on the subject area, I believe this is an important factor for LLM training in production settings, and should be followed up with a more varied class of hyperparameters, and perhaps even their interplay. The authors did a fantastic job introducing, analysing, and validating their hypothesis.

Secondly, the analysis of ensembling as an extension of the Lovelace scaling law was well-motivated and especially the interplay with hyperparameter scaling made it evident that these were both useful elements of LLM production systems for attaining optimal losses.

Critical weaknesses

The only weakness I could readily identify in the entire setup and experiments was the transfer of these lower losses to improved downstream task performance, as was done in (some, not all) prior works.

Questions for authors
  • From what I understood, for a given compute budget, the ensembling technique will result in MM smaller models. I am left wondering whether this adversely affects the performance achievable, since larger models have been shown to perform better in general? I understand that your scaling laws suggest that ensembling is more effective, but I somehow do not believe that this is always the case, nor that deploying M models will be possible / useful. For instance, an ML engineer having to serve the ensembled output of 4 models may have trouble loading them all in parallel (or not) and running them. How realistic is it to expect a production LLM to be deployed in such a manner?
  • It seems that just sweeping the hyperparameters does not produce a monotonically decreasing curve. Why do you think that is? Comparing with the ensembled models, is it that the ensembled models somehow achieve such better learning performance that they do not reach the overfitting zone?
  • When will it be necessary to specifically model overfitting in these scaling laws? I assume we are in agreement that with compute capacity outrunning data, at some point we will recover overfitting?
4 comments

Thank you for your thoughtful review and kind words! We are glad you found our work interesting and important.

We respond to your comments below. In particular, we add new results confirming that our lower losses translate to improved performance on downstream tasks. We also clarify our contributions around ensembling, monotonic gains with compute, and how our scaling law captures overfitting.

W1: The only weakness I could identify was the transfer of lower losses to improved downstream task performance, as was done in (some, not all) prior works.

Below we provide new results confirming that our lower losses translate to improved downstream task performance.

We evaluate downstream performance for 12 configurations with varying U,N,R,M,CU, N, R, M, C. For each configuration, we measure error rate on PIQA, SciQ, and ARC-Easy, following Kim et al. (2026), and BPB on the OLMES suite, following Lovelace et al. (2026).

Our validation loss is very strongly correlated with downstream task performance, with correlation coefficients of 0.96 (PIQA), 0.99 (SciQ), 0.98 (ARC-Easy), and over 0.99 (OLMES). These results confirm that our improved losses translate to improved downstream task performance. Additionally, our models compare favorably against the performance of models trained in Kim et al. (2026) and Lovelace et al. (2026) on comparable dataset sizes and compute budgets. We provide our full results table below.

ConfigPIQA (error %)SciQ (error %)ARC-Easy (error %)OLMES (BPB)
U=200M, M=2, N=1.17B, R=3632.524.052.21.174
U=200M, M=1, N=1.17B, R=3632.926.053.21.210
U=100M, M=16, N=848M, R=7634.927.855.51.276
U=100M, M=8, N=848M, R=7634.827.455.21.278
U=100M, M=4, N=848M, R=7635.029.055.51.291
U=100M, M=2, N=848M, R=7634.329.657.01.296
U=100M, M=1, N=848M, R=7636.030.757.71.359
U=50M, M=7, N=458M, R=10437.835.560.01.434
U=50M, M=4, N=458M, R=10437.735.359.51.450
U=50M, M=2, N=458M, R=10437.536.260.01.460
U=50M, M=1, N=458M, R=10438.438.761.41.522

Q1: How realistic is it to serve an ensemble as a production LLM?

You are correct that serving an ensemble carries engineering challenges. However, we think ensembling remains a relevant axis for scaling because:

  • In practice, an ensemble can often be effectively distilled into a single model, as shown in section 6.1 of Kim et al. (2026). Distillation can greatly reduce the resource requirements while preserving performance.
  • The main alternative to ensembling MM models of size NN is to train a single overparameterized model of size NMNM, which has comparable engineering challenges.

Q1: I understand that your scaling laws suggest that ensembling is more effective, but I somehow do not believe that this is always the case

It is not always the case that ensembling is compute-optimal. Ensembling is compute-optimal only for finite dataset sizes and sufficiently large compute budgets, as reflected by our scaling laws and empirical observations. Moreover, our new results on downstream tasks above confirm that any improvement in loss we achieve via ensembling also leads to stronger downstream performance.

Q2: Just sweeping hyperparameters does not produce a monotonically decreasing curve, whereas ensembling does. Why?

Without ensembling, there is no guarantee that the predicted compute-optimal loss is monotonically decreasing with compute. Increasing compute implies we must increase epochs RR and/or model size NN, both of which exacerbate overfitting and therefore can result in worse performance. Scaling hyperparameters allows us to mitigate overfitting and observe gains for larger compute budgets, but for small dataset size UU we still get non-monotonic gains as compute grows to infinity.

With ensembling, the predicted compute-optimal loss is guaranteed to monotonically decrease. As compute grows, we can always choose to increase the ensemble size MM while holding NN and RR fixed. Since increasing MM does not exacerbate overfitting, the optimal loss at a larger compute budget must be lower than the optimal loss at any smaller compute budgets.

Q3: When will it be necessary to specifically model overfitting in these scaling laws? I assume we are in agreement that with compute capacity outrunning data, at some point we will recover overfitting?

Our scaling laws, as well as previous works on data-constrained scaling, already model “overfitting” in the sense that the validation loss will be higher than the (Chinchilla) train loss. This is exactly what is captured by the term P(R−1)δ(NUγ)κP(R-1)^\delta \left( \frac{N}{U^\gamma} \right)^\kappa in Eq. (2). Indeed, for all our runs with R>1R > 1, we observe overfitting. If you are referring to non-monotonic loss as compute grows, please see our responses above.

Thank you for this detailed response! The table you present really solidified my confidence in and understanding of the work. I especially liked the nuance with respect to when ensembling is actually theoretically optimal. Very interesting!

AuthorsOpenReview ↗

Thank you!

Thank you! We put a lot of effort into these new experiments inspired by your thoughtful questions, and would really appreciate if you could consider raising your score in light of our response.

I unfortunately don't feel confident enough in my knowledge of the field to increase my score.

Reviewer_eHK5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The authors study compute-optimal training setups in the data-constrained regime through the lens of scaling laws.
  • This tackles the question, given a certain compute budget, what is the best compromise of model size and number of training tokens. While in the classical infinite-data regime of LLMs, they authors assume a finite dataset size. Then diminishing returns and potential non-monotonic overfitting affect the picture.
  • The authors choose a sophisticated experimentation setup. The vary the dataset size U, model size and number of epochs R. For each setup, they perform an optimization of effective learning rate at weight decay. The optimal hyperparameters are fitted with a functional form A U^\alpha N^\beta R^\gamma. Interestingly this implies the optimal weight decay grows with the number of training tokens NR. This fit is shown to be (slightly) better than some heuristics, and much better than others.
  • The authors introduce ensembling as an additional scaling axis. Here, M independent models are trained, and their prediction is averaged. The authors introduce a functional form of the scaling law with an additive correction for the number of ensemble members over the form by Lovelace et al, which is the established chinchilla form plus an additive correction accounting for the finite size of the considered dataset. If I'm not mistaken, the resulting functional form depends on four variables (U, N, R, M) and contains a total of 9 power-law exponents and 5 prefactors.
  • The authors evaluate the extrapolative power quantitatively in a "close" regime using the huber loss and qualitatively when increase the flops by one order of magnitude. Here, the functional form of Lovelace et al start producing artifacts as the loss starts going up when increasing the computational budget, even when the applied hyperparameter transfer methodology is applied. The fourteen-parameter function extrapolates without artifacts.
  • In consequence, their model predicts that when increasing the computational budget, N, R and M should grow together.
Critical strengths
  • The paper is written very clearly and experiments are described in sufficient detail without clutter. The methodology is very sound and the authors provide extensive original experimentation, state of the art technology and provide a technically very well executed paper. Graphs have high quality, ideas flow naturally, additional information is in available in the appendix but not necessary for reading. I only miss an estimate how many GPU hours were required and I have some fitting criticism - see below.
  • They authors tackle the classical tradeoff of model size and dataset repetitions and add the extra axis of deep ensembling, which I have, to be honest never thought of as a scaling axis. They show that under the considered conditions, it can be beneficial to trade repetitions of the data for a new ensemble member when scaling up compute - in principle a viable option even though I can't really imagine this will be an option in reality.
Critical weaknesses
  • I see two core weaknesses in the paper. Weakness 1 involves the practicability of ensembling. Is this really a realistic scaling axis? This involves increasing the number of weights proportionally, and correspondingly the required GPU memory requirement at inference time. Would it not be more realistic to look at the number of experts in MoE models, that might even affect the compute footprint in a weaker way? Alternatively scaling the model size of a dense model would scale the activation GPU memory only with the square-root of the scaling factor.
  • The second criticism I have is against the functional form of the power law. As we have learned from John von Neumann: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.". The authors use enough parameters for an entire herd of elephants, and this makes me very skeptical about the predictive power of the functional form. The author check the extrapolation over one order of magnitude qualitatively and find that no artifacts are found. I think this is not enough: The chinchilla paper evaluates extrapolating over two orders of magnitude, and this is justified as the this results in about 1M H100 hours, which is not very large compared to frontier model training budgets. And I'd even say: extrapolating over one order of magnitude is likely not very useful in practice: Then the prep-runs are easily as costly as the original runs. To show a methodology is viable it requires extrapolating further, and should include at least some kind of estimate how stable the predictions remain assuming a resampling measured losses.

The paper is very well executed, therefore, I'm okay overlooking the first point. The second point however is something were I think it would important pick up in the discussion, and therefore, also the questions I ask below.

Questions for authors
  • What happens if you extrapolate over two orders of magnitude of compute and perform an error analysis of the prediction regarding what is the best hyperparameter set? Maybe you could experiment with a synthetic model that generates data similar to your experimental data which you use to fit your model.
  • Is there really no better way than a 14-parameter-model? I see that delta, kappa, nu, epsilon and rho are all pretty small. Can you maybe absorb them? Key to a good fit is not the performance of the training data but only the reliability of the extrapolation.
3 comments

Thank you for your thoughtful review and kind words! We especially enjoyed the elephant analogy.

We respond to your questions below. In short, we explain why we believe ensembling is a realistic scaling axis and provide multiple new pieces of empirical evidence justifying our choice to use a 14-parameter scaling law form. We believe that this new evidence greatly strengthens our paper’s contributions, and would appreciate if you would consider raising your score. We are also happy to answer any further questions you have.

W1: Is ensembling a realistic scaling axis?

We believe it is. It is true that ensembling increases the total parameters, inference cost, and GPU memory requirement at inference time. However, we still believe this is a realistic scaling axis as:

  • In practice, an ensemble can often be effectively distilled into a single model, as shown in section 6.1 of Kim et al. (2026). Distillation can greatly reduce the resource requirements while preserving performance.
  • The main alternative to ensembling MM models of size NN is to train a single overparameterized model of size NMNM, which requires comparable resources.

In our experience, we found MoE models to be less effective than dense models for data-constrained training, but we agree with you that this would be an interesting direction for future work. We believe surfacing the renewed relevance of ensembles for multi-epoch is a key contribution.

W2 / Q1: What happens if you extrapolate over two orders of magnitude of compute?

We present new results demonstrating that our scaling law extrapolates accurately over two orders of magnitude of compute.

Specifically, we evaluate our scaling law’s prescription at a production budget of C=C = 6.2e20 FLOPs, representing a 75x larger run than our sweep budget. For the 100M token dataset, our scaling law prescribes the configuration N=N = 848M, R=R = 76, M=M = 16, and predicts a loss of 3.168. Running this configuration, we obtain a loss of 3.181. Compared to the loss of 3.195 (predicted 3.185) we obtained at a compute budget of 1e20 FLOPs, this result confirms that our prescriptions continue yielding monotonic improvements for significantly larger compute budgets, and that our scaling law predicts these improvements accurately.

W2 / Q2: Do we really need a 14-parameter-model? Can you absorb some parameters?

We found that all 14 parameters are needed to achieve the best extrapolation. Here, we provide new empirical results justifying this choice by comparing the extrapolation of our 14-parameter form to simpler forms, including the ones you propose.

Consider the Heldout Huber Loss in Figure 10, where we evaluate how well our form extrapolates to withheld runs at higher compute budgets. Below, we report the Heldout Huber Loss for various simplified forms. In addition to absorbing the five parameters you identified by setting them to 0, we also consider the two simplifications described in Lovelace et al. (2026): setting \delta \= \gamma \= 1 or setting \delta \= \gamma \= \kappa \= 1. The extrapolation error of our 14-parameter scaling law is at least 39% lower compared to any simplified form.

FormHeldout Huber Loss (×10−5\times 1{0}^{-5})
Ours (eq. 3)0.120
ν=0\nu =00.167
ϵ=0\epsilon =00.170
ρ=0\rho =00.268
δ=0\delta =00.449
κ=0\kappa =01.101
δ=γ=1\delta =\gamma =13.011
δ=γ=κ=1\delta =\gamma =\kappa =13.573

While we believe the full 14-parameter form is the best one for our data, we would encourage practitioners to use a similar cross-validation pipeline to identify the best form for their setting.

W2: You should estimate how stable the predictions remain when resampling measured losses.

Below we provide new results demonstrating that our predictions remain stable when resampling the observed runs.

Following previous work such as Hoffmann et al. (2022) and Lovelace et al. (2026), we use bootstrapping to measure the stability of the scaling law fits. Specifically, we create 250 bootstraps of the rungs used to fit our scaling law, each sampled with replacement. We then fit the scaling law parameters to each bootstrap. For 98% of bootstraps, our law achieves the best Heldout Huber Loss (as computed in Figure 10), verifying that the extrapolation remains accurate.

Additionally, we measure prescriptive regret as follows. For each bootstrap, we compute the prescribed configuration for $U=$100M tokens and $C=$1e20 FLOPs, and score the predicted loss of this configuration according to our main scaling law. For 60% of bootstraps, the prescribed configuration’s MM is identical to the optimal configuration of the main law and the prescribed NN is within 10%. For 80% of bootstraps, the prescribed configuration is predicted to score within 0.001 nats of the optimal configuration, and for 99% of bootstraps it is predicted to score within 0.004 nats, roughly the seed-to-seed noise of a single run.

Thank you for the kind comments.

Regarding the statistical analysis, I'm however still struggling. Maybe I miss it but I see nowhere a sign of the statistical analysis in the paper. I think bootstrapping is a great method for this task, and I think it would be worth mentioning that the method proves robust under bootstrapping. Why don't you mention your loss extrapolation result including bootstrapping error estimates? Why don't you provide error bars or a covariance matrix of the 14 parameters?

It might be that I'm to worried about all of this because your functional form is just a sum of monotonic functions, where not much can go wrong. But with a background in Physics, I have been taught that knowing what you know about fitted parameters is of prime importance.

Regarding weakness 1, I still have reservations against ensembling. I find it quite surprising to think that LLM pretraining is very convincing scenario regarding studying data-constrained (and infinite compute - shoutout to Kim et al.) strategies. But who am I to foresee if training replicas and then distilling them is maybe the best strategy. Anyways, your paper is done well and I'm ok going to weak accept if you add a bit of statistical analysis to the manuscript.

Thanks for your follow-up suggestions and for agreeing to raise your score!

We provide the requested results below, and will add all new experimental evidence and discussion points from the rebuttal, including bootstrap robustness, to the camera-ready version.

 

loss extrapolation result including bootstrapping error estimates

In the table below, we summarize the bootstrapped loss predictions for the 5 production budgets. At all budgets, the bootstrapped estimates are fairly tight around the main law’s prediction, with a standard deviation of less than 0.01 nats, and the 95% confidence intervals cover the actual observed loss.

CC (FLOPs)ObservedPredicted (main law)Bootstrap MedianBootstrap StdBootstrap 95% CI
1×10191\times 1{0}^{19}3.2703.2673.2670.002[3.263, 3.272]
2×10192\times 1{0}^{19}3.2383.2343.2350.003[3.229, 3.241]
4.5×10194.5\times 1{0}^{19}3.2103.2043.2050.005[3.195, 3.214]
1×10201\times 1{0}^{20}3.1953.1853.1850.007[3.171, 3.198]
6.2×10206.2\times 1{0}^{20}3.1813.1683.1680.009[3.149, 3.184]

 

error bars of the 14 parameters

In the table below, we summarize the bootstrapped estimates for each parameter in the scaling law. Naturally - for example, see Alabdulmohsin et al. (2022) and Mlodozeniec and Frei (2026) - some parameters are highly codependent, resulting in seemingly large uncertainty intervals. For example, the pairs (log⁡A, α)(\log A,\ \alpha ), (log⁡B, β)(\log B,\ \beta ), and (log⁡P, γ)(\log P,\ \gamma ) have Spearman correlations of 1.00, 1.00, and 0.88 across bootstraps. As our previous table shows, these fluctuations do not substantially change the extrapolated losses.

ParamMain LawBootstrap MedianBootstrap StdBootstrap 95% CI
EE2.182.160.07[2.03, 2.29]
AA16515773[89.2, 375]
(log⁡10A)({\log }_{10}A)2.222.200.16[1.95, 2.57]
α\alpha0.2980.2950.025[0.258, 0.352]
BB175914381309[531, 4419]
(log⁡10B)({\log }_{10}B)3.253.160.23[2.72, 3.65]
β\beta0.4080.3980.029[0.342, 0.459]
PP11710341[57.0, 222]
(log⁡10P)({\log }_{10}P)2.072.010.15[1.76, 2.35]
δ\delta0.07910.08100.0101[0.0665, 0.104]
κ\kappa0.1620.1620.007[0.147, 0.175]
γ\gamma3.043.020.18[2.69, 3.40]
FF192150915[9.50, 1630]
(log⁡10F)({\log }_{10}F)2.282.180.58[0.974, 3.21]
μ\mu1.121.130.11[0.904, 1.37]
ν\nu0.2240.2190.065[0.102, 0.361]
ϵ\epsilon0.1370.1290.043[0.0377, 0.215]
ρ\rho0.2630.2610.051[0.164, 0.357]

Ultimately, we believe the most important indicator in practice is whether the uncertainty in the scaling law could result in substantially worse prescriptions. Our prescriptive regret analysis above confirms that our results are robust to uncertainty in the scaling law fit. We will certainly highlight this analysis and discussion in the camera-ready version!

 

Alabdulmohsin et al. (2022), Revisiting Neural Scaling Laws in Language and Vision, https://arxiv.org/abs/2209.06640

Mlodozeniec and Frei (2026), Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation, https://arxiv.org/abs/2610.34514

Reviewer_GvMzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a data-constrained scaling law for language model pretraining. Chinchilla scaling laws assume infinite data (single pass over unique data), which is unrealistic as high quality becomes sparse. While previous work have derived data-constrained scaling laws, the main contributions of the paper is a more careful account (and thus more data efficient methodology) for training, which includes (i) deriving a law for how hyperparameters should scale with amount of data and compute, as well as (ii) deriving a law for ensembling (how the number of ensemble members affects loss).

Critical strengths
  • The problem setting is clean, interesting, and well-motivated.
  • Overall, the paper is pretty careful with experimentation in terms of hyperparameter tuning and proposing a sensible form for the scaling law.
  • This paper is well-written, and provides enough details about the setting to understand what's happening, and I appreciate the authors's non-use of AI for writing.
Critical weaknesses
  • This paper frames previous work as holding the hyperparameters fixed across scales and not considering ensembling. It is true for some prior work, but Kim et al. (2026) varies the hyperparameters and also considers ensembling. This paper considers a parametric form for hyperparameter scaling rather than tuning hyperparameters at each scale, so there is novelty in this paper, but I think the exact differences can be made clearer throughout the paper. For example, I would have expected a more direct comparison to Kim et al. (2026) in Figure 4, given that both tune hyperparameters for each scale and use ensembling. Presumably the results should be slightly worse given that hyperparameters aren't optimized per scale.
  • The basic result (Figure 4) is only with a token budget of U = 100M. I'd be curious how things work out for other values of U, just to make sure we test the scale dependence of the law (not just looking at varying U/C). Figure 3 has multiple values of U, but this only shows sweeping with M. In practice, data and compute are both resources and can be traded off, and the scaling law proposed in this paper seems not "just" for the data-constrained setting, but potentially helpful for more generally understanding the interaction between data and compute.
  • The ideas in the paper are fairly well-established, though the specific combination of them and the result is new.
Questions for authors
  • In practical LLM training settings, one has a data mixture where different datasets are epoched a different number of times (high quality data is smaller and can be epoched more times than web). Could this method be adapted to that setting?
  • Figure 5 shows that the prescribed R* is growing up to 32 or 64, which seems quite high; in practice, it doesn't seem intuitive that you'd want to epoch that many times (though I know the weight decay is also increasing). Perhaps you're in a regime where increasing compute really has diminishing gains (which maybe isn't an interesting regime). It would be good to get some commentary on this.
2 comments
AuthorsOpenReview ↗

Rebuttal 1/2

Thank you for your thoughtful review and kind words! We are glad you appreciate our human-written paper, and hope you will also appreciate our human-written responses to your questions below!

Specifically, inspired by your comments, we provide new results demonstrating that our scaling law extrapolates accurately for other values of UU and that we continue seeing substantial gains when increasing compute by another order of magnitude. We also clarify our paper’s novelty compared to Kim et al. (2026) and other prior work, and defend the practicality of ensembling and training for many epochs.

W1: Kim et al. (2026) already varies the hyperparameters and considers ensembling

True – however, they do not give a prescriptive scaling recipe. Instead, they sweep learning rate, weight decay, and epoch count at each parameter count. This approach does not support making compute-optimal prescriptions that trade-off model size and epoch count, and also provides limited guidance for scaling hyperparameters. We will specify this more clearly in this final paper.

W2: I'd be curious how things work out for other values of U, just to make sure we test the scale dependence of the law

Below we present new results verifying that our scaling law extrapolates accurately at other values of UU.

We evaluate our recipe’s prescribed configuration at a production compute budget of $C=$1e20 FLOPs (the largest budget evaluated in our original submission) for two new dataset sizes UU: 200M tokens and 50M tokens. We compare the performance against prescriptions following the scaling recipe from prior work.

URecipePrescriptionPredicted LossObserved Loss
200MPrior RecipeN=2.22B, R=383.0983.068
200MOursM=2, N=1.17B, R=363.0093.004
50MPrior RecipeN=398M, R=8364.0604.202
50MOursM=7, N=458M, R=1043.4233.441

Our scaling recipe yields significant gains over the prior recipe across all dataset sizes $U$, and our scaling law accurately predicts the observed losses. These results confirm that our contributions hold across dataset sizes.

W2: In practice, data and compute are both resources and can be traded off, and the scaling law proposed in this paper seems not "just" for the data-constrained setting, but potentially helpful for more generally understanding the interaction between data and compute.

We agree that our paper has prescriptive value beyond the data-constrained setting (for example, our scaling law accommodates setting R=1R=1 and scaling UU as desired). As our primary contribution is improved prescriptive recipes for scaling under data constraints, we focused our exposition around this scenario.

W3: The ideas in the paper are fairly well-established, though the specific combination of them and the result is new.

Hyperparameter scaling was neglected in previous work on data-constrained scaling prescriptions such as Lovelace et al. (2026) and Bryant et al. (2026) where hyperparameters are held fixed across scales. Additionally, existing heuristics for hyperparameter transfer such as μ\muP are ineffective in the data-constrained regime, calling for new prescriptions for hyperparameter scaling. Our work provides effective prescriptions for hyperparameter scaling under data constraints and applies them for the first time within a prescriptive scaling recipe.

Likewise, while ensembling is well-studied, it has not been considered as a scaling axis in the context of compute-optimal scaling prescriptions for training language models under data constraints. We find that ensembling resolves an important pathology in prior work, enabling monotonic gains in loss as the compute budget grows.

Combining these two contributions leads to a significantly improved prescriptive recipe for scaling under data constraints.

Q1: In practical LLM training settings, one has a data mixture where different datasets are epoched a different number of times (high quality data is smaller and can be epoched more times than web). Could this method be adapted to that setting?

Definitely! Many recent papers explore scaling laws for data mixtures, and some of these also take repetitions into account. For example, Akimoto et al. (2026), Longpre et al. (2026) and Sedova et al. (2026) each extend Muennighoff et al. (2023)’s data-constrained scaling law to accommodate data mixtures. Extending our ensembled scaling recipe to account for data mixtures is a promising direction for future work, allowing practitioners to trade-off additional axes of scaling (epoch counts for other domains).

AuthorsOpenReview ↗

Rebuttal 2/2

Q2: Figure 5 shows that the prescribed R∗R^* is growing up to 32 or 64, which seems quite high; in practice, it doesn't seem intuitive that you'd want to epoch that many times (though I know the weight decay is also increasing).

When our compute budgets dwarf the size of the training dataset, it is natural that we would want to train for many epochs. In the classic data-constrained regime it was common to train models for hundreds of epochs (for example see Krizhevsky et al. 2012 and He et al. 2015). Our recipe is carefully designed to enable effective training at high epoch counts, including careful hyperparameter scaling, snapshot ensembling, and a cyclic learning rate schedule (see B.4 to B.6).

Q2: Perhaps you're in a regime where increasing compute has diminishing gains

Below we provide new results showing that we continue observing substantial gains up to 75x our sweep budget.

Specifically, we evaluate our scaling law’s prescription for the 100M token dataset at a production budget of C=C = 6.2e20 FLOPs. Our scaling law prescribes the configuration N=N = 848M, R=R = 76, M=M = 16, and predicts a loss of 3.168. Running this configuration, we obtain a loss of 3.181. Compared to the loss of 3.195 (predicted 3.185) we obtained at a compute budget of 1e20 FLOPs, this result confirms that our prescriptions continue yielding substantial improvements for significantly larger compute budgets, and that our scaling law predicts these improvements accurately.

 

Akimoto et al. (2026), M3 Scaling Law: Optimizing Multi-Epoch, Multi-Lingual, and Multi-Stage Training for Low-Resource Language Models, https://arxiv.org/abs/2410.12325

Longpre et al. (2026), ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality, https://arxiv.org/abs/2510.22037

Sedova et al. (2026), Scaling Laws for Mixture Pretraining Under Data Constraints, https://arxiv.org/abs/2605.12715

Krizhevsky et al. (2012), ImageNet Classification with Deep Convolutional Neural Networks, https://papers.nips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html

He et al. (2015), Deep Residual Learning for Image Recognition, https://arxiv.org/abs/1512.03385

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of discussion period + thanks!

As the discussion period is ending, we’d like to thank all three reviewers for their thoughtful comments and support of our paper.

Our work establishes improved recipes for scaling under data constraints, a timely contribution as growing compute budgets outpace the growth of high-quality training data. We identify and resolve an important limitation of prior work by developing effective prescriptions for hyperparameter scaling, and incorporate ensembling as a new scaling axis to unlock monotonic gains with compute.

All reviewers highlighted the clarity of our presentation and soundness of our experiments. During the discussion period, we have also added several valuable new empirical results inspired by reviewer feedback, including:

  1. Demonstrating that our scaling law extrapolates accurately at even larger compute budgets - now up to 75x the scaling sweep budget - and other dataset sizes (see response to reviewer GvMz)
  2. Conducting a sensitivity analysis of our scaling law fits using bootstrapping error estimates, confirming that our prescriptions are robust to resampled observations (see response to reviewer eHK5)
  3. Evaluating the extrapolation of more alternative scaling law forms, strengthening the evidence in favor of our scaling law (see response to reviewer eHK5)
  4. Verifying that our improvements in pre-training loss translate to stronger performance on downstream tasks (see response to reviewer N4H3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.