acceptodds
Under review as a conference paper at ICLR 2027

A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

Abstract

In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.

Primary area:
learning theory
Keywords:
scaling laws, multi-epoch training, ensembling, dynamical mean-field theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_RdskOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate additional compute when training data are fixed: should one train a single model longer or train multiple models and ensemble their predictions? The authors develop a dynamical mean-field theory (DMFT) of multi-epoch random-feature regression. The analysis separates errors shared across models from model-specific prediction variance, which ensembling reduces. Under specific assumptions, including noiseless gradient flow at the interpolation threshold, the authors derive ensemble-dependent optimal stopping times and compute-allocation laws. Experiments with Transformer language models on FineWeb show that ensembling reduces late-training deterioration and delays optimal stopping.

Critical strengths

The paper connects data reuse, ensembling, and optimal stopping within a unified framework for compute allocation under fixed data. In particular, it explains how reducing model-specific variance changes the optimal training duration, and then translates this relationship into quantitative allocation rules at fixed individual model sizes. This addresses a practically relevant question. The language-model experiments provide supporting evidence for the proposed mechanism.

Critical weaknesses

1.The advantage over sufficiently tuned single-model baselines remains unclear. The main language-model experiments use constant learning rates and zero weight decay. Although the best checkpoint is selected, learning-rate decay and regularization could change both late-training deterioration and the performance gap between single models and ensembles. It would strengthen the empirical conclusions to establish whether the advantage persists when these choices are tuned separately for both approaches. 2.The experimental budget comparison does not directly validate the proposed optimal compute allocation. The theory charges the compute required for each ensemble to reach its own optimal stopping time. In contrast, the language-model experiments charge the full 40-epoch training budget and report the best checkpoint within the available evaluation horizon. This comparison does not account for compute saved by stopping earlier. Consequently, the experiments support the benefits of ensembling under a common training horizon, but provide less direct evidence for the theoretical allocation strategy.

Questions for authors

1.Does the ensemble advantage persist after separately tuning learning-rate schedules and regularization for single models and ensembles? How do these choices affect the shift in optimal stopping time? 2.Could the authors compare the attainable validation loss at matched budgets while charging only the compute used up to the selected checkpoint? Allowing both approaches to choose their stopping times would provide a closer test of the proposed allocation strategy.

3 comments

We thank the reviewer for suggesting these two comparisons. We ran both, and in each the ensemble outperforms wider and deeper single models at matched compute. The revised PDF reports them in Section 2, Section 3.8 and Appendix A.4 (Figures 2B and 10, Table 6), with settings in Table 3.

[W1, Q1] Separately tuned single models. We retrained the ensemble members and the wider and deeper single models of the matched-compute comparison with AdamW weight decay, swept separately for each model, and a learning rate decayed to zero over the last 20% of training. Tuning narrows the ensemble advantage but does not remove it: at four times the base-model compute, four ensemble members still beat the twice-wider and four-times-deeper single models, by 0.037 and 0.064 (Figure 10, solid). Deeper single models saturate under tuning, while each added member keeps lowering the loss.

Without weight decay, ensembling moves the optimal stopping time later in every configuration we tested; for the base model, from epoch 9 for one model to epoch 14 for four (Figure 6B). With tuned weight decay and a cooldown, every model reaches its minimum near the end of training, so this shift disappears. The stopping time becomes the training horizon, which is the same for both approaches.

[W2, Q2] Compute charged up to the selected checkpoint. We now use this accounting for every compute comparison in the revised paper. At a constant learning rate, the checkpoint at epoch kk is identical to a run stopped at epoch kk, so each configuration is charged only up to its own best checkpoint; both approaches select that checkpoint by validation loss. Within the compute of the twice-wider model, the best ensemble is three members stopped at epoch 9, which is 0.099 lower (Figure 2B); the advantage also holds against deeper models (Figure 10, dashed). Across all ensemble-member sizes and ensemble sizes we tested, every ensemble beats the best single model at no greater compute (Table 6). Within a fixed budget it pays to stop members before their own optimum and add another member, which is the tradeoff between training longer and adding members that our theory analyzes. Under the tuned recipe, every model reaches its minimum near the end of training, so this accounting changes the tuned comparison by at most 3% in compute.

We hope these additional experiments answer the reviewer’s questions, and we are happy to clarify anything further.

Thank you for running both comparisons. The additional weight-decay tuning and learning-rate cooldown help address my concern about the single-model baselines. It is useful to see that the ensemble advantage persists, even though the shift in optimal stopping time disappears under the tuned recipe. This distinction should be made clear in the revised paper.

Charging compute only up to each selected checkpoint also addresses my concern about using the full 40-epoch budget. Allowing both approaches to select their stopping times brings the experiments closer to the compute-allocation tradeoff studied in the theory.

Based on the reported results, my main concerns are substantially addressed. I maintain my rating of 3 (weak acceptance).

We thank the reviewer for the follow-up and are glad that the additional comparisons address the main concerns. Following the suggestion, the revised paper now makes this distinction explicit (Section 3.8 and Appendix A.4): the later optimal stopping of ensembles is measured without weight decay (Figure 6B), while the ensemble advantage also holds with weight decay tuned per model and a learning-rate cooldown (Figure 10), under which every model reaches its minimum near the end of training. We thank the reviewer again for the constructive review.

Reviewer_WzBqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how repeated use of a fixed dataset interacts with model ensembling. It extends a dynamical mean-field theory (DMFT) framework to fixed-order, multi-epoch SGD in random-feature models, tracking correlations across training steps, epochs, and ensemble members. Under particular assumptions, it derives ensemble-dependent optimal stopping times and corresponding compute-allocation laws. Language-model experiments on FineWeb demonstrate reduced late-training deterioration and later optimal stopping with logit averaging. The main contribution is a theoretical analysis of these interactions, with quantitative validation primarily within the random-feature setting.

Critical strengths

1 Nontrivial analysis of ensemble-dependent stopping.

The paper analyzes how the shared error and the variance across members evolve over training, and uses their competition to derive different stopping regimes. The finite-model result in Theorem 3, establishing an E2/3E^{2/3} stopping law under specified assumptions, provides a concrete contribution beyond the standard variance-reduction identity.

2 A potentially reusable extension of the dynamical framework.

Explicitly accounting for correlations induced by repeated minibatch use is useful for studying training on fixed datasets. The general equations accommodate a broader range of widths, spectra, targets, and noise levels than the assumptions used for the headline scaling laws. This provides a basis for further theoretical investigation, although extension to realistic feature-learning models remains substantial work.

Critical weaknesses

1 The originality is concentrated in a relatively specific theoretical extension.

Multi-epoch training, logit ensembling, approximate inverse-ensemble-size improvements, and the benefit of training ensemble members longer have substantial precedent. Consequently, the contribution should be assessed primarily through the new temporal asymptotics and stopping laws. The empirical phenomena and the general variance-reduction explanation contribute limited additional novelty.

2 The connection to language-model training remains qualitative.

The LM experiments establish that ensembling can improve validation loss and delay stopping under the tested recipe. They do not quantitatively validate the theoretical stopping laws. No correspondence between Transformer dynamics and the required spectral conditions is established.

Questions for authors

How robust are the stopping laws to deviations from N=PN=P, perturbations of the teacher structure, and label noise?

4 comments

We thank the reviewer for their careful reading and helpful questions.

[W1] We agree that these empirical observations have precedent. In fixed-data language modeling, for example, Kim et al. (2025) found that training ensemble members for more epochs with less weight decay improved the estimated infinite-ensemble loss relative to settings tuned for a single model.

To study how the optimal stopping time changes with ensemble size, we derive dynamical mean-field equations for fixed-order multi-epoch SGD in the random-feature model of Bordelon et al. (2024). To our knowledge, multiple passes over the same random data have not been treated exactly before. Our formalism requires a two-timescale analysis (epochs and optimizer steps), whereas prior analyses do not capture this two-timescale structure. Our equations track correlations between updates when the same minibatch is reused across epochs and between the prediction errors of models trained on the same data (Sec. 3.3 and Appendices B and C). In the noiseless gradient-flow limit of the power-law model at N=PN=P, we find that the expected variance across feature maps grows as τ\sqrt{\tau} at late times. Averaging EE predictions divides this variance by EE at a fixed checkpoint. Whether it pays to train longer also depends on how the error shared across members evolves.

At fixed PP, the optimal stopping time grows as E2/3E^{2/3} for equal squared target coefficients but can stay bounded for other targets in the same power-law family (Table 2). For equal squared target coefficients, Theorem 3 also proves the E2/3E^{2/3} law directly in the finite Gaussian model without requiring a power-law feature spectrum. We then use the cost of training each ensemble to its own loss minimum to derive how to split a compute budget between ensemble size and training time per member, holding data and member size fixed (Sec. 3.6). We have clarified this distinction from prior work in the Introduction.

[W2] We agree that the connection to language modeling is qualitative. Our theory gives a solvable model for studying how data reuse and ensembling affect the optimal stopping time. We test its predictions against random-feature simulations with different target spectra (Sec. 3.7 and Appendix K.3). In the FineWeb experiments, we see a similar trend: with a constant learning rate and no weight decay, ensembling reduces the late rise in validation loss and shifts the best checkpoint later (Sec. 3.8).

A quantitative connection would require understanding how the spectra in our model relate to Transformer training. It would also require accounting for feature learning, next-token prediction, and data mixtures in which different sources are repeated at different rates. We have added these points to the discussion (Sec. 4).

We continue our response with the reviewer’s question.

[Q] The mechanism behind the stopping laws carries over to all three settings. At a fixed checkpoint, averaging EE models divides the variance across feature maps by EE. The optimal stopping time depends on how this variance and the shared error evolve during training. Width, teacher structure, and label noise change these dynamics and therefore the stopping law.

Width. At N=PN=P, the low-frequency feedback is marginal, making training sensitive to weakly constrained directions and leading to τ\sqrt{\tau} variance growth. For fixed N/P≠1N/P\neq 1, this growth is eventually cut off on the regular gradient-flow branch. In the a=b=2a=b=2 example below interpolation, the critical behavior can persist over a finite range as N/PN/P approaches one (Sec. 3.4 and Appendix I.1). We expect that restriction of this growth to the vicinity of N=PN=P can be relaxed if models are trained on other loss functions such as cross-entropy, where exact empirical risk minimizers have infinite norm. Analysis of alternative loss functions would be significantly more involved, so we leave this for future work.

Teacher. In the noiseless power-law model at fixed N=PN=P, the stopping law depends on how the shared error approaches its limit. For a>ba>b, the shared error approaches its limit from above with a τ−1/2\tau^{-1/2} correction, and the optimal stopping time grows linearly with EE. For a=ba=b, this correction cancels, leaving a τ−1\tau^{-1} term and giving τ⋆≍E2/3\tau_\star\asymp E^{2/3}. For a<ba<b, the shared error reaches a finite-time minimum, so τ⋆\tau_\star stays bounded as EE grows (Table 2). When aa is close to bb, the E2/3E^{2/3} scaling can still be a good approximation over a finite range of ensemble sizes (Appendix H.3).

Label noise. We have added an analysis of label noise and supporting random-feature simulations (Appendix I.4 and Figure 11). The members fit the same noisy labels, so part of the resulting error is shared and remains after averaging. In the a=b=2a=b=2, N=PN=P continuum model, any fixed nonzero noise keeps the optimal time bounded as E→∞E\to\infty at fixed PP. With weak noise, the noiseless E2/3E^{2/3} law can still describe a finite range. Our simulations show a smaller shift in stopping time with EE at higher noise levels (Appendix I.4 and Figure 11).

We hope these clarifications and additions address the reviewer’s concerns, and we would be glad to clarify anything further.

Hi authors, thanks for your feedbacks. I am fine with the explanation and the new added senstitivity analysis. Make sure you include them in the revised paper. I will keep my original weak accept score.

Thank you for reviewing our responses. We have included the clarifications in the Introduction and Section 4, and the additional label-noise example and supporting simulations in Appendix I.4 and Figure 11 of the revised PDF. All changes are marked in blue. Thank you again for your time and constructive feedback.

Reviewer_fMSQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the impact of model ensembling on scaling behavior in data constrained settings. Specifically they want to know which components of late-training error can be reduced by averaging independently trained models and how this might effect compute optimal scaling. Through their analysis they find that ensemble size should grow with compute at least as fast as training time per member.

Critical strengths

[S1] Interesting and intuitive result that scaling compute initially grows as expected increasing model and data, however past a certain scale compute is better allocated towards ensembling.

[S2] Experiments conducted in both noiseless random-feature simulations and real language model training on fineweb.

Critical weaknesses

[W1] There is more recent work on data constrained scaling laws.

Justin Lovelace, Christian Belardi, Srivatsa R Kundurthy, Shriya Sudhakar, & Kilian Q Weinberger (2026). Prescriptive Scaling Laws for Data Constrained Training. In Third Conference on Language Modeling.

Questions for authors

[Q1] Can you clarify the learning rate schedule for the fineweb experiments? Are you using warmup stable decay? Or is there no warmup? And if there is no warmup why isnt there a warmup?

4 comments

[W1] Related Work. We thank the reviewer for pointing us to Lovelace et al. (2026). Both papers ask how to spend additional compute when the training data are fixed. Lovelace et al. fit and compare candidate scaling laws for language-model losses. Their additive repetition penalty predicts when further passes become less useful than increasing model size.

Our DMFT characterizes the full training and test loss trajectories of a random-feature model under fixed-order multi-epoch SGD. The equations track both updates within each epoch and correlations created by revisiting the same minibatches. This gives a dynamical account of how the gap between training and test loss develops. In the gradient-flow setting, we then derive how the shared error and the variance across feature maps evolve, and how their balance determines the optimal stopping time for an ensemble. We now cite Lovelace et al. and discuss the relationship between the two approaches in the Introduction.

[Q1] Learning-rate schedule. We thank the reviewer for this question and are happy to clarify. We do not use warmup: in all FineWeb experiments in the main text (not including the appendix), the learning rate is constant from the first step to the last, with no warmup (Table 4).

Why no warmup. Warmup affects only the first one to two epochs, while our results concern late training: the validation loss reaches its minimum after 6–21 epochs and then rises as the data are repeated (Figures 1 and 8A). Following the reviewer’s question, we checked this directly by adding a linear warmup of up to two epochs. The loss still rises after its minimum, and later in training the curves with and without warmup overlap (Figure 9, Appendix A.4).

Why a constant learning rate. We study how the validation loss depends on the number of training epochs. With a constant learning rate, stopping a run at epoch kk gives exactly the model that a kk-epoch run would produce. One run therefore measures every training length, and each model can be charged only the compute up to its selected checkpoint. A decaying schedule would need a separate run for each length. This setting also matches our theory, which analyzes SGD at a fixed learning rate without weight decay.

We hope these additions address the reviewer’s questions, and we would be glad to clarify anything further.

Thank you again for your time and feedback. We wanted to follow up to see whether our responses and revisions address your questions and concerns. Please let us know if there is anything further we can clarify before the discussion period ends.

Thank you for answering my questions. I will maintain my already positive score.

Thank you for your time and constructive feedback. We appreciate your careful review of our work.

Comments on the paper

3 comments

Dear Reviewers,

The authors have finished uploading their responses to reviewers, and the discussion phase is now underway. If you haven't already, please carefully review the authors' responses to understand their perspectives. Please engage in thoughtful and constructive discussions with the authors, and update your review or rating if necessary.

The discussion period lasts until November 18th AoE, which means you have less than five days left to interact with the authors. Please start engaging well before the period ends, so that the authors will have sufficient time to respond to any follow-up questions/comments.

Your active participation and contribution to the ongoing discussion are highly encouraged. Thank you very much for your contribution to ICLR.

Best regards,

AC

AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF in which all changes are shown in blue. The individual responses give the details, and below we list the changes and the comments they address.

New experiments and analyses

Learning-rate warmup (Appendix A.4, Figure 9; Reviewer fMSQ, Q1): with warmup, the validation loss still reaches a minimum and then rises as the data are repeated, and later in training the curves with and without warmup overlap. Warmup changes early training, but not the late-time overfitting that our analysis concerns.

Weight decay tuned per model with a learning-rate cooldown (Appendix A.4, Figure 10; Reviewer Rdsk, W1/Q1; Reviewer fMSQ, Q1): tuning narrows the ensemble advantage, but the ensemble still outperforms wider and deeper single models at matched compute. With tuned weight decay and a cooldown, every model reaches its minimum near the end of training, so the stopping time becomes the training horizon.

Compute charged up to the selected checkpoint (Sections 2 and 3.8, Figure 2B, new Table 6, Appendix A.2; Reviewer Rdsk, W2/Q2): the matched-compute comparisons at a constant learning rate now charge each configuration only up to its selected checkpoint, as in our theoretical allocation. Under this accounting, ensembles beat wider and deeper single models at no greater compute in every configuration we tested. The gap reported in Figure 2B is updated accordingly.

Table 3 gives the settings of the new runs. We removed the earlier appendix figure that compared matched budgets with every run charged the full 40 epochs, because Figure 10 and Table 6 supersede it.

Theory and positioning

Related work and contributions (Section 1; Reviewer fMSQ, W1; Reviewer WzBq, W1): we add a comparison with Lovelace et al. (2026) and expand the discussion of Kim et al. (2025). We clarify how our two-timescale DMFT captures cross-epoch correlations beyond the one-pass SGD analysis of Bordelon et al. (2024), and specify when the optimal stopping time grows as EE, as E2/3E^{2/3}, or stays bounded.

Robustness of the stopping laws (Section 3.5, Appendix I.4, Figure 11; Reviewer WzBq, Q): we add an explicit label-noise calculation for the a=b=2a=b=2, N=PN=P continuum model, including the weak-noise crossover, with supporting random-feature simulations.

Connection to language models (Section 4; Reviewer WzBq, W2): we expand the discussion of what a quantitative connection between our random-feature model and Transformer training would require.

We thank the reviewers again for their time and thoughtful comments, which have strengthened the paper.

AuthorsOpenReview ↗

Final discussion summary

We thank the Area Chair and the reviewers for their time and feedback. The revised PDF shows the changes in blue. Our earlier “Summary of revisions” comment lists the changes for each reviewer.

 

What the paper does

We study multi-epoch pretraining with ample compute and a fixed dataset of PP distinct examples. For a random-feature model of width NN, we derive a two-timescale dynamical mean-field theory (DMFT) for fixed-order multi-epoch SGD. The equations track updates within each epoch and correlations between updates on the same minibatch across epochs. At the interpolation threshold N=PN=P, the optimal stopping time grows as EE, grows as E2/3E^{2/3}, or stays bounded, depending on the target (Table 2). We use these laws to split a fixed compute budget between ensemble size EE and training time per member.

At matched compute on a fixed 100M-token FineWeb corpus, ensembles have lower validation loss than wider or deeper single models. With a constant learning rate and no weight decay, ensemble validation loss continues to fall after a single base model’s validation loss starts to rise.

 

Strengths recognized by reviewers

  • Compute allocation. Reviewer fMSQ highlighted ensembling as an alternative to increasing model size when more compute is available but the dataset is fixed. Reviewer Rdsk emphasized that the stopping laws give a quantitative rule for splitting that compute between more members and longer training per member.
  • Theory and experiments. Reviewer fMSQ noted that the paper includes both random-feature simulations and FineWeb experiments. Reviewer Rdsk said that the FineWeb results support the proposed mechanism.
  • Ensemble-dependent stopping. Reviewers WzBq and Rdsk highlighted how the competition between shared error and member variance determines how long an ensemble should train. Reviewer WzBq pointed to the distinct stopping regimes and identified Theorem 3’s E2/3E^{2/3} law as a concrete contribution beyond the standard variance-reduction identity.
  • Reusable multi-epoch DMFT. Reviewer WzBq noted that the equations track correlations across repeated uses of a minibatch. The reviewer saw their applicability to different widths, spectra, targets, and noise levels as a basis for further theoretical work.

 

Revisions and clarifications

Robustness of the stopping laws. We explain how width affects the late-time τ\sqrt{\tau} variance growth seen at N=PN=P (Section 3.4, Appendix I.1). Table 2 and Appendix H.3 show how the target changes the stopping law. We added an explicit label-noise calculation and supporting simulations (Appendix I.4, Figure 11). In the a=b=2a=b=2, N=PN=P continuum model at fixed PP, the optimal stopping time stays bounded as E→∞E\to\infty for any fixed nonzero noise level. With weak noise, the E2/3E^{2/3} law can still describe a finite range of ensemble sizes.

FineWeb experiments. We now count only the compute that each run uses to reach its selected checkpoint. The three-member ensemble at epoch 9 uses no more compute than a single model with twice the base width. The ensemble’s validation loss is 0.099 nats per token lower than that of the single model (Figure 2B). Each ensemble in Table 6 also attains lower validation loss than the best single-model checkpoint within its compute budget. With weight decay tuned per model and a learning-rate cooldown, ensembles still have lower validation loss than wider and deeper single models at matched compute. With these settings, all models reach their lowest validation loss near the end of training (Figure 10). A separate warmup experiment shows that validation loss still increases late in training (Figure 9).

Relation to prior work and language modeling. Section 1 explains how our results differ from those of Lovelace et al. (2026) and Kim et al. (2025). Lovelace et al. fit scaling laws with a repetition penalty to language-model losses. Our DMFT derives how training and test loss evolve as a random-feature model makes repeated passes over the same data. Section 4 distinguishes the quantitative stopping laws for the random-feature model from the qualitative trends in our FineWeb experiments.

 

Reviewer follow-up

Reviewer fMSQ noted that our response answered their questions. Reviewer WzBq found our explanation and the added sensitivity analysis satisfactory. Reviewer Rdsk said that counting compute up to the selected checkpoint better matched the theory and that our revisions substantially addressed their main concerns.

 

We thank the reviewers and the Area Chair for the constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.