- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to allocate additional compute when training data are fixed: should one train a single model longer or train multiple models and ensemble their predictions? The authors develop a dynamical mean-field theory (DMFT) of multi-epoch random-feature regression. The analysis separates errors shared across models from model-specific prediction variance, which ensembling reduces. Under specific assumptions, including noiseless gradient flow at the interpolation threshold, the authors derive ensemble-dependent optimal stopping times and compute-allocation laws. Experiments with Transformer language models on FineWeb show that ensembling reduces late-training deterioration and delays optimal stopping.
Critical strengths
The paper connects data reuse, ensembling, and optimal stopping within a unified framework for compute allocation under fixed data. In particular, it explains how reducing model-specific variance changes the optimal training duration, and then translates this relationship into quantitative allocation rules at fixed individual model sizes. This addresses a practically relevant question. The language-model experiments provide supporting evidence for the proposed mechanism.
Critical weaknesses
1.The advantage over sufficiently tuned single-model baselines remains unclear. The main language-model experiments use constant learning rates and zero weight decay. Although the best checkpoint is selected, learning-rate decay and regularization could change both late-training deterioration and the performance gap between single models and ensembles. It would strengthen the empirical conclusions to establish whether the advantage persists when these choices are tuned separately for both approaches. 2.The experimental budget comparison does not directly validate the proposed optimal compute allocation. The theory charges the compute required for each ensemble to reach its own optimal stopping time. In contrast, the language-model experiments charge the full 40-epoch training budget and report the best checkpoint within the available evaluation horizon. This comparison does not account for compute saved by stopping earlier. Consequently, the experiments support the benefits of ensembling under a common training horizon, but provide less direct evidence for the theoretical allocation strategy.
Questions for authors
1.Does the ensemble advantage persist after separately tuning learning-rate schedules and regularization for single models and ensembles? How do these choices affect the shift in optimal stopping time? 2.Could the authors compare the attainable validation loss at matched budgets while charging only the compute used up to the selected checkpoint? Allowing both approaches to choose their stopping times would provide a closer test of the proposed allocation strategy.
3 comments
Thank you for running both comparisons. The additional weight-decay tuning and learning-rate cooldown help address my concern about the single-model baselines. It is useful to see that the ensemble advantage persists, even though the shift in optimal stopping time disappears under the tuned recipe. This distinction should be made clear in the revised paper.
Charging compute only up to each selected checkpoint also addresses my concern about using the full 40-epoch budget. Allowing both approaches to select their stopping times brings the experiments closer to the compute-allocation tradeoff studied in the theory.
Based on the reported results, my main concerns are substantially addressed. I maintain my rating of 3 (weak acceptance).
We thank the reviewer for the follow-up and are glad that the additional comparisons address the main concerns. Following the suggestion, the revised paper now makes this distinction explicit (Section 3.8 and Appendix A.4): the later optimal stopping of ensembles is measured without weight decay (Figure 6B), while the ensemble advantage also holds with weight decay tuned per model and a learning-rate cooldown (Figure 10), under which every model reaches its minimum near the end of training. We thank the reviewer again for the constructive review.
We thank the reviewer for suggesting these two comparisons. We ran both, and in each the ensemble outperforms wider and deeper single models at matched compute. The revised PDF reports them in Section 2, Section 3.8 and Appendix A.4 (Figures 2B and 10, Table 6), with settings in Table 3.
[W1, Q1] Separately tuned single models. We retrained the ensemble members and the wider and deeper single models of the matched-compute comparison with AdamW weight decay, swept separately for each model, and a learning rate decayed to zero over the last 20% of training. Tuning narrows the ensemble advantage but does not remove it: at four times the base-model compute, four ensemble members still beat the twice-wider and four-times-deeper single models, by 0.037 and 0.064 (Figure 10, solid). Deeper single models saturate under tuning, while each added member keeps lowering the loss.
Without weight decay, ensembling moves the optimal stopping time later in every configuration we tested; for the base model, from epoch 9 for one model to epoch 14 for four (Figure 6B). With tuned weight decay and a cooldown, every model reaches its minimum near the end of training, so this shift disappears. The stopping time becomes the training horizon, which is the same for both approaches.
[W2, Q2] Compute charged up to the selected checkpoint. We now use this accounting for every compute comparison in the revised paper. At a constant learning rate, the checkpoint at epoch k is identical to a run stopped at epoch k, so each configuration is charged only up to its own best checkpoint; both approaches select that checkpoint by validation loss. Within the compute of the twice-wider model, the best ensemble is three members stopped at epoch 9, which is 0.099 lower (Figure 2B); the advantage also holds against deeper models (Figure 10, dashed). Across all ensemble-member sizes and ensemble sizes we tested, every ensemble beats the best single model at no greater compute (Table 6). Within a fixed budget it pays to stop members before their own optimum and add another member, which is the tradeoff between training longer and adding members that our theory analyzes. Under the tuned recipe, every model reaches its minimum near the end of training, so this accounting changes the tuned comparison by at most 3% in compute.
We hope these additional experiments answer the reviewer’s questions, and we are happy to clarify anything further.