- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies language model pretraining in the finite-data regime, where training data must be reused across multiple epochs. The author shows that optimizing batch size during this scenario reveals a U-shaped relationship between batch size and validation loss, with smaller batches might achieve better generalization. They further explore repetition-aware training strategies, including dynamic sequence repacking, weight tying, and embedding weight decay, to improve performance. Finally, the paper investigates the trade-offs between GPU availability and training time, showing that training with more GPUs does not always lead to better validation performance under fixed time constraints.
Critical strengths
- Scaling law with finite training data for LLM pretraining is a crucial problem. If proven effective, this work can guide the data composition and pretraining instrumentation for LLM pretraining.
Critical weaknesses
-
This paper is pure empirical experience-based tuning without any theoretical intuition and reasoning, neither in convex or non-convex. The selection of the batch size and the learning rate schedule determines convergence and generalization. But the authors completely overlook this part.
-
The conclusion of this paper, e.g., the U-shape cross entropy across global batch sizes (Figure 1a and Figure 2 ) depends on the model size, learning rate schedule, and data distribution. It is unclear if the observed pattern can generalize to other LLMs with larger sizes, LR schedule with cosine decay, or other datasets, such as C4 or Pile.
-
It is unclear for the learning rate and weight decay selection. The U-shape observation can be an experimental artifact due to misconfiguration of the learning rate schedule. Please justify why the learning rate is not linearly scaled with the global batch size.
-
The evaluation only shows HellaSwag, with a summarized 8-task average. Please justify the selection of HellaSwag but not other downstream tasks? What if the tasks' optimal batch sizes differ from each other?
Questions for authors
Please respond to the concerns in Critical Weaknesses.
10 comments
Response (2/2)
W4: The evaluation only shows HellaSwag, with a summarized 8-task average. Please justify the selection of HellaSwag but not other downstream tasks? What if the tasks' optimal batch sizes differ from each other?
We chose cross entropy loss and Hellaswag as primary metrics, because they provide clearer signals at our experimental scale, whereas other benchmark metrics fluctuate substantially, as explained in the paper. This choice is consistent with common practice in the literature, including many influential papers:
- The DCLM paper chose benchmarks based on whether they provide a low-variance signal even at small scales, and explicitly noted that they used Hellaswag as a main metric “given their popularity as metrics for language model performance at the 7B scale.”
- The FineWeb paper selected benchmarks based on low variance across runs, nearly monotonic improvement during training, and performance above chance and explicitly noted that Hellaswah is “one of the benchmarks in our aggregate group of tasks with the highest signal-to-noise ratio.”
- “Quantifying Variance in Evaluation Benchmarks” compared Hellaswag with many other benchmarks and directly reported that Hellaswag has the most stable and monotonic signal, which they identify as desirable properties for pre-training evaluation.
- “Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation” reported that HellaSwag has the highest signal-to-noise ratio, even compared to benchmark averages, at every model scale examined.
Can the authors provide the scores for all eight benchmarks?
Response (1/2)
Thank you for the detailed review and for recognizing the importance of pre-training with finite data and the potential practical value of this work.
We agree that our contribution is empirical: we systematically study how batch size, regularization, data processing, and training duration interact with each other when unique data is limited. We would also emphasize that many purely empirical papers have had a significant influence on our field, so being purely empirical does not necessarily mean the contribution is limited.
The choice of batch size and learning rate schedule are indeed confounding factors; accordingly, our work tunes peak learning rate and weight decay jointly with batch size and repetition count in the paper.
We held the schedule family fixed to WSD, because in the early stage of the project, we find it performing significantly better than alternatives (mainly cosine), in both repeated-data and unique-data settings. We report those results for the reviewer’s reference. (The batch size is fixed at 1,024, and the peak learning rate is tuned separately for each schedule.)
Validation CE across epochs
Downstream performance after five epochs
Given these results and the fact that WSD is the industry standard, we adopted WSD throughout. We think evaluating alternative learning rate schedules can be an avenue for future work.
The paper shows that the U-shaped relationship is consistent across three model sizes, indicating that the finding is not confined to a single model size.
Regarding the learning rate schedule, please see the answer to W1.
Regarding the data distribution: we acknowledge that we used a single training dataset. However, the paper’s contribution is to establish the value of tuning batch size in repeated-data training, rather than to identify a universally optimal batch size that holds across all training data, model architectures, and other choices.
We select learning rate and weight decay jointly with batch size and the number of repetitions. Direct tuning allows each batch size to use the best learning rate found by the search, so we don’t believe the finding is due to misconfigured learning rate.
The reason our optimal learning rate is not linearly scaled with the global batch size is that we find learning rates of 4e-03 or higher lead to unstable loss trajectories.