acceptodds
Under review as a conference paper at ICLR 2027

More GPUs Are Not Always Better: Scaling Pretraining with Finite Data

Abstract

We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim et al. 2025) and show that its frontier depends critically on choices previously fixed or studied only in isolation. In particular, optimizing batch size together with epoch counts reveals a U-shaped relationship between validation loss and batch size, favoring smaller batches than typical pretraining, likely because small-batch gradient noise provides implicit regularization. We further find that training recipes, previously optimized under a single-pass assumption, can be tuned, e.g., repacking documents into new sequences every epoch, weight tying, and positive embedding weight decay. We then relax the infinite-compute assumption. In particular, smaller-than-typical batch sizes can require prohibitively long sequential training, motivating to treat GPUs and time as separate resources. Tighter deadlines favor larger batches, and latency-oriented parallelism can trade additional GPUs for shorter training. Nevertheless, even under a fixed time budget, a smaller batch with fewer GPUs can achieve lower validation loss than a larger batch with more GPUs, overturning the conventional preference for using as many GPUs as possible under a deadline. Together, our results identify underexplored factors in repeated-data training and provide concrete guidance on how training recipes and resource allocation should change from single-pass pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pre-training, data-constrained pre-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iaiWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies language model pretraining in the finite-data regime, where training data must be reused across multiple epochs. The author shows that optimizing batch size during this scenario reveals a U-shaped relationship between batch size and validation loss, with smaller batches might achieve better generalization. They further explore repetition-aware training strategies, including dynamic sequence repacking, weight tying, and embedding weight decay, to improve performance. Finally, the paper investigates the trade-offs between GPU availability and training time, showing that training with more GPUs does not always lead to better validation performance under fixed time constraints.

Critical strengths
  1. Scaling law with finite training data for LLM pretraining is a crucial problem. If proven effective, this work can guide the data composition and pretraining instrumentation for LLM pretraining.
Critical weaknesses
  1. This paper is pure empirical experience-based tuning without any theoretical intuition and reasoning, neither in convex or non-convex. The selection of the batch size and the learning rate schedule determines convergence and generalization. But the authors completely overlook this part.

  2. The conclusion of this paper, e.g., the U-shape cross entropy across global batch sizes (Figure 1a and Figure 2 ) depends on the model size, learning rate schedule, and data distribution. It is unclear if the observed pattern can generalize to other LLMs with larger sizes, LR schedule with cosine decay, or other datasets, such as C4 or Pile.

  3. It is unclear for the learning rate and weight decay selection. The U-shape observation can be an experimental artifact due to misconfiguration of the learning rate schedule. Please justify why the learning rate is not linearly scaled with the global batch size.

  4. The evaluation only shows HellaSwag, with a summarized 8-task average. Please justify the selection of HellaSwag but not other downstream tasks? What if the tasks' optimal batch sizes differ from each other?

Questions for authors

Please respond to the concerns in Critical Weaknesses.

10 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the detailed review and for recognizing the importance of pre-training with finite data and the potential practical value of this work.

W1: This paper is pure empirical experience-based tuning without any theoretical intuition and reasoning, neither in convex or non-convex. The selection of the batch size and the learning rate schedule determines convergence and generalization. But the authors completely overlook this part.

We agree that our contribution is empirical: we systematically study how batch size, regularization, data processing, and training duration interact with each other when unique data is limited. We would also emphasize that many purely empirical papers have had a significant influence on our field, so being purely empirical does not necessarily mean the contribution is limited.

The choice of batch size and learning rate schedule are indeed confounding factors; accordingly, our work tunes peak learning rate and weight decay jointly with batch size and repetition count in the paper.

We held the schedule family fixed to WSD, because in the early stage of the project, we find it performing significantly better than alternatives (mainly cosine), in both repeated-data and unique-data settings. We report those results for the reviewer’s reference. (The batch size is fixed at 1,024, and the peak learning rate is tuned separately for each schedule.)

Validation CE across epochs

ScheduleEpoch 1Epoch 2Epoch 3Epoch 4Epoch 5
Cosine5.2614.3223.8973.6443.517
Cosine (α_f = 0)5.4014.4853.9823.7403.567
WSD5.0003.9973.6583.4663.368

Downstream performance after five epochs

ScheduleHellaSwag accuracy (%) ↑HellaSwag BPB ↓Eight-task average accuracy (%) ↑
Cosine28.381.05138.50
Cosine (α_f = 0)27.871.06737.37
WSD31.510.99439.27

Given these results and the fact that WSD is the industry standard, we adopted WSD throughout. We think evaluating alternative learning rate schedules can be an avenue for future work.

W2: The conclusion of this paper, e.g., the U-shape cross entropy across global batch sizes (Figure 1a and Figure 2 ) depends on the model size, learning rate schedule, and data distribution. It is unclear if the observed pattern can generalize to other LLMs with larger sizes, LR schedule with cosine decay, or other datasets, such as C4 or Pile.

The paper shows that the U-shaped relationship is consistent across three model sizes, indicating that the finding is not confined to a single model size.

Regarding the learning rate schedule, please see the answer to W1.

Regarding the data distribution: we acknowledge that we used a single training dataset. However, the paper’s contribution is to establish the value of tuning batch size in repeated-data training, rather than to identify a universally optimal batch size that holds across all training data, model architectures, and other choices.

W3: It is unclear for the learning rate and weight decay selection. The U-shape observation can be an experimental artifact due to misconfiguration of the learning rate schedule. Please justify why the learning rate is not linearly scaled with the global batch size.

We select learning rate and weight decay jointly with batch size and the number of repetitions. Direct tuning allows each batch size to use the best learning rate found by the search, so we don’t believe the finding is due to misconfigured learning rate.

The reason our optimal learning rate is not linearly scaled with the global batch size is that we find learning rates of 4e-03 or higher lead to unstable loss trajectories.

AuthorsOpenReview ↗

Response (2/2)

W4: The evaluation only shows HellaSwag, with a summarized 8-task average. Please justify the selection of HellaSwag but not other downstream tasks? What if the tasks' optimal batch sizes differ from each other?

We chose cross entropy loss and Hellaswag as primary metrics, because they provide clearer signals at our experimental scale, whereas other benchmark metrics fluctuate substantially, as explained in the paper. This choice is consistent with common practice in the literature, including many influential papers:

  • The DCLM paper chose benchmarks based on whether they provide a low-variance signal even at small scales, and explicitly noted that they used Hellaswag as a main metric “given their popularity as metrics for language model performance at the 7B scale.”
  • The FineWeb paper selected benchmarks based on low variance across runs, nearly monotonic improvement during training, and performance above chance and explicitly noted that Hellaswah is “one of the benchmarks in our aggregate group of tasks with the highest signal-to-noise ratio.”
  • “Quantifying Variance in Evaluation Benchmarks” compared Hellaswag with many other benchmarks and directly reported that Hellaswag has the most stable and monotonic signal, which they identify as desirable properties for pre-training evaluation.
  • “Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation” reported that HellaSwag has the highest signal-to-noise ratio, even compared to benchmark averages, at every model scale examined.

Can the authors provide the scores for all eight benchmarks?

Reviewer_Bf5ROpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents a nice empirical study of various training decisions in the data-limited infinite-compute regime. Prior work found that tuning weight decay significantly improves performance in this regime. This work extends that study to consider batch size, data processing decisions, weight tying, and embedding weight decay. They find that compared to standard pre-training smaller batch sizes, weight tying, and embedding weight decay all improve performance. For data processing, they find that Kim et al.’s choice to fix the data order is suboptimal and both shuffling and dynamically repacking across epochs is better. Shuffling across epochs is pretty standard practice so this seems like mostly an oversight in prior work, but the repacking is easily overlooked. Tuning these various components of the recipe lowers the achievable loss in the infinite data regime for fixed model sizes and compute budgets.

The paper also presents some analysis and results in a time-budget constrained setting. Typically people try to maximally scale GPUs to improve total token throughput and reduce pretraining wall clock time, but large batch sizes are harmful in the limited data regime. The paper shows how the time budget can alter the optimal pre-training recipe such that more GPUs are actually not desirable. Large batch sizes are optimal for short budgets and smaller batch sizes for long budgets. Results are also presented under joint GPU-time constraints showing similar findings.

The experimental setup and evaluation is pretty standard for LLM pre-training work. Validation loss/perplexity with some downstream benchmarks (accuracy and BPB) included as well.

Critical strengths

The data-constrained regime is becoming increasingly relevant as compute grows faster than data, so the paper is relevant. The paper is well written and the various findings are easy to follow. The presented experimental results are quite comprehensive and the methodology is sound.

This is a very sound empirical study that presents clear adjustments to the pre-training recipe to improve performance in this setting. All of the proposed regularizations are quite natural, and the cumulative effect leads to a meaningful effect. I think Figure 8 is a nice illustration of this, showing the tuned recipe achieves the same performance as 16x unique tokens. Of course this generally requires much more sequential time, so there’s not a free lunch here.

While shuffling is pretty standard in multi-epoch training, repacking is a nice improvement that could be easily overlooked. Most pre-training pipelines just preprocess and pack things once.

Critical weaknesses

This paper is a well-written empirical study whose findings are well-supported. I don’t think there are any critical weaknesses. Arguably the effects reported are “obvious” in the sense that regularization helps with limited data, but I don’t hold this against the paper. It's non-obvious a priori which way some of these would go (embedding weight decay, embedding tying) and quantifying the actual effects of these various choices is valuable.

The findings are all very empirical and the optimal parameters are found through search. It would be nice if there was some sort of scaling law framework to select hyperparameters for a given setting, but I think that is reasonably beyond the scope.

Questions for authors

I think this is reasonably beyond the scope of this work, so just curious. From your empirical sweeps, do you observe any predictable trends in the optimal hyperparameters? Does it seem like they may be predictable in some scaling law framework in the future?

1 comment

Thank you for the supportive and constructive review. We appreciate that you find the paper sound, comprehensive, and clearly presented, and that the value of quantifying the combined recipe improvements was recognized.

The findings are all very empirical and the optimal parameters are found through search. It would be nice if there was some sort of scaling law framework to select hyperparameters for a given setting, but I think that is reasonably beyond the scope.

I think this is reasonably beyond the scope of this work, so just curious. From your empirical sweeps, do you observe any predictable trends in the optimal hyperparameters? Does it seem like they may be predictable in some scaling law framework in the future?

We agree that predicting hyperparams from the training setting is a promising next step! We made an initial attempt (in Appendix D.1) by fitting repetition count and batch size as a function of model size and unique-data pool size; however, we found it difficult to establish reliable laws based on three model sizes and three data pools. We believe the first step is likely to collect more datapoints (more model sizes and data pools) so that a more reliable law can be fitted. We view our current work as empirical groundwork for this direction.

Reviewer_vtnvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies pretraining when the unique data pool is fixed and must be repeated. When batch size is tuned jointly with learning rate, weight decay, and epoch count, validation loss is U-shaped in batch size. The best result is around B=64, much smaller than typical pretraining batches. A recipe adjusted for repetition improves results further. The final part treats GPU count and training time as separate constraints and asks how batch size, repetitions, and recipe should be chosen under each.

Critical strengths
  1. The paper offers a practical training guide for data-limited settings that can be useful to many researchers.
  2. Prior work fixed the batch size, whereas this paper runs and analyzes experiments over a wide range of batch sizes.
  3. The experimental details are well reported. All configurations are listed in Tables 7–15, and B=64 remains optimal across three different data pools (Table 4).
Critical weaknesses
  1. It is already known that small batches generalize better [1], and the paper itself explains its U-curve with this known mechanism in Section 3.2. The novelty beyond applying it to the repeated-data setting is unclear.
  2. It is not clear that the small-batch advantage is specific to repetition. Fig. 8 shows that single-pass training also prefers B=64 over larger batches, so the optimum may simply reflect token efficiency at this scale rather than an effect of repeated data.
  3. The batch comparison has some confounds. Warmup is fixed at 100M tokens, which is about 1,500 steps for B=16 but fewer than 50 steps for B=512. The LR selected for B=16 is also at the bottom of the search grid, so part of the left side of the U-curve may be due to under-tuning.
  4. The GPU conclusions depend on how the extra GPUs are used. In Section 5 the microbatch is fixed, so more GPUs always means a larger B. Using extra GPUs for latency instead, through smaller microbatches or more TP/PP, is discussed in Section D.2 but left out of the main results.

*[1] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, ICLR 2017

Questions for authors
  1. What is new here beyond applying the known small-batch generalization effect to the repeated-data setting?
  2. Could the authors run a single-pass batch sweep at the same total token count (e.g., 20B tokens for the 1.5B model) to determine whether the optimum is specific to repeated data?
  3. Does the optimal batch size change if warmup is matched in steps, or if the LR grid is extended lower?
  4. Does the y-axis in Figs. 7 and 10 mean GPUs used or GPUs available?
2 comments
AuthorsOpenReview ↗

Response (1/2)

Thank you for the detailed review. We appreciate the reviewer’s recognition of the paper’s practical value for data-limited pre-training, its systematic exploration of batch size, and its detailed experimental reporting. Below, we address related concerns/questions together.

W1: It is already known that small batches generalize better [1], and the paper itself explains its U-curve with this known mechanism in Section 3.2. The novelty beyond applying it to the repeated-data setting is unclear.

W2: It is not clear that the small-batch advantage is specific to repetition. Fig. 8 shows that single-pass training also prefers B=64 over larger batches, so the optimum may simply reflect token efficiency at this scale rather than an effect of repeated data.

Q1. What is new here beyond applying the known small-batch generalization effect to the repeated-data setting?

We agree that the generalization benefits of small batches are known in the literature, as we discussed in the paper. Our contribution is to characterize how batch size affects the best achievable performance when a fixed data can be repeatedly reused until the performance saturates.

Prior work showed (and our experiments validated) that, without repetition, which batch size is better depends on which variable is held fixed. With a fixed number of epochs, small batches perform better because of more optimizer steps (verified in Figure 8). With a fixed number of optimizer steps, large batches perform better because they process more data.

What was unknown is: When neither epoch count or optimizer step is fixed, but we can repeat the data until the performance saturates, how to choose the batch size? This regime is the focus of our work.

Q2. Could the authors run a single-pass batch sweep at the same total token count (e.g., 20B tokens for the 1.5B model) to determine whether the optimum is specific to repeated data?

We have not run this additional sweep due to limited compute; but we expect that the result would hold consistent with prior work: given a fixed epoch count, small batches would perform better; given fixed optimizer steps, large batches would be better. Again, our work instead focuses on a regime in which the number of epochs or optimizer steps can be optimized until performance saturates.

W3: The batch comparison has some confounds. Warmup is fixed at 100M tokens, which is about 1,500 steps for B=16 but fewer than 50 steps for B=512. The LR selected for B=16 is also at the bottom of the search grid, so part of the left side of the U-curve may be due to under-tuning.

Q3: Does the optimal batch size change if warmup is matched in steps, or if the LR grid is extended lower?

We ran the requested ablations that match the number of warmup steps rather than warmup tokens, with a 1.5B model. Specifically, we fixed the warmup steps to be the same as batch size 64’s warmup steps (given that it achieves the best performance) and used that for other batch sizes. Results are as follows.

1.5B model

WarmupBS16BS32BS64BS128BS256BS512
Token-matched3.017 (E20) // 1,536 steps; 100.7M tokens3.001 (E20)// 768 steps; 100.7M tokens2.977 (E20) // 384 steps; 100.7M tokens2.998 (E16)//192 steps; 100.7M tokens3.037 (E12) // 96 steps; 100.7M tokens3.151 (E8) // 48 steps; 100.7M tokens
Step-matched3.020 (E20) // 384 steps; 25.2M tokens3.002 (E20) // 384 steps; 50.3M tokens2.977 (E20) // 384 steps; 100.7M tokens2.999 (E16) // 384 steps; 201.3M tokens3.004 (E12) // 384 steps; 402.7M tokens3.149 (E8) // 384 steps; 805.3M tokens

474M model

WarmupBS32BS64BS128BS256BS512
Token-matched3.113 (E32) // 768 steps; 100.7M tokens3.093 (E32) // 384 steps; 100.7M tokens3.101 (E40) // 192 steps; 100.7M tokens3.102 (E32) // 96 steps; 100.7M tokens3.136 (E24) // 48 steps; 100.7M tokens
Step-matched3.112 (E32) // 384 steps; 50.3M tokens3.093 (E32) // 384 steps; 100.7M tokens3.101 (E40) // 384 steps; 201.3M tokens3.112 (E32) // 384 steps; 402.7M tokens3.153 (E24) // 384 steps; 805.3M tokens

153M model

WarmupBS32BS64BS128BS256BS512
Token-matched3.350 (E96) // 768 steps; 100.7M tokens3.332 (E112) // 384 steps; 100.7M tokens3.356 (E80) // 192 steps; 100.7M tokens3.360 (E48) // 96 steps; 100.7M tokens3.417 (E48) // 48 steps; 100.7M tokens
Step-matched3.354 (E96) // 384 steps; 50.3M tokens3.332 (E112) // 384 steps; 100.7M tokens3.368 (E128) // 384 steps; 201.3M tokens3.369 (E48) // 384 steps; 402.7M tokens3.407 (E48) // 384 steps; 805.3M tokens

(continued)

AuthorsOpenReview ↗

Response (2/2)

(Continued from Response (1/2))

In general, we find that the original way of token-matching is slightly better than step-matched or is marginally different; but with a larger batch size (batch size 256 or 512), step-matched give benefits, likely because prior steps were too short; however, they do not change the U-shape relationship between the batch size and the infinite-compute frontier.

We would also like to note that matching warmup tokens rather than steps is a standard experimental choice, e.g., “Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler”, “Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training”, “Resolving Discrepancies in Compute-Optimal Scaling of Language Models” all made this choice. In particular, the third paper explicitly noted that the warmup budget relative to available data matters more than warmup steps.

Regarding the LR selected for B=16 being at the bottom of the search grid: We also tested a lower learning rate and it wasn’t selected, which was what we intended by “values outside these ranges were never selected”. We recognize this wording was ambiguous and will revise the paper to clarify it.

W4: The GPU conclusions depend on how the extra GPUs are used. In Section 5 the microbatch is fixed, so more GPUs always means a larger B. Using extra GPUs for latency instead, through smaller microbatches or more TP/PP, is discussed in Section D.2 but left out of the main results.

We agree that the conclusion depends on how additional GPUs are used. Additional GPUs can be better used through model parallelism rather than data parallelism, as we discussed in the last paragraph of Section 5.2, “Parallelism can instead be spent on latency.” The reviewer's point is reasonable: the general claim that “more GPUs can be worse” is too broad and can be misleading. We will revise the paper to clarify that the claim is “more data parallelism can be worse,” rather than generally using more GPUs.

Q4: Does the y-axis in Figs. 7 and 10 mean GPUs used or GPUs available?

It is GPUs used, rather than GPUs available. We will revise the captions to make this clear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.