acceptodds
Under review as a conference paper at ICLR 2027

Scaling Domain Data Repetition in LLM Pretraining

Abstract

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(TPP\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(TPP\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(TPP\) can provide a practical estimate for larger models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, data mixture, scaling law, data repetition

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_oUSQOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how often to reuse a high-quality domain corpus during language-model pretraining when the token budget grows with model size. With N parameters, the training budget is D_N = TPP × N. The authors repeat a subset of αD_N unique domain tokens e times and fill the remaining budget with fresh web data. They study Code, Math, Wiki, and Medical data across four model sizes from 348M to 1.85B, three values of α, and a repetition grid from 1 to 7.

The preferred repetition count depends strongly on the domain, increases slightly with model size at fixed TPP, and changes little with α over the tested range. Figure 4 reports correlations of −0.944 with minimum validation loss, 0.400 with model size, and 0.018 with α, based on quadratic fits. A stylized linear-regression analysis links excessive repetition to fitting sample-specific noise. Further experiments compare repeated and fresh domain tokens at fixed mixture fractions and show that the learning-rate schedule also affects how much repetition is beneficial.

Critical strengths
  1. Keeping TPP fixed is a useful way to study repetition during model scaling, rather than extrapolating from experiments with a fixed total token budget. The setup in Sections 1 and 2 is clear, and Figure 2 makes the mixture construction easy to follow. Testing four domains, four model sizes, and three unique-data fractions gives a much more informative picture than a single recommended repetition count.

  2. The comparison with fresh data is one of the more convincing parts of the paper. Figure 6 keeps the total domain fraction fixed, so repeated tokens replace unique tokens rather than simply displacing web data. Math loses relatively little with up to four presentations, while Wiki degrades much more sharply. This shows why the cost of reuse needs to be assessed separately for each domain.

  3. The analysis offers a plausible explanation for the observed domain differences, and the schedule ablation adds an important qualification. Section 3 separates fitting signal from fitting sample-specific noise. In Figure 7, delaying or removing learning-rate decay allows more repetitions. I find this useful because it shows that the best repetition count is not just a property of the data, which matters when transferring a recipe from a proxy model.

Critical weaknesses
  1. I am not convinced that minimum validation loss is yet a predictor that generalizes across domains. The points in Figure 4 fall into four visibly separated domain clusters, so much of the pooled correlation could reflect domain identity. Both the minimum loss and the estimated optimum also come from the same curve and quadratic fit. A within-domain analysis or a held-out-domain prediction would help distinguish a transferable relationship from these dependencies. Theorem 3.2 gives a plausible account under fixed N, α, retained support, and signal assumptions, but does not establish that language-model losses measure comparable noise levels across the four domains.

  2. The paper may place too much emphasis on the repetition count when the repetition policy itself could be equally important. Figure 7 already shows that changing the learning-rate schedule shifts the point at which repeated data begins to hurt, even when the data recipe is otherwise unchanged. This suggests that an optimal count partly depends on when repeated examples are encountered under a particular optimization regime. The paper does not compare different ways of distributing the same repeated subset through training, such as uniform mixing, early reuse, or later-stage reuse. Without that comparison, it is unclear whether the count is the main transferable quantity or a proxy for a broader repetition schedule.

  3. The practical scope of the scaling recommendation remains unclear beyond the small models and data-constrained setting studied here. The experiments cover 348M to 1.85B parameters, so the mild increase in the preferred repetition count does not by itself justify extrapolation to models with tens of billions of parameters. Without uncertainty estimates, even the stability of the trend within this range is difficult to assess. It is also unclear when repetition remains the right choice if more high-quality unique domain data can be collected or produced through quality-controlled synthesis. Figure 6 already shows that fresh domain tokens are generally preferable to repeated ones at a fixed domain fraction, but it does not establish the conditions under which reuse remains worthwhile once corpus expansion is feasible. Repetition may still be useful when domain data is genuinely scarce. The broader recommendation needs clearer boundaries on model scale and data availability, along with domain-task results showing that the selected counts improve useful capabilities rather than validation loss alone.

  4. Although I can follow the paper’s main ideas, the main figure appears to have been generated using GPT-Image. This presentation weakens my confidence in the paper and raises concerns about whether the manuscript itself may also be AI-generated. I encourage the authors to redraw the figure in a more precise and conventional academic style in a future revision.

Questions for authors
  1. How much of the −0.944 correlation in Figure 4 remains after accounting for domain identity? Could you report a within-domain or domain-adjusted analysis and test prediction on a held-out domain? Please also assess the dependence introduced by deriving both axes from the same validation-loss curve.

  2. Could you report repeated runs and uncertainty intervals for the loss curves and fitted optima, along with downstream domain results for the selected repetition counts?

3 comments

We sincerely thank the reviewer for the comments and suggestions. We address the questions below.

Q1: How much of the −0.944 correlation in Figure 4 remains after accounting for domain identity?

A1: We thank the reviewer for this comment. On the 48 settings in Figure 4, we report (i) the partial correlation controlling for domain, (ii) the within-domain correlation, and (iii) the held-out-domain error of a linear fit from minimum loss to optimal count on the other three domains, compared with predicting their mean optimal count. To address the dependence between the two axes, we also replace the minimum loss with the loss at e=1e=1, which does not depend on where the curve reaches its minimum.

MetricCodeMathWikiMedicalAverage
Within-domain correlation−0.802−0.759−0.917−0.962−0.860
Held-out MAE (loss-based)0.270.440.190.400.32
Held-out MAE (mean of other domains)0.281.460.740.790.82

The correlation (−0.944-0.944 pooled) remains strongly negative after controlling for domain (−0.837-0.837) and within each domain (−0.860-0.860 on average), and the loss-based prediction has a much lower average held-out error than the baseline (0.320.32 vs. 0.820.82), suggesting that the relationship is not merely driven by domain identity. The two predictors are comparable only on Code, whose mean optimal count is close to that of the other domains. Using the loss at e=1e=1 gives a similar correlation (−0.945-0.945), so this reduces the concern that the correlation is induced by extracting both quantities from the same fitted minimum. This predictor is also more practical, as it requires only one model per domain.

Within each domain, the relationship mainly comes from model size, as larger models reach lower loss and tolerate more repetitions (correlation of the optimal count with log⁡\log model size: 0.860.86 to 0.920.92; with α\alpha: 0.020.02, pooled). Increasing α\alpha lowers the loss without changing the optimal count (Figure 3), consistent with Theorem 3.2: since UNlog⁡(⋅)U_N \log(\cdot) in the denominator is approximately constant for large UNU_N, the leading term of eN∗(σ2)e^*_N(\sigma^2) depends on σ2\sigma^2 but not on the amount of unique data.

Finally, we agree that Theorem 3.2 does not directly show that the losses of different domains reflect comparable noise levels; we use the loss as an empirical proxy and will clarify this in the next version.

Q2: The repetition policy itself could be as important as the repetition count.

A2: We thank the reviewer for this insightful comment. Our paper studies the repetition count under one fixed policy, and Section 4.2 shows that changing the learning-rate schedule substantially shifts the optimal count, which supports the reviewer's point. A direct comparison of policies, e.g., spreading repeated data uniformly versus concentrating it early or late, would be valuable; we cannot run it within the discussion period but will add it in the next version.

Q3: Could you report repeated runs, uncertainty intervals, and downstream domain results?

A3: We thank the reviewer for this comment. We did not train the same configuration with multiple seeds, since each run is costly at these scales. As an alternative, the three unique-data fractions at each model size correspond to three independently trained models, and Theorem 3.2 predicts the optimal count to be independent of α\alpha. The fitted optimal repetition counts across the three fractions (mean ± std) are:

Domain348M619M1.25B1.85B
Code3.90 ± 0.114.34 ± 0.204.42 ± 0.164.68 ± 0.21
Math4.65 ± 0.065.50 ± 0.125.64 ± 0.195.72 ± 0.15
Wiki3.19 ± 0.083.76 ± 0.083.89 ± 0.094.07 ± 0.11
Medical3.16 ± 0.203.50 ± 0.143.96 ± 0.274.12 ± 0.10

The spread at a fixed model size (mean std of 0.140.14 over the 16 configurations) is much smaller than the increase across model sizes, and the monotone increase holds in all four domains. In the next version, we will add explicit seed repetitions with uncertainty intervals, as well as downstream evaluation, which we agree would be more decisive than validation loss.

Q4: I encourage the authors to redraw Figure 2 in a more precise style.

A4: We appreciate the suggestion. As stated in our AI use statement, we used generative AI tools to assist with the visualization of the experimental framework in Figure 2, while all experiments and results were produced by the authors. We have redrawn Figure 2 in a conventional academic style. You can see it in the paper.

We thank the reviewer once again for the valuable and helpful suggestions. We are happy to provide further clarifications if the reviewer has any additional questions.

The author responded to my concerns quite well, candidly admitted the shortcomings of some of the discussions, and promised to make revisions. In the latest revised version, the main image I suggested has also been modified. Therefore, I raise my rating to 3.

Thank you for your feedback! We will incorporate the relevant experiments and additional discussions into the next revision of the paper.

Reviewer_HDU6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the optimal repetition of high-quality domain-specific data during LLM pretraining. The main finding is that, while keeping the tokens-per-parameter (TPP) ratio fixed, the optimal repetition count increases slightly with model size. This contrasts with some previous observations under a fixed training-token budget. The authors also find that optimal repetition depends strongly on the domain and correlates negatively with validation loss.

Critical strengths
  • Interesting and practically relevant findings. The paper provides useful empirical observations about data repetition and model scaling. In particular, the most important observation is that, when maintaining a TPP ratio, the optimal repetition count increases slightly with model size.

  • Useful practical implications. The results suggest that smaller proxy models can help select conservative repetition counts for larger models, potentially reducing the cost of very expensive ablations on large models.

Critical weaknesses
  • Insufficient experimental details and limited generalization. I think important implementation details are missing, including the model architectures, exact TPP value, and detailed dataset specifications. These details are crucial for reproducibility. This makes me wonder whether the conclusions are general or highly dependent on the specific models or training configurations. Furthermore, the largest model has only 1.85B parameters, leaving open the generalization to current larger LLMs.

-Limitations of the theoretical explanation and experimental interpretation. I don't find the theoretical explanation particularly convincing. It relies on strong simplifying assumptions, including orthogonal domain representations and additive Gaussian noise, whose real relevance to realistic LLM pretraining is questionable. I think it is much better and safer to limit the paper to the empirical findings rather than build an artificial theory to justify the observed results. I think that part of the paper should be revised, and the limits of the theoretical analysis should be stated more clearly in the paper.

Questions for authors
  • Could you provide the exact model architectures, TPP value, dataset sources, and training hyperparameters? What motivated choosing a TPP greater than 100? Are the results consistent across random seeds?
  • The proposed approach seems useful for identifying repetition counts unlikely to cause overfitting at larger scales, but it does not necessarily predict the optimal repetition count for the larger model. Could you make this distinction more explicit?
  • How realistic are the theoretical assumptions, particularly the orthogonality of domains and the connection between validation loss and noise variance? Also, would the conclusions hold when the absolute amount of unique high-quality data remains fixed as model size increases, which seems closer to the motivating scenario?
  • Do the authors have more results on larger models? Any insights on how general the authors think these findings are?
4 comments

We thank the reviewer for the positive feedback and constructive suggestions. In the following, we address the main concerns raised.

Q1: Could you provide the exact model architectures, TPP value, dataset sources, and training hyperparameters? What motivated choosing a TPP greater than 100? Are the results consistent across random seeds?

A1: We thank the reviewer for these questions. Due to company confidentiality requirements, we cannot disclose the exact model architectures, TPP value, dataset sources, or training hyperparameters. We sincerely apologize for this limitation and appreciate the reviewer's understanding.

Nevertheless, to characterize the datasets without disclosing their sources, we profile them with the domain and genre taxonomies of WebOrganizer [1]. The largest (domain, genre) categories of the Math data are tutorials on science and technology (39.6%) and structured data on science and technology (12.2%), while those of the Medical data are academic writing on health (23.0%), academic writing on science and technology (16.0%), and knowledge articles on health (12.3%).

Regarding TPP, we use MoE models and calculate TPP using the number of activated parameters. Thus, TPP greater than 100 means that the training token count exceeds 100 times the activated parameter count, rather than the total parameter count. We choose this setting because modern LLMs are typically trained far beyond the compute-optimal TPP of about 20 [2] to reduce inference cost. For example, Llama 3 8B is trained with a TPP of about 1,900 [3], and DeepSeek-V3 is trained with a TPP of about 400 based on its activated parameters [4]. In this regime, high-quality domain data becomes increasingly scarce relative to the training budget, which is exactly the scenario we study.

Regarding random seeds, we did not repeat the same configuration with different seeds, since each run in our grid is costly at these scales. Our grid does, however, provide a related form of evidence. At each model size, the three unique-data fractions correspond to three independently trained models, and Theorem 3.2 predicts that the optimal repetition count should not depend on α\alpha. The fitted optimal repetition counts for the Math dataset are as follows:

Model Sizeα=1/40\alpha = 1/40α=1/20\alpha = 1/20α=1/10\alpha = 1/10Mean ± Std
348M4.594.594.694.694.684.684.65±0.064.65 \pm 0.06
619M5.595.595.365.365.555.555.50±0.125.50 \pm 0.12
1.25B5.805.805.705.705.425.425.64±0.195.64 \pm 0.19
1.85B5.835.835.795.795.555.555.72±0.155.72 \pm 0.15

The spread across the three independent runs at each model size, with a mean standard deviation of 0.130.13, is much smaller than the increase across model sizes, which is 1.071.07 from 348M to 1.85B. The same monotone increase holds in all four domains:

Domain348M619M1.25B1.85B
Code3.90±0.113.90 \pm 0.114.34±0.204.34 \pm 0.204.42±0.164.42 \pm 0.164.68±0.214.68 \pm 0.21
Math4.65±0.064.65 \pm 0.065.50±0.125.50 \pm 0.125.64±0.195.64 \pm 0.195.72±0.155.72 \pm 0.15
Wiki3.19±0.083.19 \pm 0.083.76±0.083.76 \pm 0.083.89±0.093.89 \pm 0.094.07±0.114.07 \pm 0.11
Medical3.16±0.203.16 \pm 0.203.50±0.143.50 \pm 0.143.96±0.273.96 \pm 0.274.12±0.104.12 \pm 0.10

Since the trend is reproduced in 16 independent configurations covering four domains and four model sizes, we believe it is unlikely to be an artifact of a particular random seed. We will add explicit seed repetitions in the next version of the paper.

Q2: The proposed approach seems useful for identifying repetition counts unlikely to cause overfitting at larger scales, but it does not necessarily predict the optimal repetition count for the larger model. Could you make this distinction more explicit?

A2: We appreciate the reviewer's question. We agree that our approach provides a conservative estimate of the repetition count for the larger model, rather than an exact prediction of its optimal repetition count. In practice, this conservative estimate already achieves performance close to the optimum. As shown in our response to Reviewer JPsP (Q2), using the repetition count selected on the 348M proxy model achieves nearly the same performance as using the optimal repetition count for the 1.85B models. We will make this distinction explicit in the next version of the paper, and leave a quantitative rule for predicting the exact optimal repetition count to future work.

Q3: How realistic are the theoretical assumptions? Would the conclusions hold when the absolute amount of unique high-quality data remains fixed as model size increases?

A3: We thank the reviewer for raising this important question. Our theoretical analysis aims to explain the observed relationship between validation loss and the optimal repetition count through the trade-off between learning the signal and fitting the noise. The assumptions are mainly introduced to facilitate theoretical analysis and are not required to strictly hold in practice.

The orthogonality assumption allows us to analyze the high-quality domain independently. It approximates the setting where different domains mainly rely on different features, which is consistent with our observation in Appendix D that repeating one domain has little effect on the OOD loss.

Regarding the connection between validation loss and noise variance, σ2\sigma^2 determines the irreducible risk in our model, which corresponds to the intrinsic entropy of a domain in language modeling. Since the validation loss also contains a reducible part, we use it as an empirical proxy of the noise level when comparing different domains. In addition, the Gaussian assumption is not necessary, since the proof only uses the first two moments of the noise and thus holds for any i.i.d. zero-mean noise with variance σ2\sigma^2. Empirically, the theoretical prediction is consistent with our results across all domains, model sizes, and unique-data fractions (Figure 4). We will clarify these assumptions in the next version of the paper.

Regarding a fixed amount of unique high-quality data, the unique-data fraction α\alpha decreases as the model size increases. Since the optimal repetition count is largely insensitive to α\alpha within the tested range (Figure 4), we expect our conclusions to still hold. To verify this, we select settings with nearly the same amount of unique high-quality data across model sizes from our experiments, i.e., α=1/10\alpha = 1/10 for 348M, α=1/20\alpha = 1/20 for 619M, and α=1/40\alpha = 1/40 for 1.25B. The optimal repetition counts are as follows:

Model Size (α\alpha)CodeMathWikiMedical
348M (1/101/10)3.883.884.684.683.283.283.343.34
619M (1/201/20)4.464.465.365.363.673.673.423.42
1.25B (1/401/40)4.254.255.805.803.893.893.653.65

We find that the optimal repetition count generally increases with model size even when the amount of unique data is fixed, which is consistent with our main findings. We will include this discussion in the next version of the paper.

Q4: Do the authors have more results on larger models? Any insights on how general the authors think these findings are?

A4: We thank the reviewer for the question. We provide theoretical insights into why larger models may benefit from more repetitions in our response to Reviewer z6fw's Q1.

We also conducted additional experiments on the Math dataset at a larger scale, using the same TPP and the same unique-data fractions as our main experiments. The fitted optimal repetition counts are as follows:

Model Sizeα=1/40\alpha = 1/40α=1/20\alpha = 1/20
1.85B5.835.835.795.79
3.2B5.975.975.905.90

Combined with our main experiments, where the optimal count rises from 4.654.65 at 348M to 5.725.72 at 1.85B, this suggests that the optimal repetition count is still increasing and is consistent with our main conclusion. We will report them in the next version of the paper.

Regarding generality, our findings are consistent across four domains with different characteristics, four model sizes, and three unique-data fractions, and are further supported by the theoretical analysis. Therefore, we believe that they generally apply to practical LLM pretraining where the training budget grows with model size, as long as the proxy and target models share the same TPP and training configuration.

We thank the reviewer once again for the valuable and helpful suggestions. We would be happy to provide further clarifications if the reviewer has any additional questions.

References

[1] Wettig, Alexander, et al. "Organize the Web: Constructing Domains Enhances Pre-Training Data Curation." The Forty-second International Conference on Machine Learning.

[2] Hoffmann, Jordan, et al. "Training Compute-Optimal Large Language Models." The Thirty-sixth Annual Conference on Neural Information Processing Systems.

[3] Dubey, Abhimanyu, et al. "The Llama 3 Herd of Models." arXiv preprint arXiv:2407.21783 (2024).

[4] Liu, Aixin, et al. "DeepSeek-V3 Technical Report." arXiv preprint arXiv:2412.19437 (2024).

[5] Xue, Fuzhao, et al. "To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis." The Thirty-seventh Annual Conference on Neural Information Processing Systems.

[6] Chudnovsky, Jessica, et al. "Internal Data Repetition Destroys Language Models." arXiv preprint arXiv:2606.24998 (2026).

I'm only partially satisfied with the author's response. In my view, the lack of detail provided, due to company policy limitations, undermines a core scientific value: reproducibility. We will discuss this issue internally with the other reviewers and the area chair. For now, I keep my score unchanged.

We thank the reviewer for the response. We can release all numerical results reported in the paper and the plotting code, but company confidentiality requirements prevent us from disclosing the specific training data and model architectures. We appreciate your understanding. Our main goal is to propose empirical findings, and we plan to further validate our findings using open models such as Qwen.

Reviewer_z6fwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies scaling repetitions of high-quality domain data in the data-constrained regime. It proposes to study smaller proxy models with the same TPP (tokens-per-parameter) as the target model and presents three main findings: (1) larger models may benefit from slightly more repetitions under constant TPP scaling, so tuning on smaller models gives a conservative estimate; (2) domains with higher validation loss tolerate fewer repetitions than domains with lower validation loss; (3) the amount of unique data for a domain is not a significant determining factor of optimal repetition counts. The findings are mostly empirical. For finding (2), some theoretical motivation is provided as well. Some additional findings presented, e.g., on the impact of the learning rate decay schedule, are also quite interesting.

Critical strengths

The paper offers simple but important insights into scaling data mixtures in data-constrained regimes. It also offers very practical advice and recipes.

Critical weaknesses

The finding (1) that larger models may benefit from more repetitions seems underexplored. The finding is not motivated and not quantified. No actual scaling law is presented and in the figures some flattening is apparent, but not significant shifts of the optimal repetition count.

Questions for authors

Do you have motivation or quantifications for your finding that larger models may benefit from more repetitions than smaller models?

3 comments

We thank the reviewer for the constructive feedback and suggestion. Below, we address the main question.

Q1: Do you have motivation or quantifications for your finding that larger models may benefit from more repetitions than smaller models?

A1: Thank you for the question. We provide a theoretical analysis that motivates and quantifies this trend in a simplified setting where the model learns only from the high-quality domain. Under linear scaling of dataset size with model size, the optimal repetition count grows sublinearly with model size.

Setup. We consider one-hot linear regression, where each coordinate represents a knowledge unit. Knowledge frequencies and expected signal energies follow power laws:

\qquad \mathbb{E}[p_k(\theta_k^*)^2]=k^{-\beta}, \qquad \beta>\alpha>1.$$ Here, the expectation is over a prior on the target predictor. A model of size $N$ can represent only the first $N$ coordinates, through the projection $S_N$. We draw $D$ independent examples: $$y_i=\langle x_i,\theta^*\rangle+\epsilon_i, \qquad \epsilon_i\sim\mathcal N(0,\sigma^2).$$ Starting from zero, we run full-batch gradient descent with a fixed learning rate: $$L_N(\theta)=\frac{1}{2D}\sum_{i=1}^{D} (\langle S_Nx_i,\theta\rangle-y_i)^2, \qquad \theta_{r+1}=\theta_r-\eta\nabla L_N(\theta_r), \quad 0<\eta<1.$$ Each iteration reuses the same dataset, so r serves as the repetition count in this simplified model. $D$ counts sampled examples, distinct from the total repeated exposure $rD$. We define the expected population excess risk and its earliest optimal integer stopping time as $$R_{D,N}(r) =\frac{1}{2}\mathbb{E}\left[ \left(\langle S_Nx,\theta_r\rangle-\langle x,\theta^{\ast}\rangle\right)^2 \right].$$ $$r^{\ast}(D,N)=\min\underset{r=0,1,2,\ldots}{\operatorname{argmin}} R_{D,N}(r).$$ **Result and proof.** For fixed positive noise variance and $$D/N\to C_0\in(0,\infty),$$ we obtain $$r^*(D,N)=\Theta(D^{\alpha/\beta}) =\Theta(N^{\alpha/\beta}).$$ Since the exponent lies between zero and one, this gives sublinear growth of the optimal repetition count. To see why, let $M_k$ be the number of occurrences of knowledge unit $k$ and define $$M_k\sim\operatorname{Binomial}(D,p_k), \qquad q_k=1-\eta M_k/D.$$ Solving the coordinate-wise gradient descent updates gives the exact decomposition: $$2R_{D,N}(r)=A_N+B_{D,N}(r)+C_{D,N}(r).$$ where the unrepresented knowledge contributes $$A_N=\sum_{k>N}k^{-\beta},$$ the knowledge-acquisition error is $$B_{D,N}(r)=\sum_{k=1}^{N}k^{-\beta}\mathbb{E}\left[q_k^{2r}\right],$$ and the noise-fitting error is $$C_{D,N}(r)=\sigma^2\sum_{k=1}^{N}p_k \mathbb{E}\left[ \frac{(1-q_k^r)^2}{M_k}\mathbf{1}_{\{M_k>0\}} \right].$$ The integrand in the last expression is defined as zero when the occurrence count is zero. The first term is independent of repetition. Repetition decreases the knowledge-acquisition error but increases the noise-fitting error, and their balance determines the optimal repetition count. Denote their one-step decrease and increase by $S_D(r)$ and $V_D(r)$, respectively. Then $$2[\mathcal R_{D,N}(r+1)-\mathcal R_{D,N}(r)] =-S_D(r)+V_D(r).$$ For intermediate repetitions, the dominant coordinates satisfy $$rp_k\asymp1,\qquad k\asymp r^{1/\alpha}.$$ Using binomial generating functions and sum-integral comparisons gives, uniformly for sufficiently large $r$ up to $D$, $$S_D(r)\asymp r^{(1-\alpha-\beta)/\alpha}, \qquad V_D(r)\asymp\frac{\sigma^2}{D}r^{(1-\alpha)/\alpha}.$$ The truncation at N does not change these orders because D and N grow proportionally. Therefore, $$\frac{S_D(r)}{V_D(r)} \asymp\frac{D}{\sigma^2}r^{-\beta/\alpha}.$$ The balance between learning signal and fitting noise consequently occurs at $$r\asymp D^{\alpha/\beta}$$ for fixed noise variance. More precisely, the risk decreases below a sufficiently small constant times this scale and increases above a sufficiently large constant times this scale, up to $D$. To exclude a later, better minimum, the exact binomial expressions also yield $$\sup_{r\ge D}\frac{S_D(r)}{V_D(r)} \lesssim \frac{D^{1-\beta/\alpha}}{\sigma^2}\longrightarrow0.$$ Thus, the risk remains increasing beyond $D$ for sufficiently large $D$, establishing the stated order of the global optimum. The intuition is that, as model size and dataset size grow together, the larger dataset reduces the noise-fitting penalty. Additional repetitions can therefore remain useful for longer, allowing the model to learn less frequent knowledge units. We also tried fitting a scaling law to predict the optimal repetition count, but omitted it due to large prediction errors. We believe this discrete optimum is difficult to predict reliably, so we focus on trends rather than exact values. We thank the reviewer once again for the valuable suggestion. We are happy to engage in further discussions to clarify any concerns.

Thank you for the response and toy model. It could be interesting to explore that idea and those toy model assumptions more in-depth in the future. I also appreciate the candid and clear response that "We also tried fitting a scaling law to predict the optimal repetition count, but omitted it due to large prediction errors". Overall, as the current paper stands, I still don't feel the claim (1) is sufficiently supported and explored, and I maintain my rating.

We thank the reviewer for the response. We will incorporate the points discussed here into the next version of our paper.

Reviewer_JPsPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work discusses the trade-off between training-token budget and model size under both fixed-domain and diverse-domain scenarios. The main results are empirical observations on the relationship on Token-Per-Parameter (TPP) ratio, optimal repletion count, and model size. These results provide beneficial implications on how to configure the optimal repetition counts for high-quality domain data.

Critical strengths
  1. This work addresses an important topic, i.e., the data repetition in LLM pretraining. As the model scales, some high-quality data is inevitably re-used. Exploring data repletion provides beneficial understanding on model optimization.
  2. The provided observations could provide useful insight into practice, if correct.
  3. Theoretical discussions are provided to explain the phenomenon, but I do not check the math.
Critical weaknesses

The paper reads more like a technical report than a research paper. Sections 2 and 3 are devoted almost entirely to experimental design and empirical phenomena: Section 2 defines the training grid, token budgets, and data fractions, while Section 3 reports how the optimal repetition count varies with domain loss, model size, and the unique-data fraction, followed by a theoretical explanation. The manuscript does not propose a substantive method for the problem it studies. It only offers a simple procedure in the end of Section 3.1 using one paragraph. Besides, nor does the paper systematically compare against existing data-repetition and data-constrained scaling approaches. As a result, the contribution is primarily descriptive: it documents a set of correlations under a fixed-TPP protocol and offers a qualitative explanation

Questions for authors
  1. The authors are suggested to extend the current work into a methodological framework, rather than presenting only empirical observations.

  2. Could the authors explain why repeating one domain leaves OOD performance largely unchanged (Section 4.1 and Appendix D), despite the intuition that repetition may distort representations outside the repeated domain? A brief mechanistic analysis, such as representation drift or gradient interference, would help justify this finding.

3 comments

We sincerely appreciate the reviewer's feedback and suggestions. Below, we provide responses to the questions.

Q1: The authors are suggested to extend the work into a methodological framework.

A1: We thank the reviewer for the question. Our work aims to provide practical guidance on configuring the repetition count of high-quality domain data. A key finding is that the conclusion depends on the scaling protocol. When the training-data size is fixed across model sizes as in prior work, the optimal repetition count decreases with model size. In contrast, under a fixed TPP, which better reflects practical LLM scaling, the optimal repetition count increases with model size (Figure 1).

Based on these findings, we summarize a framework for selecting the repetition count. We first train a smaller proxy model with the same TPP as the target model, using any unique-data fraction α\alpha within the tested range, and sweep the repetition count for each high-quality domain. We then fit a quadratic function to the final validation loss and use its minimizer as the selected repetition count. Since the optimal repetition count increases with model size under a fixed TPP and is insensitive to α\alpha, the selected count can be safely transferred to the target model. For a new domain, its validation loss on the proxy model further indicates a reasonable range of repetition counts to sweep. We will present it as an explicit algorithm in the next version of the paper.

Q2: The paper does not systematically compare against existing approaches.

A2: We thank the reviewer for this question. Existing data-repetition studies [1, 2, 3] compare model sizes under a fixed training-data size and conclude that larger models are more susceptible to overfitting, suggesting that repetition should be minimized for large models. We directly compare this setting with ours in Figure 1. Under a fixed training-data size, we reproduce the conclusion of prior work, where the optimal repetition count of the 1.85B model drops to 2. Under a fixed TPP, however, the optimal repetition count of the same model increases to 6. In Section 4.1, we also follow the setup of Muennighoff et al. [1], which shows that repeated data is nearly as effective as unique data for up to about four repetitions. We find that this observation holds for Math but not for Wiki (Figure 6), indicating that the repetition count should be selected for each domain rather than following a universal threshold.

To quantify the benefit of our framework, we compare the validation loss of the 1.85B models under (i) no repetition, as suggested by [2, 3]; (ii) 4 repetitions, as suggested by [1]; (iii) the repetition count selected by our framework on the 348M proxy model; (iv) the optimal repetition count of the 1.85B models. The results averaged over the three values of α\alpha are as follows:

MethodCodeMathWikiMedical
No repetition [2, 3]1.3391.3390.8490.8491.7851.7852.0402.040
4 repetitions [1]1.1981.1980.6970.6971.7431.7431.9871.987
Ours1.180\mathbf{1.180}0.689\mathbf{0.689}1.712\mathbf{1.712}1.953\mathbf{1.953}
Optimal1.1751.1750.6870.6871.6981.6981.9331.933

The results show that our method achieves lower validation loss than the baseline methods, and that the repetition count selected on a small proxy model can be effectively transferred to a larger model under the same TPP. We will include this comparison in the next version of the paper.

Q3: Could the authors explain why repeating one domain leaves OOD performance unchanged?

A3: We thank the reviewer for raising this important question. In the experiments of Appendix D, we fix the total fraction of Math tokens, and thus also the amount of web data, while only replacing unique Math tokens with repeated ones. As a result, all models are trained on the same amount of fresh web data, which accounts for at least 80% of the training tokens and largely determines the OOD performance.

Mechanistically, repetition mainly leads the model to fit the sample-specific noise of the repeated data (Section 3.2). In our theoretical model, different domains occupy different coordinates, so the noise fitted from the repeated Math data stays within the Math coordinates and does not affect the OOD risk.

We thank the reviewer again for the helpful feedback. Please feel free to let us know if there are any further questions.

References

[1] Muennighoff, Niklas, et al. "Scaling Data-Constrained Language Models." The Thirty-seventh Annual Conference on Neural Information Processing Systems.

[2] Xue, Fuzhao, et al. "To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis." The Thirty-seventh Annual Conference on Neural Information Processing Systems.

[3] Chudnovsky, Jessica, et al. "Internal Data Repetition Destroys Language Models." arXiv preprint arXiv:2606.24998 (2026).

I appreciate the careful responses from the authors, which somehow address my concerns. After reading the responses and other reviewers' comments, I have decided to maintain my rating. The authors' responses are not strong enough for me to change it. As I mentioned in my initial review, the paper is really hard to understand (to myself), which may be partially because I have no direct research experience on this topic. Even as a paper focusing on empirical discovery, its current shape is not satisfactory. I am also not very convinced of the extent to which the theoretical results can be effective in the background of llm training. Overall, I maintain my score and will let the AC downweight my rating.

We thank the reviewer for taking the time to carefully read our rebuttal and consider our clarifications. We will take the reviewer's suggestions and do our best to further improve the paper in future revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.