acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws for Mixture Pretraining Under Data Constraints

Abstract

As language models scale, the amount of data they require grows - yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality-filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15–20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data mixtures, LLM pretraining, scaling laws, data quality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EuAvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies several scenarios of data-constrainedness at small scale: a low resource language together with an abundant language; a tunable quality threshold (scarce high-quality data vs. abundant low-quality data); a multi-domain setting. The paper proposes two scaling laws LfixL_{fix} and LsizeL_{size} (one fit per model size, the other fit jointly) to predict optimal repetition counts for data-constrained domains and shows they achieve better fits than some other baselines.

Critical strengths

The paper covers several interesting scenarios representative of real-world challenges: e.g., (simulated) data-constrained low-resource language + unconstrained English text; trade-off between less unique tokens at higher quality thresholds versus more unique tokens at lower quality thresholds; and a multi-domain setting.

Critical weaknesses

The paper throughout studies fairly small models (up to 500M parameters) and fixed training horizons up to 10B tokens. Usually, scaling laws are meant to enable transferring conclusions from smaller proxy models such as these to models at larger scales at, e.g., constant TPP (tokens-per-parameter). However, in this paper, the focus seems to be more on an LfixL_{fix} scaling law that must be fit for each model size separately. (Another LsizeL_{size} scaling law is also presented but with little discussion and results.) The utility of this is a bit unclear to me: for small models and small training horizons, there are usually enough unique tokens and data-constrainedness is not a big concern; but for big models where data-constrainedness is a significant concern, it may be too expensive to fit a LfixL_{fix} scaling law for that model size.

Questions for authors

How do you practically suggest to use your LfixL_{fix} scaling laws for real-world large models trained for large token horizons, where data constrainedness is a real concern? Or else, do you have more to say on the utility of your LsizeL_{size} scaling law?

3 comments

Thank you for your review! We address the main concerns you raised below.

First, we would like to clarify the scale of our experiments: as stated in Section 3 (L208–210), our models span not up to 500M but up to 1.2B parameters and are trained for up to 54B tokens for the multilingual models and up to 65.5B for the Maths models (see Appendix B). The 10B-token horizons in the main-text examples (e.g., Figure 3) correspond only to our smaller models.

Indeed, fitting L_fix at the target scale is impractical for large models, and L_size is designed exactly for this: it is fitted on a ladder of small models and extrapolated to a larger target. Its data-size coupling term N^δ captures that larger models overfit faster yet reach lower minima (Figure 5), and at fixed N it reduces exactly to L_fix. Fitted on all but the largest model, L_size predicts held-out loss on the unseen largest model with wR² 0.65 on German, 0.73 on Maths (Table 8), 0.93 on French and 0.69 on Swahili (Table 17), higher than all multi-size baselines on German, Maths and French (e.g., 0.65 vs 0.56 for utility decay+N on German). Moreover, fitted on only the three smallest model sizes, it remains predictive, extrapolating 2.7× in parameters to 539M (wR² 0.59 / 0.88 / 0.64 on German / French / Swahili, Table 16). On the unseen model, L_size is also the only formula whose recommended mixture avoids catastrophic failure on both German and Maths (Table 15). We used the additional page to bring these results into the main text: Section 6 ("Size extrapolation", L423–457, marked in blue) summarises the held-out-size accuracy (Table 8), the wasted-token comparison (Table 15) and the three-model fitting ladder (Table 16), which remain in the appendix, together with the cost of fitting the law relative to a direct search.

To demonstrate the practical usefulness of L_size and address the concern about scale, we validated it at 2B scale: L_size fitted on 101M–539M models predicted the optimal German weight for a 2B model trained on 100B tokens (h* = 0.199), and we trained 2B models at h = 0.025, 0.199 and 0.625.

Table A: Target (German) validation loss:

RegimehGerman (target) loss
low0.0252.13
optimal0.1991.93
high0.6252.49

Table B: Target (German) downstream 0-shot accuracy at the same checkpoints.

Regimeharc_carc_ehellaswaglambadapiqasciqwinograndeavg
low0.0250.2710.3530.3680.2540.5620.6900.5140.430
optimal0.1990.2600.3930.4260.2910.6150.6680.5330.455
high0.6250.2580.3660.3630.1840.5950.5560.4990.403

The predicted optimum gives the best target-language loss, and downstream accuracy reproduces the loss results, so the effect reported in the paper persists at the larger 2B scale, a practically relevant scale used by widely adopted open model families (e.g., Qwen3.5-2B, Gemma-2-2B) and common in multilingual and on-device settings, where target-language data is scarce and repetition matters most. We have added the 2B validation to Section 6 (paragraph "Validation at 2B scale", L459–469 and Table 3, marked in blue).

Thank you again for your constructive feedback, we hope our responses and the revised paper have addressed your concerns. As the discussion period is coming to a close, please let us know if any questions remain, and we will be happy to address them.

Thank you for your detailed reply and updating the paper. The material you added to the main text significantly improves the merits of the paper and I have raised my rating to 3.

Reviewer_KkxeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mixture pretraining under data constraints, focusing on the trade-off between emphasizing scarce target data and repeatedly exposing the model to the same examples. Through extensive experiments across model sizes, target-data budgets, languages, domains, and quality-filtered datasets, the authors investigate how repetition affects performance. They propose a repetition-aware scaling law to predict target-domain loss and recommend mixture proportions.

Critical strengths

The paper addresses an important practical problem: how to balance data quality, target-domain exposure, and repetition when valuable training data is limited.

The empirical study is extensive and covers a broad range of settings. In particular, the experiments that vary data quantity at a fixed quality level and compare quality levels at approximately equal dataset sizes help disentangle quality and quantity effects.

The experiments provide useful observations about how optimal repetition depends on the training budget, dataset size, and model scale. These findings offer practical insights even independently of the proposed scaling law.

Critical weaknesses

The relationship between the proposed formula and the observed overfitting dynamics is unclear. For fixed model size, target-pool size, and mixture proportion, Eqs. (2)–(4) appear to predict monotonically decreasing loss as the number of training tokens increases, assuming positive parameters consistent with their intended interpretation. The term γh is constant along such a training trajectory and cannot produce a subsequent increase in loss. This appears inconsistent with the non-monotonic learning curves shown in Figures 3 and 5. Although this does not necessarily invalidate the formula as an approximate mixture-selection tool, it limits its ability to explain the training dynamics that motivate the paper.

The practical advantage of formula-based mixture selection is not sufficiently established. The proposed method improves over several baselines, but its advantage over the stronger utility-decay baseline is modest or inconsistent. For example, Table 2 reports median wasted-token fractions of 26% and 31%, respectively, while Table 12 reports 24% for the proposed method versus 22% for utility decay on the held-out Maths model. Furthermore, its advantage over non-scaling-law strategies, such as fixed-repetition heuristics or a coarse mixture search under a comparable calibration budget, remains unclear. More direct training-based validation of the recommended mixtures would strengthen the practical claims.

The scaling-law comparisons could be more comprehensive. InfoLaw [1] jointly considers data quality, mixture weights, repetition, and model scale, making it particularly relevant to the quality-filtering experiments. A comparison under aligned fitting and evaluation settings, or a clear explanation of why such a comparison is not applicable, would help establish the additional value of the proposed formulation.

[1] Zhou et al. InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition. ICML 2026. https://proceedings.mlr.press/v306/zhou26af.html

Questions for authors

How do Eqs. (2)–(4) account for the increase in validation loss observed during training at a fixed mixture proportion? Please clarify the parameter constraints and provide representative predicted-versus-observed learning curves, including the overfitting region.

Under comparable calibration and training budgets, how do the recommended mixtures perform against the strongest scaling-law baseline and simple non-scaling-law strategies? Can the authors directly validate representative recommended mixtures through training, rather than relying primarily on interpolation within the existing experimental grid?

How does the proposed method relate to InfoLaw, particularly in the quality-filtering setting? What additional predictive or mixture-selection benefits does the proposed formulation provide?

3 comments

Thank you for the detailed review as well as suggested ablations! We now address the specific weaknesses and questions raised below:

W1/Q1. Formula vs overfitting dynamics. Following standard practice for data-constrained scaling laws (Kaplan et al., 2020; Muennighoff et al., 2023), our law models the best loss reachable with a given mixture: it decreases monotonically in D_total at fixed h (A, α, r₁, τ, γ > 0 in all fits), and each run is fitted only up to its loss minimum. Our goal is to predict the optimal h at a given budget, and the optimal mixture has typically not yet started to overfit at that budget, so its loss lies in the regime that the law estimates accurately. Over-repeated mixtures plateau as ρ(r) saturates, while lower-h mixtures keep improving through (1−h)·D_total. The predicted h* is within ~1.17× of the empirical optimum on held-out German checkpoints (Table 9). We added this clarification to the Appendix K (L1879–1886) in the updated paper version.

W2. Practical advantage of formula-based mixture selection. Your observation is correct; however, other experimental results in the paper demonstrate that our formulas are both:

  • more robust. Utility decay is competitive in places but fails elsewhere, while our formulas are more consistent across setups. L_fix predicts held-out loss better on every setup (wR² German 0.95 vs 0.72, Maths 0.88 vs 0.55, Quality 0.71 vs −0.64, Wiki/peS2o 0.80 vs 0.79, Table 1), wastes fewer training tokens (26% vs 31% median), and its worst-case recommendations are far less costly (76% vs 95% at the 90th percentile, Table 2). In size extrapolation, the 2pp Maths gap contrasts with German, where utility decay+N recommends mixtures outside the tested range whereas L_size wastes 58%; L_size also has higher held-out wR² on both German and Maths (0.65 vs 0.56 and 0.73 vs 0.69, Table 8).
  • more data-efficient. Halving the fitting data (first 25% of steps) preserves the mixture recommendation of L_fix on German, Maths and Wiki/peS2o (h* error 0.08, 0.13, 0.08; Table 11). On German, where we ran this sweep for all formulas (Table 12), L_fix is the best at every fitting fraction (12.5–50%) on both metrics and the only formula with positive held-out wR² throughout; at 25%, utility decay's error triples (0.26 vs 0.08) and its wR² turns negative.

In order to better address your questions and further verify efficiency and robustness of our model, we ran additional experiments:

  • Q2a. Performance of the recommended mixture against the utility decay and non-scaling-law strategies. The comparison in Table 2 is already at a matched budget: both formulas are fitted on the first 50% of training steps and evaluated on the same held-out steps. To test even a tighter budget, we refitted both on only the first 25% of steps and evaluated them as in Table 2 (pooled over German, Maths and Quality), adding two fixed-repetition heuristics: | Method (calibration data) | Median | Mean | p90 | |---|---|---|---| | L_fix (50%, Table 2) | 26 | 34 | 76 | | Utility decay (50%, Table 2) | 31 | 41 | 95 | | L_fix (25%) | 34 | 35 | 64 | | Utility decay (25%) | 61 | 56 | 89 | | Fixed r = 1 | 95 | 86 | 98 | | Fixed r = 4 | 82 | 70 | 95 |

Reducing the calibration data to the first 25% of steps barely affects L_fix, while utility decay's median waste doubles and fixed-repetition rules waste most of the budget regardless, confirming that the advantage of L_fix grows when calibration data is scarce.

Next, following your suggestion, we compared against a mixture search under the same calibration budget. Using the same runs, we keep the mixture with the lowest loss at the end of the calibration window for the full budget (median wasted training tokens in %, pooled over German, Maths and Quality):

Target budget ÷ calibration budget1.5–2×2–3×3–4.5×
L_fix262936
Direct search153380

As the budget grows, the target data is repeated more and the optimal mixture shifts, which search cannot anticipate but L_fix can. Search is thus near-optimal only within 2×, which requires training the full mixture grid to half the target budget; beyond that, in the practical regime, L_fix is clearly preferable. We added these results to Appendix K (Tables 13–14, L1992-2015).

  • Q2b. Direct validation of recommended mixtures through training. Using L_size fitted on 101M–539M models, we predicted the optimal German weight for a 2B model trained on 100B tokens (h* = 0.199) and trained 2B models at this weight and at a lower (0.025) and a higher (0.625) one. The predicted mixture gives the best German loss (1.93 vs 2.13 and 2.49) and wins 5 of 7 German downstream tasks (full results in our response to Reviewer JHDW). We added these results to the end of Section 6 (L455-467, Table 3 at L468-474).

W3/Q3. Comparison with InfoLaw. Thank you for the pointer! We have now added InfoLaw as a baseline, using the published formula with target and generic data as its two buckets. We report it under two fitting procedures: (i) an exact reproduction of their Section 5.2 (θ shared across model sizes, λ per size, selected by random search over 100k candidates to minimize the summed Spearman correlation, then α, β in log-log space); and (ii) our recipe, as for all other formulas (weighted Huber loss, basin-hopping). Two differences from the original setup apply to both: we fit target-domain validation cross-entropy rather than perplexity derived from downstream accuracies, and we fit on multiple checkpoints per run rather than one final loss per run.

Held-out loss prediction (test wR², higher is better; same fit/test split as in Table 1):

GermanMathsQualityWiki/peS2o
L_fix (ours)0.950.880.710.80
InfoLaw, exact0.40−1.38−4.81−2.77
InfoLaw, our recipe0.960.80−2.000.52

Mixture selection on held-out checkpoints (mixture error: median |log10 h*_pred − log10 h*_emp|, lower is better; wasted tokens: median / p90, pooled over German, Maths and Quality):

GermanMathsQualityWiki/peS2oWasted tokens (median / p90)
L_fix (ours)0.070.130.470.1026% / 76%
InfoLaw, exact0.310.630.290.3288% / 98%
InfoLaw, our recipe0.300.210.400.1244% / 97%

Under either procedure, our law recommends better mixtures on German, Maths and Wiki/peS2o and predicts loss better almost everywhere. On Quality, InfoLaw selects better mixtures across most pools, as expected for a law designed for quality buckets of a single corpus; ours targets the general case of a constrained domain mixed with generic data (languages, domains and quality tiers alike) and still predicts loss far better on Quality (wR² 0.71 vs −4.81 and −2.00). In the updated paper version, we added InfoLaw to related work (L129-131) and the experimental results to Tables 1, 2 and 9.

Thank you again for your constructive feedback, we hope our responses and the revised paper have addressed your concerns. As the discussion period is coming to a close, please let us know if any questions remain, and we will be happy to address them.

Reviewer_JHDWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates optimal data mixing strategies for large language model pretraining under data constraints. Through more than 2,000 training experiments, the authors analyze how target dataset size, data repetition, model size, and training compute budget affect model performance. The results show that abundant general-domain data in mixture pretraining can mitigate overfitting caused by repeated exposure to limited target-domain data. Based on these findings, the authors propose a repetition-aware scaling law for mixture pre-training that accounts for the diminishing returns of repeated data. The proposed scaling law models effective data volume to predict target-domain loss and optimize data mixture proportions. Experiments demonstrate that the method outperforms existing baselines across multilingual, multi-domain, and quality-filtered data settings, effectively reducing computational waste caused by suboptimal data mixing strategies.

Critical strengths
  1. The paper addresses the challenge of limited high-quality or domain-specific data in large language model pretraining by investigating the optimal configuration of data mixture proportions and repetition frequencies. It accounts for both the diminishing returns of repeated data and the contribution of general-domain data, addressing limitations of existing scaling laws.

  2. The paper conducts over 2,000 training experiments across different model sizes, languages, domains, and data quality settings, demonstrating the effectiveness and generalizability of the proposed method. These extensive experiments also provide valuable insights into the trade-offs among model size, data quality, and data quantity.

  3. Compared with traditional scaling laws, the proposed method more accurately predicts target-domain loss and optimal data mixture proportions, while supporting extrapolation from small-scale experiments to larger model scales.

Critical weaknesses
  1. Limited model scale and insufficient validation of scaling law extrapolation. The cross-model-scale prediction experiments are primarily conducted on models ranging from 101M to 805M parameters. The applicability of the proposed method to large language models with billions or even tens of billions of parameters remains unverified. However, the substantial computational cost of conducting such experiments is understandable, so this is not a major weakness.

  2. Limited validation in complex multi-domain mixing scenarios. The paper primarily investigates mixtures consisting of one data-constrained target domain and one abundant general domain. Although multi-domain experiments are included, the evaluation remains relatively limited. In practical large language model pretraining, mixtures often involve multiple data-constrained domains, complex inter-domain relationships, and potentially competing training objectives. It would be valuable to further investigate the effectiveness and generalizability of the proposed scaling law under these more complex scenarios.

Questions for authors
  1. The experiments are primarily conducted on relatively small-scale models, and cross-model-scale predictions still exhibit some errors. Could the authors further discuss the applicability of the proposed method to models with billions or even tens of billions of parameters? Is there any additional evidence supporting its extrapolation capabilities?

  2. The paper primarily adopts uniform data mixing and fixed training hyperparameters. Although the appendix evaluates cosine learning rate decay, the optimal number of repetitions still changes. Have the authors considered how dynamic data mixing, staged training, or other training hyperparameters might affect the proposed scaling law?

  3. The proposed method requires a certain number of small-scale training experiments to fit the scaling law parameters. Could the authors further quantify the computational cost of fitting the scaling law and compare it with the cost of directly searching for the optimal data mixture proportions?

3 comments

Thank you for the thorough review! We now address the specific weaknesses and questions raised below:

W1/Q1. Limited model scale and scaling law extrapolation. Following your suggestion, we have trained 2B-parameter models on German (1B)-English mix under three data-repetition regimes: the optimal target weight predicted by our scaling law (h=0.199), a lower weight (h=0.025), and a higher weight (h=0.625). The optimal weight was predicted with L_size on our bilingual German-English experiments: 5 model sizes (101M–539M) × 5 German pools, fitted against German validation loss, extrapolating those fitted parameters to N=2B, T=100B (~3.7× extrapolation in parameters and ~1.9× in training tokens beyond the largest model in the German fit, i.e. ~6.9× in training compute). The three models differ only in the German mixture fraction h; we compare all three at a common checkpoint of ~100B training tokens. Table A: Target (German) validation loss:

RegimehGerman (target) loss
low0.0252.13
optimal0.1991.93
high0.6252.49

Table B: Target (German) downstream 0-shot accuracy at the same checkpoints.

Regimeharc_carc_ehellaswaglambadapiqasciqwinograndeavg
low0.0250.2710.3530.3680.2540.5620.6900.5140.430
optimal0.1990.2600.3930.4260.2910.6150.6680.5330.455
high0.6250.2580.3660.3630.1840.5950.5560.4990.403

The predicted optimum gives the best target-language loss, and downstream accuracy reproduces the loss results, so the effect reported in the paper persists at the larger 2B scale, a practically relevant scale used by widely adopted open model families (e.g., Qwen3.5-2B, Gemma-2-2B) and common in multilingual and on-device settings, where target-language data is scarce and repetition matters most. We have added these results to Section 6 of the revised paper ((L455-467 and Table 3, marked in blue).

W2. Multi-domain scenarios. In practical pretraining mixtures, the number of data-constrained domains is typically small: in the Dolma 3 mix (Olmo 3 paper, Table 4), three of six sources are repeated in the 5.93T-token mix: FineMath (4.5x), Stack-Edu (3x), arXiv (2.4x), and the other three are consumed in under one epoch. This is close to the regime we study: one or two constrained domains plus unlimited generic data. Our three-domain experiments test the most direct practical question: whether the two-domain law can be applied per domain without a joint sweep. Setting each domain's r independently from the bilingual law outperforms the best grid-searched proportional weighting (Section 7, Figure 10c). We do not claim to cover mixtures with many simultaneously constrained domains. We added this clarification to Section 7 (L480-482).

Q2. Learning-rate schedule, dynamic mixing, and other HPs. The cosine-LR shift is modest and doesn't affect the law: optimal repetition r* is 13–20% smaller than under a constant schedule, but the qualitative picture remains unchanged: r* still grows with the budget and stays well above the ~4-epoch rule of thumb, and L_fix still fits well (test wR² = 0.81); r* thus remains predictable from small-scale runs under a realistic schedule (now stated in footnotes 3 and 4). As for the other training modifications, we use uniform sample-level mixing because it is the regime where mixture weight and repetition separate cleanly, and keep other HPs fixed since, in data-constrained settings, the mixing ratio matters more than HP tuning (as shown, e.g., in [1]). Since in our formulation D_eff depends only on the number of passes over the target data, not on when they occur, testing staged schedules is a natural next step; we added this to the Appendix N.

Q3. Computational costs quantification. Our law is cheaper than direct search along two axes:

  • Model size: L_size fitted on the smaller model sizes (101M–197M; 222 configurations at 40–100 GPU-hours each; see Table 5 and App.B) extrapolates to the larger model (test wR² = 0.59; Table 16), whereas a direct search at the 539M size would cost ~3× more in GPU-hours (82 configurations × ~600 GPU-hours), 4.1x in FLOPs. This gap grows with the target model size, since per-run cost grows quickly with N while the fitting cost stays fixed (this comparison is now also summarised in the main text, Section 6, "Size extrapolation", L451-453).
  • Training budget: fitting L_fix on only the first 25% of training steps preserves the mixture recommendation on German, Maths and Wiki/peS2o (median error ~1.2–1.4× in h; Tables 11–12). Moreover, a direct search answers a single (N, D_total, D_target) configuration, whereas the fitted law can be reused for any budget and pool size; e.g., our 2B prediction required no 2B runs at all. We added this to App.K (L2060-2086).

Follow-up on Q3. cost of fitting vs. direct search

To further address your question on the cost of fitting the scaling law compared with directly searching for the optimal mixture, we added a matched-compute comparison. For the German 539M model, we keep only m target fractions per data pool (m ∈ {3, 4, 6, 8, 12}) and compare two ways of choosing the mixture from these runs: (1) direct search, picking the run with the lowest loss, and (2) L_fix fitted on the first 50% of training only, predicting the optimal mixture for the full budget.

Main conclusions (new Figure 34 in Appendix K):

  • Wasted tokens: the law reaches the same level at 2-4× less compute (~32% below 2×10²¹ FLOPs; the direct search only gets there above 4×10²¹), and is much more stable across choices of target fractions.
  • A large direct search (m = 12) is more accurate, as expected, but significantly more costly.
  • Distance to the optimal mixture: the law gets within ~1.4× of h* below 2×10²¹ FLOPs, cheaper than the smallest direct search we tested, and ties with it at 2×10²¹; beyond that, the direct search gets closer.

Overall, our scaling law is more compute-efficient, allows choosing the mixture at a fraction of the compute of a direct search, and predicts the optimal mixture beyond the scales it was fitted on, which a direct search cannot do. These results are now in the revised paper (Appendix K, L2060–2086, Figure 34).

Thank you again for your constructive feedback, we hope our responses and the revised paper have addressed your concerns. As the discussion period is coming to a close, please let us know if any questions remain, and we will be happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and new results

We thank all reviewers again for their thoughtful and constructive feedback, which helped us strengthen the paper. Below we summarize how we addressed each concern. We have also uploaded an updated PDF version with all changes marked in blue.

  • Scale (JHDW, EuAv, Kkxe): we verified our law at a larger scale, on 2B-parameter models. Fitted only on 101M–539M models, L_size predicted the optimal mixture for a 2B model trained on 100B tokens, i.e., 3.7× more parameters and 6.9× more compute than any model in the fit. Training 2B models at the predicted mixture and at a lower and a higher one confirmed the prediction: the predicted mixture achieves the best German held-out loss and wins 5 of 7 downstream tasks. This demonstrates that the recommendation transfers to a much larger model, on loss as well as downstream performance.
    • In the revised paper: Sec. 6, L455–467; Table 3; setup in Tables 4–5 and App. B.
  • Utility of L_size (EuAv): we moved additional clarification about L_size from the appendix to the main text and explicitly showed that L_size, fitted once on small models, reliably predicts larger ones. On the unseen largest model it achieves wR² of 0.65 on German and 0.73 on Maths, it is the only formula that avoids catastrophic wasted-token failure, and three model sizes suffice to fit it.
    • In the revised paper: Sec. 6, L425–453; Tables 8, 15, 16.
  • Cost of fitting vs. direct search (JHDW): we added a new matched-compute comparison between the law and a direct search on the same runs (new Figure 34), together with a cost breakdown of fitting across model sizes. The comparison shows that the law, fitted on half-length runs, matches the wasted tokens of a direct search with 2–4× less compute and is much less sensitive to which mixture ratios are available. We further show that fitting L_size on small models is 2.9× cheaper than a grid search at the target size, and that the 2B prediction required no target-size runs at all.
    • In the revised paper: App. K, L2060–2086; Figure 34; Sec. 6, L421–423 and L451–453.
  • Advantage over baselines (Kkxe): we demonstrated that the advantage of L_fix grows when calibration data is scarce. With 25% of the fitting data, L_fix stays robust (median waste 26→34%) while utility decay doubles; fixed-repetition rules waste 82–95%, and a direct search calibrated at a smaller budget wastes 80% at 3–4.5× that budget, against 36% for L_fix.
    • In the revised paper: App. K, L1992–2015; Tables 13–14.
  • Comparison with InfoLaw (Kkxe): we added the comparison to InfoLaw under both its original and our fitting procedure and showed that L_fix selects mixtures better on German, Maths and Wiki/peS2o and predicts loss better almost everywhere. InfoLaw selects better mixtures only on Quality, consistent with its design for quality buckets of a single corpus.
    • In the revised paper: L130–131; Sec. 6, L390–392 and L403–406; Tables 1, 2, 9; App. J, L1852–1868; Table 18.

Additional clarifications: (i) overfitting dynamics: the law models the best reachable loss for each mixture, as is standard for data-constrained scaling laws, and predicts the optimal mixture within ~1.17× on held-out German checkpoints (App. K, L1880–1886); (ii) multi-domain mixtures: real pretraining mixtures typically contain only a few constrained domains, which is the regime of our three-domain experiments (Sec. 7, L480–482); (iii) learning-rate schedule: under cosine decay, all conclusions hold, with the optimal repetition only 13–20% lower, and dynamic/staged mixing and hyperparameter interactions are discussed as limitations (footnotes 3–4; App. N).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.