acceptodds
Under review as a conference paper at ICLR 2027

Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

Abstract

The repetition count that works best for a small language model may not remain best at a larger scale. We study this effect in pretraining with a finite target corpus mixed with generic data at a fixed target fraction. On Wikipedia-derived data and Proof-Pile-2, the ranking of measured repetition counts changes with model size, and a 520M Proof-Pile-2 experiment confirms that reducing repetition from six- teen to eight improves loss while using fewer training tokens. We use loss curves from several smaller models to retain a short list of promising repetition counts for evaluation at a larger scale. On PubMed and Caselaw, candidate sets fixed before target-model training retain the lowest-loss measured count on the original evaluation grids at both 200M and 520M. This supports candidate retention as a practical alternative to exact point prediction. We also relate the pruning regres- sion to an empirical scaling model with two opposing repetition-dependent loss terms. A first-order expansion in log model size yields the linear form used by the selection rule, providing a scaling-based interpretation of the candidate-selection procedure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model pretraining, scaling laws, data-constrained training, data repetition, repetition count selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xsxtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This study investigates how preferred repetition counts change across model scales in data-constrained language model pretraining. The authors demonstrate that repetition counts selected using smaller models may not remain optimal at larger scales. To facilitate repetition selection at larger scales, they propose a candidate selection approach that fits linear regressions of excess loss against log model size and selects three promising repetition counts for evaluation at larger scales. The authors also develop empirical scaling models to describe how loss changes with repetition and estimate loss-minimizing repetition counts.

Critical strengths
  • The paper addresses a practically relevant problem in data-constrained pretraining, where selecting an appropriate repetition count can affect both training cost and held-out target loss.
  • The proposed candidate selection method identifies a small set of promising repetition counts using smaller-model loss curves, potentially reducing the number of configurations that need to be evaluated at larger scales.
Critical weaknesses
  • The main experiments are limited to models up to 520M parameters within a single Transformer family, using a fixed 100M-token target corpus and a target-data fraction of 0.8. Although the authors acknowledge these limitations, broader evaluations across model scales, architectures, and data configurations would strengthen the empirical evidence for generalizability. Additionally, the evaluation relies primarily on held-out cross-entropy loss without assessing downstream task performance.

  • The proposed candidate selection method is evaluated under a fixed target-data fraction, where increasing repetition also increases generic-data exposure and the total training budget. In contrast, Li et al. (2026) report mildly increasing preferred repetition counts with model size under fixed TPP, where the total token budget is held constant across repetition counts at each model size. Although the authors acknowledge this difference, the proposed candidate selection method has not been evaluated under fixed-TPP settings. Further evaluation would provide stronger evidence for its generalizability.

  • While the empirical scaling models capture the observed repetition-loss trends, their predictions of preferred repetition counts appear less reliable at 520M (Section 6.2 and Appendix E.3.3). Allowing the scaling exponents to vary or introducing saturation does not consistently improve these predictions. Although the authors acknowledge this limitation and report more favorable results for the simple model at 200M, further discussion of the factors affecting cross-scale prediction reliability would be helpful.

  • Given the high proportion of target data in the training mixture, the experimental setup appears relevant to domain-focused pretraining. However, practical domain adaptation often involves continued pretraining of existing LLMs rather than training models from scratch. It would therefore be useful to investigate whether the proposed repetition selection method remains effective when starting from pretrained checkpoints.

Questions for authors
  • Is there any additional evidence supporting the generalizability of the proposed method to larger models, different data configurations, or downstream tasks?
  • Would the proposed candidate selection method remain effective under fixed-TPP settings, where varying repetition changes the target-data fraction rather than the total training budget?
  • What factors might explain the more favorable point predictions at 200M compared with the less reliable extrapolations to 520M?
  • Would the proposed repetition selection method remain applicable to continued pretraining from existing pretrained checkpoints?
Reviewer_Z9GgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies how to choose repetition counts for a limited target corpus mixed with generic data as model size increases. It uses smaller model loss curves to shortlist three counts and fits empirical loss models to estimate the best count and the extra loss from nearby choices. Experiments show changes in repetition rankings across model sizes, with the preselected sets retaining low-loss choices on PubMed and Caselaw at 200M and 520M.

Critical strengths

The paper studies a practical problem. A repetition count that works well for a small model may be a poor choice for a larger one. This matters because small LMs experiments are often used to plan more expensive training runs. Getting the choice wrong can mean spending more compute and still getting a higher loss. The idea of keeping a few promising counts also makes sense. Predicting one exact best count is difficult, and a short list leaves some room for error. The authors then train larger models on multiple corpora to check whether the selected counts are actually useful.

Critical weaknesses
  1. The contribution seems incremental. Sedova et al. already study scarce target data mixed with abundant generic data and show that repetition preferences change with model size. The main additions here seem to be separately completed training runs for each repetition count and a rule that uses smaller-model loss curves to retain three candidates for a larger model. Although the training protocols differ, the added value over the closest work is not clear to me.
  2. More importantly, the comparisons do not establish a clear practical advantage. There is no direct comparison of the proposed selection rule with Sedova et al. under the same decision setting. Table 2 compares against an adapted Lovelace model, but the improved curve predictions do not translate into better repetition choices. On both 520M test corpora, the fixed set {8,12,32} achieves the same lowest measured loss as the selected sets (Tables C.2 and C.6). Including the smaller-model sweeps, the estimated GPU time is also 11.1% higher than this fixed baseline (Appendix C.6).
  3. The evidence for generalization is limited. The experiments only go up to 520M parameters, and the main selection study fixes the target pool at 100M tokens and the target fraction at 80%. Whether the method remains useful for substantially larger models or different data mixtures is unclear.
Questions for authors

See the above section content.

2 comments

Thank you for the clarifications and additional experiments. The comparison with Sedova et al. addresses part of my concerns. However, the practical advantage over the fixed candidate set remains unclear, and the evidence is still limited to a narrow setting. I will raise my soundness score to 3, but maintain my overall rating.

AuthorsOpenReview ↗

Response to Reviewer Z9Gg

Thank you for the follow-up and for raising the soundness score. We appreciate your recognition of the comparison with Sedova et al. We would like to clarify the practical role of the fixed-set comparison.

The set {8,12,32} was an empirical choice informed by development-stage reversals: we expected the valley to shift toward 8–12 at 520M and retained 32 as a higher-repetition check. This set includes the measured minima on the seven-count grid at 520M, but misses both minima in the retrospective 80M check, incurring 0.043/0.062 nat excess loss on the selection split (Table C.6).

Our rule retains the measured grid minima at both target sizes by adapting its candidates. With the same 35M/50M source curves, it selects {16,24,32} for 80M and {8,12,16} for 520M on both corpora (Table C.5). Section 6.1 now explicitly connects this change in candidates to adaptation to the target size.

The cost tradeoff for the main 35M/50M/80M pipeline is reported in §6.1 and Table C.7. At 520M, our target stage costs 187.8 versus 325.2 GPU-hours for the fixed set (−42.3%); including source scans gives 361.4 versus 325.2 (+11.1%). These deployment estimates exclude earlier development for both approaches. A simple source-only baseline—keeping the three best 80M counts—incurs 0.213/0.658 nat excess at 520M and costs 461.3 GPU-hours including its source scan. Our pipeline achieves lower loss at 21.7% lower cost than this baseline.

Beyond candidate retention, the quantitative model supports finer selection: at 200M, applying the frozen smaller-model prediction functions to integer counts without refitting yields choices only 0.003/0.008 nat above the measured minima on the refined twelve-count grids (§6.2, Table 1).

These results demonstrate scale-adaptive candidate selection and quantitative refinement, providing a reproducible procedure when a suitable empirical candidate set is not already available.

Reviewer_sd3pOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how preferred repetition counts change with model size when a limited target corpus is mixed with generic data during pretraining. It uses smaller-model loss curves to retain a short list of candidates for a larger model, and develops empirical models of repetition-dependent loss. The main selection experiments use PubMed and Caselaw.

Critical strengths
  1. The independently completed training schedules and repeated seed experiments give fairly clear evidence of ranking changes across model sizes.

  2. The candidate sets are fixed before target-model training and retain the best measured original-grid choices on PubMed and Caselaw. The full sweeps and appendix provide useful detail for assessing these results.

Critical weaknesses
  1. The benefit over a simple baseline is unclear. The fixed candidate set in Table C.2 reaches the same best loss as the proposed sets, while Appendix C.6 estimates lower training time for that baseline. This makes the additional proxy search difficult to justify. Please bring this comparison into the main paper and explain how the fixed set was chosen.

  2. The experiments say little about downstream usefulness. All quality evaluations use cross-entropy. Table B.11 even shows target and generic loss favoring different repetition counts, so the recommendation involves a practical tradeoff that remains unexplored. Evaluating existing checkpoints on a domain task and a general task would help explain what the selected count actually improves.

  3. The analysis adds limited insight. Much of it derives properties of an assumed loss curve. The local expansion also assumes a fixed minimizing count, while the minima change across the source sizes used for regression. The practical selector then adds bootstrap resampling, clipping, and candidate retention heuristics. Please clarify what new finding the analysis contributes beyond existing repetition models and the observed curves.

  4. The analytical models give inconsistent guidance for selection. Table E.11 shows a substantial extrapolation error on PubMed. In Table 2, the richer model fits losses better but selects worse configurations; this comparison also uses target-scale measurements during fitting. The candidate sets do succeed in the reported tests, but the separate curve models leave me unsure when their predictions would be useful. Their failure cases need a clearer discussion.

Overall I do not see this paper as a manuscript that meets the publication standard of a top ML venue such as ICLR.

Questions for authors
  1. Please clarify whether the fixed candidate set was chosen before seeing target results, and compare its full training cost with the proposed pipeline. Under what conditions would the proxy search be worth using?

  2. Could the existing checkpoints be evaluated on a small domain/general task pair, particularly where target and generic losses disagree? This would directly address the practical meaning of the recommendation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall response and revised PDF

We thank the reviewers for their constructive feedback. We have revised the manuscript to integrate the clarifications, comparisons, and downstream evaluation discussed in our responses. The revisions make the theoretical contribution and its empirical tests explicit in the main text. The table below locates the changes corresponding to each concern.

Reviewer concernClarification and evidence in the revised manuscriptLocation
sd3p W1/Q1; Z9Gg W2 — Fixed candidates and computational valueWe explain the empirical choice of {8,12,32}: observed reversals suggested a further shift toward 8–12 at 520M, with 32 retained as a higher-repetition check. We distinguish target-stage costs from deployment costs including source scans, and report the retrospective 80M comparison and actual candidate changes across target sizes.§6.1; Appendix C.1; Tables C.5–C.7
sd3p W3; Z9Gg W1/W2 — Contribution beyond existing repetition lawsAt fixed mixture fraction and target-pool size, the positive-parameter effective-data forms imply decreasing loss with repetition. Our increasing loss component captures the measured high-repetition rise. The three-candidate comparison gives 2.380/2.492 nat excess for the law-implied sets versus 0/0 for our retained sets.§5.1; Appendix F.3–F.4; Table F.2
sd3p W3 — Theoretical insight and moving minimaUnder the stated assumptions, each pair reverses at most once, toward lower repetition. All 42 pairwise mean-loss trajectories across five sizes satisfy these constraints. We explicitly state that the source minima move across 35M–80M and that the per-count regression spans these changes. The local expansion motivates the regression form; target-scale experiments evaluate its effectiveness.§§5.1–5.2; Appendix D.1; Propositions D.4–D.5; Appendix I
sd3p W3/W4; Z9Gg W2; xsxt W3/Q3 — Fitting accuracy versus transferProposition D.3 separates source-fit error from variation in cross-scale loss changes. We distinguish the frozen 200M predictions from Table 2’s curve-description comparison using target-scale measurements. The frozen alternatives and subsequent local-error diagnostic show why additional fitting flexibility need not improve selection.§§5.1, 6.2–6.3; Proposition D.3; Tables C.4 and 2
Z9Gg W1 — Value of independently completed runsThe matched-exposure decomposition explains why intermediate states and completed endpoints can give different rankings: schedule effects and additional training contribute terms of opposite sign in the measured example.§3.1; Appendix B.4; Eq. (12); Table B.11
sd3p W2/Q2; xsxt W1/Q1 — Downstream relevanceExploratory evaluation covers all 98 completed 520M checkpoints. Domain word-completion accuracy peaks at each corpus’s mean target-loss minimum in every seed, while generic-task scores reveal the accompanying tradeoff. Complete multiple-choice results and the CaseHOLD scoring diagnosis are reported alongside these findings.Introduction; §7; Appendix H
xsxt W2/Q2 — Fixed TPP and the training decisionWe clarify the practical decision: choosing duration after fixing the mixture. At fixed TPP, h=rSt/(κN)h=rS_t/(\kappa N), so varying repetition also changes composition. The revised discussion distinguishes these experimental axes.§7; Appendices B.5 and F.1
Z9Gg W3; xsxt W1/Q1 and W4/Q4 — Scale and continued pretrainingWe specify the validated model sizes, corpus pool, mixture, and architecture, and identify larger-scale and continued-pretraining evaluations as extensions requiring additional experiments.§7

Together, these revisions connect the observed reversals, the model’s testable directional constraints, and prospective candidate retention, while making the downstream tradeoff and the conditions for practical use explicit.

Dear Reviewers sd3p and xsxt,

As the discussion period closes on November 18 (AoE), we would be grateful for any feedback on our response and the revised PDF. Our general response includes a summary of revisions that maps each of your concerns to the corresponding section. We are happy to clarify anything further before the deadline.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.