acceptodds
Under review as a conference paper at ICLR 2027

Data Schedules and Learning Rate Schedules Should Be Designed Together

Abstract

Mid-training (MT) or continual pretraining has become a common stage in modern large language model training pipelines. It follows the pretraining (PT) on web data, and includes smaller but higher-value data such as mathematics, code, reasoning traces, synthetic demonstrations, or long-context corpora. Although this practice is now widespread, it is still mostly treated as a practical technique rather than as a principal way of evolving training data distributions over time. In this work, we propose to view PT and MT as two phases of a unified data scheduling problem, in which the training distribution gradually shifts from broad web-scale coverage toward increasingly targeted objectives. This temporal choice is inherently coupled with optimization. The data schedule determines which distribution supplies the gradient at each point in training, while the learning rate (LR) schedule determines how strongly that gradient changes the model. Under this view, we reveal a critical but underexplored factor of data scheduling: the data schedule should be coordinated with the learning rate (LR) schedule. We find that while partially replaying PT data in MT substantially outperforms directly shifting data distribution from PT to MT data in the commonly used warmup-stable-decay LR schedule, its advantage diminishes when using a constant LR. Motivated by this contrast, we align the onset of LR decay with the increase in the MT data ratio. This matched decay range produces a better tradeoff between PT and MT loss than the other LR schedules we compare. Finally, we develop a tractable theoretical model that explains why the optimal allocation of a fixed MT data budget changes across analytically solvable LR schedules. Our results show that data and LR schedules should be designed jointly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Pretraining, Data Mixing, Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GbBVOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors investigate how to schedule data mixtures and LR decay jointly during MT stage, which is an important and interesting topic in the LLM training community. The main result demonstrate different outcomes under different settings. In specific, replaying PT data while introducing MT earlier can lower final MT loss under full-range linear LR decay, while concentrating MT data at the end often works better under constant LR.

Critical strengths

This paper raises an important question about how to optimize the MT stage during LLM training. The experuments are comprehensive and attractive.

Critical weaknesses

None

Questions for authors

Do the ranking reversals and delayed-decay recommendation persist under common nonlinear schedules, such as cosine decay or warmup-stable-decay?

Reviewer_Za2AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study investigates the joint optimization of data mixing and learning rate scheduling in LLM mid-training. Under fixed training compute and mid-training token budgets, the optimal data schedule depends on the learning rate schedule. Introducing and mixing mid-training data earlier improves performance when the learning rate decays throughout training, while a constant learning rate reverses this result. Experiments further show that starting learning rate decay after mid-training data enters training yields better results. A theoretical model explains this dependence.

Critical strengths
  1. The paper reveals a coupling between data scheduling and learning rate scheduling. It shows that the relative performance of data scheduling strategies can reverse under different learning rate schedules, indicating that the two should be jointly optimized. This finding provides valuable guidance for designing training strategies.
  2. The paper demonstrates the potential for making better use of limited data. Under continuous learning rate decay, introducing and mixing MT data earlier can achieve a lower final MT loss under the same budget.
  3. The experimental comparisons are fair. By fixing the total training compute and MT token budget, the experiments more clearly isolate the effects of the scheduling strategies.
  4. The paper addresses a clearly defined research question and is well written and logically organized. It also combines experimental validation with theoretical explanations.
Critical weaknesses
  1. The coverage of models, datasets, and training starting points remains limited. The experiments primarily use a 630M dense model and an MoE model with 2.99B total parameters, with DCLM and FineMath as the main datasets. Whether the conclusions generalize to larger models and more complex data from multiple domains requires further validation.
  2. The paper lacks downstream task evaluation. Including downstream task results would further strengthen the paper. The evaluation primarily relies on validation loss in the target domain, without sufficiently reporting downstream performance, such as mathematical reasoning accuracy or code generation success rates. Thus, the current results support improvements in language modeling loss, but do not establish improvements in practical task capabilities.
  3. The diversity of data types evaluated is insufficient. The experiments mainly focus on mathematical data, with code data used for supplementary validation. It remains unclear whether the same patterns hold for multilingual, medical, legal, or other types of data. The scope of the conclusions therefore requires further validation.
Questions for authors
  1. Do the relative rankings of data scheduling strategies and the advantages of delaying learning rate decay remain consistent when the peak learning rate or the final learning rate is changed?
  2. Can the authors provide guidance for selecting schedules for new models and datasets while reducing the number of trial runs?
  3. Does the similarity between new and existing data affect the optimal schedule? When the two differ substantially, should the recommended strategy be adjusted?
  4. When target domain data is scarce and must be reused across multiple training epochs, does delaying learning rate decay remain effective?
Reviewer_vhS5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how data mixing schedules and learning-rate schedules interact during continued pretraining under fixed total training and target-domain token budgets. Its main finding is that changing the learning-rate schedule can reverse the relative performance of data schedules. It further recommends maintaining a high learning rate after target data is introduced, followed by a shorter final decay.

Critical strengths

The topic is practically relevant. Understanding how to use limited, high-value data effectively is an important problem in language model training. The comparisons across dense and MoE architectures, data budgets, and training horizons provide useful evidence that data schedules should be evaluated together with optimization choices.

Critical weaknesses
  1. The contribution relative to closely related work needs clarification.

Prior work, particularly Kotha & Liang (2026) and Luo et al. (2026), has already discussed interactions between data ordering and learning-rate schedules, including the limitations of placing valuable data late under strong learning-rate decay. The paper should more precisely identify what new understanding it provides beyond these observations. In particular, it is unclear how much the proposed recommendation adds beyond appropriately tuning a warmup-stable-decay schedule. The systematic comparisons and theoretical characterization may offer additional value, but their contribution should be distinguished more clearly from the broader claim that data and learning-rate schedules should be designed together.

  1. The experiments do not establish robustness across model scales.

Although the paper evaluates both dense and MoE models, architectural diversity does not substitute for a model-size sweep. The preferred data schedule and decay window may change with model capacity and training dynamics. Consequently, it remains unclear whether the reported patterns, particularly the preferred relative decay durations, transfer to larger models. Even a modest scale sweep would strengthen the empirical conclusions.

  1. The evaluation does not establish downstream benchmark benefits.

The results primarily concern held-out language modeling loss on the pretraining and target distributions. These metrics are useful for controlled comparisons, but they do not establish whether the recommended schedules improve mathematical reasoning, code generation, or general capabilities. Representative target-domain and general benchmarks would help determine whether the loss improvements translate into practically meaningful gains.

  1. The trade-off between PT and MT performance remains insufficiently resolved.

The paper acknowledges that PT loss matters and presents PT–MT loss comparisons, but its scheduling recommendations primarily optimize final MT loss. It does not provide a clear criterion for selecting a schedule when improved target-domain performance comes at the expense of pretraining-domain performance. Comparisons at matched PT loss, or under an explicit retention constraint, would make the recommendations more actionable and clarify whether they remain preferable when both objectives matter.

Questions for authors

n/a

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our rebuttal

Dear Area Chair,

Thank you for your time and for overseeing the review of our paper. We have provided detailed responses and additional experiments, but have not yet received follow-up feedback from the reviewers as the deadline is approaching. To assist your assessment, we summarize below the main concerns raised by Reviewers vhS5 and Za2A and how we addressed them all point by point, and details are in our rebuttal.


  1. Contribution beyond prior work.
  • A: We clarified the relationship to Kotha & Liang and Luo et al. Our contributions include showing how changing the LR schedule can reverse the preference between cross-domain data schedules, and theoretically characterizing how the optimal data schedule depends on LR. We revised the introduction and related work discussion accordingly. (Response to vhS5, W1.)
  1. Limited model coverage.
  • A: We added experiments with a larger 1.4B dense model. (Responses to vhS5, W2; Za2A, W1.)
  1. Whether validation loss findings extend to downstream evaluation.
  • A: We added MATH benchmark evaluations. (Responses to vhS5, W3; Za2A, W2.)
  1. Limited target-domain diversity and the role of dataset similarity.
  • A: We added multilingual MT experiments, which reproduce both the change in preferred data schedule and the benefit of late LR decay. (Responses to vhS5, W3; Za2A, W3 and Q3.)
  1. How to select schedules when both PT and MT performance matter.
  • A: We expanded Section 4.1 and clarified the guidance from the existing PT–MT frontier comparisons. (Response to vhS5, W4.)
  1. Sensitivity to peak and final LR.
  • A: We added experiments varying these LR values. The tested changes preserve both the reversal in data-schedule preference and the benefit of late decay for MT loss. (Response to Za2A, Q1.)
  1. Data reuse/Multi epoch and practical schedule selection.
  • A: We added new experiments with multi epochs show that greater target-data reuse can favor earlier and longer LR decay. (Response to Za2A, Q2 and Q4.)

We would appreciate your consideration of these clarifications and new results when assessing the remaining concerns. If possible, we would also welcome your help in encouraging discussion with the reviewers. We remain happy to address further questions.

Thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.