- The contribution relative to closely related work needs clarification.
Prior work, particularly Kotha & Liang (2026) and Luo et al. (2026), has already discussed interactions between data ordering and learning-rate schedules, including the limitations of placing valuable data late under strong learning-rate decay. The paper should more precisely identify what new understanding it provides beyond these observations. In particular, it is unclear how much the proposed recommendation adds beyond appropriately tuning a warmup-stable-decay schedule. The systematic comparisons and theoretical characterization may offer additional value, but their contribution should be distinguished more clearly from the broader claim that data and learning-rate schedules should be designed together.
- The experiments do not establish robustness across model scales.
Although the paper evaluates both dense and MoE models, architectural diversity does not substitute for a model-size sweep. The preferred data schedule and decay window may change with model capacity and training dynamics. Consequently, it remains unclear whether the reported patterns, particularly the preferred relative decay durations, transfer to larger models. Even a modest scale sweep would strengthen the empirical conclusions.
- The evaluation does not establish downstream benchmark benefits.
The results primarily concern held-out language modeling loss on the pretraining and target distributions. These metrics are useful for controlled comparisons, but they do not establish whether the recommended schedules improve mathematical reasoning, code generation, or general capabilities. Representative target-domain and general benchmarks would help determine whether the loss improvements translate into practically meaningful gains.
- The trade-off between PT and MT performance remains insufficiently resolved.
The paper acknowledges that PT loss matters and presents PT–MT loss comparisons, but its scheduling recommendations primarily optimize final MT loss. It does not provide a clear criterion for selecting a schedule when improved target-domain performance comes at the expense of pretraining-domain performance. Comparisons at matched PT loss, or under an explicit retention constraint, would make the recommendations more actionable and clarify whether they remain preferable when both objectives matter.
Summary of our rebuttal
Dear Area Chair,
Thank you for your time and for overseeing the review of our paper. We have provided detailed responses and additional experiments, but have not yet received follow-up feedback from the reviewers as the deadline is approaching. To assist your assessment, we summarize below the main concerns raised by Reviewers vhS5 and Za2A and how we addressed them all point by point, and details are in our rebuttal.
We would appreciate your consideration of these clarifications and new results when assessing the remaining concerns. If possible, we would also welcome your help in encouraging discussion with the reviewers. We remain happy to address further questions.
Thank you for your time and consideration.
The Authors