- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the effect of pre-training with curriculum learning on the mathematical and reasoning ability of small LLMs. The authors train Qwen3.5-0.8B/1.7B architectures for 1 billion tokens on the DCIM pretraining dataset and find limited benefit with a curriculum over the document compression ratio. A similar experiment with synthetically curated grade-school-level mathematical data shows that a curriculum over grade level leads to improved mathematical ability after 10 billion tokens of pretraining, over randomly ordered data.
Critical strengths
Clarity
- The paper is generally well-written. The motivation and research question are well-posed, the related work is appropriate, and the figures and tables are attractive and easy to parse.
- The overall structure of the paper is solid, and the appendices contain ample detail on the data generation, methodology, and results.
Setting
- The paper studies a few curricula at the scale of a small pre-training run (1B--10B tokens). This is a fairly new frontier for the curriculum learning literature due to the expense of training with multiple curricula.
- The choice of the popular Qwen3.5 architecture at two scales (0.8/1.7B) is both relevant to practitioners and useful for understanding curriculum learning.
Critical weaknesses
By my assessment, the paper contains significant weaknesses do not address the broadly posed question in the paper title ('Does data order matter?').
Significance
- Much of the paper replicates results that I consider to be well-explored in the literature on curriculum learning. For example, the observation that curricula are of limited benefit on non-naturally-ordered pretraining data has been made in the BabyLM challenge [1], across a much wider range of curriculum strategies.
- The choice of difficulty metric (compression ratio) is not justified beyond an intuitive sense, and the usefulness of curricula with multiple other language-oriented difficulty measures has been established at exactly this data scale [2]. There is also no comparison to other difficulty measures (even simple ones like sequence length). While expensive, this is necessary for a robust claim.
- The choice of evaluation datasets, exclusively focused on commonsense/math reasoning ability, is odd for the assessment of a pre-trained model. The goal of pre-training is typically to learn a linguistic distribution, in addition to limited reasoning primitives. These datasets are typically used to evaluate models subject to extensive SFT/RL post-training, and a stronger choice would be to study linguistic or general knowledge characteristics (e.g., MMLU [4]).
- For such a broadly posed question, the choice of ordering and pacing plays a large role, and is not explored beyond a simple strategy.
Soundness
- I have strong concerns about the soundness of the synthetic data generation process (prompting a relatively weak LLM — GPT-OSS-20B — with a few concepts) used for the mathematics pre-training. LittleLearner [3] (cited by the authors themselves) has an extensive pipeline for ensuring that selected data actually adheres to the grade level that is required, and shows that a significant fraction of examples are rejected due to a lack of consistency. Other work has shown that models are poorly calibrated across grade levels. Combined with the need for interleaving, this undermines the conclusions made about the natural difficulty progression benefiting performance.
- Corroborating the above, the grade-ordered curriculum sees a very small benefit at 10B tokens (Figure 2) with the random ordering actually performing better at lower budgets — counter to nearly every observation on curriculum speedup in the literature.
- Pre-training exclusively on mathematical data for 10B tokens leads to only a modest performance on GSM8k and Math500. This is expected, as the model needs to learn linguistic primitives alongside the mathematical information, from only synthetic data. This is why most studies on mathematical reasoning are in post-training settings. Instead of starting from scratch, a continual pre-training setting would be a more sound approach.
References
[1] Warstadt, Alex, et al. "Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora." Proceedings of the babylm challenge at the 27th conference on computational natural language learning. 2023.
[2] Zhang, Yang, et al. "Beyond random sampling: Efficient language model pretraining via curriculum learning." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[3] Li, Fanfei, et al. "LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure." arXiv preprint arXiv:2608.13545 (2026).
[4] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).
Questions for authors
Answers to my questions are unlikely to push my score towards acceptance, particularly as my primary suggestions require new pretraining-scale experiments. However, I welcome the opportunity to discuss with the authors and encourage them to iterate on this work.
Clarification requests
- For the DCIM pretraining experiments, can you justify your choice of evaluation datasets (math/commonsense reasoning), and present results on tests of linguistic ability or general knowledge?
- Do you have an explanation for the fact that the random curriculum outperforms the grade-wise ordering up to 6 billion tokens in Figure 2? This runs counter to the long-standing result that curriculum learning can speed up early learning, despite limited long-term benefit.
- Can you provide statistics on the quality of the synthetic data (e.g., passing it through a stronger model to verify the grade level)?
Suggestions for additional experiments and future work
- To make a broad claim about whether data ordering matters requires a more extensive and principled exploration, particularly w.r.t. the evaluation metric. A good example of a well-posed question and a scoped answer is the When do Curricula Work? paper [1]. An exploration of a few more more orderings, pacing types, and difficulty measures at this scale would amount to a significant contribution in my view.
- LittleLearner is a great example of the care required to curate a data that is aligned according to grade level. Can you winnow out your pre-training data and iterate on the data generation process to make it more sound, and try a similar experiment?
- Reasoning ability is typically not a key goal of pretraining. Several works explore post-training curricula, but fewer focus on continued pretraining. Can you show that a curriculum with your synthetic data mixture results in improved performance after domain-adaptive continual pre-training?
References
[1] Wu, Xiaoxia, Ethan Dyer, and Behnam Neyshabur. "When do curricula work?." arXiv preprint arXiv:2012.03107 (2020).
2 comments
Response to Reviewer 9TAk (2/2)
"Pre-training exclusively on mathematical data for 10B tokens leads to only a modest performance on GSM8k and Math500. ... Reasoning ability is typically not a key goal of pretraining."
We respectfully disagree with both points.
On reasoning as a goal of pretraining: recent work finds that post-training mostly sharpens abilities the base model already has. The probability a pretrained model puts on good responses determines whether post-training can succeed (Chen et al., 2025, arXiv:2510.15020). RL converges on behaviours already present in the pretraining data (Zhao et al., 2025, arXiv:2504.07912), and a base model given the right opening tokens matches its RL-trained counterpart on math and code (Wang et al., 2026, arXiv:2610.06851). So what the model learns about mathematics in pretraining decides what post-training can reach, and this is why we study it there.
On the scores: we want to clarify that they are high for a 10B-token budget. The table compares our single-sample accuracy (avg@8, Appendix E.1) with published few-shot scores of base models of similar size.
| Model | Pretraining tokens | GSM8K | MATH |
|---|---|---|---|
| Llama-3.2-1B | 9T | 7.6 | 3.3 |
| SmolLM2-1.7B | 11T | 31.1 | 11.6 |
| Qwen2.5-0.5B | 18T | 41.6 | 19.5 |
| Qwen2.5-1.5B | 18T | 68.5 | 35.0 |
| Ours, 0.6B, grade order | 10B | 22.0 | 19.8 |
| Ours, 1.7B, grade order | 10B | 31.5 | 25.0 |
Baseline numbers are from Allal et al. (2025, arXiv:2502.02737, Table 4) and the Qwen2.5 technical report (arXiv:2412.15115, Table 5), with 4 or 5 shots. Ours use 1 shot on GSM8K and MATH-500. Our 1.7B model matches SmolLM2-1.7B on GSM8K and is 13 points above it on MATH, with about a thousand times fewer tokens (we are only training on math).
We also want to emphasize that in our experiments models learn a reasonable amount from very little data. After only 1B tokens of general web text they are well above chance on every benchmark we report:
| Benchmark | Random | 0.6B | 1.7B |
|---|---|---|---|
| PIQA (accuracy) | 50.0 | 61.5 | 63.4 |
| ARC-Easy (accuracy) | 25.0 | 46.2 | 49.3 |
| GSM8K (pass@8) | ~0 | 12.7 | 12.5 |
| MATH-500 (pass@8) | ~0 | 10.9 | 9.5 |
"Instead of starting from scratch, a continual pre-training setting would be a more sound approach."
We have not run a continual-pretraining experiment, and we want to explain why we think training from scratch is the right setting for our question. We are asking how the data a model sees early changes what it learns from the data it sees later. We can only answer that when we control everything the model has seen. A continually pretrained model has already seen the web, including mathematics at every level in no particular order, and this would confound any curriculum applied afterwards.
We agree that mixing with other data is an important experiment to do. Since the submission, we have added this experiment with a 50:50 mixture of web text and generated mathematics. We are still only ordering the mathematics part of the data, and leaving the web part as it is. Interestingly, we still see improvements on MATH-500 of 2.9 and 5.2, while ARC-Easy and PIQA are unchanged or slightly better (Table 16).
Response to Reviewer 9TAk (1/2)
We thank the reviewer for their detailed feedback. We recognize that the original title, "Does Data Order Matter?", led the reviewer to expect a general answer to this question. The goal of our work is more specific. We want to show that when we are already generating data for pretraining, we can use that step to create a progression, and that simply following this progression can work better than shuffling the same data. We have revised the framing to make this goal clear.
We agree that the inconsistency of sorting existing data has already been observed, including in BabyLM (Warstadt et al., 2023, arXiv:2504.08165) and Wu et al. (2020, arXiv:2012.03107). This is the motivation for our work. Rather than trying to find a useful order in an existing corpus, we are proposing to generate the data in a way that creates a progression.
We want to clarify that the original submission did include comparisons with length, compression ratio, perplexity and PDPC. We ran these on both web corpora as well as the generated data, using sequential, interleaved and reversed schedules. These comparisons are in Tables 1 and 2 of the revision.
We want to clarify how the data is generated. We do not just give the model a few concept names and ask for a question. We start from a source problem in OpenMathInstruct-2 or NuminaMath-1.5, and ask the model to rewrite it using a full rubric for the grade. This rubric specifies the concepts, the types of problems and the solutions, as shown in Appendix A.3.
We also want to emphasize that rewriting with relatively small models is already common in pretraining. BeyondWeb (arXiv:2508.10975) reports a gain of +1.8 over its real-data baseline with a 1B rewriter, and only 0.4 more from going from a 3B to an 8B rewriter. WRAP (arXiv:2401.16380) finds improvements over real data alone with Qwen-1.8B and Mistral-7B rewrites. Nemotron-CC (arXiv:2412.02595) generated about 1.9T tokens using Mistral-NeMo-12B, and SmolLM's Cosmopedia v2 did not see significant gains from larger generators. Roberts et al. (2026, arXiv:2610.31867) also fit scaling laws for how much compute to spend on rewriting.
We did not verify the grade level of every question. However, we want to emphasize that both the ordered run and Uniform use exactly the same generated data. We find that showing this data in grade order improves MATH-500 by 2.4 to 7.2, while showing it in reverse order reduces performance by 2.5 to 3.1. So the result is about the order in which the model sees the data.
We also want to distinguish this from LittleLearner (arXiv:2608.13545). There, the authors filter web text into a single K–5 band, and they do not compare different orderings of the data.
We think this is actually an important part of the result. The model is going from grade 1 to grade 7, so early on it has not seen the same range of material as the model trained uniformly. For example, at 4B tokens it has mostly seen grades 1–3. But we are testing it on GSM8K and MATH-500, including material from the grades it has not reached. We would therefore expect it to trail at this point.
What we want to emphasize is what happens after the model has gone through the curriculum. Until 6–8B tokens, it is behind. By the end, it is ahead on all six seeds for the 1.7B model, by 5.1 on MATH-500 and 3.5 on GSM8K (Figure 2, Tables 11–14). Both models have used the same data and compute. We find this interesting because the model that was initially only seeing easier material ends up doing better on the harder benchmarks than the model that had access to the harder material throughout.
Our claim is therefore about the model being better at the end of training, rather than about learning faster at every stage. If early performance is important, we also have the interleaved schedule, which cycles through the grades ten times, exposing the model to all grades earlier. That schedule is ahead of Uniform at every token budget (Tables 11–14).