acceptodds
Under review as a conference paper at ICLR 2027

Curriculum Learning: Does Data Order Matter?

Abstract

Curriculum learning, training on data in a defined order rather than at random, is one of the oldest ideas in machine learning, yet its gains in language-model pretraining have been unreliable. In this work, we ask whether data order matters at all. Holding the data mix fixed, we find that the same curriculum can help or hurt depending on the corpus. Compression-ratio interleaving improves GSM8K and MATH-500 by 5–7 percentage points (pp) on mathematical web text, but on general web text reduces ARC-Easy by 3–7 pp and GSM8K by 6–8 pp. These results suggest that curriculum effectiveness depends on whether an ordering aligns with useful structure in the corpus. We therefore introduce such structure during data generation, rewriting mathematical problems under grade-specific pedagogical rubrics. When a 1.7B model sees these 10B tokens of synthetic math, presenting grades 1 through 7 in pedagogical order raises GSM8K and MATH-500 by 4 pp over randomly sampling that data. Our results suggest that when natural data lacks a useful order, synthetic rewriting offers a general way to create one.

Primary area:
foundation or frontier models, including LLMs
Keywords:
curriculum learning, pretraining, data centric machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9TAkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the effect of pre-training with curriculum learning on the mathematical and reasoning ability of small LLMs. The authors train Qwen3.5-0.8B/1.7B architectures for 1 billion tokens on the DCIM pretraining dataset and find limited benefit with a curriculum over the document compression ratio. A similar experiment with synthetically curated grade-school-level mathematical data shows that a curriculum over grade level leads to improved mathematical ability after 10 billion tokens of pretraining, over randomly ordered data.

Critical strengths

Clarity

  • The paper is generally well-written. The motivation and research question are well-posed, the related work is appropriate, and the figures and tables are attractive and easy to parse.
  • The overall structure of the paper is solid, and the appendices contain ample detail on the data generation, methodology, and results.

Setting

  • The paper studies a few curricula at the scale of a small pre-training run (1B--10B tokens). This is a fairly new frontier for the curriculum learning literature due to the expense of training with multiple curricula.
  • The choice of the popular Qwen3.5 architecture at two scales (0.8/1.7B) is both relevant to practitioners and useful for understanding curriculum learning.
Critical weaknesses

By my assessment, the paper contains significant weaknesses do not address the broadly posed question in the paper title ('Does data order matter?').

Significance

  • Much of the paper replicates results that I consider to be well-explored in the literature on curriculum learning. For example, the observation that curricula are of limited benefit on non-naturally-ordered pretraining data has been made in the BabyLM challenge [1], across a much wider range of curriculum strategies.
  • The choice of difficulty metric (compression ratio) is not justified beyond an intuitive sense, and the usefulness of curricula with multiple other language-oriented difficulty measures has been established at exactly this data scale [2]. There is also no comparison to other difficulty measures (even simple ones like sequence length). While expensive, this is necessary for a robust claim.
  • The choice of evaluation datasets, exclusively focused on commonsense/math reasoning ability, is odd for the assessment of a pre-trained model. The goal of pre-training is typically to learn a linguistic distribution, in addition to limited reasoning primitives. These datasets are typically used to evaluate models subject to extensive SFT/RL post-training, and a stronger choice would be to study linguistic or general knowledge characteristics (e.g., MMLU [4]).
  • For such a broadly posed question, the choice of ordering and pacing plays a large role, and is not explored beyond a simple strategy.

Soundness

  • I have strong concerns about the soundness of the synthetic data generation process (prompting a relatively weak LLM — GPT-OSS-20B — with a few concepts) used for the mathematics pre-training. LittleLearner [3] (cited by the authors themselves) has an extensive pipeline for ensuring that selected data actually adheres to the grade level that is required, and shows that a significant fraction of examples are rejected due to a lack of consistency. Other work has shown that models are poorly calibrated across grade levels. Combined with the need for interleaving, this undermines the conclusions made about the natural difficulty progression benefiting performance.
  • Corroborating the above, the grade-ordered curriculum sees a very small benefit at 10B tokens (Figure 2) with the random ordering actually performing better at lower budgets — counter to nearly every observation on curriculum speedup in the literature.
  • Pre-training exclusively on mathematical data for 10B tokens leads to only a modest performance on GSM8k and Math500. This is expected, as the model needs to learn linguistic primitives alongside the mathematical information, from only synthetic data. This is why most studies on mathematical reasoning are in post-training settings. Instead of starting from scratch, a continual pre-training setting would be a more sound approach.

References

[1] Warstadt, Alex, et al. "Findings of the BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora." Proceedings of the babylm challenge at the 27th conference on computational natural language learning. 2023.

[2] Zhang, Yang, et al. "Beyond random sampling: Efficient language model pretraining via curriculum learning." Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

[3] Li, Fanfei, et al. "LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure." arXiv preprint arXiv:2608.13545 (2026).

[4] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).

Questions for authors

Answers to my questions are unlikely to push my score towards acceptance, particularly as my primary suggestions require new pretraining-scale experiments. However, I welcome the opportunity to discuss with the authors and encourage them to iterate on this work.

Clarification requests

  • For the DCIM pretraining experiments, can you justify your choice of evaluation datasets (math/commonsense reasoning), and present results on tests of linguistic ability or general knowledge?
  • Do you have an explanation for the fact that the random curriculum outperforms the grade-wise ordering up to 6 billion tokens in Figure 2? This runs counter to the long-standing result that curriculum learning can speed up early learning, despite limited long-term benefit.
  • Can you provide statistics on the quality of the synthetic data (e.g., passing it through a stronger model to verify the grade level)?

Suggestions for additional experiments and future work

  • To make a broad claim about whether data ordering matters requires a more extensive and principled exploration, particularly w.r.t. the evaluation metric. A good example of a well-posed question and a scoped answer is the When do Curricula Work? paper [1]. An exploration of a few more more orderings, pacing types, and difficulty measures at this scale would amount to a significant contribution in my view.
  • LittleLearner is a great example of the care required to curate a data that is aligned according to grade level. Can you winnow out your pre-training data and iterate on the data generation process to make it more sound, and try a similar experiment?
  • Reasoning ability is typically not a key goal of pretraining. Several works explore post-training curricula, but fewer focus on continued pretraining. Can you show that a curriculum with your synthetic data mixture results in improved performance after domain-adaptive continual pre-training?

References

[1] Wu, Xiaoxia, Ethan Dyer, and Behnam Neyshabur. "When do curricula work?." arXiv preprint arXiv:2012.03107 (2020).

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9TAk (1/2)

We thank the reviewer for their detailed feedback. We recognize that the original title, "Does Data Order Matter?", led the reviewer to expect a general answer to this question. The goal of our work is more specific. We want to show that when we are already generating data for pretraining, we can use that step to create a progression, and that simply following this progression can work better than shuffling the same data. We have revised the framing to make this goal clear.

"Much of the paper replicates results that I consider to be well-explored in the literature."

We agree that the inconsistency of sorting existing data has already been observed, including in BabyLM (Warstadt et al., 2023, arXiv:2504.08165) and Wu et al. (2020, arXiv:2012.03107). This is the motivation for our work. Rather than trying to find a useful order in an existing corpus, we are proposing to generate the data in a way that creates a progression.

"There is also no comparison to other difficulty measures (even simple ones like sequence length)."

We want to clarify that the original submission did include comparisons with length, compression ratio, perplexity and PDPC. We ran these on both web corpora as well as the generated data, using sequential, interleaved and reversed schedules. These comparisons are in Tables 1 and 2 of the revision.

"I have strong concerns about the soundness of the synthetic data generation process."

We want to clarify how the data is generated. We do not just give the model a few concept names and ask for a question. We start from a source problem in OpenMathInstruct-2 or NuminaMath-1.5, and ask the model to rewrite it using a full rubric for the grade. This rubric specifies the concepts, the types of problems and the solutions, as shown in Appendix A.3.

We also want to emphasize that rewriting with relatively small models is already common in pretraining. BeyondWeb (arXiv:2508.10975) reports a gain of +1.8 over its real-data baseline with a 1B rewriter, and only 0.4 more from going from a 3B to an 8B rewriter. WRAP (arXiv:2401.16380) finds improvements over real data alone with Qwen-1.8B and Mistral-7B rewrites. Nemotron-CC (arXiv:2412.02595) generated about 1.9T tokens using Mistral-NeMo-12B, and SmolLM's Cosmopedia v2 did not see significant gains from larger generators. Roberts et al. (2026, arXiv:2610.31867) also fit scaling laws for how much compute to spend on rewriting.

We did not verify the grade level of every question. However, we want to emphasize that both the ordered run and Uniform use exactly the same generated data. We find that showing this data in grade order improves MATH-500 by 2.4 to 7.2, while showing it in reverse order reduces performance by 2.5 to 3.1. So the result is about the order in which the model sees the data.

We also want to distinguish this from LittleLearner (arXiv:2608.13545). There, the authors filter web text into a single K–5 band, and they do not compare different orderings of the data.

"Do you have an explanation for the fact that the random curriculum outperforms the grade-wise ordering up to 6 billion tokens?"

We think this is actually an important part of the result. The model is going from grade 1 to grade 7, so early on it has not seen the same range of material as the model trained uniformly. For example, at 4B tokens it has mostly seen grades 1–3. But we are testing it on GSM8K and MATH-500, including material from the grades it has not reached. We would therefore expect it to trail at this point.

What we want to emphasize is what happens after the model has gone through the curriculum. Until 6–8B tokens, it is behind. By the end, it is ahead on all six seeds for the 1.7B model, by 5.1 on MATH-500 and 3.5 on GSM8K (Figure 2, Tables 11–14). Both models have used the same data and compute. We find this interesting because the model that was initially only seeing easier material ends up doing better on the harder benchmarks than the model that had access to the harder material throughout.

Our claim is therefore about the model being better at the end of training, rather than about learning faster at every stage. If early performance is important, we also have the interleaved schedule, which cycles through the grades ten times, exposing the model to all grades earlier. That schedule is ahead of Uniform at every token budget (Tables 11–14).

AuthorsOpenReview ↗

Response to Reviewer 9TAk (2/2)

"Pre-training exclusively on mathematical data for 10B tokens leads to only a modest performance on GSM8k and Math500. ... Reasoning ability is typically not a key goal of pretraining."

We respectfully disagree with both points.

On reasoning as a goal of pretraining: recent work finds that post-training mostly sharpens abilities the base model already has. The probability a pretrained model puts on good responses determines whether post-training can succeed (Chen et al., 2025, arXiv:2510.15020). RL converges on behaviours already present in the pretraining data (Zhao et al., 2025, arXiv:2504.07912), and a base model given the right opening tokens matches its RL-trained counterpart on math and code (Wang et al., 2026, arXiv:2610.06851). So what the model learns about mathematics in pretraining decides what post-training can reach, and this is why we study it there.

On the scores: we want to clarify that they are high for a 10B-token budget. The table compares our single-sample accuracy (avg@8, Appendix E.1) with published few-shot scores of base models of similar size.

ModelPretraining tokensGSM8KMATH
Llama-3.2-1B9T7.63.3
SmolLM2-1.7B11T31.111.6
Qwen2.5-0.5B18T41.619.5
Qwen2.5-1.5B18T68.535.0
Ours, 0.6B, grade order10B22.019.8
Ours, 1.7B, grade order10B31.525.0

Baseline numbers are from Allal et al. (2025, arXiv:2502.02737, Table 4) and the Qwen2.5 technical report (arXiv:2412.15115, Table 5), with 4 or 5 shots. Ours use 1 shot on GSM8K and MATH-500. Our 1.7B model matches SmolLM2-1.7B on GSM8K and is 13 points above it on MATH, with about a thousand times fewer tokens (we are only training on math).

We also want to emphasize that in our experiments models learn a reasonable amount from very little data. After only 1B tokens of general web text they are well above chance on every benchmark we report:

BenchmarkRandom0.6B1.7B
PIQA (accuracy)50.061.563.4
ARC-Easy (accuracy)25.046.249.3
GSM8K (pass@8)~012.712.5
MATH-500 (pass@8)~010.99.5

"Instead of starting from scratch, a continual pre-training setting would be a more sound approach."

We have not run a continual-pretraining experiment, and we want to explain why we think training from scratch is the right setting for our question. We are asking how the data a model sees early changes what it learns from the data it sees later. We can only answer that when we control everything the model has seen. A continually pretrained model has already seen the web, including mathematics at every level in no particular order, and this would confound any curriculum applied afterwards.

We agree that mixing with other data is an important experiment to do. Since the submission, we have added this experiment with a 50:50 mixture of web text and generated mathematics. We are still only ordering the mathematics part of the data, and leaving the web part as it is. Interestingly, we still see improvements on MATH-500 of 2.9 and 5.2, while ARC-Easy and PIQA are unchanged or slightly better (Table 16).

Reviewer_nqJmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how training order affects language-model pretraining while holding the selected data fixed. It evaluates document-based ordering signals and sequential or interleaved schedules on general web text and mathematical web text, then constructs synthetic mathematical questions and solutions using grade-specific generation instructions. Experiments train Qwen3-0.6B and Qwen3-1.7B architectures from random initialization. The strongest control is a 10B-token comparison in which every condition uses exactly the same training sequences, each once, with different presentation orders. Grade-based schedules improve mean mathematical reasoning scores over random presentation, while generic ordering methods can help on one corpus and hurt on another. The paper suggests that synthetic generation can provide useful structure for organizing training, in addition to supplying training examples.

Critical strengths
  1. The research question is interesting and practically relevant. Data order is a potentially useful training choice even when the dataset and total training budget are fixed. Connecting data generation with the construction of a training curriculum is a promising perspective. The paper also reports outcomes that resist a simple universal prescription: the same ordering method can have opposite effects across corpora, and the preferred schedule depends on the evaluation task.

  2. The fixed-data experiment provides a meaningful control. In the 10B-token comparison, all schedules consume the same 2,441,408 sequences of length 4,096 exactly once. This separates presentation order from changes in total data exposure or overall mixture. In particular, the random baseline also uses the same teacher-generated mathematical data, so teacher knowledge transfer or improved synthetic-data quality alone cannot explain the difference between the curriculum and random conditions. This is an important strength of the experimental design.

  3. The reported gains are encouraging across more than one setting. In the 10B-token experiment, sequential grade order improves GSM8K pass@8 from 59.06 to 61.94 and MATH-500 pass@8 from 46.40 to 53.13; grade interleaving reaches 63.18 and 50.33, respectively. Sequential grade order also improves mean pass@8 on both tasks at both model sizes in the 1B-token experiments. The comparisons across natural and synthetic corpora provide useful evidence that the effect of a curriculum depends on the data it organizes.

  4. The reporting includes several good practices and acknowledges important limits. The primary comparisons use final checkpoints and report three training seeds, with full absolute scores in the appendix. The authors explicitly distinguish the reported method standard deviation from uncertainty in the difference from the baseline. The additional uniform-ending experiment addresses a relevant alternative explanation, and the paper acknowledges that it does not fully isolate recency from other schedule changes.

Critical weaknesses

I find the study interesting, particularly its fixed-data comparison. My main concerns involve the explanation and applicability of the observed effects. The evidence that order can matter is stronger than the evidence that a particular pedagogical progression is the reason it matters. A focused scope is acceptable, but the distinction between these claims is important for assessing the contribution.

  1. The contribution of pedagogical progression remains difficult to separate from changing topic exposure over time. Grade labels jointly change mathematical concepts, problem complexity, and potentially solution length and style. A progression through grades therefore also changes the proportions of mathematical subdomains during training. Keeping the total data fixed rules out differences in overall exposure, but does not distinguish prerequisite learning from benefits of presenting more evaluation-relevant topics later. Section 5.4 is informative: after a uniform ending, the GSM8K advantage remains, while the MATH-500 advantage falls from 6.35 to 1.40 points in that separate experiment. However, the grade phases are also shortened and the uniform phase overlaps learning-rate decay. The authors appropriately acknowledge this limitation. A comparison with a common final training segment, or controls that maintain topic proportions while changing order within topics, would better identify what the grade structure contributes.

  2. Sensitivity to batch size is not established. Appendix B uses a global batch of 32 sequences of 4,096 tokens, approximately 131K tokens per optimizer update, in both training-budget settings. Larger batches need not eliminate a curriculum because the data distribution can still change across stages. Nevertheless, batch size changes gradient noise and, at a fixed token budget, the number of updates within each stage. The paper does not test whether the gains remain under different batch sizes or whether schedule duration interacts with this choice. This limits the guidance the results provide for pretraining configurations with substantially larger batches.

  3. The interaction between curriculum and overall domain proportions remains open. General web, mathematical web, and synthetic mathematics are evaluated as separate corpus settings. The paper does not systematically vary the proportions of mathematics, code, and general text within a shared pretraining mixture. A curriculum effective in a mathematics-focused corpus could behave differently when mathematics occupies a small share of training and interacts with other domains. The reversal across the two web corpora makes this question especially relevant, although the current results do not identify domain composition as its cause. Fixing the overall mixture within a comparison is the right control for an order effect, but does not establish that the effect is robust to other mixtures.

  4. The uncertainty in the gains and in the ranking of schedules could be characterized more directly. Three-seed reporting is useful, but some methods have substantial variation. For example, at 10B tokens, grade interleaving gives GSM8K pass@8 of 63.18 ± 4.08, compared with 59.06 ± 1.09 for Uniform and 61.94 ± 1.65 for sequential grade order. These means alone do not establish that interleaving is preferable. The tables explicitly report method standard deviations rather than uncertainty in the differences; per-seed gains and paired uncertainty estimates would help assess the robustness of each comparison. Practical interpretation should also distinguish pass@8 gains from single-sample performance: for sequential grade order, GSM8K Avg@8 increases from 28.60 to 30.10, a smaller absolute gain than the pass@8 change.

Questions for authors
  1. What does the grade progression contribute? Can the authors compare schedules with the same final training segment and the same learning-rate schedule, or maintain topic proportions at each stage while changing order within topics? A random permutation of grade blocks would also help distinguish the intended progression from simply grouping similar examples together.

  2. Batch-size robustness: Do the curriculum-versus-random gains persist at larger global batch sizes? A small sweep with the same corpus and total token budget, curriculum phases aligned by tokens, and comparable learning-rate tuning would clarify whether the finding depends strongly on the current batch and number of updates per phase.

  3. Domain-mixture interaction: What happens when the grade-conditioned mathematics is mixed with general text or code at different fixed total proportions? Within each mixture, can the same-data ordering comparison be repeated while also tracking non-mathematical capabilities?

  4. Statistical reliability: Can the authors provide per-seed curriculum-minus-Uniform differences and uncertainty estimates for the principal 10B-token comparisons? It would be useful to distinguish robust improvements over random order from less certain differences between curriculum schedules.

  5. Construction and selection of grades: Is there a check that generated examples follow the intended grade requirements, and how do topic and length distributions vary across grades? Section 4.1 selects Grades 1–7 using GSM8K performance; was this choice made on a development partition separate from the final reported GSM8K evaluation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer nqJm

We thank the reviewer for their feedback on this draft. First, we want to address a common confusion among the reviewers. The goal of this paper is not to benchmark existing curriculum learning algorithms, but rather to expose their inconsistency and offer an entirely new axis of curriculum learning algorithms, which we call Generative Curriculum Learning.

We recognize that the original title, "Does Data Order Matter?", led the reviewers to believe otherwise. Since the original paper submission, we have put significant effort into revising our framing to portray the actual goal of the work.

All gains below are in percentage points of pass@8 unless stated otherwise.

Q1. What does the grade progression contribute?

We have added an experiment towards this question now, for both the model sizes at 10B tokens. The idea is to train with the curriculum and then, for the last 20% of the run, just show the model all the data uniformly.

As we show in Table 5, on GSM8K, Generative Curriculum Learning still works quite well and brings consistent improvements over uniform data (+4.8 for 0.6B, +3.7 for 1.7B).

On MATH-500, this experiment still improves over uniform by 3.7 for 1.7B, and by 1.7 for 0.6B.

As another evidence around the importance of the progression, when we reverse the order of the grades, we see that the model performs worse than Uniform on MATH-500 (Table 2).

Q2. Batch size.

All the experiments here are with a global batch size of 32 sequences. We have not done the batch-size ablation, so we do not have a result to report for larger batches. Under a fixed experimental budget, we decided to prioritize ablations where we expected results to change more. In this case, we had less reason to believe that a change in batch size will alter the performance of curriculum learning.

Q3. Domain mixture.

We agree that mixing with other data is an important experiment to do. Since the submission, we have added this experiment with a 50:50 mixture of web text and generated mathematics. We still only order the mathematics part of the data, and leave the web part as it is. Interestingly, we still see improvements on MATH-500 of 2.9 and 5.2, and there is no drop in ARC-Easy or PIQA (Table 16).

Q4. Statistical reliability.

We agree that looking at just the mean is not sufficient here. We have therefore added the paired differences from Uniform, together with their SD, to every table. The appendix has the scores for the individual seeds too. In the main experiment, we now have six seeds for the 1.7B model trained on 10B tokens, and grade order is better on all six paired comparisons (Figure 2).

For GSM8K, we get an improvement of 3.5 when measured as pass@8, and 2.3 when measured as avg@8. For MATH-500, the same numbers are 5.1 and 3.2. Regarding interleaving, yes, we should not have ranked it above sequential order. This was based on three seeds, with a +4.1 gain on GSM8K, and we agree with the reviewer that this is not enough to make that comparison. We have removed the ranking now.

Q5. Grades.

We used an LLM as a judge to verify a sample of 1% of the examples row by row, but did not do a full analysis. There was significant iteration in an autoresearch style to get to the final prompts, but much of this knowledge is hard to codify.

We do see that length generally increases with grade, but it is not a monotonic relation (Spearman 0.60, Appendix A.6).

We used the grade information for selecting grades 1–7, which were more aligned with the contents of GSM8K. No information about MATH-500 was used for this selection. Also, Uniform gets the exact same grades, so we are not giving the ordered model a different set of data to learn from.

Reviewer_ykmzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies curriculum learning in the pretraining stage. The authors find that a curriculum can either help or hurt performance depending on the corpus. They further show that when mathematical problems are rewritten into grade-specific datasets and used for curriculum training, performance improves on GSM8K and MATH-500.

Critical strengths
  • The paper is clearly written and easy to follow.
  • Generating synthetic data to create a curriculum is an interesting direction that merits further study.
Critical weaknesses

Novelty

  • Most of the methods are borrowed from Zhang et al.
  • The main finding, that the effectiveness of a curriculum depends on the corpus, is not surprising.
  • The authors do not analyze why DCLM and MegaMath show conflicting results.

Weak baselines

  • The approach is verified on only one downstream domain (math).
  • There is no baseline that presents the G1–G7 buckets in a random order.
  • There is no comparison with curriculum methods based on synthetic dataset generation, such as [1] or [2].

Motivation for pretraining curricula

  • [3] shows that the gains from curricula shrink at larger scales, and [4] shows that a high degree of overparameterization can limit the benefit of curricula. Training a 1.7B model on 10B tokens is relatively small-scale to claim that the curriculum is effective. Moreover, comparing Tables 3 and 4, the gain decreases as the token budget increases.
  • Furthermore, grouping the dataset requires additional cost, and the gain from the curriculum does not seem large enough to justify it.

[1] Learning to solve complex problems via dataset decomposition
[2] MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learning
[3] Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
[4] Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

Questions for authors

na

1 comment
AuthorsOpenReview ↗

Response to Reviewer ykmz

Thank you for your feedback. We are glad that you find generating synthetic data to create a curriculum "is an interesting direction that merits further study". First, we want to address a common confusion among the reviewers. The goal of this paper is not to benchmark existing curriculum learning algorithms, but rather to expose their inconsistency and offer an entirely new axis of curriculum learning algorithms, which we call Generative Curriculum Learning.

We recognize that the original title, "Does Data Order Matter?", led the reviewers to believe otherwise. Since the original paper submission, we have put significant effort into revising our framing to portray the actual goal of the work.

"Most of the methods are borrowed from Zhang et al."

Yes, these methods are our baselines. We use them to show that the benefits of existing scoring methods is quite inconsistent, and changes from one corpus to the other. This becomes important motivation for entirely rethinking a new curriculum learning approach. It might also be worth noting that under this new form of curriculum learning: Generative Curriculum Learning, the Spearman correlation of existing methods with the human pedagogical grades is at most 0.60, again suggesting that existing methods do not prioritize human pedagogical learning signal (Appendix A.6).

"The main finding, that the effectiveness of a curriculum depends on the corpus, is not surprising."

We agree with the reviewer here. This is the motivation for our work rather than the main finding. The main contribution of the work is to first establish the inconsistency in an easily reproducible experimental setup, and then show that modifying the approach towards how we think about curriculum (creating order, rather than finding it) changes the outcome of curriculum learning in the same settings where these methods fail.

"There is no baseline that presents the G1–G7 buckets in a random order."

We have not tried randomly permuting the grades. We did try reversing them, so that the model sees the same groups for the same number of phases but starts from the higher grades. This gives worse results than Uniform on MATH-500 (−3.1 and −2.5). In comparison, starting from the lower grades gives gains of 2.4 and 7.2 (Table 2). For having high confidence in the results under an experimental compute budget, we prioritized having more seeds (up to 6) per setup we chose, rather than having several low confidence ablations.

"There is no comparison with curriculum methods based on synthetic dataset generation, such as [1] or [2]."

Both of these works fine-tune pretrained models on a few thousand to 34.5K derived problems. In our work, we pretrain from scratch on billions of tokens, and we specify the stages before generating the data. While these might seem related, they are not directly comparable baselines for our setting. Our work aims to understand what happens when models are taught from the beginning by developing the right priors and features in the same order as how humans learn.

"Comparing Tables 3 and 4, the gain decreases as the token budget increases."

For the 1.7B model, you are right that the gain on MATH-500 goes from 7.2 at 1B tokens to 5.1 at 10B tokens. However, Uniform itself goes from 18% to 48%, and we still see a gain in each of the six seeds. The performance delta reducing is a natural consequence of there being lesser headroom of performance improvement, and is a general phenomenon in machine learning. We also find the opposite trend for the 0.6B model, where the gain goes from 2.4 to 6.0. So the decrease is not something we see across both sizes.

We want to distinguish our setting from [3], which orders an existing web corpus using lexical scores, and [4], which studies a two-layer network. In our case, we create the progression while generating the data. Neither of those works studies this setting. Of course, we cannot say what happens beyond the 1.7B parameters and 10B tokens we have tested, and this is stated in the limitations.

"Grouping the dataset requires additional cost."

The additional cost of grouping would be a concern if we were taking an existing dataset and trying to assign a grade to every document. Here, we are asking the model to generate a document for a particular grade in the first place. We therefore already know the assigned grade, without having to score the document separately.

"The approach is verified on only one downstream domain (math)."

Yes, mathematics is the only domain where we have generated a curriculum so far. This was a limitation of our work. During the rebuttal period, we have added a new experiment where the model learns from web text as well. In this new experiment, half the data is web text and the other half is the pedagogically generated mathematics. Even when only the math subset is ordered, this still gives improvements of 2.9 and 5.2 on MATH-500, as shown in Table 16.

Comments on the paper

1 comment

We thank all the reviewers for their feedback on this draft. First, we want to address a common confusion among the reviewers. The goal of this paper is not to benchmark existing curriculum learning algorithms, but rather to expose their inconsistency and offer an entirely new axis of curriculum learning algorithms, which we call Generative Curriculum Learning.

We recognize that the original title, "Does Data Order Matter?", led the reviewers to believe otherwise. Since the original paper submission, we have put significant effort into revising our framing to portray the actual goal of the work. We show that many of the curriculum learning methods so far work inconsistently across different datasets, and the reason for this is that often we are trying to find order in datasets that do not have one that is useful for learning.

We have now titled the revision Generative Curriculum Learning. We want to emphasize that the method, the corpus we generated and the main 10B-token experiment were already in the original submission. We have revised the framing and added some ablations, but the goal of the work has remained the same.

Generative Curriculum Learning is a general-purpose recipe that can use synthetic data to create a curriculum for pretraining in a pedagogically inspired way. We want to emphasize that rewriting data has become extremely common in language model training, such as in WRAP [1], Nemotron-CC [2], SmolLM [3], Kimi-K2 [4], Reflection-Beam [5], Laguna [6], BeyondWeb [7] etc. Frontier model training recipes use synthetic data extensively, and pretraining has moved to a new paradigm where data is actively generated, rather than merely being sourced. This opens a new avenue of unexplored curriculum learning approaches.

Rather than generating synthetic data at random, we could actually use it to build a progression that is pedagogically inspired. In this work, we prompt the gpt-oss-20b model with very diverse rubrics of what students in grades 1 to 12 learn in the math curriculum in U.S. schools to create new question-answer pairs given some fixed seed text. We find that by relying on human pedagogy, models learn significantly better when they are shown the data in a progression, as compared to throwing in all the data together. The method consistently works across the model sizes, training scales, and unseen benchmarks we test.

  • Final performance. At 1.7B and 10B tokens, grade order beats uniform sampling of the same data on all six seeds: +5.1 on MATH-500 and +3.5 on GSM8K (Table 5, Figure 2).
  • Direction. At 1B tokens the reverse grade order drops MATH-500 below Uniform (−3.1 and −2.5), while grade order gains 2.4 and 7.2 (Table 2).
  • Recency. One might be concerned that the curriculum gains arise only because the model sees the hardest data near the end of training. To test this, we use a schedule that ends with 2B tokens of uniform sampling. At 1.7B, this retains the GSM8K gain and about half of the MATH-500 gain, suggesting that the benefits extend beyond recency alone (Table 5).
  • Non-Math Mixtures. We also test a scenario where half the corpus is replaced by web text and only the math half is ordered. MATH-500 still gains 2.9 and 5.2 (Table 16).

[1] Maini et al., 2024. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling (WRAP). arXiv:2401.16380

[2] Su et al., 2024. Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. arXiv:2412.02595

[3] Allal et al., 2025. SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model. arXiv:2502.02737

[4] Kimi Team, 2025. Kimi K2: Open Agentic Intelligence. arXiv:2507.20534

[5] Reflection Team, 2026. Beam Model Technical Report. reflection.ai/documents/beam-model-tech-report

[6] Abadji et al., 2026. Laguna M.1/XS.2 Technical Report. arXiv:2605.27605

[7] DatologyAI, 2025. BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining. arXiv:2508.10975

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.