- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the relative contributions of training-recipe and pretraining-data improvements to small-scale LLM progress from 2019 to 2025, visible in small dense models trained on web text corpora. They select recipes and corpora associated with each year, and conduct a pretraining scaling study over combinations of these. Progress is measured as the reduction in training FLOPs required to match a reference model’s accuracy on OLMES10 (10 downstream QA benchmarks). They find that data improvements contribute more to the reduction than the training-recipe, attributing 64% of the efficiency gain to data.
Critical strengths
Clear experimental comparison. The paper systematically compares the selected methods across different combinations, controlling for confounds from hyperparameters as much as they can and clearly stating their limitations. The evidence is thus robust and useful for their selected methods.
Critical weaknesses
1. Narrow training recipe scope. The selected training recipes cover only a small range of changes to dense Transformers, especially in optimization. That's why I find it not surprising that their performance differences are small (eg, on FineWeb-Edu, Figure 3 shows about the same accuracy for most models after GPT-2, suggesting little progress within 6 years of recipe improvements). Also, several recipe changes target large-scale training stability, long-context support, or inference efficiency, whose benefits are poorly captured by the small-scale training-FLOP comparison. For example, GQA reduces KV-cache overhead, so maintaining accuracy while improving inference efficiency is already a valuable outcome and an important part of "software progress" that is not measured here. The selected datasets on the other side change substantially in preprocessing, domain composition, and sources. For me, the results thus insufficiently justify interpreting this comparison as a decomposition of historical software progress.
2. Limited additional insight. The authors study only a very general and narrow view on LLM training and data. The paper measures general progress through training FLOPs and downstream QA accuracy. Although the experiments quantify the relative effects of the selected recipes and corpora, they provide little explanation of which components drive the gains or when the findings would change. The cited works studying only data or recipe improvements such as DataDecide and the recipe ablations by Gundlach et al. did not explicitly connect the two domains but provided a deep look into the different components that are driving progress and giving explicit recommendations for practitioners and researchers. Additionally, this paper excludes important dimensions that could substantially increase the study’s informativeness, including training throughput instead of FLOPs (eg, flash attention, CUDA kernels, etc.), inference compute, and larger-scale training. Overall, I find that the high-level view in this work provides little additional insight or practical guidance.
Questions for authors
- What was the selection criteria of training recipes and data, and how do these choices make them representatives of historical progress in a very large field like LLMs?
- What actionable conclusion or new understanding does this study establish beyond prior findings on data curation and recipe improvements?
- Appendix F shows different gains and regressions across tasks. To what extent do these differences reflect the domain mix of training data, such as educational or mathematical content? How dependent is the main conclusion on the alignment of data mix to the downstream task domains?
- Were weight decay and batch size also swept? If not, did you test if they do not systematically favor recipes or model sizes?
3 comments
Response to Reviewer h3ds (2 of 2)
2. What was the selection criteria of training recipes and data, and how do these choices make them representatives of historical progress in a very large field like LLMs?
This requirement explains our choice of scale - with the compute available to an academic group, small dense Transformers allowed us to train all four historical endpoint pairings at five budgets with three seeds each, and to cross seven recipes with seven corpora at a common budget.
For each year from 2019 through 2025 we chose a documented dense-Transformer recipe that we could implement and a released English pretraining corpus. This gives us a reproducible series of changes that we could implement and pretrain from scratch. These were all reasonably popular and well documented model and corpus choices in their years of publication.
3. Task alignment
We compared the composition of all ten corpora by applying the same FineWeb-Edu educational-content classifier to their held-out corpora. In the five historical corpora from 2019–2023, the share of documents scoring at least 3 ranged from 4% in RefinedWeb to 10% in OpenWebText and The Pile. That share reached 100% in FineWeb-Edu, which uses this cutoff to select documents, and 44% in Ultra-FineWeb (Appendix G.1, Table 21).
Given that difference in educational content, we tested whether the corpus advantage depended on including knowledge and science questions in OLMES10. At the 10¹⁹-FLOP reference, the corpus multiplier was 2.54 times the recipe multiplier on the five knowledge and science tasks. Restricting the evaluation to the four commonsense tasks reduced the ratio to 1.44, compared with 2.41 on the full suite. In all cases though, the conclusion still stands that the corpus upgrade accounts for a larger compute multiplier gain than the model upgrade. (Appendix H.1, Figure 16 and Appendix F, Table 19).
4. Weight decay and batch size
Yes, we swept both. We varied one at a time for GPT-2 and OLMo 2 on both endpoint corpora at 3.16 × 10¹⁸ FLOPs. We find little evidence that the defaults systematically favored either recipe at this budget. (Appendix K, Table 28).
Thank you for the response and revision. However, my main concerns are not resolved and I keep my score. The insights and generalisability are very limited and in my opinion fall well short of what I would expect for a major conference like ICLR.
The paper claims to decompose improvements from data and training recipes with a single metric, but uses only a small set of similar training recipes and a simplified metric based on training FLOPs, while many important architectural or training recipe improvements target wall-clock time and inference cost (like GQA and FlashAttention). As the authors state in Appendix A, the choices "were mostly driven by prioritizing the ability for us to reproduce and implement these recipe.", which in my opinion led to a very narrow and not generalisable study. There are also more informative metrics than training FLOPs that would clearly strengthen the paper, e.g., inference-aware scaling laws (Sardana et al., 2024) or wall-clock time on fixed hardware, which would capture improvements like GQA and FlashAttention.
Additionally, the main takeaway is that data is very important for LLMs, especially small models, which is established knowledge. The quantitative breakdown does not add much beyond this simple takeaway. The paper gives no guidance on how and which data to scale, which would be much more valuable given this established knowledge.
Response to Reviewer h3ds (1 of 2)
Reply to Reviewer h3ds
We thank the reviewer for their productive and helpful comments. In particular, we appreciate your recognition of the strength of our controlled comparison and experiments.
We respectfully disagree with the conclusion that our training results add little beyond prior work. The broader question is an ambitious one: how much of historical LLM progress came from better training data, and how much from better model recipes? This question is easily stated, but extremely difficult to answer given the wide ranging advancements in LLM algorithms (on both the model architecture side, and also on the systems side).
To begin answering this question, we need to study model recipes and pretraining corpora jointly. Successive releases typically change both at once, so comparing them reveals only their combined gain. To estimate each contribution, we trained the earlier recipe on the later corpus and the later recipe on the earlier corpus. This lets us measure how much training compute each upgrade saves, alone and in combination, at the same downstream accuracy.
We agree that this measure captures only part of LLM progress, as we discuss in Section 7. Recipe changes can improve large-scale training stability, enable longer contexts, or reduce inference costs even when they yield little improvement on our pretraining metric.
I wanted to clarify that our intention was to provide a quantitative breakdown of LLM progress between model and data progress. In light of that goal, we sought to use compute multipliers as the common metric that would allow for a direct quantitative comparison. It was otherwise not clear to us what would be a reasonable common metric that could account additionally for the large scale-dependent model improvements, and also the systems optimizations or inference efficiency improvements. Within the scope we could test directly, our study provides a quantitative estimate of the relative contributions of model and data progress.
1. Key insights of our study, and actionable conclusions for practitioners
To our knowledge, this is the first controlled estimate of the relative shares of historical corpus and recipe improvements in small-scale pretraining efficiency at matched downstream accuracy. DataDecide compares corpora under a common recipe, and Gundlach et al. isolate recipe changes under a common corpus, and neither gives the historical breakdown and decomposition into the two factors.
The result that the scales we have investigated, and even accounting for the limited scope, that data improvements have actually driven more of the compute efficiency gains than model improvements, is already a very surprising finding, and hints that data engineering and synthetic data might be a critical driver of future LLM progress (more than we might otherwise expect).
More concretely, for practitioners, our work suggests that they should be spending significant attention on investing in data quality. The results suggest that for highly compute-constrained practitioners who are seeking to maximize downstream performance given a fixed budget of training FLOPs, investing in data quality is the most impactful direction that can greatly magnify their effective compute. The results also support the hypothesis that distillation would be a very effective way for such compute-constrained practitioners to rapidly improve on their downstream performance, as distillation from stronger models would be a very effective way of generating such high quality data.
We also obtain novel results on the interaction effects of the compute multipliers from data improvements and model improvements. It is also notable that the gains from data improvements and model improvements are largely independent and additive, which carries important implications for practitioners. The grid of recipes crossed with corpora shows that a separate effect for each recipe and each corpus captures much of the variation in aggregate accuracy. This makes independently measured effects useful as a first approximation when predicting a new combination.
Moreover, we also compare the compute multipliers from model and data improvements across a variety of compute scales (see Figure 2).