acceptodds
Under review as a conference paper at ICLR 2027

Most small-scale software progress in pretraining is data progress

Abstract

How much of the rapid progress in language models observed over the last few years has come from model versus data improvements? Current observational estimates from model releases have only provided measurements of joint improvements from both components. We conduct a novel investigation into the improvements in language model compute efficiency for pretraining specifically from 2019 to 2025, by experimentally decomposing these improvements into algorithmic progress in the training procedure and data progress in the pretraining corpora. We do so by retraining year-representative model training recipes (GPT-2 to OLMo 2) and corpora (OpenWebText to Ultra-FineWeb) at FLOPs. We evaluate these models on a common downstream OLMES10 benchmark suite and compare the observed compute multipliers. From GPT-2 trained on OpenWebText at FLOPs, upgrading the corpus to Ultra-FineWeb yields a multiplier of ×12.0, upgrading the model to OLMo 2 yields a multiplier of ×5.0, and upgrading both ×21.1. A Shapley decomposition attributes 64% of the joint gain to data progress. We conclude that from 2019 to 2025, at the scales we study, most software progress in pretraining is accounted for by data progress.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling laws, compute multipliers, algorithmic progress, pretraining data quality, pretraining, data curation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_h3dsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the relative contributions of training-recipe and pretraining-data improvements to small-scale LLM progress from 2019 to 2025, visible in small dense models trained on web text corpora. They select recipes and corpora associated with each year, and conduct a pretraining scaling study over combinations of these. Progress is measured as the reduction in training FLOPs required to match a reference model’s accuracy on OLMES10 (10 downstream QA benchmarks). They find that data improvements contribute more to the reduction than the training-recipe, attributing 64% of the efficiency gain to data.

Critical strengths

Clear experimental comparison. The paper systematically compares the selected methods across different combinations, controlling for confounds from hyperparameters as much as they can and clearly stating their limitations. The evidence is thus robust and useful for their selected methods.

Critical weaknesses

1. Narrow training recipe scope. The selected training recipes cover only a small range of changes to dense Transformers, especially in optimization. That's why I find it not surprising that their performance differences are small (eg, on FineWeb-Edu, Figure 3 shows about the same accuracy for most models after GPT-2, suggesting little progress within 6 years of recipe improvements). Also, several recipe changes target large-scale training stability, long-context support, or inference efficiency, whose benefits are poorly captured by the small-scale training-FLOP comparison. For example, GQA reduces KV-cache overhead, so maintaining accuracy while improving inference efficiency is already a valuable outcome and an important part of "software progress" that is not measured here. The selected datasets on the other side change substantially in preprocessing, domain composition, and sources. For me, the results thus insufficiently justify interpreting this comparison as a decomposition of historical software progress.

2. Limited additional insight. The authors study only a very general and narrow view on LLM training and data. The paper measures general progress through training FLOPs and downstream QA accuracy. Although the experiments quantify the relative effects of the selected recipes and corpora, they provide little explanation of which components drive the gains or when the findings would change. The cited works studying only data or recipe improvements such as DataDecide and the recipe ablations by Gundlach et al. did not explicitly connect the two domains but provided a deep look into the different components that are driving progress and giving explicit recommendations for practitioners and researchers. Additionally, this paper excludes important dimensions that could substantially increase the study’s informativeness, including training throughput instead of FLOPs (eg, flash attention, CUDA kernels, etc.), inference compute, and larger-scale training. Overall, I find that the high-level view in this work provides little additional insight or practical guidance.

Questions for authors
  • What was the selection criteria of training recipes and data, and how do these choices make them representatives of historical progress in a very large field like LLMs?
  • What actionable conclusion or new understanding does this study establish beyond prior findings on data curation and recipe improvements?
  • Appendix F shows different gains and regressions across tasks. To what extent do these differences reflect the domain mix of training data, such as educational or mathematical content? How dependent is the main conclusion on the alignment of data mix to the downstream task domains?
  • Were weight decay and batch size also swept? If not, did you test if they do not systematically favor recipes or model sizes?
3 comments
AuthorsOpenReview ↗

Response to Reviewer h3ds (1 of 2)

Reply to Reviewer h3ds

We thank the reviewer for their productive and helpful comments. In particular, we appreciate your recognition of the strength of our controlled comparison and experiments. 

We respectfully disagree with the conclusion that our training results add little beyond prior work. The broader question is an ambitious one: how much of historical LLM progress came from better training data, and how much from better model recipes? This question is easily stated, but extremely difficult to answer given the wide ranging advancements in LLM algorithms (on both the model architecture side, and also on the systems side). 

To begin answering this question, we need to study model recipes and pretraining corpora jointly. Successive releases typically change both at once, so comparing them reveals only their combined gain. To estimate each contribution, we trained the earlier recipe on the later corpus and the later recipe on the earlier corpus. This lets us measure how much training compute each upgrade saves, alone and in combination, at the same downstream accuracy.

We agree that this measure captures only part of LLM progress, as we discuss in Section 7. Recipe changes can improve large-scale training stability, enable longer contexts, or reduce inference costs even when they yield little improvement on our pretraining metric. 

I wanted to clarify that our intention was to provide a quantitative breakdown of LLM progress between model and data progress. In light of that goal, we sought to use compute multipliers as the common metric that would allow for a direct quantitative comparison. It was otherwise not clear to us what would be a reasonable common metric that could account additionally for the large scale-dependent model improvements, and also the systems optimizations or inference efficiency improvements. Within the scope we could test directly, our study provides a quantitative estimate of the relative contributions of model and data progress.

1. Key insights of our study, and actionable conclusions for practitioners

To our knowledge, this is the first controlled estimate of the relative shares of historical corpus and recipe improvements in small-scale pretraining efficiency at matched downstream accuracy. DataDecide compares corpora under a common recipe, and Gundlach et al. isolate recipe changes under a common corpus, and neither gives the historical breakdown and decomposition into the two factors. 

The result that the scales we have investigated, and even accounting for the limited scope, that data improvements have actually driven more of the compute efficiency gains than model improvements, is already a very surprising finding, and hints that data engineering and synthetic data might be a critical driver of future LLM progress (more than we might otherwise expect). 

More concretely, for practitioners, our work suggests that they should be spending significant attention on investing in data quality. The results suggest that for highly compute-constrained practitioners who are seeking to maximize downstream performance given a fixed budget of training FLOPs, investing in data quality is the most impactful direction that can greatly magnify their effective compute. The results also support the hypothesis that distillation would be a very effective way for such compute-constrained practitioners to rapidly improve on their downstream performance, as distillation from stronger models would be a very effective way of generating such high quality data. 

We also obtain novel results on the interaction effects of the compute multipliers from data improvements and model improvements. It is also notable that the gains from data improvements and model improvements are largely independent and additive, which carries important implications for practitioners. The grid of recipes crossed with corpora shows that a separate effect for each recipe and each corpus captures much of the variation in aggregate accuracy. This makes independently measured effects useful as a first approximation when predicting a new combination.

Moreover, we also compare the compute multipliers from model and data improvements across a variety of compute scales (see Figure 2).

AuthorsOpenReview ↗

Response to Reviewer h3ds (2 of 2)

2. What was the selection criteria of training recipes and data, and how do these choices make them representatives of historical progress in a very large field like LLMs?

This requirement explains our choice of scale - with the compute available to an academic group, small dense Transformers allowed us to train all four historical endpoint pairings at five budgets with three seeds each, and to cross seven recipes with seven corpora at a common budget. 

For each year from 2019 through 2025 we chose a documented dense-Transformer recipe that we could implement and a released English pretraining corpus. This gives us a reproducible series of changes that we could implement and pretrain from scratch. These were all reasonably popular and well documented model and corpus choices in their years of publication. 

3. Task alignment

We compared the composition of all ten corpora by applying the same FineWeb-Edu educational-content classifier to their held-out corpora. In the five historical corpora from 2019–2023, the share of documents scoring at least 3 ranged from 4% in RefinedWeb to 10% in OpenWebText and The Pile. That share reached 100% in FineWeb-Edu, which uses this cutoff to select documents, and 44% in Ultra-FineWeb (Appendix G.1, Table 21).

Given that difference in educational content, we tested whether the corpus advantage depended on including knowledge and science questions in OLMES10. At the 10¹⁹-FLOP reference, the corpus multiplier was 2.54 times the recipe multiplier on the five knowledge and science tasks. Restricting the evaluation to the four commonsense tasks reduced the ratio to 1.44, compared with 2.41 on the full suite. In all cases though, the conclusion still stands that the corpus upgrade accounts for a larger compute multiplier gain than the model upgrade. (Appendix H.1, Figure 16 and Appendix F, Table 19).

4. Weight decay and batch size

Yes, we swept both. We varied one at a time for GPT-2 and OLMo 2 on both endpoint corpora at 3.16 × 10¹⁸ FLOPs. We find little evidence that the defaults systematically favored either recipe at this budget. (Appendix K, Table 28).

Thank you for the response and revision. However, my main concerns are not resolved and I keep my score. The insights and generalisability are very limited and in my opinion fall well short of what I would expect for a major conference like ICLR.

The paper claims to decompose improvements from data and training recipes with a single metric, but uses only a small set of similar training recipes and a simplified metric based on training FLOPs, while many important architectural or training recipe improvements target wall-clock time and inference cost (like GQA and FlashAttention). As the authors state in Appendix A, the choices "were mostly driven by prioritizing the ability for us to reproduce and implement these recipe.", which in my opinion led to a very narrow and not generalisable study. There are also more informative metrics than training FLOPs that would clearly strengthen the paper, e.g., inference-aware scaling laws (Sardana et al., 2024) or wall-clock time on fixed hardware, which would capture improvements like GQA and FlashAttention.

Additionally, the main takeaway is that data is very important for LLMs, especially small models, which is established knowledge. The quantitative breakdown does not add much beyond this simple takeaway. The paper gives no guidance on how and which data to scale, which would be much more valuable given this established knowledge.

Reviewer_9RyvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relative contributions of training-recipe and data improvements to small-scale language-model pretraining efficiency from 2019 to 2025. Through controlled experiments crossing historical recipes and corpora, it measures compute savings on downstream benchmarks and attributes approximately 64% of the joint log-compute gain to data improvements in its primary comparison. Additional recipes, corpora, evaluation tasks, and compute-accounting checks support the finding that data improvements contribute more than recipe improvements within the studied setting.

Critical strengths
  1. Important research question and controlled design. The paper addresses a meaningful question about the sources of pretraining progress. Crossing historical training recipes and corpora provides a stronger basis for separating their contributions than comparisons of independently released models.

  2. Substantial experimental effort. Multiple compute budgets, model-size selection, learning-rate calibration, and repeated seeds for the main comparisons strengthen the empirical evidence.

  3. Useful robustness checks. Additional recipes and corpora, a second evaluation suite, and fuller compute accounting consistently support the qualitative conclusion, while the paper acknowledges the limits of its scope.

Critical weaknesses
  1. Limited architectural coverage. The evaluated recipes primarily capture incremental changes within dense Transformers, leaving larger architectural changes, such as mHC [1], AttnRes [2], and hybrid attention [3], unexplored. Where these fall within the study’s temporal scope, including representative comparisons would strengthen the attribution analysis; otherwise, the conclusions should remain explicitly limited to the tested architecture families.

  2. Limited evaluation of generation and post-SFT performance. Evaluation primarily relies on multiple-choice or candidate-ranking tasks, including most tasks in the supplementary suite. It remains unclear whether the observed gains extend to open-ended generation or persist after SFT under matched data and training budgets.

  3. Potential task alignment and aggregation effects. The benchmarks emphasize English knowledge and question answering, which may favor educationally filtered corpora. The task-level results show heterogeneous gains and some regressions, suggesting that aggregate improvements do not necessarily reflect broad capability gains. Per-task attribution and sensitivity analyses across task subsets would help establish the robustness of the conclusions.

  4. Sensitivity to answer-format selection. OLMES10 takes the better score from cloze and multiple-choice formulations for each task. Although this follows the established protocol, differences in format sensitivity across configurations could affect relative performance and the resulting compute multipliers. Reporting the main comparisons separately under each fixed format would provide a useful robustness check.

[1] mHC: Manifold-Constrained Hyper-Connections

[2] Attention Residuals

[3] Kimi Linear: An Expressive, Efficient Attention Architecture

Questions for authors

Seeing Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Ryv

Reply to Reviewer 9Ryv

We thank the reviewer for their valuable insights. We thank the reviewer for their acknowledgement that we are answering important research questions about the historical sources on pretraining progress, and their recognition of our in-depth and substantial experimental efforts.

1. How far does the result extend beyond the historical dense Transformers?

We acknowledge explicitly in our limitations section that our results primarily apply to historical dense transformer architectures. And so, our conclusions are explicitly limited to the tested architecture families, and do not generally translate to more scale-dependent architectures.

That being said, we also performed further experiments in the directions you suggested. We trained a hybrid-attention variant of OLMo 3, replacing three of every four attention layers with Gated DeltaNet while holding the model size, schedule and learning rate fixed. We also considered other optimizers: we also replaced AdamW with Muon on OLMo 2’s matrix weights. (Appendix J, Tables 25–26).

We did not explicitly reproduce mHC or Attention Residuals. However, in Table 27, we report the reported numbers from the relevant research papers for these other architectures. At 27B, mHC lowered training loss by 0.021 with 6.7% more training time. Block Attention Residuals and Kimi Linear report fitted compute advantages of ×1.25 and ×1.16 against their own baselines. 

It’s important to note that while these are useful numbers in context, they cannot be compared to our experimentally obtained compute multipliers in an apples-to-apples fashion because these compute multipliers were obtained at different scales and with other data corpora.

2. Do the corpus gains extend to generated answers or persist after instruction tuning?

We definitely agree that it's unclear whether these observed gains transfer or persist after instruction tuning, or other further model post-training. This would certainly be a really interesting direction for future research, but unfortunately we did not have sufficient compute to tackle these questions in the current paper, and we made the conscious choice to focus on purely pretraining. 

3. Investigating task alignment and sensitivity to answer-format selection

Following your suggestion, we tested sensitivity to task choice by omitting each OLMES10 task in turn and comparing task subsets. We also repeated the analysis with the answer format fixed to cloze or to lettered multiple choice. We report these checks in Appendix H.1 and Figure 16.

Thank you for your response! I will keep my positive score.

Reviewer_CEqjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this study, the authors conduct an analysis to measure how much of the pre-training gains (from 2019 to 2025) come from data improvements (de-dup, quality filtering, diversity improvements etc.) vs. algorithmic improvements (architectural changes, pre-training recipe, etc.). The authors have conducted extensive experiments with models spanning from 6M to 1B parameters in a controlled setup. The results from the experiments show that the pre-training benefits observed over the years can be attributed more to data improvements than otherwise.

Critical strengths
  • The authors clearly separate their contributions from the related studies, i.e., isolating the effect of algorithmic vs. data vs. combined improvements
  • The authors have conducted extensive experiments to provide the results that they promised. These experiments, from my experience, are expensive and might be out of budget for many labs. Hence, the community can benefit from the presented findings.
  • Results are interesting; e.g., performance gains from model and data improvements are independent
Critical weaknesses
  • A lot of effort has gone into the experiments, but the unclear writing makes it very difficult to navigate the paper and understand the takeaway message. Please refer to the questions below.
  • The main manuscript is not self-contained. A lot of important details are in the appendix, which spans 17 pages.
  • Related work needs to be more comprehensive and accessible for a general reader.
  • Limitations are not discussed separately
  • I am focusing on the last line in the abstract "We conclude that from 2019 to 2025, at the scales we study, most software progress in pretraining is accounted for by data progress.", can you help me understand how this question is or isn't answered by scaling-laws studies? Our view of pre-training improvements have changed from Kaplan et al. (model-heavy) to Hoffman et al. (balanced scaling) to more data and higher quality is important (https://neurips.cc/virtual/2023/81185, https://openreview.net/forum?id=x54wwB6QvL). Can you help me understand what the presented study addresses that scaling studies don't.
  • I am curious to know the authors thought process behind selecting tokenizer. The fixed tokenizer used in experiments is GPT-2 (published in year 2019). The datasets published after GPT-2 do use code (Table 3) and additional data from different sources. Q1: Is any of the pre-training corpus multi-lingual?, Q2: How different are the new sources from the GPT-2 corpus? Q3: Did you check how does tokens-per-word change with every year? My main reason for asking this question is that I want to understand whether the changes in the tokens-per-word would underestimate the pre-training benefits from the data. I don't know how feasible is this but, can authors train 4 models with suitable vocabulary for "Ultra-FineWeb" (GPT-2 with new vocab on two dataset, and Olmo with new vocab on 2 datasets).
Questions for authors
  • Abstract: The abstract claims read a little broader to me becuase of three reason: (1) experimentation is with <1B models and authors speculate that the algorithmic improvements might get realized more in the larger models space (I understand that the FLOPs range is provided but it does not clearly convey the size of the models which is more familiar yardstick for the reader), (2) 64% of the benefits are attributed to data improvements, it is difficult to put judge this number in isolation i.e., how high or low is this, (3) data progress here predominantly means quality improvement via filtering. It would be great if authors can write the claims in more grounded fashion.
  • line-46: there are a few sentences here that read as if there were supposed to be headers, such as "Observational estimates measure only the joint effect.", "Controlled model ablations measure algorithmic progress only."
  • line-93: why 68%?
  • section-2: The language and brevity of the section 2 may make this section not very intuitive for a reader who is not familiar with this domain. I would suggest making this section more readable with a bit more explaination.
  • line-153: can authors please break the flops score in model and data size (range). I understand flops is arguably more accurate but model/data size provides some familiar metric.
  • line-156/157: I understand the iso-flop line but I don't understand "one with the lowest loss on held-out text from the training corpus (its native validation loss)". Did authors pre-train a bunch of models and then took the model with lowest loss on validation?
  • Line-157/158: This sentence is confusing for me, especially the "relatively constant" part. What are you comparing against? Do you mean the size is same for most corpora and for some the size changes?
  • line-260: Why is 3.16e18 flops is important? And attain what?
  • line-86/349: what is 2x2 design?
  • line-354: this pragraph reads as limitations and not results
  • line-370: some error in the reference
  • line-374: might be helpful to have some explanation for "DCLM-QC7/FW3"
2 comments
AuthorsOpenReview ↗

Response to Reviewer CEqj (1 of 2)

Reply to Reviewer CEqj

We thank the reviewer for the valuable comments. We appreciate the reviewer’s acknowledgement of our contributions in isolating the effect of algorithmic versus data versus combined improvements. We also appreciate the reviewer’s acknowledgement of the substantial and extensive experiments that we have conducted. We certainly hope to share the insightful results with the community. Much of the focus was certainly on completing the very extensive suite of experiments, and we have taken the opportunity to improve the writing and presentation of the paper to communicate the main insights.

  1. Main revisions to writing and presentation in the paper

We have tried to make the main manuscript more self-contained (to the extent we can). We seek the reviewers’ understanding that there are many important technical details in an ambitious experiment suite like ours for which there is insufficient space to explain fully in the main manuscript, and must therefore be relegated to the appendix.

The abstract now directly gives the scale of the primary sweeps in units that should be more familiar to the reader (as opposed to FLOPs): 5.5M–462M non-embedding parameters and 0.4B–8.3B training tokens per run. Section 3 separately gives the range across all runs, including calibration and higher-budget checks. Table 2 brings the previously separate recipe and corpus tables together, so readers can see side by side what changed in each year.

We have also made the related work section much more comprehensive, discussing scaling laws and pretraining algorithmic improvements in a more accessible manner. We have also drafted a separate limitations section that explicitly lists the limitations of our study.

  1. What does our study address that scaling studies don’t.

Scaling studies have certainly been extremely impactful in the field of frontier LLMs. Scaling laws like Kaplan et al. and Hoffmann et al. ask how to allocate a compute budget between model size and training tokens. As you point out, more recent scaling laws also account for training-data quality to a limited extent. These studies however do not consider the interaction effects between improvements in model algorithmic recipes and the improvements in data corpus quality.

On the other hand, we provide a novel investigation of the decomposition of historical compute efficiency gains between these two factors. In order to answer this question, we’ve had to retrain historically SOTA model recipes on various data corpuses, in contrast to scaling law studies that typically seek to understand how model size and data corpus scale for one particular model recipe or data corpus.

We also investigate whether a corpus improvement measured under one recipe carries over to others (as part of our study on the interaction effects). Across 49 recipe–corpus pairings at fixed compute, separate recipe and corpus effects explain 88.8% of the variation in downstream accuracy. Adding a product interaction does not improve prediction for held-out pairings (Section 4.3, Figure 6 and Appendix E.3).

The conclusion of our study is also something that exceeds what can be obtained from a scaling law study - that historical compute multiplier progress in small scale LLMs has mostly been accounted for by data quality improvements.

  1. Questions about the choice of tokenizer and relation to data corpuses.

We used the same GPT-2 tokenizer so that a change of corpus would leave the vocabulary fixed. All seven historical corpora were selected for English pretraining, although they contain some small proportion of non-English text. Using regular expressions on held-out text, we estimated that non-English text accounted for 0.4% of GPT-2 tokens in OpenWebText, 0.3% in C4, 3.4% in The Pile, and 0.1% in RefinedWeb. For OLM-CC, FineWeb-Edu, and Ultra-FineWeb, the estimates rounded to 0.0% (Appendix G.1, Table 21).

The sources themselves change in pretty substantial ways: OpenWebText follows links shared on Reddit, several later corpora filter Common Crawl, and The Pile combines 22 sources including code and scientific writing. The latest corpora use learned classifier filters.

Your proposed tokens-per-word check gives a direct answer about the direction of the bias. For the representative corpuses of 2019 through 2025 we measured 1.380, 1.317, 1.923, 1.313, 1.375, 1.345 and 1.336 GPT-2 tokens per whitespace-delimited word. Ultra-FineWeb uses 3.2% fewer tokens per word than OpenWebText. Normalizing by words therefore moves the corpus compute multiplier estimate slightly downward, from ×12.0 to ×11.6. Recounting the same fixed runs with the OLMo 2, GPT-NeoX and Qwen2.5 tokenizers gives ×11.71, ×12.09 and ×11.79, respectively. We see little evidence of an underestimate from token accounting. (Appendix G.2, Tables 21 and 23).

AuthorsOpenReview ↗

Response to Reviewer CEqj (2 of 2)

  1. Questions about the abstract. 

We have revised the abstract to present our claims in a more grounded fashion. We have explicitly stated our model sizes that we have experimented with. The 64% figure implies that data progress is the main contributor to pretraining compute multipliers at our scale (if both data and model upgrades contributed equally, the share would be 50%). I would further point out that it is precisely because of the lack of prior research into decompositions of historical AI progress that we lack clear priors on what percentage we should expect here. And that the 64% figure by itself might already be incredibly surprising to researchers in the field who have mostly seen it as being driven by model algorithmic progress.

While we agree that data progress here predominantly means quality improvement via filtering, this is not the only dimension of data progress, and we feel it would be overly restrictive to assert that this is the primary (or future expected) driver of data progress. 

  1. Line-156/157: I understand the iso-flop line but I don't understand "one with the lowest loss on held-out text from the training corpus (its native validation loss)". Did authors pre-train a bunch of models and then took the model with lowest loss on validation?

For each model recipe x data corpus configuration, we need to determine the compute-optimal model size and training token count. To do this, we train an isoFLOP series of the same configuration but across different model sizes and training token counts. We then look at the validation loss of these pretrains and choose the one that is lowest - this is the compute optimal model size for each configuration. 

  1. Line-157/158: This sentence is confusing for me, especially the "relatively constant" part. What are you comparing against? Do you mean the size is same for most corpora and for some the size changes?

As mentioned previously, we have a bunch of compute optimal model sizes for each configuration. Basically, our point is that usually this compute optimal model size does not vary much based on the corpus we use. Rather it is more a function of the model recipe. 

  1. line-260: Why is 3.16e18 flops is important? And attain what?

This is to present the compute multipliers at different compute scales. We’ve provided a clearer presentation of the compute multipliers of recipes and corpora, and how they change at different compute scales in Figure 2.

Reviewer_5cXxOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper tests whether purported algorithmic progress measured by other papers comes from better optimization or if it instead comes from better data sets. This is done by testing a variety of model recipes and a variety of data sets. As with previous work, these are tested for only small-scale amounts of compute. The conclusion is that much of the progress comes from better data rather than better optimization.

Critical strengths

This is a deeply important question for the future of AI.

The paper does an admirable job of reconstructing state-of-the-art systems and data sets from across time and providing a broad set of empirical tests on these topics. The strongest parts of the paper are 4.2 and 4.3, which start to provide a more integrated view of the work.

The linear model, while poorly presented, captures an important finding that further justifies the broader claims of the paper.

Critical weaknesses

The writing in the paper is choppy and abrupt. This makes it difficult to read.

The paper is much too focused on presenting methodological choices (which could be relegated to the appendix) and not enough on providing a framework for interpreting their results or clear graphical representations of them. For example, the results in Figure 2 are almost unreadable and would be better expressed in other ways. Figure 3 is the exception to this, and is a nice presentation.

While not explicitly needed in this paper, which is more focused on establishing this phenomenon, these results beg for an analysis of the data themselves - what changed, how much overlap is there, etc.

Questions for authors

This paper feels like the empirical results came in only shortly before the submission deadline and so the writing focused on incorporating them rather than building the overall flow of the paper. I encourage you to correct this in the review process.

Please clarify early on, what size model and how much data is being used for each experiment. Since they are trained to be compute-optimal, this might mean only parts of the data are being used. Is that happening?

Where possible, it would be good to compare estimates here to those that full-scale models get to see if there are any scale-effects or other differences.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5cXx

Reply to Reviewer 5cXx

Thank you for your productive comments. We appreciate the reviewer’s recognition that we are answering a deeply important question for the future of AI through an ambitious and novel effort of reconstructing multiple historical state-of-the-art systems and datasets, as well as comprehensive empirical tests.

We have substantially revised the writing, flow and presentation of the paper as suggested by the reviewer. We have also further improved on the linear model study of the interactive effects of model and data improvements. We made the grid’s interpretation explicit and added a check that asks whether it can predict a pairing left out of the fit. Separate recipe and corpus effects explain 88.8% of accuracy variation across the 49 pairings. Adding a product interaction increases held-out RMS error from 0.756 to 0.790 percentage points in the 7 × 7 grid and from 0.659 to 0.672 in the 8 × 10 grid (Section 4.3, Figure 6 and Appendix E.3). These further show the independent nature of the model and data improvement effects.

1. Main revisions to the figures

We understand your concern with the readability of Figure 2 (now Figure 3). We understand that it might be hard to decipher given the large number of series presented together on the same compute scaling graph. We have increased the size and clarity of the graph to hopefully make it less crowded. We’ve decided to keep this graph in the paper for now as we think it is important to be able to see the compute scaling properties of both the model recipe improvements as well as the corpus improvements at a glance.

We have introduced new figures that are more intuitive. For example, the new Figure 2 shows you at a glance what are the compute multipliers at the various compute scales, across both corpus and recipe upgrades. Figure 4 is also a clear presentation of the compute multipliers of the various recipes and corpuses (much easier to see as opposed to the previous table we had).

2. How large were the models, and did they use the whole corpus?

The primary sweeps used 5.5M–462M non-embedding parameters and 0.4B–8.3B tokens per run (which we have now made explicit in the abstract and main paper). Including calibration, size sweeps and higher-budget checks gives 2.7M–1.1B parameters and 41M–20B tokens (Introduction and Section 3).

To answer your question whether only parts of the data corpus are used: Yes, the models are trained to compute-optimal points, and so most runs only use part of their training split.

3. What changed in the corpora, and how much text did they share?

The clearest change we measured was in educational content. Applying the same classifier to held-out documents from each corpus, we found that 10% of OpenWebText documents scored at least 3, compared with 44% of Ultra-FineWeb documents. Appendix G.1, Table 21 also compares the prevalence of code, mathematical text, question–answer passages, and non-English text.

To measure shared text, we compared FineWeb-Edu and Ultra-FineWeb, which use different filters to select documents from the same FineWeb collection. We sampled 512 held-out windows from each corpus and searched the other corpus’s training splits for exact matches to every consecutive 13-token sequence. Of the 13-token positions sampled from Ultra-FineWeb, 18.6% matched text in FineWeb-Edu. In the reverse direction, 11.2% matched.

Shared passages were widespread: 93% of the sampled Ultra-FineWeb windows contained at least one match in FineWeb-Edu, and 85% of the FineWeb-Edu windows contained at least one match in Ultra-FineWeb.

We also had much more detailed discussions of the changes among the corpora and other related questions in section G of the Appendix.

4. What comparison to larger models is supported?

Figure 2 provides a comparison of the changes in compute multipliers over the compute scales that we were able to study. In Table 27 of Appendix J, we looked at compute multipliers reported for larger scale models / architectural changes. These numbers were mostly reported numbers from literature as opposed to reproductions.

Appendix J, Table 27 gives examples from published larger-scale experiments. Kimi Linear’s fitted scaling law gives a ×1.16 compute multiplier relative to its full-attention baseline. For block Attention Residuals, the fitted comparison gives ×1.25 relative to the authors’ baseline.

It is important to note that while useful contextually, these published numbers do not provide an apples-to-apples comparison with our obtained multipliers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.