acceptodds
Under review as a conference paper at ICLR 2027

Upcycling the Low-Quality Corners of the Web for Pretraining

Abstract

Modern language model pretraining pipelines collect far more web text than they ultimately use, discarding large fractions through quality filtering. We call this **token waste**: potentially useful tokens are acquired during data collection but removed before training. Existing synthetic data methods mostly *recycle* data by rewriting high-quality documents that already survive filtering. We instead ask whether data deemed too low-quality for training must actually be discarded; can they instead be **upcycled** by transforming low-quality documents into useful pretraining data. Concretely, we train a small language model with preference optimization over learned quality and influence-based utility signals, then use it to upcycle the lowest-quality 14B tokens from C4 (i.e., wasted tokens). Across language models from 19M to 1.4B parameters, upcycled tokens consistently outperform wasted tokens on in-domain and out-of-distribution language modeling, as well as downstream performance. When combined with high-quality data, upcycled tokens recover much of the benefit of adding more high-quality data. Additional experiments show that upcycling preserves factual knowledge lost through filtering, improves existing synthetic data recipes, remains effective with substantially smaller synthesizers, and extends over-training gains beyond what is achieved by repeating high-quality data. Together these results show that **token waste can be prevented,** with benefits extending well beyond recovering pretraining utility.

Primary area:
datasets and benchmarks
Keywords:
Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K8DkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to reuse the filtered-out tokens from pre-training datasets, because they do actually contain information that is useful. They use a small fine-tuned language model to rephrase pruned tokens. The rephraser model is fine-tuned using DPO on synthetically generated pairs of wasted tokens and their high scoring (quality and influence) rephrase. They have experiments to show the efficacy of upcycling wasted tokens and some ablations.

Critical strengths

The paper is well written and structured well. The idea is intuitive, to rephrase things that do not look important, but do indeed have crucial information. that rephrasing wasted tokens indeed improves performance as the pretrained model size increases, it can preserve knowledge that is present in the wasted tokens

Critical weaknesses
  1. They don't have baselines besides their own ablations, so it's not clear whether their method of upcycling is helping or whether their experimental results are just an effect of adding more pre-training data. If I were writing this paper, I'd run Megadocs [1] and/or Rephrasing the Web [2] to show that using wasted tokens is as good (or, maybe better) than the summarization-based pretraining methods.

  2. Can the authors explain why Retained + Retained is a low performing method in Table 1? It seems counter intuitive that it performs worse than Retrained + Wasted.


[1] Kim et al, Data-efficient pre-training by scaling synthetic megadocs [2] Maini et al, Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Questions for authors
  1. In Section 4.1, what is the knowledge that the wasted/upcycled tokens contain?
2 comments

Thank you for the constructive review, and for finding the paper well written and the idea intuitive. We have uploaded a revised PDF (changes in blue), including two new external baselines, WRAP [9] and Megadocs [7] (Sec. 2.1, Figs. 1-2), and address your concerns below.

W1. Baselines.

TL;DR: We added WRAP [9] and Megadocs [7] as baselines. Upcycling outperforms both, despite their 4 times larger generators.

Our gains do not simply come from adding more data. Every mixture comparison is token-matched, with the same Retained tokens and total budget, and differs only in the source of the second half (i.e., Wasted, Upcycled, or Retained) (Sec. 2.1).

Our prompt follows [7], which adapts the Wikipedia-style prompt of WRAP [9] to full documents. Applying it to an off-the-shelf model is clearly weaker than training the synthesizer (App. C, Table 3), which motivates our approach. Training the synthesizer also improves the strongest recipe of [15] (Table) at both 670M and 1.4B (Sec. 4.2).

Following your suggestion, we add two external baselines applied to the same source documents that make up Wasted Tokens, each in its best-performing reported configuration (Sec. 2.1, App. G):

  • WRAP [9]: Mistral-7B-Instruct with its question-answer prompt, which yields their strongest zero-shot results, on chunks of at most 300 tokens.
  • Megadocs [7]: Stitched Megadocs with G=32 and Llama-3.1-8B-Instruct, keeping their prompt, decoding, context length, and document ordering.

We train three seeds per scale from 19M to 670M and will extend to 1.4B in the camera-ready. Upcycled achieves lower loss than both baselines at every scale, alone and mixed with Retained, and matches or outperforms them on CORE (Figs. 1-2, App. I). At 670M, both baselines even trail raw Wasted data on held-out loss when mixed with Retained, suggesting that off-the-shelf rewriting does not reliably recover the rejected low-quality tail, while our trained synthesizer does.

W2. Why Retained+Retained scores low on NoKLB.

TL;DR: NoKLB probes knowledge that exists in filtered documents, which more high-quality data does not supply.

This is the main finding of Sec. 4.1. NoKLB measures knowledge contained in documents that quality filtering removes. Factual recall depends strongly on how often a fact appears in pretraining data [1, 5, 10], so filtering, which can remove content unevenly [2, 21], makes already rare facts even harder to learn. Retained+Retained trains on more high-quality data, but that data does not contain this knowledge, so the extra tokens bring little gain on NoKLB. In contrast, Retained+Wasted and Retained+Upcycled include the filtered documents, in raw or upcycled form, and therefore the models acquire such knowledge. The result shows that the filtered data carries knowledge complementary to the retained data, and that upcycling preserves it (Retained+Upcycled scores highest at both scales).

On held-out loss and CORE, Retained+Retained remains strongest or on par with Retained+Upcycled (Figs. 1 and 2), as expected. We made this explanation explicit in Sec. 4.1.

Q1. What knowledge do the wasted tokens contain?

NoKLB keeps only self-contained, durable, encyclopedic facts from Wasted documents with high educational value, each verified against external sources (App. F). We added representative questions with answers spanning computing, natural sciences, geography, history, public health, and psychology (App. F, Table 6).

These documents are discarded because of how they are written, not what they contain. The DCLM classifier [8] favors text resembling its instruction and ELI5-style positive examples, so informative content wrapped in boilerplate, catalog listings, or forum formatting scores low (App. H). [13] makes the same observation about classifier-based filtering. Upcycling keeps the content and fixes the form.

We hope this addresses your concerns and would be grateful if you would reconsider your score.

All my concerns are addressed. I've raised by score.

Reviewer_BUcxOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper hypothesizes that low-quality data that is filtered out during pre-preprocessing can be upcycled and turned into useful pre-training data. Given that current pre-training pipelines utilize a quality filter to construct pre-training corpora, a bulk of samples can be discarded during the process. The paper proposes to utilize these samples given the potential limited availability of high-quality pre-training resources. To achieve this, the paper trains a synthesizer model (i.e., generator) and uses it to upcycle a low-quality document. The paper conducts extensive experiments to verify the effectiveness of this upcycling method, varying data mixtures, model scales, and generator sizes, by pre-training models on upcycled and/or high-/low-quality original data. Results consistently show the effectiveness of the proposed upcycling method, suggesting that previously discarded tokens can become a vast reserve of new valuable resources.

Critical strengths
  1. The motivation of trying to upcycle low-quality samples is clear, and the overall experimental design is sound.

  2. The paper has conducted quite extensive experiments to examine the efficacy of upcycling as well as its implications, such as better knowledge coverage.

  3. The paper has also built a new probe dataset: NoKLB, which is a grounded multiple-choice benchmark useful for analyzing knowledge coverage.

Critical weaknesses
  1. The major weakness of this paper is its limited novelty. While the motivation of using low-quality data and upcycle sounds reasonable, this has been already discussed in Niklaus et al. (COLM 2026). The paper currently lacks the discussion on this point, claiming that previous work largely operates on data that already survives upstream selection (L53; Introduction). In my opinion, this is not true given the existence of the work: Niklaus et al. (COLM 2026). At least, the paper needs to clearly discuss what are the new findings compared to this work.

    Niklaus et al. (COLM 2026) How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data, https://openreview.net/forum?id=fnqFKR2Y4T

    For instance, I found that factual knowledge-related analysis is something new. However, ablating generator model sizes is not, as Niklaus et al. (COLM 2026) has already conducted more extensive grid experiments on this. The corresponding takeaway is also quite similar to Niklaus et al. (COLM 2026) on this generator size experiment. For this, the paper should at least acknowledge this work.

  2. Another weakness is its experimental scale and its practical usefulness. The tested model range is quite small, up to 1.4B parameters. Also, the target raw low-quality tokens are just 14B tokens. Given that typical pre-training data sizes are over trillions of tokens, I am entirely not sure how reliable and useful this proxy setting is. Training budgets are also quite small, less than 28B tokens in total for each configuration by default. With this setup, it is not clear whether the efficacy properly scales and will be retained for a longer period of training.

  3. I am not sure how useful the knowledge probe (4.1) is. Particularly, the paper does not clearly discuss why it is important to retain “tail” knowledge in the model weights. In a practical scenario, practitioners should use RAG-based or tool calling methods to let LLMs answer knowledge-specific questions. That is, the importance of retaining tail knowledge appears small. Therefore, while the finding itself sounds interesting, I do not think this aspect matters a lot in reality.

  4. Finally, I found the motivation regarding over-training experiments is misleading. The paper describes over-training as if typical pre-training pipelines employ many repetitions over the same pre-training data (L359-361). I think this is not entirely true. For instance, Olmo3 only repeated training data for a fraction of data, mainly for STEM data, and they capped this at most 7 times. In contrast, the paper repeats this process over 37 times at 55M.

Questions for authors
  1. Figure 1: each model is independent. So, it should not use a line plot. Also, it is really hard to read TPR=20. What does each dashed line mean? Please be explicit. The same applies to Figure 2.

  2. Table 2 says the paper used DCLM 7B, and the corresponding text also says DCLM 7B is trained on 2.5T tokens (L261-263). I don’t think the comparison here in Table 2 is fair, as DCLM is the only vanilla causal language model and the rest are post-trained. Therefore, the corresponding discussion that DCLM filtering reduces knowledge coverage seems to be not supported well as it involves a confounding factor.

2 comments

Thank you for the insightful review, and for noting that the experimental design is sound and that NoKLB is a useful benchmark. We have uploaded a revised PDF (changes in blue), including two new external baselines, WRAP [9] and Megadocs [7] (Sec. 2.1, Figs. 1-2), and address your concerns below.

W1. Novelty relative to Niklaus et al. [15].

TL;DR: [15] treats low-quality data as one axis of a broad design study. We isolate it, compare it against a token-matched high-quality ceiling, and optimize the synthesizer itself.

We already cite and build on [15] (Sec. 4.2), and now discuss it in revised Sec. 4.3 and Related Work (App. A). Their finding that source quality matters little given a strong mix-in supports our premise, and our work goes further in three ways:

  1. Matched-token ceiling. [15] compares mix-in corpora but not against spending the same tokens on more high-quality data. This ceiling comparison is our main result, as we find that Retained+Upcycled closes 73-80% of the loss gap to Retained+Retained at 1.4B.

  2. A trained synthesizer that improves on them. [15] varies prompts and uses off-the-shelf generators, whereas we optimize the synthesizer itself. Applied to their strongest recipe (Table), it improves CORE at both 670M and 1.4B (Sec. 4.2).

  3. New consequences. We show that filtering removes knowledge that upcycling preserves (Sec. 4.1), as the reviewer notes, and that upcycling's benefits extend to over-training (Sec. 4.4). Neither finding appears in [15].

Generator size. We agree that Sec. 4.3 is consistent with [15] beyond 1B, and do not claim it as a separate contribution. It shows that the gains do not require a synthesizer larger than the pretrained model. Notably, their off-the-shelf 360M generator trails its 1.7B counterpart, whereas our trained 360M synthesizer remains close (Sec. 4.3).

W2. Experimental scale.

TL;DR: Our largest scale exceeds [15], and we add more scales, seeds, and baselines.

We emphasize that our largest models and per-run budgets are larger than those of [15], who train 1.2B models on about 21B tokens per run. In contrast, we span five scales from 19M to 1.4B (~2 orders of magnitude), with multiple seeds and budgets up to 28B tokens. Our runs show a clear trend across multiple scales, a key step in establishing scaling behavior without running extremely costly runs. For example, training a single 1.5B model at a trillion token scale would take over 8000 H100 hours, for a cost of more than 20K USD at cloud H100 rates.

Moreover, our over-training runs (Sec. 4.4) use 56B and 150B tokens. The gains also hold up as models grow, and at 1.4B upcycling still cuts DCLM loss from 3.47 to 3.07. The 14B-token Wasted pool is a token-matched sample, not a limit of the method. Upcycling operates document by document, so its cost grows linearly with the amount of data, and Sec. 4.3 shows that a 360M synthesizer is enough.

W3. Knowledge coverage as a goal of pretraining.

TL;DR: Factual knowledge remains a central function of pretraining, and retrieval complements rather than replaces it.

Knowledge acquisition is a direct focus of related work [1, 5, 10], and [15] also uses factual knowledge as a main metric. It therefore matters whether filtering, a standard step in every modern pipeline, quietly removes knowledge, which NoKLB measures.

Retrieval complements parametric knowledge rather than replacing it. A model relies on what it knows to decide what to retrieve and to interpret what comes back. Many deployments also run without retrieval. This is why knowledge benchmarks such as MMLU [4] remain standard.

Coverage is also a question of fairness. Filtering can remove text about particular topics unevenly, which introduces biases to the models [2]. Our results show that upcycling recovers this coverage without giving up the utility.

W4. Motivation for the over-training experiments.

We agree that current pipelines repeat data only moderately, e.g., Olmo 3 caps repetition at seven [16]. Sec. 4.4 deliberately anticipates a data-constrained future, following established prior work [3, 6, 7, 11, 12], as token-to-parameter ratios already exceed 1,000 [19] while unique high-quality data stays fixed. We revised the motivation in Sec. 4.4 accordingly. Notably, Retained+Upcycled already outperforms the alternatives within 7 repetitions (Fig. 6), the range the reviewer considers realistic.

Q1. Figures 1 and 2.

Thank you. We added per-scale markers and clarified that each marker is an independently trained model.

Q2. Fairness of Table 2.

We replaced Table 2 with base checkpoints. DCLM 7B remains the only model scoring lower on NoKLB than on MMLU, and we frame this comparison as suggestive, with Table 1 as our main controlled evidence.

We hope this addresses your concerns and would be grateful if you would reconsider your score.

Thanks for the response and clarifications. While the updates to Table 2 and Figures 1-2 address my presentation-related questions, my main concerns remain unresolved.

First, the response mentions adding WRAP and Megadocs. I never asked for new external baselines in my review.

On the first weakness:

  • Appendix B groups Niklaus et al. (2026) among works operating "on curated or already-filtered data". In fact, Niklaus et al. explicitly evaluate a low-quality split (FineWeb-LQ, classifier scores 0-1). So this point is incorrect. Furthermore, Appendix A acknowledges this, so the two appendices are inconsistent now.
  • The response claims Niklaus et al. do not compare against spending the same tokens on high-quality data. However, their Section 4.3 and Tables 8/9 explicitly compare rephrased FWLQ + FWHQ against 100% FWHQ and rephrased FWHQ + FWHQ at a matched 21B-token budget. So, this claim in the response is simply false. The paper needs to correct this and clarify what is new relative to that comparison.
  • Section. 4.3 (L398-402) contrasts their off-the-shelf SmolLM2-360M with the proposed trained 360M synthesizer. Their scaling curve uses the tutorial prompt on FWHQ. This paper uses a Wikipedia-style prompt on C4. Without an untrained SmolLM2-360M baseline under this paper's exact setup, attributing the smaller gap to the proposed DPO training is unsupported.

On the second weakness, the response claims that the largest scale (1.4B) exceeds Niklaus et al. (1.2B). While 1.2B is their default proxy, Niklaus et al. scale student models up to 2.9B and 6.2B parameters (Section 4.2, Table 5) and run a 72-run multi-seed grid up to 6.2B (Table 21). In contrast, the 1.4B results in the paper remain single seed and over-training runs (56B/150B tokens) remain restricted to up to 152M models. The response therefore overlooks this difference and reports an incorrect fact.

On the third weakness, the response claims without citation that "many deployments also run without retrieval" and points to MMLU to justify parametric knowledge retention. However, MMLU evaluates broad academic and textbook subjects rather than long-tail web facts. Therefore, its use does not support the practical necessity of memorizing long-tail facts in model weights.

On the fourth weakness, the motivation in Section 4.4 is still unconvincing. First, models exceeding TPR = 1k (e.g., Qwen3 on 36T tokens, cited in footnote 2) achieve this with trillions of unique tokens, not by repeating a 1.5B token pool up to 100 times. Second, pointing to the first 7 repetitions in Figure 6 is misleading. 7 repetitions of 1.5B tokens is only 10.5B tokens (7% of the schedule), meaning Retained + Upcycled (15.5B unique tokens in total) has not even completed one epoch.

For these unfaithful responses, I have decided to lower my rating, as the work does not accurately reflect its contribution, and I do not see a willingness to present it faithfully.

Reviewer_3TgaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The main idea is to use discarded real data in LM pretraining. Data that would be normally discarded in quality filtering can be "upcycled" by rewriting and used in pretraining. Rewriting is done by training a separate model via a preference classifier.

The authors create three buckets of data: Retained (regular good quality), Upcycled (bad quality turned into better quality by rewriting), and Wasted (bad quality data as-is). Experiments are done for LM pretraining at various scales from 19M to 1.4B parameter count: on the Llama-2 model family.

Three claims are made:

  1. [Figure 1 and Figure 2] The main claim is that Retained + Upcycled can almost match the performance of Retained + Retained (i.e. at matched token budget), and significantly outperforming Retained + Wasted. Secondly (and less surprisingly), Upcycled outperforms Wasted.
  2. [Table 1] On a benchmark (called NoKLB, created by the authors) which measures factual recall from the Wasted tokens, Retained+Upcycled outperforms all other data mixtures (including Upcycled only).
  3. [Figure 6] In regimes where data is repeated a lot, using Upcycled data can improve over just repeating Retained data.

Additionally, as a fourth thing, the authors initially used a 1.7B model for rewriting, but also performed experiments with a 360M model to demonstrate a smaller model can also perform rewriting (Section 4.3).

Critical strengths
  1. The paper performs detailed experiments.
  2. The paper is well-written and easy to follow.
Critical weaknesses
  1. Novelty is unclear. At a result level, Nemotron-CC (cited by the manuscript) already compared raw and rewritten low-quality web text in matched pretraining mixtures. Even at a methodology level, RePro (also cited) already trained a rewriter and explored both fastText quality scores and training-data influence as rewards. I might be missing something -- can the authors please highlight which of their contributions is truly novel over past work?

  2. What happens if high-quality data is rewritten? Only use case of Upcycled data is when Retained data needs to be repeated a lot of times (Section 4.4). But given the same budget, how would rewriting the available retained documents, or a mid-quality portion of the crawl, compare? This matters for deciding which documents a practitioner should rewrite. Can this experiment be done? Only this will provide an apples-to-apples comparison.

  3. NoKLB benchmark is bogus. The benchmark is explicitly checking for facts from the rejected data. It is obvious that including this in pretraining will help performance on this benchmark. But it is unclear if the questions are actually useful. Can you provide some examples of questions from the NoKLB benchmark? I could not find it anywhere in the paper. Additionally, there is no control of a similar benchmark on the Retained questions. Is the increase in utility on NoKLB coming from decrease in utility on high-quality factual recall questions? Manuscript does not show this experiment.

Questions for authors
  1. In the Retained+Retained setting, is the second half a repeat of the first half or fresh high-quality data?
  2. Any particular reason for choosing Llama-2? That model family seems quite old.
2 comments

Thank you for the careful review and for finding the experiments detailed and the paper clear. We have uploaded a revised PDF (changes in blue), including two new external baselines, WRAP [9] and Megadocs [7] (Sec. 2.1, Figs. 1-2), and address your concerns below.

W1: Novelty relative to Nemotron-CC and RePro

TL;DR: No prior work isolates the rejected tail or compares it against a token-matched high-quality ceiling.

Source distribution is the core difference. Prior work transforms broad or quality-stratified pools, whereas our sources are the lowest-scoring 14B of 200B C4 tokens, all below the DCLM rejection threshold (App. B, Fig. 8). Our central question is whether this tail can be made competitive with high-quality data, and no prior work isolates it.

Nemotron-CC [18] never isolates the low-quality tail. (i) Its 1T-token ablation (their Table 10) adds rephrasings to the full Common Crawl distribution, where low-quality documents are only ~9% of Common Crawl tokens (their Table 2), so rephrased low-quality data is a small addition to a mostly medium-to-high-quality mixture. In contrast, our low-quality data makes up 100% (TPR=10) or 50% (TPR=20) of training. (ii) It never compares rephrased low-quality data against spending the same amount of tokens on high-quality data. This ceiling comparison is our main result: at 1.4B, Retained+Upcycled closes 73–80% of the loss gap to Retained+Retained and matches it on CORE (0.103 vs. 0.106). (iii) Its flagship 15T run uses only medium-to-high-quality data (their App. E), so rephrased low-quality data is never used at scale.

RePro [20] uses different rewards. It tries fastText and influence as rewards and finds that both fail, where the former saturates and the latter collapses to no rephrasing (their Sec. 5.3), so its final method uses DataMan and faithfulness rewards. We use both signals successfully. RePro also rewrites DCLM-RefinedWeb, a higher-quality pool, rather than the rejected tail (Fig. 8).

We also provide contributions not in either paper: (i) filtering removes knowledge that upcycling preserves (Sec. 4.1), (ii) optimizing the synthesizer adds to the gains of an existing recipe (Sec. 4.2), and (iii) upcycling extends over-training beyond what repeating high-quality data achieves (Sec. 4.4).

W2: Why not rewrite high-or-mid-quality data?

Only ~1.4% of Common Crawl survives DCLM filtering [8]. Rewriting it is well studied [9, 14, 18], and gains shrink as source quality rises [17]. Making the discarded 98.6% usable would enlarge the usable pool by an order of magnitude, and our results show this is possible even at the very bottom of the distribution.

Moreover, rewriting retained documents re-expresses content the model already sees, while upcycling adds content it would never see. Its value is also not limited to over-training, as Sec. 3 shows it under compute-optimal budgets.

W3: Is NoKLB a meaningful benchmark?

TL;DR: NoKLB gains come at no cost on general knowledge and show that upcycling preserves the knowledge that filtering removes.

NoKLB does not replace general knowledge benchmarks, which we measure with CORE. Retained+Upcycled matches Retained+Retained (0.078 vs. 0.073 at 670M; 0.103 vs. 0.106 at 1.4B), so the NoKLB gain does not come at the cost of established benchmarks. We believe this comparison answers the question of "Is the increase in utility on NoKLB coming from decrease in utility on high-quality factual recall questions?".

Instead, NoKLB is an additional diagnostic for whether quality filtering removes factual knowledge. Some gain is expected once Wasted Tokens is included, but our results remain informative in two ways. First, Retained+Wasted beats Retained+Retained, so this knowledge is missing from high-quality data rather than redundant with it. Second, Upcycled matches or exceeds Wasted, and Retained+Upcycled beats Retained+Wasted at both scales (32.05 vs. 30.07; 39.41 vs. 37.41), so rewriting keeps these facts rather than dropping or hallucinating them.

Are the questions useful? Our pipeline (App. F) keeps only self-contained, durable, encyclopedic questions, and checks every answer against external sources with web search. Strong LLMs score 70-81% (Table 2), so these are broadly known facts. We added representative questions with answers (App. F, Table 6).

Q1: In Retained+Retained, is the second half repeated or fresh?

The second half is fresh, as we use no repetition in the compute-optimal setting, so it is the strongest baseline available. We made this explicit in Sec. 2.1.

Q2: Why Llama-2?

We use Llama-2 architecture and tokenizer, trained from scratch, following prior work [6, 7, 14]. The architecture (RoPE, SwiGLU, RMSNorm) remains the standard modern decoder, and our conclusions concern the data, not the architecture.

We hope our response addresses your concerns and would be grateful if you would consider updating your score.

W1. Thanks for clarifying this. I understand that no prior work might have focused on the lowest quality data (the "tail"). If this is the central contribution, I advise to make this more clear in the writing of the paper. In particular, in the second last paragraph of Introduction (before Contributions), it should be made clear that prior works study similar techniques but never apply them to the lowest quality part of the filtered out data.

W2. I am still not satisfied with the response. Firstly, your response never explains why mid-quality data is not better than using low-quality data. Secondly, this needs to be shown experimentally, not just mentioned anecdotally. The experiment: take the same quantity of data from rejected (filtered out) part of the data but from the head instead of the tail, and perform Upcycling on this part. If the authors can provide such an experiment, I will reconsider my rating.

There are two conflating claims going on:

C1. There is value in lowest quality subset of the rejected data (the tail).

C2. The tail is the best additional data to include in pretraining.

I believe the experiments show C1, but the writing of the paper leans towards C2, and the NoKLB benchmark also tries to show C2. I don't think C2 is true, and I see not enough experimental evidence to establish C2.

W3. Thanks for providing the example questions. I agree that questions are reasonable. However, the controlled study (for proper comparison) would involve creating a similar benchmark (say, ControlKLB) that measures performance on Retained data. Just reporting CORE does not create a proper comparison. So again, I am not fully satisfied with the response.

Due to these reasons, I will maintain my score and lean towards rejection of this paper.


Thanks for answering my questions Q1 and Q2. I am satisfied with Q1's answer, and I understand Q2's answer (however I still think ideal experimental setup should use newer models).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Reviewers found the idea intuitive and well motivated (BUcx, K8Dk), the experiments detailed and sound (3Tga, BUcx), and the paper clearly written (3Tga, BUcx, K8Dk). We have uploaded a revised PDF with changes in blue and summarize the main revisions below.

New external baselines.

Following Reviewer K8Dk's suggestion, we compare against WRAP [9] and Megadocs [7], applied to the same Wasted source documents in their best-performing reported configurations (Sec. 2.1). Both use generators more than 4 times larger than our 1.7B synthesizer. The new sweep covers 19M - 670M parameters with three seeds per scale, totaling 48 models. We are running 1.4B and will report results before the end of the discussion period if compute allows, and in the camera-ready version otherwise.

Upcycling achieves lower loss than both baselines at every scale, alone and mixed with Retained Tokens, and matches or outperforms them on CORE (Sec. 3, Figs. 1–2, App. I). We provide the results at 670M model scale below:

CorpusDCLM loss (↓)FineWeb-Edu loss (↓)CORE (↑)
Wasted3.6673.3870.037
WRAP3.6923.6000.054
Megadocs3.6513.5530.044
Upcycled (ours)3.2293.1010.061
Retained + Wasted3.0112.9810.060
Retained + WRAP3.1793.2250.074
Retained + Megadocs3.0893.1030.063
Retained + Upcycled (ours)2.9492.9020.078

Other revisions.

  • Related work: Expanded comparison with Nemotron-CC, RePro, and Niklaus et al. (App. A). We will move Related Work back into the main paper in the camera-ready version using the additional page.
  • NoKLB: Representative questions with answers (App. F, Table 6). Table 2 now reports base checkpoints.
  • Clarifications: Fresh second half in Retained+Retained (Sec. 2.1), why Retained+Retained scores low on NoKLB (Sec. 4.1), generator size relative to Niklaus et al. (Sec. 4.3), and the motivation for over-training (Sec. 4.4).
  • Figures: Per-scale markers and explicit captions (Figs. 1–2).

References.

[1] Chang et al. How Do Large Language Models Acquire Factual Knowledge During Pretraining? NeurIPS, 2024.

[2] Dodge et al. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. EMNLP, 2021.

[3] Fang et al. Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality. 2025.

[4] Hendrycks et al. Measuring Massive Multitask Language Understanding. ICLR, 2021.

[5] Kandpal et al. Large Language Models Struggle to Learn Long-Tail Knowledge. ICML, 2023.

[6] Kim et al. Pre-training under Infinite Compute. 2025.

[7] Kim et al. Data-Efficient Pre-training by Scaling Synthetic Megadocs. 2026.

[8] Li et al. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. NeurIPS, 2024.

[9] Maini et al. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL, 2024.

[10] Mallen et al. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. ACL, 2023.

[11] Mohri et al. A Bitter Lesson for Data Filtering. 2026.

[12] Muennighoff et al. Scaling Data-Constrained Language Models. NeurIPS, 2023.

[13] Nait Saada et al. The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining. 2025.

[14] Nguyen et al. Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. 2025.

[15] Niklaus et al. How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. COLM, 2026.

[16] Olmo Team. Olmo 3. 2025.

[17] Pieler et al. Rephrasing Natural Text Data with Different Languages and Quality Levels for Large Language Model Pre-training. 2024.

[18] Su et al. Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. ACL, 2025.

[19] Yang et al. Qwen3 Technical Report. 2025.

[20] Yu et al. RePro: Training Language Models to Faithfully Recycle the Web for Pretraining. 2025.

[21] Longpre et al. A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity. NAACL, 2024.

AuthorsOpenReview ↗

Thank You to the Reviewers

We thank all reviewers for the time and care they put into reviewing our paper and engaging in the discussion. The feedback has been valuable, and we will carry it into future versions of this work. We would like to close by briefly noting where we respectfully see things differently, in the hope that it is useful to readers of this thread.

Different notions of "low quality" (re: Reviewer BUcx, comparison with Niklaus et al.). Niklaus et al. define FWLQ by FineWeb-Edu scores of 0 and 1 within FineWeb, a corpus that is already filtered and deduplicated. Our Wasted Tokens are the lowest-scoring 7% of C4 (14B of 200B tokens), all below the DCLM rejection threshold. We see these as distinct settings, where one targets the bulk of a filtered corpus, the other the tail that filtering itself discards.

Scale (re: Reviewer BUcx, model scale relative to Niklaus et al.). The 1.2B scale we referred to is that of the LQ vs. HQ source experiments in Niklaus et al., the experiments highlighted in the discussion. Their largest student there is 1.2B, comparable to our 1.4B models. We acknowledge that our 1.4B results are single-seed.

Parametric knowledge still matters (re: Reviewer BUcx, relevance of long-tail knowledge given retrieval, and NoKLB vs. MMLU). In Menlo Ventures' 2024 survey of 600 enterprise decision-makers, RAG had 51% adoption, so roughly half of deployments rely on parametric knowledge alone [1]. NoKLB questions are also encyclopedic rather than obscure web trivia (App. F, Table 6).

Over-training (re: Reviewer BUcx, epoch count in Fig. 6). Fig. 6 compares data mixtures at matched compute. That Retained + Upcycled has not completed an epoch by the time Retained has been repeated 7 times is the intended comparison, since it shows a larger unique pool against repetition at the same budget. Repeating the highest-quality data is also current practice (e.g., Olmo 3 resamples its top web data up to 7 times).

Synthesizer training (re: Reviewer BUcx, untrained-synthesizer baseline). App. C (Table 3) compares the untrained and trained synthesizer in our exact setup at 1.7B. We agree the same comparison at 360M would make Sec. 4.3 stronger.

We also agree with several points raised, especially the inconsistency between App. A and App. B (Reviewer BUcx), the value of a retained-data counterpart to NoKLB (Reviewer 3Tga), and the C1/C2 distinction (Reviewer 3Tga).

Thank you again for the thoughtful reviews and discussion.

References.

[1] T. Tully, J. Redfern, and D. Xiao. 2024: The State of Generative AI in the Enterprise. Menlo Ventures, November 2024. https://menlovc.com/2024-the-state-of-generative-ai-in-the-enterprise/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.