- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes to reuse the filtered-out tokens from pre-training datasets, because they do actually contain information that is useful. They use a small fine-tuned language model to rephrase pruned tokens. The rephraser model is fine-tuned using DPO on synthetically generated pairs of wasted tokens and their high scoring (quality and influence) rephrase. They have experiments to show the efficacy of upcycling wasted tokens and some ablations.
Critical strengths
The paper is well written and structured well. The idea is intuitive, to rephrase things that do not look important, but do indeed have crucial information. that rephrasing wasted tokens indeed improves performance as the pretrained model size increases, it can preserve knowledge that is present in the wasted tokens
Critical weaknesses
-
They don't have baselines besides their own ablations, so it's not clear whether their method of upcycling is helping or whether their experimental results are just an effect of adding more pre-training data. If I were writing this paper, I'd run Megadocs [1] and/or Rephrasing the Web [2] to show that using wasted tokens is as good (or, maybe better) than the summarization-based pretraining methods.
-
Can the authors explain why Retained + Retained is a low performing method in Table 1? It seems counter intuitive that it performs worse than Retrained + Wasted.
[1] Kim et al, Data-efficient pre-training by scaling synthetic megadocs [2] Maini et al, Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
Questions for authors
- In Section 4.1, what is the knowledge that the wasted/upcycled tokens contain?
2 comments
All my concerns are addressed. I've raised by score.
Thank you for the constructive review, and for finding the paper well written and the idea intuitive. We have uploaded a revised PDF (changes in blue), including two new external baselines, WRAP [9] and Megadocs [7] (Sec. 2.1, Figs. 1-2), and address your concerns below.
W1. Baselines.
TL;DR: We added WRAP [9] and Megadocs [7] as baselines. Upcycling outperforms both, despite their 4 times larger generators.
Our gains do not simply come from adding more data. Every mixture comparison is token-matched, with the same Retained tokens and total budget, and differs only in the source of the second half (i.e., Wasted, Upcycled, or Retained) (Sec. 2.1).
Our prompt follows [7], which adapts the Wikipedia-style prompt of WRAP [9] to full documents. Applying it to an off-the-shelf model is clearly weaker than training the synthesizer (App. C, Table 3), which motivates our approach. Training the synthesizer also improves the strongest recipe of [15] (Table) at both 670M and 1.4B (Sec. 4.2).
Following your suggestion, we add two external baselines applied to the same source documents that make up Wasted Tokens, each in its best-performing reported configuration (Sec. 2.1, App. G):
We train three seeds per scale from 19M to 670M and will extend to 1.4B in the camera-ready. Upcycled achieves lower loss than both baselines at every scale, alone and mixed with Retained, and matches or outperforms them on CORE (Figs. 1-2, App. I). At 670M, both baselines even trail raw Wasted data on held-out loss when mixed with Retained, suggesting that off-the-shelf rewriting does not reliably recover the rejected low-quality tail, while our trained synthesizer does.
W2. Why Retained+Retained scores low on NoKLB.
TL;DR: NoKLB probes knowledge that exists in filtered documents, which more high-quality data does not supply.
This is the main finding of Sec. 4.1. NoKLB measures knowledge contained in documents that quality filtering removes. Factual recall depends strongly on how often a fact appears in pretraining data [1, 5, 10], so filtering, which can remove content unevenly [2, 21], makes already rare facts even harder to learn. Retained+Retained trains on more high-quality data, but that data does not contain this knowledge, so the extra tokens bring little gain on NoKLB. In contrast, Retained+Wasted and Retained+Upcycled include the filtered documents, in raw or upcycled form, and therefore the models acquire such knowledge. The result shows that the filtered data carries knowledge complementary to the retained data, and that upcycling preserves it (Retained+Upcycled scores highest at both scales).
On held-out loss and CORE, Retained+Retained remains strongest or on par with Retained+Upcycled (Figs. 1 and 2), as expected. We made this explanation explicit in Sec. 4.1.
Q1. What knowledge do the wasted tokens contain?
NoKLB keeps only self-contained, durable, encyclopedic facts from Wasted documents with high educational value, each verified against external sources (App. F). We added representative questions with answers spanning computing, natural sciences, geography, history, public health, and psychology (App. F, Table 6).
These documents are discarded because of how they are written, not what they contain. The DCLM classifier [8] favors text resembling its instruction and ELI5-style positive examples, so informative content wrapped in boilerplate, catalog listings, or forum formatting scores low (App. H). [13] makes the same observation about classifier-based filtering. Upcycling keeps the content and fixes the form.
We hope this addresses your concerns and would be grateful if you would reconsider your score.