- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies which source documents should be selected for rewriting in data-constrained language model pretraining. It compares five strategies, including quality-based selection, random selection, filtering after rewriting, topic-balanced selection, and selection based on disagreement among quality scorers. The authors train 1.5B models on mixtures of original and rewritten text over three epochs. Results show that early commonsense gains from rewriting largely diminish with repeated training. Topic-balanced selection achieves the best average commonsense performance, while disagreement-aware selection achieves the highest MMLU scores in later epochs. The study suggests that selecting high-scoring source documents alone is insufficient, and that the benefits of rewriting depend on both source selection and training duration.
Critical strengths
- The paper addresses a practical and well-motivated question: which documents are worth rewriting when the generation budget is limited. This focus makes the study relevant to improving data efficiency in pretraining.
- Evaluating models across three training epochs provides useful insights into how rewriting benefits change with repeated data use. The results show that improvements observed early in training may not persist, highlighting a limitation of evaluation at a single training stage.
- The analysis of quality scorers and topic distributions provides useful evidence that different selection criteria favor different types of content. The topic-balanced and disagreement-aware strategies offer simple, interpretable approaches for investigating how these differences affect downstream performance.
Critical weaknesses
- The contribution needs to be positioned more precisely relative to prior work. Existing studies have already examined how rewriting affects pretraining performance. For example, Synthetic Rewriting as a Quality Multiplier [1] investigates the interaction between source quality and rewriting benefits, while FinePhrase [2] systematically studies rewriting prompts, generator models, and source data. Although the paper discusses these studies, it should more clearly distinguish its specific contribution from findings already established in the literature.
- The experimental scope limits the generalizability of the conclusions. All rewriting relies on Qwen2.5-7B-Instruct, a relatively dated generator whose particular errors, stylistic preferences, and compression behavior may influence the observed results. The preliminary comparison within the Qwen2.5 family does not establish robustness across generators. Furthermore, all downstream pretraining experiments use a 1.5B model, whereas related work such as REWIRE [3] includes experiments at the 7B scale. Testing a more recent generator and a larger pretrained model would strengthen the claims, particularly whether the strategy rankings and diminishing gains from rewriting persist across model scales.
- Insufficient control of experimental variables makes it difficult to extract clear, actionable insights. Several factors change simultaneously across strategies. For example, Diversity-Oriented introduces both topic quotas and a three-scorer consensus, while Quality-First uses only fastText. Disagreement-Aware also changes the candidate pool and filtering rules, yet no downstream experiment with λ = 0 isolates the contribution of the disagreement bonus. Moreover, the proportion of retained distillation outputs varies from 16.9% to 53.1% across strategies, introducing differences in the final rewriting mixture. Consequently, the observed performance differences cannot be confidently attributed to the intended selection principles. Controlled ablations that independently vary topic constraints, scorer aggregation, and the disagreement bonus while matching the rewriting mixture would provide more convincing evidence.
[1] Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining
[2] How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
[3] Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
Questions for authors
- What specific new insights does this work provide beyond prior studies on source quality and rewriting strategies? In particular, how do the findings about source selection and repeated training extend existing results?
- Do the strategy rankings and diminishing benefits of rewriting persist when using a newer rewriting model or training a larger model, such as 7B? How dependent are the conclusions on Qwen2.5-7B-Instruct and the 1.5B training setup?
- Can the authors provide controlled ablations to isolate the contributions of topic quotas, scorer aggregation, and the disagreement bonus? For example, what happens when comparing λ = 0 with λ = 0.5, or adding topic quotas while keeping the quality scorer and rewriting mixture fixed?
5 comments
Response to Reviewer LNbW (2/3): Generalizability Across Generators and Model Scales (W2, Q2)
(1) Our scale is comparable to prior controlled studies.
WRAP [5] evaluates models up to approximately 1.3B parameters, FinePhrase [1] trains 1.2B models on approximately 21B tokens per ablation, and REWIRE [4] evaluates 1B, 3B, and 7B models. We train 1.5B models on 30B tokens across three seeds and three epochs, enabling systematic comparison under matched training budgets. We are also extending the experiments to 3B-parameter models and will include those results in the revised paper.
(2) Our preliminary evaluation supports the choice of rewriter.
We selected Qwen2.5-7B-Instruct based on Appendix C. Within the Nemotron prompt family, the 7B rewriter shows a 5% major hallucination rate and negligible truncation, compared with a 21% major hallucination rate and 8% truncation for the 1.5B rewriter. The same rewriter and prompt templates are used across Quality-First, Diversity-Oriented, and Disagreement-Aware, so that comparisons isolate the source-selection policy.
(3) Scope of our findings.
Our findings identify source selection and repeated training as important factors in evaluating rewriting pipelines, and the new raw-data controls show that carefully selected raw documents can outperform quality-first rewriting on certain benchmarks without any generation compute. Robustness across generator families and larger model scales remains to be established, and we plan to extend the experiments to 3B. Our central claims concern the importance of source selection under constrained training budgets, not the superiority of any particular rewriting model. We will state these conditions and limitations in the revised paper.
References
[1] Niklaus, J., et al. (2026). How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. arXiv:2604.13977. https://arxiv.org/abs/2604.13977
[4] Nguyen, T., et al. (2025). Recycling the Web: A Method to Enhance Pre-Training Data Quality and Quantity for Language Models. arXiv:2506.04689. https://arxiv.org/abs/2506.04689
[5] Maini, P., et al. (2024). Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024, pp. 14044–14072. https://doi.org/10.18653/v1/2024.acl-long.757
Response to Reviewer LNbW (3/3): Controlled Ablations and Experimental Variables (W3, Q3)
(1) Disagreement bonus: the requested λ = 0 ablation is complete.
We ran the requested λ = 0 versus λ = 0.5 comparison during the rebuttal period, with all other components fixed, across three seeds and three epochs. The two settings achieve comparable downstream performance, with no consistent advantage for either. This is expected at the current scale, where they share 88.6% of their selected documents (Appendix D, Figure 9). Our purpose in designing Disagreement-Aware was to obtain a source set that differs substantially from those of Quality-First and Diversity-Oriented, and we chose λ = 0.5 for that contrast rather than tuning it for downstream performance. We expect the bonus to matter more with a larger candidate pool, where it changes a larger share of the selection, and we leave a dedicated study of λ to future work. We will add the full results to the appendix and state this in the Limitations.
(2) Our Diversity-Oriented strategy jointly considers quality and topic coverage.
Section 3 shows that individual quality scorers have different ranking preferences and systematic topic biases. Diversity-Oriented (Section 4.2) extends single-scorer quality selection in response: three-scorer consensus reduces reliance on any one scorer, and topic-proportional quotas preserve coverage under a fixed budget. The two are combined deliberately to obtain a source set whose distribution differs as much as possible from the fastText-based selections behind Quality-Base and Quality-First. Our source-selection analysis shows that quotas alone do not achieve this: applying topic quotas to fastText-based selection still gives 80.6% source-token overlap with Quality-First, whereas Diversity-Oriented has only 37.3% overlap. These comparisons characterize the selected source sets rather than each component's contribution to performance. Our objective is to evaluate the complete policy, not to attribute differences to topic quotas or scorer aggregation individually.
(3) The rewriting-prompt mixture varies naturally under our fixed-budget pipeline.
Our three main strategies use the same rewriter, the same two rewriting prompts, and approximately 10B source-token and 5B rewritten-output-token budgets. Both rewriting passes are generated over the selected source documents. Wikipedia-style outputs are retained first, and Distill outputs fill the remaining budget. Under this budget-constrained procedure, different source documents naturally yield different retained prompt proportions (24.4%–34.9% Distill). These proportions are outcomes of data construction, not hyperparameters tuned on downstream performance. We will clarify the composition of the retained prompts, actual generation costs, and budget implementation in the revised paper.
Overall, our study evaluates complete source-selection policies under matched rewriting procedures and training budgets, rather than claiming to isolate the contribution of every component. Further factorial ablations could add mechanistic insight but would not change the policy-level comparisons that are the focus of this work.
Response to Reviewer LNbW (1/3): Novelty and Contributions Beyond Prior Work (W1, Q1)
We thank the reviewer for the careful evaluation and constructive suggestions. We address the three critical weaknesses and their corresponding questions in three parts.
Prior rewriting pipelines mostly select or partition source documents by quality alone. In this paper, we argue that quality should not be the only criterion for source selection in rewriting pipelines, and that diversity should also be considered. This leads to three contributions beyond prior work:
(1) Beyond quality-based source selection.
FinePhrase [1] studies source corpora, prompts, and generators, while Almeida et al. [2] and Nemotron-CC [3] examine how rewriting interacts with document quality. We instead ask which individual source documents should be selected under a fixed rewriting budget, without assuming that quality-based selection is optimal. We compare quality-based, topic-aware, and scorer-disagreement policies under the same rewriting framework. Our rebuttal experiments strengthen this distinction: carefully selected raw documents can outperform Quality-First Rewritten on certain benchmarks, with no rewriting compute.
(2) Rewriting under data constraints and repeated training.
Each setting reuses its approximately 10B-token mixture for three epochs, allowing comparison at matched training horizons. For example, Quality-First starts with a +0.67-point mean commonsense advantage over Quality-Base, which declines to −0.21 by epoch 3. Our new raw-data controls further show that the advantages of source selection and of rewriting can evolve differently under repetition. This multi-epoch perspective goes beyond single-horizon comparisons and tests whether early gains persist as data is reused.
(3) Benchmark-dependent source selection, with and without rewriting.
Among the rewriting strategies, Diversity-Oriented leads on commonsense at all three epochs and Disagreement-Aware leads on MMLU at epochs 2 and 3, with some comparisons sensitive to seed variability. During the rebuttal period, we added matched non-rewritten controls for both strategies, across three seeds and three epochs. The same split holds without rewriting: at epoch 3, Diversity-Oriented Raw leads by +1.03 points on commonsense Mean6 and Disagreement-Aware Raw leads by +0.74 points on MMLU, consistent across all three paired seeds. Source selection itself therefore shapes downstream performance.
References
[1] Niklaus, J., et al. (2026). How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. arXiv:2604.13977. https://arxiv.org/abs/2604.13977
[2] Almeida, T. S., et al. (2026). Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining. arXiv:2603.24826. https://arxiv.org/abs/2603.24826
[3] Su, D., et al. (2025). Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. ACL 2025, pp. 2459–2475. https://doi.org/10.18653/v1/2025.acl-long.123