acceptodds
Under review as a conference paper at ICLR 2027

Know Your Sources: Data Selection Matters When Rewriting for Data-Constrained Pretraining

Abstract

High-quality data is a cornerstone of large language model pretraining, and document rewriting with instruction-tuned models offers a promising way to expand pretraining corpora amid data scarcity. Existing pipelines rewrite documents uniformly or gate rewrite candidates with a quality measure built for raw-data filtering. However, it is not evident whether quality scoring measures, built to select from web-data, are also suitable signals for corpus rewriting. We study this in the data-constrained pretraining setting, exactly where synthetic documents may be most valuable. We investigate by framing source selection as balancing two competing objectives: quality and diversity. We compare 5 strategies on this balance, including a novel Disagreement-Aware strategy that accounts for inter-scorer disagreements. We pretrain 1.5B-parameter models from scratch for a multi-epoch, 30B token trajectory, across multiple seeds for a total of 18 models. Diversity-Oriented selection ranks first on aggregated commonsense tasks at every epoch, and Disagreement-Aware selection leads on knowledge-intensive benchmarks by the final epoch. These findings establish source selection for rewriting as a problem distinct from raw-data filtering.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data, pretraining, large language models, rewriting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LNbWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which source documents should be selected for rewriting in data-constrained language model pretraining. It compares five strategies, including quality-based selection, random selection, filtering after rewriting, topic-balanced selection, and selection based on disagreement among quality scorers. The authors train 1.5B models on mixtures of original and rewritten text over three epochs. Results show that early commonsense gains from rewriting largely diminish with repeated training. Topic-balanced selection achieves the best average commonsense performance, while disagreement-aware selection achieves the highest MMLU scores in later epochs. The study suggests that selecting high-scoring source documents alone is insufficient, and that the benefits of rewriting depend on both source selection and training duration.

Critical strengths
  1. The paper addresses a practical and well-motivated question: which documents are worth rewriting when the generation budget is limited. This focus makes the study relevant to improving data efficiency in pretraining.
  2. Evaluating models across three training epochs provides useful insights into how rewriting benefits change with repeated data use. The results show that improvements observed early in training may not persist, highlighting a limitation of evaluation at a single training stage.
  3. The analysis of quality scorers and topic distributions provides useful evidence that different selection criteria favor different types of content. The topic-balanced and disagreement-aware strategies offer simple, interpretable approaches for investigating how these differences affect downstream performance.
Critical weaknesses
  1. The contribution needs to be positioned more precisely relative to prior work. Existing studies have already examined how rewriting affects pretraining performance. For example, Synthetic Rewriting as a Quality Multiplier [1] investigates the interaction between source quality and rewriting benefits, while FinePhrase [2] systematically studies rewriting prompts, generator models, and source data. Although the paper discusses these studies, it should more clearly distinguish its specific contribution from findings already established in the literature.
  2. The experimental scope limits the generalizability of the conclusions. All rewriting relies on Qwen2.5-7B-Instruct, a relatively dated generator whose particular errors, stylistic preferences, and compression behavior may influence the observed results. The preliminary comparison within the Qwen2.5 family does not establish robustness across generators. Furthermore, all downstream pretraining experiments use a 1.5B model, whereas related work such as REWIRE [3] includes experiments at the 7B scale. Testing a more recent generator and a larger pretrained model would strengthen the claims, particularly whether the strategy rankings and diminishing gains from rewriting persist across model scales.
  3. Insufficient control of experimental variables makes it difficult to extract clear, actionable insights. Several factors change simultaneously across strategies. For example, Diversity-Oriented introduces both topic quotas and a three-scorer consensus, while Quality-First uses only fastText. Disagreement-Aware also changes the candidate pool and filtering rules, yet no downstream experiment with λ = 0 isolates the contribution of the disagreement bonus. Moreover, the proportion of retained distillation outputs varies from 16.9% to 53.1% across strategies, introducing differences in the final rewriting mixture. Consequently, the observed performance differences cannot be confidently attributed to the intended selection principles. Controlled ablations that independently vary topic constraints, scorer aggregation, and the disagreement bonus while matching the rewriting mixture would provide more convincing evidence.

[1] Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining

[2] How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data

[3] Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

Questions for authors
  1. What specific new insights does this work provide beyond prior studies on source quality and rewriting strategies? In particular, how do the findings about source selection and repeated training extend existing results?
  2. Do the strategy rankings and diminishing benefits of rewriting persist when using a newer rewriting model or training a larger model, such as 7B? How dependent are the conclusions on Qwen2.5-7B-Instruct and the 1.5B training setup?
  3. Can the authors provide controlled ablations to isolate the contributions of topic quotas, scorer aggregation, and the disagreement bonus? For example, what happens when comparing λ = 0 with λ = 0.5, or adding topic quotas while keeping the quality scorer and rewriting mixture fixed?
5 comments
AuthorsOpenReview ↗

Response to Reviewer LNbW (1/3): Novelty and Contributions Beyond Prior Work (W1, Q1)

We thank the reviewer for the careful evaluation and constructive suggestions. We address the three critical weaknesses and their corresponding questions in three parts.

Prior rewriting pipelines mostly select or partition source documents by quality alone. In this paper, we argue that quality should not be the only criterion for source selection in rewriting pipelines, and that diversity should also be considered. This leads to three contributions beyond prior work:

(1) Beyond quality-based source selection.

FinePhrase [1] studies source corpora, prompts, and generators, while Almeida et al. [2] and Nemotron-CC [3] examine how rewriting interacts with document quality. We instead ask which individual source documents should be selected under a fixed rewriting budget, without assuming that quality-based selection is optimal. We compare quality-based, topic-aware, and scorer-disagreement policies under the same rewriting framework. Our rebuttal experiments strengthen this distinction: carefully selected raw documents can outperform Quality-First Rewritten on certain benchmarks, with no rewriting compute.

(2) Rewriting under data constraints and repeated training.

Each setting reuses its approximately 10B-token mixture for three epochs, allowing comparison at matched training horizons. For example, Quality-First starts with a +0.67-point mean commonsense advantage over Quality-Base, which declines to −0.21 by epoch 3. Our new raw-data controls further show that the advantages of source selection and of rewriting can evolve differently under repetition. This multi-epoch perspective goes beyond single-horizon comparisons and tests whether early gains persist as data is reused.

(3) Benchmark-dependent source selection, with and without rewriting.

Among the rewriting strategies, Diversity-Oriented leads on commonsense at all three epochs and Disagreement-Aware leads on MMLU at epochs 2 and 3, with some comparisons sensitive to seed variability. During the rebuttal period, we added matched non-rewritten controls for both strategies, across three seeds and three epochs. The same split holds without rewriting: at epoch 3, Diversity-Oriented Raw leads by +1.03 points on commonsense Mean6 and Disagreement-Aware Raw leads by +0.74 points on MMLU, consistent across all three paired seeds. Source selection itself therefore shapes downstream performance.

References

[1] Niklaus, J., et al. (2026). How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. arXiv:2604.13977. https://arxiv.org/abs/2604.13977

[2] Almeida, T. S., et al. (2026). Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining. arXiv:2603.24826. https://arxiv.org/abs/2603.24826

[3] Su, D., et al. (2025). Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. ACL 2025, pp. 2459–2475. https://doi.org/10.18653/v1/2025.acl-long.123

AuthorsOpenReview ↗

Response to Reviewer LNbW (2/3): Generalizability Across Generators and Model Scales (W2, Q2)

(1) Our scale is comparable to prior controlled studies.

WRAP [5] evaluates models up to approximately 1.3B parameters, FinePhrase [1] trains 1.2B models on approximately 21B tokens per ablation, and REWIRE [4] evaluates 1B, 3B, and 7B models. We train 1.5B models on 30B tokens across three seeds and three epochs, enabling systematic comparison under matched training budgets. We are also extending the experiments to 3B-parameter models and will include those results in the revised paper.

(2) Our preliminary evaluation supports the choice of rewriter.

We selected Qwen2.5-7B-Instruct based on Appendix C. Within the Nemotron prompt family, the 7B rewriter shows a 5% major hallucination rate and negligible truncation, compared with a 21% major hallucination rate and 8% truncation for the 1.5B rewriter. The same rewriter and prompt templates are used across Quality-First, Diversity-Oriented, and Disagreement-Aware, so that comparisons isolate the source-selection policy.

(3) Scope of our findings.

Our findings identify source selection and repeated training as important factors in evaluating rewriting pipelines, and the new raw-data controls show that carefully selected raw documents can outperform quality-first rewriting on certain benchmarks without any generation compute. Robustness across generator families and larger model scales remains to be established, and we plan to extend the experiments to 3B. Our central claims concern the importance of source selection under constrained training budgets, not the superiority of any particular rewriting model. We will state these conditions and limitations in the revised paper.

References

[1] Niklaus, J., et al. (2026). How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. arXiv:2604.13977. https://arxiv.org/abs/2604.13977

[4] Nguyen, T., et al. (2025). Recycling the Web: A Method to Enhance Pre-Training Data Quality and Quantity for Language Models. arXiv:2506.04689. https://arxiv.org/abs/2506.04689

[5] Maini, P., et al. (2024). Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024, pp. 14044–14072. https://doi.org/10.18653/v1/2024.acl-long.757

AuthorsOpenReview ↗

Response to Reviewer LNbW (3/3): Controlled Ablations and Experimental Variables (W3, Q3)

(1) Disagreement bonus: the requested λ = 0 ablation is complete.

We ran the requested λ = 0 versus λ = 0.5 comparison during the rebuttal period, with all other components fixed, across three seeds and three epochs. The two settings achieve comparable downstream performance, with no consistent advantage for either. This is expected at the current scale, where they share 88.6% of their selected documents (Appendix D, Figure 9). Our purpose in designing Disagreement-Aware was to obtain a source set that differs substantially from those of Quality-First and Diversity-Oriented, and we chose λ = 0.5 for that contrast rather than tuning it for downstream performance. We expect the bonus to matter more with a larger candidate pool, where it changes a larger share of the selection, and we leave a dedicated study of λ to future work. We will add the full results to the appendix and state this in the Limitations.

(2) Our Diversity-Oriented strategy jointly considers quality and topic coverage.

Section 3 shows that individual quality scorers have different ranking preferences and systematic topic biases. Diversity-Oriented (Section 4.2) extends single-scorer quality selection in response: three-scorer consensus reduces reliance on any one scorer, and topic-proportional quotas preserve coverage under a fixed budget. The two are combined deliberately to obtain a source set whose distribution differs as much as possible from the fastText-based selections behind Quality-Base and Quality-First. Our source-selection analysis shows that quotas alone do not achieve this: applying topic quotas to fastText-based selection still gives 80.6% source-token overlap with Quality-First, whereas Diversity-Oriented has only 37.3% overlap. These comparisons characterize the selected source sets rather than each component's contribution to performance. Our objective is to evaluate the complete policy, not to attribute differences to topic quotas or scorer aggregation individually.

(3) The rewriting-prompt mixture varies naturally under our fixed-budget pipeline.

Our three main strategies use the same rewriter, the same two rewriting prompts, and approximately 10B source-token and 5B rewritten-output-token budgets. Both rewriting passes are generated over the selected source documents. Wikipedia-style outputs are retained first, and Distill outputs fill the remaining budget. Under this budget-constrained procedure, different source documents naturally yield different retained prompt proportions (24.4%–34.9% Distill). These proportions are outcomes of data construction, not hyperparameters tuned on downstream performance. We will clarify the composition of the retained prompts, actual generation costs, and budget implementation in the revised paper.

Overall, our study evaluates complete source-selection policies under matched rewriting procedures and training budgets, rather than claiming to isolate the contribution of every component. Further factorial ablations could add mechanistic insight but would not change the policy-level comparisons that are the focus of this work.

Reviewer_G6YdOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper investigates which documents to rewrite for LLM pretraining. The authors approach this from the perspective of the trade-off between quality and diversity, as well as the effect of using rewritten documents in training for multiple epochs. Their main experiments entail training small models with quality-filtered documents + documents selected for rewriting using 5 different methods (and one baseline selected based on quality alone). They find that the 1st epoch behaviour is not descriptive of final performance for rewritten documents, and that selecting documents for rewriting is different from quality selection.

Critical strengths

S1: The topic is timely and well handled (outside of the weakness). For example, using the topic labels to show the disagreement between quality classifiers is interesting, and it visualizes the drastic differences between them. The multi-epoch results are especially intriguing and worth taking into account in practice.

S2: Data selection, model sizes, the scale of experiments, etc., are ambitious; figures are informative, and the motivation for experiments is sound and comprehensive.

Critical weaknesses

W1: This paper seems to conflate the selection of data for rewriting with the selection of pretraining data. Only one baseline is used, so it is unclear whether the variation in evaluation scores is due to rewriting or because of what was included/omitted in the differing 5BT selected for pretraining. This could be fixed with additional baselines, but as a substantial part of the results rely on these results, I consider this a major weakness.

EDIT: major weaknesses resolved in rebuttal.

Questions for authors

None.

4 comments
AuthorsOpenReview ↗

Response to Reviewer G6Yd (1/2): Matched Raw-Data Baselines

We thank the reviewer for highlighting this important distinction. We agree that comparisons against a single shared raw-data baseline cannot fully separate the effects of source selection and rewriting.

To directly address this concern, we have completed two additional matched raw-data baselines for our Diversity-Oriented and Disagreement-Aware strategies. Each baseline trains on the strategy's selected raw documents without rewriting, under the same architecture, training-token budget, and evaluation protocol, across three seeds and three training horizons (10B, 20B, and 30B tokens). This adds 6 training runs and 18 evaluated checkpoints. We are also extending the experiments to 3B-parameter models and will include those results in the revised paper.

1. Separating Source-Selection Gains from Rewriting Gains

Let BB denote the original fastText Quality-Base, SS the corresponding raw-selected baseline, and RR the rewritten strategy. We decompose the observed performance difference as:

R−B⏟Observed gain=(S−B)⏟Source-selection contrast+(R−S)⏟Incremental rewriting contrast.\underbrace{R-B}_{\text{Observed gain}} = \underbrace{(S-B)}_{\text{Source-selection contrast}} + \underbrace{(R-S)}_{\text{Incremental rewriting contrast}}.

Here S−BS-B is the effect of changing the raw source selection, and R−SR-S is the incremental effect of rewriting those same sources. The decomposition is descriptive and does not claim that either term isolates a single causal mechanism.

Table 1. Performance differences across three epochs (E1 / E2 / E3). All values are accuracy differences in percentage points, averaged over three seeds. Positive values indicate improvements.

StrategyBenchmarkRaw − Quality-BaseRewritten − Raw
Diversity-OrientedCommonsense Mean6+0.73 / +0.52 / +0.58+0.03 / +0.08 / −0.43
Diversity-OrientedMMLU−0.20 / +0.37 / −0.23+0.23 / +0.03 / +0.45
Disagreement-AwareCommonsense Mean6+0.52 / −0.04 / −0.45−0.16 / +0.06 / +0.11
Disagreement-AwareMMLU+0.09 / +0.60 / +0.51+0.07 / +0.00 / −0.02

These additional comparisons lead to two key findings.

2. Finding 1: Source Selection Alone Can Outperform Quality-First Rewriting

Diversity-Oriented Raw consistently outperforms Quality-Base on commonsense across all three epochs, by +0.73, +0.52, and +0.58 points, with positive differences across all three seeds at every epoch. Similarly, Disagreement-Aware Raw outperforms Quality-Base on MMLU at epochs 2 and 3, by +0.60 and +0.51 points, respectively, again with positive differences across all three seeds. More importantly, these raw-selected baselines can also outperform Quality-First Rewritten, despite requiring no additional rewriting:

  • Diversity-Oriented Raw achieves higher mean commonsense accuracy than Quality-First Rewritten across all three epochs. The advantages are more pronounced at epochs 2 and 3, reaching +0.37 and +0.79 points, respectively.
  • Disagreement-Aware Raw achieves higher mean MMLU accuracy than Quality-First Rewritten at epochs 2 and 3, with a +0.46-point advantage at epoch 3.

These findings demonstrate that careful source selection can yield stronger downstream performance than quality-first rewriting, even without additional generation compute, depending on the target benchmark.

Importantly, this observation is consistent with our original motivation: selecting sources solely by a single quality ranking is not necessarily the most effective way to allocate a limited data or rewriting budget.

3. Finding 2: Rewriting Provides Additional, Benchmark-Dependent Benefits

The matched raw-data controls also clarify when rewriting provides benefits beyond source selection.

For Diversity-Oriented, rewriting improves final-epoch MMLU by +0.45 ± 0.20 points over its corresponding raw baseline, with positive paired differences across all three seeds. In contrast, the raw counterpart achieves higher commonsense accuracy at the final epoch (46.85 vs. 46.41).

For Disagreement-Aware, raw and rewritten models achieve nearly identical MMLU performance at epochs 2 and 3 (30.13 vs. 30.13 and 31.10 vs. 31.08, respectively). This indicates that much of its MMLU advantage over Quality-Base is already present in the selected raw data.

The two raw baselines also exhibit distinct benchmark preferences. At epoch 3, Diversity-Oriented Raw exceeds Disagreement-Aware Raw on commonsense by +1.03 points, while Disagreement-Aware Raw exceeds Diversity-Oriented Raw on MMLU by +0.74 points. Both differences are consistent in direction across all three paired seeds. Thus, the benchmark-dependent differences between these selection strategies are already observable without rewriting.

Together, these results show that source selection and rewriting have distinguishable effects: source selection can provide substantial improvements independently, while the additional value of rewriting depends on the downstream benchmark and training horizon.

4. Refined Conclusions and Paper Revisions

These additional matched raw-data controls strengthen the empirical foundation of our study and provide a more precise interpretation of our original findings. They show that source selection contributes substantially to the observed performance differences, while rewriting provides additional benefits in certain benchmark settings. Accordingly, we will refine our original claims to distinguish the effects associated with source selection from the incremental benefits of rewriting.

Quality-guided data curation is a common component of existing rewriting pipelines, whether through quality-based source selection, quality-dependent transformations, or post-rewriting filtering [1, 2, 3]. However, our additional controls highlight why the benefits of source selection should be distinguished from those of rewriting itself.

Our results lead to two practical takeaways:

Takeaway 1: The early advantages of quality-first rewriting diminish with repeated training.

Quality-First initially outperforms the fastText-based Quality-Base by +0.67 percentage points on commonsense, but this advantage declines to −0.21 points by the third epoch. This suggests that the gains of a quality-first rewriting pipeline over a quality-based raw baseline may not persist as training data is repeatedly reused.

Takeaway 2: Better source selection can outperform quality-first rewriting, even without rewriting.

As shown in Finding 1, raw documents chosen by Diversity-Oriented or Disagreement-Aware selection can outperform Quality-First Rewritten on their respective benchmarks, with no generation compute. At the same time, rewriting still adds benchmark-dependent gains, such as +0.45 MMLU points for Diversity-Oriented at the final epoch.

Together, these findings suggest that selecting the right source documents can be more consequential than rewriting quality-ranked documents, and that the preferred strategy depends on the downstream objective and training horizon. We will add these results to the corresponding sections of the paper, with complete per-seed and per-task results in the Appendix.

References

[1] Su, D., et al. (2025). Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), pp. 2459–2475. https://doi.org/10.18653/v1/2025.acl-long.123

[2] Nguyen, T., et al. (2025). Recycling the Web: A Method to Enhance Pre-Training Data Quality and Quantity for Language Models. arXiv:2506.04689. https://arxiv.org/abs/2506.04689

[3] Almeida, T. S., et al. (2026). Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining. arXiv:2603.24826. https://arxiv.org/abs/2603.24826

Reviewer_G6YdOpenReview ↗

Response to Authors

Thank you for addressing my review. You have run extremely diligent experiments that address my main concern. I think your paper is much stronger with these results, and I am happy to increase the soundness score and the recommendation.

My only remaining concern is that these new results will change the manuscript substantially, and a partial rewrite of the experiments and conclusions is needed. Will you be able to fit all of this into one additional page and the appendices? I am inclined to believe you do, but I can also allow for the AC's discretion on this.

Thank you again for the opportunity to read your work.

AuthorsOpenReview ↗

Round 2 Response to Reviewer G6Yd: Paper Revisions and 3B Scaling

Thank you so much for reconsidering your assessment and for your encouraging feedback!

We agree that the new results need to be properly reflected in the paper. We believe we can fit the main findings and a compact comparison table into the additional page, with the detailed experiments and analyses in the Appendix. We will also revise our conclusions to better distinguish the effects of source selection and rewriting.

We have also managed to secure some additional GPU resources and are working to complete the 3B experiments. We plan to include these results in the revised paper as well.

Thank you again for your constructive suggestions!

Reviewer_RmX7OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies which source documents should be prioritized for rewriting when the rewriting budget is limited in data-constrained language model pretraining. It compares five source-selection strategies with a non-rewritten baseline, including Diversity-Oriented and Disagreement-Aware selection. Experiments use 1.5B-parameter models trained from scratch for approximately 30B tokens across three epochs and multiple random seeds. The authors report that different benchmark families favor different selection strategies and analyze how quality-scorer disagreement and topic distributions relate to these outcomes.

Critical strengths
  1. The paper addresses a useful data-curation question: how to allocate a limited rewriting budget across source documents. Treating source selection as a distinct decision highlights a practically relevant trade-off between document quality and corpus diversity.

  2. The experimental framework provides useful empirical evidence through matched training-token budgets, multiple seeds, and multi-epoch evaluation. The analysis of scorer disagreement and topic distributions also helps explain how selection policies differ and why their benefits may depend on the benchmark family.

Critical weaknesses
  1. The evidence for stable advantages remains limited. Diversity-Oriented selection improves final-epoch commonsense accuracy by 0.14 points with a standard deviation of 0.86, and the improvement becomes negative when either HellaSwag or PIQA is removed from the aggregate. The final-epoch MMLU improvement of Disagreement-Aware selection is 0.49 points with a standard deviation of 0.42. These results provide interesting exploratory observations, but uncertainty across seeds and benchmark composition limits the strength of practical recommendations.

  2. The contribution of the selection criteria is not sufficiently isolated. Disagreement-Aware selection combines consensus quality, a disagreement bonus, and candidate constraints without a downstream comparison against the same framework with the disagreement coefficient set to zero. Differences in rewriting composition and generation costs further complicate some strategy comparisons. Moreover, non-rewritten counterparts using the same selected source subsets are needed to distinguish the effects of source selection from those of rewriting itself. These controls directly affect the interpretation of the central findings.

Post-rebuttal update Thank you for the response and the additional analyses. After reviewing the rebuttal, I still find the evidence insufficient to support the paper’s main conclusions. I consider this concern substantial enough to warrant rejection and have revised my rating from 2 to 1.

Questions for authors
  1. Which strategy comparisons remain supported by paired uncertainty estimates and leave-one-benchmark-out analyses? Given the reported seed variability and aggregate sensitivity, how narrowly should the practical recommendations be interpreted?

  2. Does the disagreement bonus improve downstream performance over the same consensus-quality framework with its coefficient set to zero, while keeping the candidate pool, quality floor, and other constraints fixed?

  3. How sensitive are the strategy rankings to rewriting-prompt composition and total generation cost, including oversampling and discarded outputs? Which comparisons isolate source selection under the same rewriting recipe and generation budget?

  4. Have non-rewritten counterparts been evaluated using the same source subsets selected by the proposed strategies and matched training-token budgets? Without these controls, what evidence distinguishes rewriting-specific selection benefits from the benefits of selecting different raw documents?

6 comments
AuthorsOpenReview ↗

Response to Reviewer RmX7 (1/4): Statistical Robustness and Scope of Conclusions (W1, Q1)

Thank you for your thorough feedback on our paper. We have structured our response to your main concerns around five points.

We agree that the conclusions need to be more precise. We extended the paired-seed analysis to all 15 strategy pairs at each epoch and performed leave-one-benchmark-out analysis on the commonsense tasks. We focus on comparisons in which all three seeds show positive differences and the mean paired difference exceeds its standard deviation.

(1) Paired-seed analysis: Strategy differences remain consistent.

On commonsense, Diversity-Oriented outperforms the other four rewriting strategies at epochs 2 and 3 by 0.35–0.78 percentage points. All eight comparisons (2 epochs × 4 strategies) show positive differences across all three seeds, with mean improvements exceeding their paired standard deviations.

On MMLU, Disagreement-Aware outperforms Quality-Base at epochs 2 and 3 by 0.60 and 0.49 percentage points, respectively, with positive differences across all three seeds. However, its lead over Diversity-Oriented is less robust to seed variability, so we will narrow the scope of this ranking claim.

(2) Leave-one-benchmark-out: Diversity-Oriented maintains its mean lead.

At epochs 2 and 3, Diversity-Oriented maintains a positive mean advantage over the other four rewriting strategies across all 48 leave-one-benchmark-out comparisons (2 epochs × 4 strategy comparisons × 6 benchmarks). This suggests that its average advantage is not driven by any single benchmark, although seed-level variability remains.

(3) Multi-epoch results: Rewriting gains weaken with repetition.

On commonsense, all five rewriting strategies improve over Quality-Base by 0.2–0.8 percentage points at epoch 1, averaged across seeds. However, these gains largely diminish by epoch 3 (e.g., Diversity-Oriented: +0.14 ± 0.86 points). We therefore do not claim a reliable final-epoch commonsense improvement over Quality-Base. On MMLU, however, Diversity-Oriented and Disagreement-Aware maintain positive gains over Quality-Base at epochs 2 and 3 across all three seeds.

AuthorsOpenReview ↗

Response to Reviewer RmX7 (2/4): Isolating the Disagreement Bonus (W2, Q2)

We have completed the requested λ=0 ablation, with three seeds and three training horizons, adding 3 training runs and 9 evaluated checkpoints. The λ=0 and λ=0.5 settings share the candidate pool, quality floor, variance cap, source-selection budget, and rewriting recipe, and differ only in the disagreement coefficient of the selection utility:

uλ(d)=q(d)+λv(d).u_\lambda(d) = q(d) + \lambda \sqrt{v(d)}.

The two settings share 88.6% of their selected documents (Appendix D, Figure 9), so at the current scale the bonus mainly changes the selection near the ranking cutoff. The λ=0.5 value was fixed independently of downstream evaluation results.

The two settings achieve broadly comparable downstream performance, with no consistent advantage for either across benchmarks and epochs. At epoch 3, λ=0.5 is slightly higher on MMLU (31.08 vs. 30.98) and λ=0 is higher on commonsense Mean6 (46.21 vs. 45.93), both small relative to paired-seed variability. The disagreement bonus therefore does not provide a reliable additional benefit at the 1.5B scale, and the broader multi-scorer selection framework is the more appropriate level for interpreting our strategy comparisons. A larger candidate pool could make selection more sensitive to λ, but whether this yields stronger downstream gains remains an open question.

AuthorsOpenReview ↗

Response to Reviewer RmX7 (3/4): Controlled Source-Selection Comparisons (W2, Q3)

Our main question is whether quality-based source selection is the best way to allocate a limited rewriting budget. Our goal is to compare source-selection strategies rather than optimize individual rewriting prompts.

(1) Our main strategies share the same rewriting framework.

Quality-First, Diversity-Oriented, and Disagreement-Aware use the same rewriter (Qwen2.5-7B-Instruct), the same two Nemotron prompts, and approximately 10B-token source and 5B-token rewritten-output budgets. All three follow the same two-pass generation procedure, prioritizing Wikipedia-style outputs and supplementing them with Distill outputs. Unlike REWIRE-Inspired, these strategies do not perform additional source-pool oversampling.

(2) Prompt composition and generation costs reflect the budget-constrained pipeline.

Both rewriting passes are generated over the selected sources, and only the outputs needed to construct the final rewritten corpus are retained. Thus, some generated Distill outputs are discarded after budget-based assembly. Differences in selected source distributions, rewriting yields, and assembly ordering lead to variations in retained prompt proportions and actual generation costs. These were not independently tuned based on downstream performance. We will clarify these differences in the revised paper.

(3) WRAP-Inspired and REWIRE-Inspired serve as method-level baselines.

WRAP-Inspired and REWIRE-Inspired are adapted from Maini et al. (2024) [1] and Nguyen et al. (2025) [2], respectively. WRAP-Inspired uses different style prompts, while REWIRE-Inspired uses 2× source oversampling and post-rewrite filtering. We therefore treat them as method-level baselines rather than controlled source-selection comparisons.

Our conclusions concern the observed performance of complete selection policies under constrained data and training budgets; the sensitivity of rankings to independently matched prompt mixtures or generation costs remains to be established.

References

[1] Maini, P., et al. (2024). Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. ACL 2024.

[2] Nguyen, T., et al. (2025). Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models. COLM 2025.

[3] Li, J., et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. Advances in Neural Information Processing Systems (NeurIPS 2024), 37, Datasets and Benchmarks Track.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We have conducted additional experiments to strengthen our empirical analysis and clarify the contributions of source selection and rewriting.

Additional 1.5B Baselines

We completed two matched raw-data baselines for Diversity-Oriented and Disagreement-Aware. These experiments disentangle the effect of source selection from that of rewriting. We also completed a λ=0 ablation of Disagreement-Aware. In total, we add 9 training runs and 27 evaluated checkpoints, covering three seeds and three training horizons.

Key Finding: Better Source Selection Can Outperform Quality-First Rewriting, Even Without Rewriting

Quality-guided data curation is common in existing rewriting pipelines, where substantial improvements have been reported [1–3]. However, these improvements do not necessarily come from rewriting alone. Our new experiments show that carefully selected raw data can outperform Quality-First rewriting, even without additional generation compute.

Table 1. Selected raw data versus Quality-First rewriting across three epochs. Scores are averaged over three seeds; differences are in percentage points (pp) and computed from unrounded scores. Bold differences indicate comparisons with positive differences across all three seeds.

BenchmarkStrategy / ComparisonEpoch 1Epoch 2Epoch 3
Commonsense Mean6Quality-Base (Raw)41.7544.8046.27
Quality-First (Rewritten)42.4144.9646.06
Diversity-Oriented (Raw)42.4845.3246.85
Diversity Raw − Quality-First Rewritten+0.07+0.37+0.79
MMLUQuality-Base (Raw)28.1629.5330.59
Quality-First (Rewritten)28.5130.0130.64
Disagreement-Aware (Raw)28.2530.1331.10
Disagreement Raw − Quality-First Rewritten−0.26+0.12+0.46

These comparisons lead to two key takeaways.

Takeaway 1: The early advantages of quality-first rewriting diminish with repeated training.

Quality-First shows a mean commonsense advantage of +0.67 points over the fastText-based Quality-Base at epoch 1 (positive in two of three seeds), but this difference declines to −0.21 points by epoch 3.

Takeaway 2: Better source selection can outperform quality-first rewriting, even without rewriting.

Diversity-Oriented Raw outperforms Quality-First Rewritten on commonsense at epochs 2 and 3, with positive differences across all three seeds. Disagreement-Aware Raw also outperforms Quality-First Rewritten on MMLU at epoch 3, again across all three seeds.

Importantly, these results do not argue against rewriting. Its benefit depends on the downstream benchmark. Our matched controls show no consistent additional commonsense gains from rewriting the same selected sources, while Diversity-Oriented rewriting still improves final-epoch MMLU by +0.45 points over its matched raw baseline.

Together, these findings indicate that source selection is a first-order factor, often more consequential than rewriting quality-ranked documents, and that the best combination of selection and rewriting depends on the downstream objective.

We will incorporate these findings into the relevant sections of the revised paper and Appendix. Detailed analyses are provided in our reviewer-specific responses.

Scaling to 3B Parameters

We are also extending our experiments to 3B-parameter models, with source selection performed over a substantially larger candidate pool of approximately 600B tokens. We will share the additional results as the runs complete during the discussion period.

References

[1] Su, D., et al. (2025). Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. ACL 2025. https://doi.org/10.18653/v1/2025.acl-long.123

[2] Nguyen, T., et al. (2025). Recycling the Web: A Method to Enhance Pre-Training Data Quality and Quantity for Language Models. https://arxiv.org/abs/2506.04689

[3] Almeida, T. S., et al. (2026). Synthetic Rewriting as a Quality Multiplier: Evidence from Portuguese Continued Pretraining. https://arxiv.org/abs/2603.24826

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.