Know Your Sources: Data Selection Matters When Rewriting for Data-Constrained Pretraining

High-quality data is a cornerstone of large language model pretraining, and document rewriting with instruction-tuned models offers a promising way to expand pretraining corpora amid data scarcity. Existing pipelines rewrite documents uniformly or gate rewrite candidates with a quality measure built for raw-data filtering. However, it is not evident whether quality scoring measures, built to select from web-data, are also suitable signals for corpus rewriting. We study this in the data-constrained pretraining setting, exactly where synthetic documents may be most valuable. We investigate by framing source selection as balancing two competing objectives: quality and diversity. We compare 5 strategies on this balance, including a novel Disagreement-Aware strategy that accounts for inter-scorer disagreements. We pretrain 1.5B-parameter models from scratch for a multi-epoch, 30B token trajectory, across multiple seeds for a total of 18 models. Diversity-Oriented selection ranks first on aggregated commonsense tasks at every epoch, and Disagreement-Aware selection leads on knowledge-intensive benchmarks by the final epoch. These findings establish source selection for rewriting as a problem distinct from raw-data filtering.