Don't Discard, Repair: A Repair-First Paradigm for Recovering Training Utility from Imperfect Pre-training Corpora

Large language model (LLM) pre-training increasingly relies on large-scale corpora, yet existing data curation pipelines predominantly follow a filter-and-discard paradigm: documents failing quality criteria are removed rather than assessed for recoverability. We show that this paradigm conflates structural corruption (e.g., encoding, parsing, and layout errors) with low semantic quality, leading to the discard of potentially useful training data—a limitation that is especially consequential in data-scarce settings where such data may be unique and irreplaceable. We introduce Repair-First, a data curation paradigm for recovering training utility from imperfect pre-training corpora through reversible repair before irreversible filtering. Repair-First makes three contributions: (1) it establishes structural integrity as a missing dimension of LLM pre-training data quality and introduces repair-before-filtering as a new curation paradigm; (2) it develops a reversible, mixed-content-aware repair framework whose bounded, logged edits restore structurally corrupted corpora without generative rewriting; and (3) it systematically demonstrates that structural repair improves LLM pre-training utility and exposes limitations of existing quality assessment methods. A controlled perturbation benchmark and fixed-budget experiments with Qwen3-1.7B across multiple Nemotron-CC quality regimes reveal three findings: (1) structural corruption constitutes a distinct failure mode in LLM data curation, with existing quality estimators exhibiting systematic blind spots, especially for layout-level anomalies; (2) reversible repair recovers measurable training utility from imperfect corpora; and (3) Repair-First improves corpus utilization across diverse quality regimes while complementing existing filtering pipelines. Our findings suggest that future LLM data curation should not only evaluate whether data is high quality, but also assess whether apparently low-quality data contains recoverable training value before irreversible deletion.