acceptodds
Under review as a conference paper at ICLR 2027

Don't Discard, Repair: A Repair-First Paradigm for Recovering Training Utility from Imperfect Pre-training Corpora

Abstract

Large language model (LLM) pre-training increasingly relies on large-scale corpora, yet existing data curation pipelines predominantly follow a filter-and-discard paradigm: documents failing quality criteria are removed rather than assessed for recoverability. We show that this paradigm conflates structural corruption (e.g., encoding, parsing, and layout errors) with low semantic quality, leading to the discard of potentially useful training data—a limitation that is especially consequential in data-scarce settings where such data may be unique and irreplaceable. We introduce Repair-First, a data curation paradigm for recovering training utility from imperfect pre-training corpora through reversible repair before irreversible filtering. Repair-First makes three contributions: (1) it establishes structural integrity as a missing dimension of LLM pre-training data quality and introduces repair-before-filtering as a new curation paradigm; (2) it develops a reversible, mixed-content-aware repair framework whose bounded, logged edits restore structurally corrupted corpora without generative rewriting; and (3) it systematically demonstrates that structural repair improves LLM pre-training utility and exposes limitations of existing quality assessment methods. A controlled perturbation benchmark and fixed-budget experiments with Qwen3-1.7B across multiple Nemotron-CC quality regimes reveal three findings: (1) structural corruption constitutes a distinct failure mode in LLM data curation, with existing quality estimators exhibiting systematic blind spots, especially for layout-level anomalies; (2) reversible repair recovers measurable training utility from imperfect corpora; and (3) Repair-First improves corpus utilization across diverse quality regimes while complementing existing filtering pipelines. Our findings suggest that future LLM data curation should not only evaluate whether data is high quality, but also assess whether apparently low-quality data contains recoverable training value before irreversible deletion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Pre-training Data, Data Curation, Data Quality, Corpus Construction, Data-Centric Machine Learning, Corpus Repair, Foundation Models, Language Model Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oZhEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a method with some rules to "repair" low quality data so that they can become more useful in LLM pre-training. A set of heuristic rules for such repair has been proposed, following a pipelined framework shown in Figure 2. The proposed method has then been evaluated in pretraining a Qwen3-1.7B model with 34.4B tokens. The results show some improvement in the overall performance when using the proposed repair method compared to without it.

Critical strengths
  • The paper could be of practical relevance for teams who pretrain LLMs.
  • The description is detailed enough to represent the proposed approach.
  • The experiment includes pretraining a real model.
Critical weaknesses
  • The paper is poorly structured. For example, RQ1 appears in its own section while RQ2 and RQ3 appear as part of the results.
  • The paper is mostly an engineering description. Some important discussions such as why the algorithm is designed in this way instead of some other way are essential for research papers but are largely missing.
  • The proposed approach may not be difficult to come up with by an engineer experienced in pretraining. Some aspects of it may also already be included in text extractors (or could be easily included there) used in practice.
  • Even from an engineering perspective, important aspects such as the compute it requires to run the repair pipeline are missing from the paper. This is important since such data processors need to be very lightweight to be able to process the huge amount of data that is used for training practical LLMs, which the current experiment with the 1.7B parameter model and 34.4B tokens may not be facing.
Questions for authors
  • Could you improve the structure of the paper?
  • Can you explain the core principles and reasons behind the design, including why these rules/algorithms instead of others?
  • What is the novel contribution that is non-obvious for a hands on person?
  • How is the compute consumption of this method compared to commonly used data processing pipelines that filter out bad quality data?
Reviewer_YS8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether pre-training data that appears to be low quality can still be useful after repairing structural issues. Rather than removing these documents through filtering directly, the authors propose a ten-stage pipeline Repair-First that identifies and corrects recoverable errors while protecting code, mathematical expressions, and other structured content from unintended modifications. The approach relies on localized, reversible edits instead of LLM-based rewriting. The authors first investigate how existing data quality estimators respond to different types of structural corruption, finding that some errors are poorly detected or even incorrectly favored. They then evaluate the impact of repair through controlled pre-training experiments, showing performance improvements across different data quality tiers, particularly on evaluations involving connected text and longer passages.

Critical strengths
  • The paper tackles an interesting and practical problem. Instead of simply throwing away low-quality data, it explores whether some of that data can still be useful after fixing formatting and structural errors.

  • The method takes a systematic approach to repairing different types of structural errors while preserving the original content. Its use of content-aware processing, conservative edits, and validation makes the repair process more reliable and easier to audit.

  • The overall paper is well-written and easy to follow.

Critical weaknesses
  • The paper discusses related work such as FineWeb2, ProX, and REWIRE, but does not directly compare against these methods in the experiments. Most results are compared with raw data, making it hard to tell whether the proposed approach is actually better than existing cleaning methods, especially simpler ones.

  • Many steps in the pipeline use existing text cleaning and correction techniques. It is not clear how much of the improvement comes from the proposed design rather than these existing tools. More detailed ablations would help show which steps are actually necessary.

  • The experiments only use a 1.7B model, and each repair setting is tested with a single training seed. More experiments are needed to show whether the improvements are consistent and extend to larger models.

  • The ten-stage pipeline adds extra work to data preprocessing, but the paper does not report detailed runtime costs for each stage or compare the overall cost with simpler alternatives.

Questions for authors
  • Which components of the ten-stage pipeline contribute most significantly to downstream performance? Can the authors provide component-level ablations?

  • What are the end-to-end preprocessing costs and throughput of Repair-First, and how do they compare with simpler cleaning or filtering strategies?

  • How does Repair-First compare with existing data cleaning pipelines under the same pre-training budget? Can simpler combinations of standard cleaning tools achieve similar improvements without the full ten-stage repair process?

Reviewer_s8t1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a new data curation paradigm REPAIR-FIRST, that recovers training utility from imperfect corpora that existing data curation pipelines may discard. Where existing paradigms may discard potentially useful training data, REPAiR-FIRST makes the distinction between structural corruption and semantic quality and argues for structural integrity to have its own curation axis.

The method is reversible and uses bounded, logged edits to restore structurally corrupted corpora without the use of generative rewriting.

Controlled fixed-budget pre-training experiments with a Qwen 3 1.7B model using retrieved texts from Common Crawl guided by Nemotron CC quality tiers shows wins across all 3 quality tiers with the lowest quality tier benefitting most from the method. Evaluations across 38 tasks reporting PPL or BPB also show +17 net wins.

Critical strengths
  1. The 10 stage pipeline for REPAIR-FIRST is sensible and well designed and the choices made are well motivated. Further, there is strong evidence showing the benefit of linguistic repair alongside surface repair across tasks and tiers.
  2. The method shows strong performance compared to raw in the LAMBADA and significant gains over a majority of the tasks measured with Paloma BPB .
  3. The supplemental sections are detailed and extensive and add meaningful arguments toward interpreting the results.
Critical weaknesses
  1. Essential portions of the pipeline are underspecified. Stage 8 and 9 have more exposition in the appendices but still critical details remain unseen and hence the pipelines sufficiently assessable.
  2. If repair before filtering is to be a new curation paradigm, that would require either a conventional cleaning control or something to establish the benefit of repairing before filtering. The results establish the benefits over untreated texts but the benefit over just standard pre-processing is unresolved.
  3. The presentation could be improved, the writing is dense in some areas and the paper moves freely between intentions, outcomes and interpretations without adequately distinguishing them.
Questions for authors
  1. Appendices B.4 and E defer some critical implementation details (stage 8 and 9) to the experiment code, can you please share these?
  2. Does there exist a comparison to conventional cleaning on the same set of source texts? If not, maybe the conclusion can distinguish the demonstrated repair benefit over raw from the proposed broader benefit over standard processes and repair before filtering.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.