acceptodds
Under review as a conference paper at ICLR 2027

RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

Abstract

The foundational capabilities of large language models (LLMs) are deeply influenced by the quality of their pretraining corpora. However, enhancing data quality at scale remains a significant challenge, primarily due to the trade-off between refinement effectiveness and processing efficiency. While rule-based filtering remains the dominant paradigm, it typically operates at the document level and lacks the granularity needed to refine specific content within documents. Inspired by emerging work such as ProX, we propose RefineX, a novel framework for large-scale, surgical refinement of pretraining data through programmatic editing tasks. RefineX enables efficient and fine-grained data refinement while reliably preserving the diversity and naturalness of raw text. The core strength of RefineX lies in its ability to distill high-quality, expert-guided end-to-end refinement results into minimal edit-based deletion programs. This high-precision distillation pipeline is used to train an efficient and reliable refine model that can systematically improve every instance in the corpus at scale. We evaluate RefineX across from-scratch pretraining at multiple model scales and find that it consistently outperforms models trained on raw, filtered, or alternatively refined data across diverse downstream tasks. On the 750M model, RefineX yields 2.6%-7.2% average gains on lighteval tasks, and achieves comparable performance using significantly fewer training tokens. Further analysis shows that RefineX reliably enhances text quality with both high efficiency and precision, outperforming prior approaches such as end-to-end generation and Prox-C. These results position RefineX as a scalable, effective, and reliable solution for optimizing pretraining data in modern LLM pipelines.

Primary area:
generative models
Keywords:
pretrain data, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RZaQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes improving language model pretraining data through deletion programs. A larger teacher first produces refined text, and the system extracts deletion operations by comparing it with the original document. These programs supervise a smaller model that refines a corpus at scale. The evaluation uses language models pretrained from scratch under matched token budgets and compares downstream performance and text quality with filtering and prior programmatic refinement methods.

Critical strengths
  • Generating text before extracting executable supervision is a practical alternative to asking a teacher to directly emit a custom program. The restricted deletion API makes edits inspectable and reduces output generation.
  • The central evidence is downstream performance after actual pretraining, rather than only a quality judge. Two model scales, multiple filtering pipelines, and ten tasks provide useful coverage.
  • Table 2 shows consistently better group-average results than Prox-C under matched training-token budgets, including 44.7 versus 43.5 in the Prox-D group.
  • The relationship to ProX is acknowledged, and the appendices provide prompts, filtering thresholds, model/training details, and editing examples.
Critical weaknesses
  • The implementation does not establish minimum Levenshtein edits. Section 3.2 describes minimal edit distance, but Appendix B.1 uses difflib/get_opcodes. SequenceMatcher does not guarantee minimal edit sequences, and its span-level replacements need not correspond to optimal character edits. Because only deletions are retained, alignment choices alter supervision. This is more than terminology, although correcting the description need not invalidate measured pretraining benefits. Python difflib
  • Deletion-only editing does not guarantee semantic preservation. Removing negation, conditions, units, or context can reverse meaning without introducing words. Table 4's new-word count therefore cannot establish absence of hallucination or semantic drift. Keeping only deletions from a broader teacher rewrite may also discard dependencies among edits.
  • The decisive supervision-pipeline ablation does not match teacher, seed documents, student, supervision quantity, and training budget. Prompting, context, filtering, and selection differences confound attribution. Several average improvements are modest and lack repeated-pretraining uncertainty. Reported percentages should be clearly distinguished from absolute points.
  • DataMan participates in seed stratification and evaluation. It is useful but not an independent guarantee of edit correctness or diversity. More documents per fixed token budget and better content within each document are also not disentangled.
  • Output-token ratios do not establish total efficiency. The reported teacher generation alone costs approximately 12,480 H800 GPU-hours. End-to-end accounting should include teacher/student training, prefill, decoding, execution, and downstream savings, with an amortization break-even point.
  • Execution semantics need clarification for changing line indices, overlapping chunks, conflicting deletions, skipped operations, and keep_all supervision after filtering small-deletion examples. Generalization beyond sub-billion-parameter models and general web text remains open. Data sampling and benchmark decontamination should be reproducible; no contamination is inferred here.
Questions for authors

Q1. Does supervision extraction use SequenceMatcher.get_opcodes or exact minimum Levenshtein alignment? If the former, what settings are used, and how much do labels and downstream results change under exact alignment?

Q2. Is there a direct-program versus text-then-program comparison matching teacher, documents, student, supervision size, and training budget? Can the current results separate this contribution from prompting, filtering, and context length?

Q3. What independent audit measures preservation of negation, units, conditions, and code semantics? How do you distinguish introducing no words from preserving factual meaning?

Q4. How many independent pretraining seeds support the main comparisons, what is the uncertainty of the central gains, and are the reported percentages relative improvements or absolute points?

Q5. Including teacher generation, student training, and corpus processing, what are the total cost and amortization break-even relative to Prox-C or simpler deletion models?

Q6. How does execution handle shifting line indices and overlapping/conflicting edits, and where does keep_all supervision come from after filtering small-deletion examples?

Reviewer_s1N8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RefineX, an LLM pretraining data refinement method. Unlike ProX, which directly generates refinement programs using expert models, RefineX derives edit-based deletion programs from expert-guided end-to-end refined texts and uses them to train a lightweight refinement model. Experimental results demonstrate that RefineX consistently outperforms ProX on both raw and various filtered pretraining datasets.

Critical strengths
  1. The proposed method introduces an intuitive yet clever improvement over ProX.
  2. From the perspective of evaluation coverage, the experiments are relatively comprehensive.
Critical weaknesses
  1. RefineX employs Qwen3-0.6B as its refinement model, whereas ProX uses a smaller 0.3B base model. This difference in model size makes it difficult to determine whether the downstream performance gains stem from the proposed methodological improvements or from the increased model capacity. Given the close methodological relationship between RefineX and ProX, as well as their relatively modest performance differences, a more controlled comparison would be important to better substantiate the effectiveness of the proposed method.
  2. Although the paper repeatedly emphasizes that restricting refinement to deletion-only operations can prevent overcorrection and potential biases, which I also find intuitively reasonable, this claim does not appear to be explicitly validated through experiments. Moreover, the paper does not investigate how much this design choice contributes to the overall performance gains. Adding a baseline that applies deletion-only operations to ProX would help address both concerns and better isolate the benefits of this design choice.
Questions for authors
  1. How does RefineX compare with ProX when using refinement models of the same parameter size?
  2. How does RefineX compare with ProX when both methods are restricted to deletion-only operations?
  3. The results in Table 3 are particularly interesting. Could the authors provide a more in-depth discussion of why RefineX achieves better refinement quality than E2E-del, despite being trained on supervision derived from E2E refinement?
Reviewer_mWaQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RefineX to improve pretraining data through fine-grained text deletion. It builds on ProX, which trains a small language model to generate executable programs for cleaning documents. RefineX generally performs better than several filtering baselines on ten downstream benchmarks. The paper also evaluates document quality, editing behavior, and processing efficiency.

Critical strengths
  1. The method avoids requiring the teacher model to generate both the desired edits and a correct executable program in one step. This may produce cleaner supervision and make it easier to train a small refiner.

  2. The deletion-only design is also practically useful. It makes the editing process interpretable and prevents the refiner from introducing new text. These are desirable properties for large-scale pretraining data processing.

  3. The authors pretrain models from scratch on the resulting corpora and compare them with multiple filtering and refinement baselines. RefineX achieves the best average downstream score in all six data settings.

Critical weaknesses
  1. The paper does not establish that the method is responsible for the improvements over ProX. RefineX mainly changes the construction of supervision data. The comparison with ProX-C changes several factors beyond this supervision strategy.

  2. The proposed editing procedure does not provide the reliability guarantees. Section 3.2 and Algorithm 2 describe extracting operations using minimum Levenshtein edit distance. How do you implement this? Also, restricting editing to deletions does not guarantee semantic preservation or prevent harmful over-editing. Removing a short phrase can change the meaning of a sentence, including its negation, qualification, or factual context.

Questions for authors
  1. Can the authors isolate the effect of the proposed two-stage supervision construction? Specifically, what happens when direct program generation and text-to-program conversion use the same teacher model, student architecture, seed data, editing functions, and chunk size?

  2. What evidence shows that deletion-only refinement preserves factual meaning and important information, rather than simply preventing the insertion of new words?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.