- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes improving language model pretraining data through deletion programs. A larger teacher first produces refined text, and the system extracts deletion operations by comparing it with the original document. These programs supervise a smaller model that refines a corpus at scale. The evaluation uses language models pretrained from scratch under matched token budgets and compares downstream performance and text quality with filtering and prior programmatic refinement methods.
Critical strengths
- Generating text before extracting executable supervision is a practical alternative to asking a teacher to directly emit a custom program. The restricted deletion API makes edits inspectable and reduces output generation.
- The central evidence is downstream performance after actual pretraining, rather than only a quality judge. Two model scales, multiple filtering pipelines, and ten tasks provide useful coverage.
- Table 2 shows consistently better group-average results than Prox-C under matched training-token budgets, including 44.7 versus 43.5 in the Prox-D group.
- The relationship to ProX is acknowledged, and the appendices provide prompts, filtering thresholds, model/training details, and editing examples.
Critical weaknesses
- The implementation does not establish minimum Levenshtein edits. Section 3.2 describes minimal edit distance, but Appendix B.1 uses difflib/get_opcodes. SequenceMatcher does not guarantee minimal edit sequences, and its span-level replacements need not correspond to optimal character edits. Because only deletions are retained, alignment choices alter supervision. This is more than terminology, although correcting the description need not invalidate measured pretraining benefits. Python difflib
- Deletion-only editing does not guarantee semantic preservation. Removing negation, conditions, units, or context can reverse meaning without introducing words. Table 4's new-word count therefore cannot establish absence of hallucination or semantic drift. Keeping only deletions from a broader teacher rewrite may also discard dependencies among edits.
- The decisive supervision-pipeline ablation does not match teacher, seed documents, student, supervision quantity, and training budget. Prompting, context, filtering, and selection differences confound attribution. Several average improvements are modest and lack repeated-pretraining uncertainty. Reported percentages should be clearly distinguished from absolute points.
- DataMan participates in seed stratification and evaluation. It is useful but not an independent guarantee of edit correctness or diversity. More documents per fixed token budget and better content within each document are also not disentangled.
- Output-token ratios do not establish total efficiency. The reported teacher generation alone costs approximately 12,480 H800 GPU-hours. End-to-end accounting should include teacher/student training, prefill, decoding, execution, and downstream savings, with an amortization break-even point.
- Execution semantics need clarification for changing line indices, overlapping chunks, conflicting deletions, skipped operations, and keep_all supervision after filtering small-deletion examples. Generalization beyond sub-billion-parameter models and general web text remains open. Data sampling and benchmark decontamination should be reproducible; no contamination is inferred here.
Questions for authors
Q1. Does supervision extraction use SequenceMatcher.get_opcodes or exact minimum Levenshtein alignment? If the former, what settings are used, and how much do labels and downstream results change under exact alignment?
Q2. Is there a direct-program versus text-then-program comparison matching teacher, documents, student, supervision size, and training budget? Can the current results separate this contribution from prompting, filtering, and context length?
Q3. What independent audit measures preservation of negation, units, conditions, and code semantics? How do you distinguish introducing no words from preserving factual meaning?
Q4. How many independent pretraining seeds support the main comparisons, what is the uncertainty of the central gains, and are the reported percentages relative improvements or absolute points?
Q5. Including teacher generation, student training, and corpus processing, what are the total cost and amortization break-even relative to Prox-C or simpler deletion models?
Q6. How does execution handle shifting line indices and overlapping/conflicting edits, and where does keep_all supervision come from after filtering small-deletion examples?