acceptodds
Under review as a conference paper at ICLR 2027

ZERO-TRAINING SURFACE STATISTICS FOR PRETRAINING-CORPUS FILTERING

Abstract

The value of pretraining corpora depends not only on their size, but also on how effectively low-quality content is filtered. Heuristic filters are efficient, interpretable, and scalable, but their hand-designed rules are pipeline-specific and do not generalize across languages. We extract the underlying measurements from seven widely used pipelines and summarize the corpus problems they target in an operational taxonomy of seven corpus-quality failure modes. Building on heuristic filters' view of language structure, we form ZenoFilter from how these measurements respond across failure modes, yielding a training-free adaptive detector that measures departures from ordinary text in redundancy and lexical composition and supports an adjustable corpus-level operating point. On real WARC documents, ZenoFilter outperforms the filtering layers of four widely used pipelines: Gopher/C4, FineWeb, Dolma, and RefinedWeb. At matched keep rates on symmetric English, it leaves 2.16–2.63× fewer document-level and 1.74–2.12× fewer token-level residual artifacts while running about six times faster. Against the closest adaptive comparator DCAD, ZenoFilter is never worse across its keep-rate curve. On non-English monolingual text it performs comparably to FineWeb 2 and DCAD, the only comparators that cover it. It processes multilingual and code-switched documents without language gates or language-specific thresholds, and discriminates mixed-language documents comparably to English ones. In downstream validation it achieves lower held-out loss than FineWeb across the six content categories and on the source-aligned Nemotron-CC validation set; the FineWeb advantage persists on the equal-weight mean of the externally defined Pile-22 suite.

Primary area:
datasets and benchmarks
Keywords:
training data quality, web data filtering, corpus filtering, data selection, training-free detection, surface statistics, LLM pretraining, anomaly detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qF1AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ZenoFilter, a training-free method for pre-training corpus quality filtering. Authors derive seven corpus-quality failure modes from the documented rules of seven widely-used data processing pipelines, remove the thresholds from them, and recast the underlying statistics as continuous surface operators. Authors organize 27 operators using controlled noise-injection experiments and labeled real web documents, organize them into three branches, compute a weighted score for each and use the maximum as the final decision. For deployment, the detector selects a reference set from the target corpus to calibrate the mean and variance of each operator, without labels or language identification. The experiments reveal that the methods leaves fewer residual artifacts than the filtering layers of C4, FineWeb, Dolma and RefinedWeb at matched keep rates, and runs faster and achieves lower validation loss than FineWeb when training 160M model.

Critical strengths
  1. The paper unifies heuristic rules scattered across pipelines into threshold-free measurements, maps them onto a shared failure-mode taxonomy, and characterizes each statistic through a controlled dose-response benchmark. This framework can serve as a tool for analyzing existing and future filtering rules, and has value independent of the detector itself.
  2. The self-reference calibration requires neither labels nor language identification, and the experiments show that a source-matched reference clearly outperforms an external curated corpus. This finding has broader implications for filtering methods based on reference distributions.
Critical weaknesses
  1. Residual artifact rates are labeled by an LLM annotator under the authors' own taxonomy, which is also used to select operator directions and weights, so the evaluation is inherently favorable to the proposed method. Under independent review, the English advantage shrinks from 2.16-2.63× to 1.32-1.48×, and the multilingual advantage over FineWeb 2 is not resolved in any language.
  2. Residual artifact rates do not track downstream performance: matched random retention carries about three times ZenoFilter's artifact rate, yet achieves the lowest validation loss and wins on 18 of 22 Pile components. This undermines the paper's core premise, and the abstract does not mention the random baseline.
  3. The confirmatory downstream result is fragile: the ZenoFilter arm has only three seeds, and its paired-t interval against FineWeb crosses zero. Moreover, FineWeb is the weakest of all trained arms, even below random sampling without deduplication.
  4. Although described as zero-training, the method selects its operators, directions, and weights using labeled data, yet no learned baseline is compared. A trained classifier reported in the appendix reaches an AUROC of 0.96 versus 0.906 for ZenoFilter, but is excluded from all comparisons.
Questions for authors
  1. The matched-random corpus carries about three times the residual artifact rate of ZenoFilter, yet achieves lower downstream loss. How do the authors explain this discrepancy, and do they think the relationship between these seven failure modes and training value needs to be re-examined?
  2. Could the authors add a simple classifier (e.g., logistic regression) trained on the same 27 features and the same development labels as a direct baseline?
  3. Since the FineWeb arm has higher loss than random sampling without deduplication, why was FineWeb rather than matched random retention pre-registered as the sole confirmatory comparator? Could the authors also add seeds to the ZenoFilter arm to match the other arms and report the paired test results?
Reviewer_C8B3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ZenoFIlter, a training-free surface statistics filter for pretraining corpus filtering that use self-referenced statistical signals computed on the corpus instead of learned quality classifiers such as DCLM, and Fine-web EDU. Each signal is direction aligned and standardized directly based on the target corpus. It computes a main anomaly score, a rare-pattern score and a content-absence score. FInal filtering is performed by using the maximum of the three branch scores against a threshold defined by keep rate.

Critical strengths
  • The self-reference calibration based on length-bucketing and corpus derived moment estimation provides a principled approach for training-free adaption of per-signal normalization, which is robust to cross-corpus and multi-lingual distribution shift.
  • The controlled experiment demonstrate a reproducible empirical procedure that illustrate operator signals, matched-keep deployment on symmetric English and reference ablation.
Critical weaknesses
  • Operator selection lacks dedicated experiment validation. Although the paper claims operators are chosen based on reduction of residual noise, no systematic ablation is provided that isolates the contribution of the operator selection, nor any experiment justifying the selection of particular 27 surface signals per document,14 anomaly ( value off-centre) and 13 absence (feature missing ).
  • Eavluation mostly on self-defined synthetic experiments not available to public. No sufficient real-world evidence is shown that zero-training filtering outperforms conventional LLM or classifier based approaches on SOTA LLM. OOD generalization is not demonstrated on out-of-distribution corpora.
Questions for authors
  • The author release the implementation in https://anonymous.4open.science/r/zenofilter-5F3C/README.md., however access to any file returns the error message {Allow this repository on GitHub using the read-only access button, then retry. Organization access may require administrator approval.", could the author release the actual code via zipped file or normal repo?

  • The synthetic experiments and DCLM comparison requires specific calibration corpora, injection templates and benchmark dataset. Will the author provide the dataset for reproducing the raw per-model scores underlying?

Reviewer_vvvHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ZenoFilter combines 27 surface statistics, extended from existing pipeline rules, into three branch scores. Operators and weights are fixed on labeled development data, and scores are calibrated on about 300 documents from the target corpus. At matched keep rates, it leaves 2.16–2.63× fewer residual artifacts than the Gopher/C4, FineWeb, Dolma, and RefinedWeb filters and runs about six times faster. In 160M-parameter training, it beats FineWeb on held-out loss, but token-matched random selection beats it on the pre-registered validation loss.

Critical strengths
  • Careful comparison. The advantage over rule-based filters survives many robustness checks, and ablations support the main design choices.
  • Cheap. Scoring takes about 0.9 ms per document on CPU, with no model inference.
Critical weaknesses
  1. Lower artifact rates do not reliably yield better pretraining data. Random selection keeps about three times the artifact rate, yet beats ZenoFilter on the pre-registered validation loss and on Pile-22, although ZenoFilter wins on WikiText and LAMBADA. Because FineWeb is worse than random on the pre-registered loss, beating FineWeb does not show that filtering helps; the abstract and conclusion omit the random comparison.
  2. Document-level label validity is uncertain, and the labels may penalize useful text. The only human gold standard is 100 sentences. LLM audits cover more documents but agree poorly with the annotator at document level, and the judge itself drifts between rounds. Random selection's advantage concentrates in code, math, and other symbol-heavy text.
  3. The headline ratios rely on labels from the rule set the detector was fitted to. Operators, directions, and weights are fitted to labels from the evaluation rule set, unlike the baselines' public thresholds. An independent review on different samples preserves the ranking (1.32–1.48×) but does not confirm the headline magnitude. The paper's own trained classifier on the same material (AUROC 0.96) is left out of the comparisons, leaving unclear what the training-free constraint buys.
  4. Two secondary claims need qualification. Word-list operators and the English-vocabulary calibration effects reported in the paper qualify the claim of operating without language-specific thresholds. The cited pre-registration is neither linked nor dated.

Minor: "the FineWeb advantage persists" (abstract) should read "the advantage over FineWeb"; Table B.1 reports raw AUC, which does not match the Table D.2 weights; the token-shuffling control in B.1 is overstated; several cross-references and headings need correction.

Questions for authors
  1. Can a small blinded human audit, stratified by content type, measure how often useful documents such as code, math, or tables are labeled as artifacts?
  2. How do the authors reconcile the random-selection result with the premise that fewer artifacts yield better pretraining data?
  3. On the Table 1 pool, how does a lightweight classifier trained on the same signals and labels compare?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.