- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ZenoFilter, a training-free method for pre-training corpus quality filtering. Authors derive seven corpus-quality failure modes from the documented rules of seven widely-used data processing pipelines, remove the thresholds from them, and recast the underlying statistics as continuous surface operators. Authors organize 27 operators using controlled noise-injection experiments and labeled real web documents, organize them into three branches, compute a weighted score for each and use the maximum as the final decision. For deployment, the detector selects a reference set from the target corpus to calibrate the mean and variance of each operator, without labels or language identification. The experiments reveal that the methods leaves fewer residual artifacts than the filtering layers of C4, FineWeb, Dolma and RefinedWeb at matched keep rates, and runs faster and achieves lower validation loss than FineWeb when training 160M model.
Critical strengths
- The paper unifies heuristic rules scattered across pipelines into threshold-free measurements, maps them onto a shared failure-mode taxonomy, and characterizes each statistic through a controlled dose-response benchmark. This framework can serve as a tool for analyzing existing and future filtering rules, and has value independent of the detector itself.
- The self-reference calibration requires neither labels nor language identification, and the experiments show that a source-matched reference clearly outperforms an external curated corpus. This finding has broader implications for filtering methods based on reference distributions.
Critical weaknesses
- Residual artifact rates are labeled by an LLM annotator under the authors' own taxonomy, which is also used to select operator directions and weights, so the evaluation is inherently favorable to the proposed method. Under independent review, the English advantage shrinks from 2.16-2.63× to 1.32-1.48×, and the multilingual advantage over FineWeb 2 is not resolved in any language.
- Residual artifact rates do not track downstream performance: matched random retention carries about three times ZenoFilter's artifact rate, yet achieves the lowest validation loss and wins on 18 of 22 Pile components. This undermines the paper's core premise, and the abstract does not mention the random baseline.
- The confirmatory downstream result is fragile: the ZenoFilter arm has only three seeds, and its paired-t interval against FineWeb crosses zero. Moreover, FineWeb is the weakest of all trained arms, even below random sampling without deduplication.
- Although described as zero-training, the method selects its operators, directions, and weights using labeled data, yet no learned baseline is compared. A trained classifier reported in the appendix reaches an AUROC of 0.96 versus 0.906 for ZenoFilter, but is excluded from all comparisons.
Questions for authors
- The matched-random corpus carries about three times the residual artifact rate of ZenoFilter, yet achieves lower downstream loss. How do the authors explain this discrepancy, and do they think the relationship between these seven failure modes and training value needs to be re-examined?
- Could the authors add a simple classifier (e.g., logistic regression) trained on the same 27 features and the same development labels as a direct baseline?
- Since the FineWeb arm has higher loss than random sampling without deduplication, why was FineWeb rather than matched random retention pre-registered as the sole confirmatory comparator? Could the authors also add seeds to the ZenoFilter arm to match the other arms and report the paired test results?