- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper introduces ReScraper, a unified 0.6B language model for preparing web data for LLM pretraining. Instead of employing separate heuristic scraping and filtering stages, ReScraper jointly performs main-content extraction and selects among four cleaning operations: keep, edit, delete, and rewrite. Its supervision is constructed from three teacher models, followed by two-stage supervised fine-tuning.
The authors evaluate the resulting corpora by pretraining language models at three scales (400M, 1.4B, and 2.8B) on fixed token budgets. ReScraper achieves DCLM Core scores of 0.15345, 0.27348, and 0.31841, respectively, outperforming the strongest reported baseline at each scale by approximately 3.8–4.7% relative. Additional experiments investigate individual operations, scraper and refiner combinations, data quality, diversity, and computational cost.
Critical strengths
(1) Strong downstream validation across multiple pretraining scales.
The most convincing aspect of the paper is its evaluation through actual language-model pretraining rather than relying exclusively on proxy measures of data quality. Table 2 shows consistent improvements over competitive rule-based, model-based, and multi-agent curation baselines at three model scales. These results establish meaningful practical value for the proposed data-curation approach.
(2) A practical unified framework supported by useful diagnostic experiments.
The combination of extraction and adaptive cleaning within a single small model is operationally appealing. The paper further investigates the contributions of individual operations, alternative scrapers, output diversity, teacher–student agreement, and processing costs. These analyses provide useful evidence that the proposed system learns nontrivial data-curation behaviors rather than merely applying a fixed filtering rule.
Critical weaknesses
(1) Model selection on the reported evaluation metric raises concerns about the reliability of the main performance margin.
Appendix B.2 (Figure 10) compares six student configurations using the 1B DCLM Core score and selects the best-performing variant (0.2735), which is also the reported result in Table 2 and the reference configuration in Table 3. The paper does not clearly establish whether this configuration was selected using an independent validation criterion or the same downstream benchmark used for the final evaluation.
This matters because the reported 1B improvement may partly reflect selection among multiple single-seed experiments. Several alternative configurations perform substantially worse, and some do not outperform UltraX. Although the 400M and 3B results provide additional evidence because they reuse the selected configuration, uncertainty remains: the five-seed analysis in Appendix B.3 is restricted to 400M and compares against UltraX, whereas FineWeb-rule is the strongest baseline at that scale. The gains at 1B and 3B are not accompanied by comparable multi-seed evaluations.
An independent model-selection protocol or repeated runs against the strongest baselines would substantially strengthen the empirical conclusions.
(2) The experiments do not sufficiently isolate the benefit of unified modeling from the teacher pipeline used to construct its supervision.
ReScraper is trained on targets generated by Dripper, Qwen3.8-27B, FineWeb-Edu, and RePro. However, the paper does not report downstream pretraining results using this complete teacher pipeline directly. Consequently, it remains unclear how much of the improvement is attributable to integrating extraction and cleaning into a single model, rather than to stronger teacher supervision, the rescue-and-rewrite strategy, or the resulting data distribution.
The Dripper+UltraX comparison in Figure 3 is informative but does not reproduce the same teacher pipeline. Similarly, the Dripper-extraction ablation in Table 3 provides partial evidence for the importance of jointly learned extraction, but does not fully separate the effects of the learned student from the supervision used to train it.
This distinction is central to the paper's claimed methodological contribution. A direct comparison with the complete teacher pipeline, or an appropriately controlled approximation, would clarify whether unification provides a benefit beyond distilling a stronger curation pipeline.
Questions for authors
Clarification 1 — Model selection and statistical reliability.
Was the final ReScraper configuration selected using the same 1B DCLM Core evaluation tasks reported in Table 2, or was an independent validation protocol employed? Can the authors provide uncertainty estimates or repeated-run results against the strongest baselines at the 1B and 3B scales?
Clarification 2 — Contribution of unified modeling.
Have the authors evaluated the complete teacher pipeline (Dripper → Qwen3.8-27B refinement → FineWeb-Edu rescue → RePro rewriting) as a direct data-curation baseline? If not, what existing evidence separates the benefit of unified student modeling from the quality of the teacher-generated supervision?
3 comments
Thank you for the thorough response and the substantial new experiments.
Model selection (W1/Q1). The clarification resolves my concern. Selecting the configuration by the DCLM-fastText score of the curated data does not depend on the downstream evaluation, and its agreement with C4 validation loss and Core is reassuring. The new multi-seed results at all three scales, with significant gains over the strongest baselines, make the main results much more reliable.
Teacher pipeline (W2/Q2). Thank you for running the complete teacher pipeline as a baseline at all three scales. This directly addresses my question. ReScraper outperforms its own teacher pipeline at 1B and 3B at about one seventh of the curation cost, which is a strong practical result and clarifies the value of the unified student. I also appreciate the honest discussion of where the teacher remains stronger, at 400M and on world knowledge.
These additions address my concerns, and I have raised my Informativeness and Soundness scores accordingly. I encourage the authors to include these results and the clarified selection procedure in the final version.
Thank you very much for recognizing our rebuttal and raising your scores! We will incorporate these new experiments and clarifications into the next revision of the paper.
Official Rebuttal to Reviewer e6rQ
Thank you for your time and insightful review of our paper! We address your comments/questions below:
Response: Thank you for the careful reading. We select the final configuration by an intermediate quality check on the curated data, the average score of DCLM-fastText, the quality scorer used to build the original DCLM corpus. As shown in the table below, the final configuration attains the highest score among the six variants. We have clarified this procedure in Appendix B.2 of the revised paper.
To further address your concern, we evaluate the six variants on the out-of-distribution C4 validation loss. The final configuration also attains the lowest C4 loss, so this loss can be used to select the best configuration as well.
For statistical reliability, the 400M gain already holds across five seeds in the submission. We further pretrain FineWeb-rule with five seeds at 400M, and ReScraper, UltraX, and DataOrchestra with three seeds at 1B and 3B given our compute budget. As shown in the table below, ReScraper significantly outperforms the strongest baselines at all three scales (p ≤ 0.007), and its lowest seed exceeds every seed of the baselines. We have included these results in the revised paper (Appendix B.3).
Response: Thank you for the suggestion. We agree this comparison is central to our contribution. We run the complete teacher pipeline (Dripper, Qwen3.8-27B, FineWeb-Edu rescue, and RePro) over the source pool and pretrain models at all three scales on its output. As shown in the table below, ReScraper outperforms the teacher pipeline at 1B and 3B at about one seventh of its cost. The teacher pipeline performs better at 400M and on world knowledge at every scale, potentially due to a distillation effect from its 27B teacher, which processes every page. These results suggest that unification can provide a benefit beyond the teacher pipeline at larger scales. We have included this comparison in the revised paper (Section 5.4).
The student differs from its teacher pipeline in two ways. It rewrites more pages (9.0% vs. 4.1%), mostly ones the teacher would delete. In the cases we examined, its rewrites also stay closer to the wording and structure of the source (Appendix C of the revised paper), suggesting that unified learning may help it better respect the original writing when rewriting. We hypothesize that these differences, together with the page-level context that ReScraper sees in every decision, may contribute to its advantage at larger scales.
In conclusion, we hope our responses resolve your concerns:
We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.