acceptodds
Under review as a conference paper at ICLR 2027

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining

Abstract

LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozens of rule-based filters then clean it, so corpus quality is capped by the coarseness and accuracy of the rules. In this work, we propose ReScraper, a unified language model of only 0.6B parameters that replaces this entire stack. To train ReScraper, we carefully curate supervised data from the outputs of three teacher models, so it learns to first extract the main content from raw data and then choose among four operations: keeping the page as extracted, editing out noisy lines and spans, deleting it entirely, or rewriting it when it is poorly written but informative. Based on the same crawled data pool, pretraining 400M, 1.4B, and 2.8B models on our curated data improves the DCLM Core score by a relative 3.8–4.7% over the strongest baseline at each scale, including the costly multi-agent curation. Our analyses show that each operation plays a distinct and complementary role, and that extracting and cleaning in one model outperforms a cascade of separate models. ReScraper also concentrates its operations on the pages that need them, raising the quality of poor pages the most while keeping the corpus diverse. These results demonstrate the feasibility and effectiveness of AI4AI for pretraining data curation, where a small learned model takes over an entire stage of the pipeline from hand-written heuristics. Our anonymized code is available at https://anonymous.4open.science/r/ReScraper-FCF0.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Pretraining, Data Curation, Synthetic Data, AI4AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_e6rQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces ReScraper, a unified 0.6B language model for preparing web data for LLM pretraining. Instead of employing separate heuristic scraping and filtering stages, ReScraper jointly performs main-content extraction and selects among four cleaning operations: keep, edit, delete, and rewrite. Its supervision is constructed from three teacher models, followed by two-stage supervised fine-tuning.

The authors evaluate the resulting corpora by pretraining language models at three scales (400M, 1.4B, and 2.8B) on fixed token budgets. ReScraper achieves DCLM Core scores of 0.15345, 0.27348, and 0.31841, respectively, outperforming the strongest reported baseline at each scale by approximately 3.8–4.7% relative. Additional experiments investigate individual operations, scraper and refiner combinations, data quality, diversity, and computational cost.

Critical strengths

(1) Strong downstream validation across multiple pretraining scales.

The most convincing aspect of the paper is its evaluation through actual language-model pretraining rather than relying exclusively on proxy measures of data quality. Table 2 shows consistent improvements over competitive rule-based, model-based, and multi-agent curation baselines at three model scales. These results establish meaningful practical value for the proposed data-curation approach.

(2) A practical unified framework supported by useful diagnostic experiments.

The combination of extraction and adaptive cleaning within a single small model is operationally appealing. The paper further investigates the contributions of individual operations, alternative scrapers, output diversity, teacher–student agreement, and processing costs. These analyses provide useful evidence that the proposed system learns nontrivial data-curation behaviors rather than merely applying a fixed filtering rule.

Critical weaknesses

(1) Model selection on the reported evaluation metric raises concerns about the reliability of the main performance margin.

Appendix B.2 (Figure 10) compares six student configurations using the 1B DCLM Core score and selects the best-performing variant (0.2735), which is also the reported result in Table 2 and the reference configuration in Table 3. The paper does not clearly establish whether this configuration was selected using an independent validation criterion or the same downstream benchmark used for the final evaluation.

This matters because the reported 1B improvement may partly reflect selection among multiple single-seed experiments. Several alternative configurations perform substantially worse, and some do not outperform UltraX. Although the 400M and 3B results provide additional evidence because they reuse the selected configuration, uncertainty remains: the five-seed analysis in Appendix B.3 is restricted to 400M and compares against UltraX, whereas FineWeb-rule is the strongest baseline at that scale. The gains at 1B and 3B are not accompanied by comparable multi-seed evaluations.

An independent model-selection protocol or repeated runs against the strongest baselines would substantially strengthen the empirical conclusions.

(2) The experiments do not sufficiently isolate the benefit of unified modeling from the teacher pipeline used to construct its supervision.

ReScraper is trained on targets generated by Dripper, Qwen3.8-27B, FineWeb-Edu, and RePro. However, the paper does not report downstream pretraining results using this complete teacher pipeline directly. Consequently, it remains unclear how much of the improvement is attributable to integrating extraction and cleaning into a single model, rather than to stronger teacher supervision, the rescue-and-rewrite strategy, or the resulting data distribution.

The Dripper+UltraX comparison in Figure 3 is informative but does not reproduce the same teacher pipeline. Similarly, the Dripper-extraction ablation in Table 3 provides partial evidence for the importance of jointly learned extraction, but does not fully separate the effects of the learned student from the supervision used to train it.

This distinction is central to the paper's claimed methodological contribution. A direct comparison with the complete teacher pipeline, or an appropriately controlled approximation, would clarify whether unification provides a benefit beyond distilling a stronger curation pipeline.

Questions for authors

Clarification 1 — Model selection and statistical reliability.

Was the final ReScraper configuration selected using the same 1B DCLM Core evaluation tasks reported in Table 2, or was an independent validation protocol employed? Can the authors provide uncertainty estimates or repeated-run results against the strongest baselines at the 1B and 3B scales?

Clarification 2 — Contribution of unified modeling.

Have the authors evaluated the complete teacher pipeline (Dripper → Qwen3.8-27B refinement → FineWeb-Edu rescue → RePro rewriting) as a direct data-curation baseline? If not, what existing evidence separates the benefit of unified student modeling from the quality of the teacher-generated supervision?

3 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer e6rQ

Thank you for your time and insightful review of our paper! We address your comments/questions below:

Weakness 1 & Question 1: Model selection on the reported 1B Core score, and missing multi-seed runs against the strongest baselines at 1B and 3B

Response: Thank you for the careful reading. We select the final configuration by an intermediate quality check on the curated data, the average score of DCLM-fastText, the quality scorer used to build the original DCLM corpus. As shown in the table below, the final configuration attains the highest score among the six variants. We have clarified this procedure in Appendix B.2 of the revised paper.

To further address your concern, we evaluate the six variants on the out-of-distribution C4 validation loss. The final configuration also attains the lowest C4 loss, so this loss can be used to select the best configuration as well.

1B Variant (Appendix B.2)DCLM-fastText Score (Avg.) ↑C4 Validation Loss (CE) ↓Core ↑
FineWeb-Edu >= 1.0, 30%, 3 ep. (final)0.030252.58750.2735
FineWeb-Edu >= 1.0, 4%, 2 ep.0.029652.58970.2660
27B classifier, 11%, 2 ep.0.028752.59130.2640
27B classifier, 3%, 2 ep.0.028022.59110.2616
FineWeb-Edu >= 1.5, 5%, 1 ep.0.027232.64330.2577
FineWeb-Edu >= 1.5, 15%, 3 ep.0.027872.64610.2542

For statistical reliability, the 400M gain already holds across five seeds in the submission. We further pretrain FineWeb-rule with five seeds at 400M, and ReScraper, UltraX, and DataOrchestra with three seeds at 1B and 3B given our compute budget. As shown in the table below, ReScraper significantly outperforms the strongest baselines at all three scales (p ≤ 0.007), and its lowest seed exceeds every seed of the baselines. We have included these results in the revised paper (Appendix B.3).

MethodModel Scale#SeedsCore (mean ± std)Welch's t-test vs. ReScraper
FineWeb-rule400M50.1414 ± 0.0040p = 0.001
UltraX400M50.1416 ± 0.0036p < 0.001
ReScraper400M50.1539 ± 0.0039-
UltraX1B30.2621 ± 0.0007p < 0.001
DataOrchestra1B30.2569 ± 0.0027p = 0.007
ReScraper1B30.2726 ± 0.0007-
UltraX3B30.3060 ± 0.0013p < 0.001
DataOrchestra3B30.3057 ± 0.0023p = 0.004
ReScraper3B30.3176 ± 0.0012-

Weakness 2 & Question 2: Separating the benefit of unified modeling from the teacher pipeline, and evaluating the complete teacher pipeline as a baseline

Response: Thank you for the suggestion. We agree this comparison is central to our contribution. We run the complete teacher pipeline (Dripper, Qwen3.8-27B, FineWeb-Edu rescue, and RePro) over the source pool and pretrain models at all three scales on its output. As shown in the table below, ReScraper outperforms the teacher pipeline at 1B and 3B at about one seventh of its cost. The teacher pipeline performs better at 400M and on world knowledge at every scale, potentially due to a distillation effect from its 27B teacher, which processes every page. These results suggest that unification can provide a benefit beyond the teacher pipeline at larger scales. We have included this comparison in the revised paper (Section 5.4).

SettingMethodCuration Cost (H200 GPU Hours)Commonsense ReasoningLanguage UnderstandingReading ComprehensionSymbolic ProblemWorld KnowledgeCore
400MTeacher3,5500.234960.197230.027120.170830.153380.16321
400MReScraper4940.224200.18741-0.010230.173470.148430.15345
1BTeacher3,5500.300550.359090.174380.184160.257780.26314
1BReScraper4940.331390.355330.204100.205630.249970.27348
3BTeacher3,5500.365860.433940.266320.168530.310260.31337
3BReScraper4940.380030.430420.244430.205980.303870.31841

The student differs from its teacher pipeline in two ways. It rewrites more pages (9.0% vs. 4.1%), mostly ones the teacher would delete. In the cases we examined, its rewrites also stay closer to the wording and structure of the source (Appendix C of the revised paper), suggesting that unified learning may help it better respect the original writing when rewriting. We hypothesize that these differences, together with the page-level context that ReScraper sees in every decision, may contribute to its advantage at larger scales.


In conclusion, we hope our responses resolve your concerns:

  1. The final configuration is selected by its DCLM-fastText quality score and also attains the lowest C4 loss, and ReScraper significantly outperforms the strongest baselines across seeds at all three scales (W1, Q1).
  2. ReScraper outperforms the complete teacher pipeline at 1B and 3B at about one seventh of its cost (W2, Q2).

We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.

Thank you for the thorough response and the substantial new experiments.

Model selection (W1/Q1). The clarification resolves my concern. Selecting the configuration by the DCLM-fastText score of the curated data does not depend on the downstream evaluation, and its agreement with C4 validation loss and Core is reassuring. The new multi-seed results at all three scales, with significant gains over the strongest baselines, make the main results much more reliable.

Teacher pipeline (W2/Q2). Thank you for running the complete teacher pipeline as a baseline at all three scales. This directly addresses my question. ReScraper outperforms its own teacher pipeline at 1B and 3B at about one seventh of the curation cost, which is a strong practical result and clarifies the value of the unified student. I also appreciate the honest discussion of where the teacher remains stronger, at 400M and on world knowledge.

These additions address my concerns, and I have raised my Informativeness and Soundness scores accordingly. I encourage the authors to include these results and the clarified selection procedure in the final version.

Thank you very much for recognizing our rebuttal and raising your scores! We will incorporate these new experiments and clarifications into the next revision of the paper.

Reviewer_Bt6wOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes ReScraper, a 0.6B parameter model built on the Qwen3-0.6B model for unified HTML text-extraction, filtering, and cleaning through the supervision of Dripper, Qwen3.8-27B, and RePro models. Authors perform multiple experiments and ablations by training 400M, 1.4B, and 2.8B models on the outputs produced by the ReScraper model and show that their unified filtering approach improves DCLM Core performance over prior heuristic, model, and multi-agent based filtering baselines, show the effects of each of the operations, as well as the analysis of the quality and diversity of the generated data.

Critical strengths
  1. The method is sound and having a single relatively small 0.6B model curate high-quality data starting from HTML rendering is a promising and useful direction for pretraining data curation.
  2. There are many experiments showing improvements of ReScraper over prior baselines, as well as detailed analyses of the operations that are performed by ReScraper and their effects on quality and diversity. Overall, the model curates high-quality data according to DCLM score evaluation with better n-gram diversity than UltraX, ProX-C, and RefinedWeb.
  3. Authors show that the ReScraper model follows the teacher operation distribution in the majority of the cases and reaches token F1 of 89.3 against the output of the teacher, indicating successful distillation.
Critical weaknesses
  1. Experiments at 400M, 1.4B, 2.8B use the same source pool, somewhat confounding data quality and repetition. While this can still be okay up to 1x and 4x repetition at 400M and 1.4B, respectively, at 2.8B scale with 7.5x repetition, repetition might skew the results. A positive result at the 2.8B scale is that ReScraper at 7.5x repetition still beats model-based baselines at 4.1-5.2x repetition, although a more fair experimental setup to quantify the improvement of ReScraper over other baselines would be to fix the repetition ratio to be the same for all datasets.
  2. I think weak-to-strong pretraining data curation claim might be too strong, since a lot of prior work could be considered to be the same (e.g., FineWeb-Edu, ProX, DataOrchestra). Additionally, since a part of the supervision comes from a 27B model, it could be closer to strong-to-weak distillation applied for training larger models.
Questions for authors
  1. Did you perform any downstream evaluation experiments where the data repetition was controlled?
  2. Did you analyze whether the rewrite operation removing noisy content affects the model’s capability to understand noisy content? Leaving some noisy content in the data distribution might be a good idea to make the model robust to this type of noise and I think this would be an interesting finding although more detailed analyses might be over the scope of your work.
  3. How is the rendering of the raw HTML performed to obtain the text?
4 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer Bt6w

Thank you for your time and insightful review of our paper! We address your comments/questions below:

Weakness 1 & Question 1: Confound between data quality and repetition (7.5x for ReScraper vs. 4.1-5.2x for model-based baselines in the 3B setting), and downstream evaluation with repetition controlled

Response: Thank you for the suggestion. ReScraper's corpus is repeated the most among the model-based methods, yet it still outperforms all baselines. To control repetition directly, we randomly subsample UltraX, ProX-C, and DataOrchestra to the size of ReScraper's corpus (7.44B unique tokens), so that every corpus is repeated equally. As shown in the tables below, every corpus is repeated 3.9x at 1B, within four epochs, where repeated data is nearly as valuable as unique data [1], and ReScraper still outperforms the strongest baseline by a relative 5.8%. At 3B, subsampling to 7.5x repetition lowers the Core of every baseline, and the gain of ReScraper over the strongest baseline grows from a relative 3.8% to 7.0%. We have included this comparison in the revised paper (Section 5.1).

1B Setting, 28.8B Training Tokens, 3.9x Repetition#UniqueCommonsense ReasoningLanguage UnderstandingReading ComprehensionSymbolic ProblemWorld KnowledgeCoreCore Change vs. Full Corpus
ProX-C (subsampled)7.44B0.306890.302400.168290.197720.217750.24169+3.3%
UltraX (subsampled)7.44B0.332760.332570.156300.203590.241190.25848-1.2%
DataOrchestra (subsampled)7.44B0.320970.329230.199720.177710.243730.25658+0.0%
ReScraper7.44B0.331390.355330.204100.205630.249970.27348-
3B Setting, 55.9B Training Tokens, 7.5x Repetition#UniqueCommonsense ReasoningLanguage UnderstandingReading ComprehensionSymbolic ProblemWorld KnowledgeCoreCore Change vs. Full Corpus
ProX-C (subsampled)7.44B0.348810.378120.080110.186460.266440.26454-10.9%
UltraX (subsampled)7.44B0.329220.387700.228240.184680.282270.28788-6.1%
DataOrchestra (subsampled)7.44B0.337280.393910.251510.190530.293370.29770-3.0%
ReScraper7.44B0.380030.430420.244430.205980.303870.31841-

[1]: Muennighoff, Niklas, et al. "Scaling Data-Constrained Language Models." NeurIPS 2023.

Weakness 2: Strength of the weak-to-strong claim, given prior small curators (FineWeb-Edu, ProX, DataOrchestra) and supervision from a 27B model

Response: Thank you for the comment. We have removed the "weak-to-strong" term from the revised paper and now state the finding directly, that a small curator can improve the pretraining of larger models. Compared with prior small curators, which score or edit the output of a heuristic scraper, ReScraper handles extraction, editing, deletion, and rewriting in a single model.

We also evaluate Qwen3-0.6B, the initialization of ReScraper, on DCLM Core and compare it against the largest model we pretrain on ReScraper's data, in the 3B setting. As shown in the table below, the Core of Qwen3-0.6B is lower (0.297 vs. 0.318), and at this scale ReScraper still outperforms the strongest baseline by a relative 3.8%.

ModelCommonsense ReasoningLanguage UnderstandingReading ComprehensionSymbolic ProblemWorld KnowledgeCore
Qwen3-0.6B (initialization of ReScraper)0.241470.282060.315380.284180.347980.29653
Model pretrained on ReScraper's data (3B setting)0.380030.430420.244430.205980.303870.31841

Although Qwen3 models may already be exposed to a larger training corpus, we would like to note that UltraX and DataOrchestra also build on Qwen3 models of the same or larger size, Qwen3-0.6B for UltraX and Qwen3-0.6B to Qwen3-4B for DataOrchestra, yet ReScraper outperforms both at every scale, suggesting that its gains are not simply due to a stronger initialization. We focus on the size of the curator because curation cost is paid for every page, and since frontier pretraining corpora contain trillions of tokens, a small curator can save a large amount of curation cost.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer Bt6w (Continued)

Question 2: Effect of removing noisy content on the model's ability to understand noisy text

Response: Thank you for the interesting question. ReScraper leaves more than half of the pages unchanged or lightly edited. <keep> returns 37.7% of the extracted pages unchanged, and <edit> operates by deleting lines or strings and changes another 17.9% by less than 10% of their words, so it is not designed to correct typos or informal language (Figure 6). The prompt of the refining teacher also instructs it to "keep all typos" and not to "fix" awkward phrasing or non-native English (Appendix D.1). <rewrite>, chosen for 9.0% of the pages, mainly recycles pages that would otherwise be deleted, so it adds text with little effect on the noisy text that is kept.

To measure robustness to noisy inputs, we evaluate the models in the 1B setting on three noisy benchmarks, AdvGLUE with its adversarial examples, and Winogrande and SQuAD with spelling noise injected (two adjacent letters swapped in about 20% of the words, with the answer options and references unchanged). As shown in the table below, ReScraper performs the best on all three, and raw text, which keeps all of the noise, falls behind it. We have included this evaluation in the revised paper (Appendix B.5).

1B SettingAdvGLUE (Accuracy)Winogrande with Spelling Noise (Accuracy)SQuAD with Spelling Noise (F1)
Raw text44.2154.3042.92
RefinedWeb-rule49.9252.7246.31
UltraX47.3254.5447.05
DataOrchestra47.5953.5941.56
ReScraper53.0556.2752.45

We agree that deliberately keeping some noise is an interesting direction, and the share of unchanged pages could be tuned for this purpose.

Question 3: How the raw HTML is rendered into text

Response: Thank you for the question. We render each page's raw HTML into line-numbered text mainly following Dripper, in three steps, and we have added these details to Appendix A.

(1) DOM parsing and cleanup. We parse the HTML into a DOM tree with lxml and BeautifulSoup, decode HTML entities into UTF-8 characters, and strip non-text elements, such as scripts, styles, images, and tracking URLs.

(2) Whitespace collapsing. We collapse redundant newlines and indentation inside inline elements into single spaces, so that words are not glued together or broken mid-sentence.

(3) Block-level segmentation. We insert line breaks at block-level tags, such as paragraphs, divisions, headings, list items, and block quotes, so that each block becomes one line, and convert headings, lists, tables, code blocks, and math formulas into Markdown.

Finally, we prefix each line with a line identifier <lid:n>. This stage does not select main content, so navigation, sidebars, and footers remain in the input, and the removals are made by ReScraper.


In conclusion, we hope our responses resolve your concerns:

  1. With repetition matched, ReScraper still outperforms all baselines at 1B and 3B, by a larger margin at 3B (W1, Q1).
  2. We have removed the "weak-to-strong" term; Qwen3-0.6B has a lower Core than the model pretrained on ReScraper's data in the 3B setting, and UltraX and DataOrchestra build on Qwen3 models of the same or larger size (W2).
  3. ReScraper leaves more than half of the pages unchanged or lightly edited and performs the best on three noisy benchmarks (Q2).

We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.

Thank you for including additional experiments, including those beyond the scope of your work (Q2 on robustness of the models on noisy inputs). All of my concerns have been resolved and I have increased my Rating (3 to 4) and Soundness (2 to 3) scores accordingly.

Thank you very much for recognizing our rebuttal and raising your scores! We will incorporate these new experiments and clarifications into the next revision of the paper.

Reviewer_BLbpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RESCRAPER, a 0.6B language model that replaces the heuristic scraping and cleaning pipeline for LLM pre-training. RESCRAPER is a fine-tuned Qwen3-0.6B using SFT data distilled from 3 teachers (Dripper, Qwen3.8-27B, and RePro) to learn to extract the main content from raw data and choose among 4 operations: keep/edit/delete/rewrite the page. The paper shows that pre-training 400M, 1.4B, and 2.8B models on the data curated by RESCRAPER outperforms existing baselines.

Critical strengths
  • The paper introduces a unified 0.6B language model that replaces the heuristic scraping and cleaning for LLM pre-training.
  • The paper demonstrates 400M, 1.4B, and 2.8B models pertained on the data curated by RESCRAPER outperforms existing baselines.
Critical weaknesses
  • The paper claims "enabling weak-to-strong pretraining data curation". However, "weak" here means nothing more than 0.6B < 1.4B / 2.8B. For every other measures (e.g., pre-training compute of Qwen vs. the 1.4B/2.8B models here; the fact that it also leverages distillation from larger LLMs, etc.), RESCRAPER looks like a stronger model.
  • The evaluation may be biased towards clean text. All 22 tasks have well-formed prompts, so they can hardly measure scenarios where the input itself is noisy.
Questions for authors
  • Why does cleaning not seem to help for reading comprehension?
4 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer BLbp

Thank you for your time and insightful review of our paper! We address your comments/questions below:

Weakness 1: Meaning of "weak-to-strong" when ReScraper is stronger by other measures (pretraining compute of Qwen3, distillation from larger LLMs)

Response: Thank you for pointing this out. We have removed the "weak-to-strong" term from the revised paper and now state the finding directly, that a small curator can improve the pretraining of larger models.

We agree that Qwen3-0.6B, the initialization of ReScraper, is pretrained with far more compute than our models. For reference, we evaluate Qwen3-0.6B on DCLM Core and compare it against the largest model we pretrain on ReScraper's data, in the 3B setting. As shown in the table below, the Core of Qwen3-0.6B is lower (0.297 vs. 0.318), and at this scale ReScraper still outperforms the strongest baseline by a relative 3.8%.

ModelCommonsense ReasoningLanguage UnderstandingReading ComprehensionSymbolic ProblemWorld KnowledgeCore
Qwen3-0.6B0.241470.282060.315380.284180.347980.29653
Model pretrained on ReScraper's data (3B setting)0.380030.430420.244430.205980.303870.31841

Although Qwen3 models may already be exposed to a larger training corpus, we would like to note that UltraX and DataOrchestra also build on Qwen3 models of the same or larger size, Qwen3-0.6B for UltraX and Qwen3-0.6B to Qwen3-4B for DataOrchestra, yet ReScraper outperforms both at every scale, suggesting that its gains are not simply due to a stronger initialization. We focus on the size of the curator because curation cost is paid for every page, and since frontier pretraining corpora contain trillions of tokens, a small curator can save substantial curation cost.

Weakness 2: Bias of the evaluation toward clean text, since all 22 tasks have well-formed prompts

Response: Thank you for the suggestion. We agree that DCLM Core, the standard suite we follow, mostly consists of clean and well-formed prompts. However, we would like to note that ReScraper still keeps natural noises, leaving 37.7% of the extracted pages unchanged and editing another 17.9% by less than 10% of their words (Figure 6). Since <edit> operates by deleting lines or strings, it is not designed to correct typos or informal language in the main content. The refining teacher's prompt also instructs it to "keep all typos" and not to "fix" awkward phrasing or non-native English (Appendix D.1), and ReScraper's own system prompt quotes the same rules.

To measure robustness directly, we evaluate the 1B-setting models on three noisy benchmarks, AdvGLUE with its adversarial examples, and Winogrande and SQuAD with spelling noise (two adjacent letters swapped in about 20% of the words, with the answer options and references unchanged). As shown in the table below, ReScraper performs the best on all three, and raw text, which keeps all the noise, falls behind it. This suggests that ReScraper's cleaning does not hurt robustness to noisy text on these benchmarks. This evaluation is in the revised paper (Appendix B.5).

1B SettingAdvGLUE (Accuracy)Winogrande with Spelling Noise (Accuracy)SQuAD with Spelling Noise (F1)
Raw text44.2154.3042.92
RefinedWeb-rule49.9252.7246.31
UltraX47.3254.5447.05
DataOrchestra47.5953.5941.56
ReScraper53.0556.2752.45

Question 1: Why cleaning does not seem to help reading comprehension

Response: Thank you for the question. ReScraper achieves the second-best reading comprehension at both 1B and 3B, close to the best method in each (Table 2).

We hypothesize that some noisy pages in raw text still help reading comprehension. Forum threads, comment sections, and FAQ pages often look messy, yet they contain question-answer exchanges and dialogue, the formats that SQuAD and CoQA test. Raw text keeps all of these pages, and at 3B its score in this category is only 0.003 below the best method.

Consistent with this hypothesis, the methods with the lowest reading comprehension score at 1B and 3B are those that delete the most pages, C4-rule and FineWeb-rule, which keep only about 5B unique tokens and whose whole-page rules likely remove many of these pages.


In conclusion, we hope our responses resolve your concerns:

  1. We have removed the "weak-to-strong" term; Qwen3-0.6B has a lower Core than the model pretrained on ReScraper's data in the 3B setting, and UltraX and DataOrchestra build on Qwen3 models of the same or larger size (W1).
  2. ReScraper leaves more than half of the pages unchanged or lightly edited and performs the best on three noisy benchmarks (W2).
  3. ReScraper is second-best in reading comprehension at 1B and 3B, and raw text stays competitive likely because some of its noisy pages still help this ability (Q1).

We sincerely thank you for your time and effort in reviewing our paper, and we are happy to address any follow-up questions.

Reviewer_BLbpOpenReview ↗

Thanks for the rebuttal

I appreciate the effort in your response and the revised manuscript. On W1, thanks for removing the overclaim. The editorial fix improves the paper but does not add evidence, and I wouldn't lean on the new Qwen3-0.6B comparison, since that model leads on 3/5 categories in your own table. W2 adds real evidence, but 2 of the 3 benchmarks test artificial spelling noise and none covers structural mess; I'd like to see evaluation on naturally noisy inputs, including the structural noise your extraction step is built to remove. Q1 remains an untested hypothesis, and the 400M case, where ReScraper's reading comprehension falls below raw text, is not discussed. I'd like to keep my score.

Thank you for engaging with our discussion. We address the remaining points below:

Weakness 1: Qwen3-0.6B comparison

Response: As in our earlier response, the Qwen3-0.6B comparison is provided for reference. We would like to note that the main baselines also build on Qwen3 models of the same or larger size. UltraX builds on Qwen3-0.6B, and DataOrchestra uses a Qwen3-1.7B orchestrator with Qwen3-0.6B and Qwen3-4B tool models. Under identical pretraining, ReScraper outperforms UltraX by a relative 3.9% to 10.0% and DataOrchestra by 3.8% to 6.6% across three scales (Table 2). Since all three start from Qwen3 models, the gains of ReScraper are unlikely to come from its initialization alone, and its core advantage lies in the unified model for scraping and cleaning.

Weakness 2: Evaluation on naturally noisy inputs, including structural noise

Response: Thank you for the suggestion. We would like to note four points.

(1) AdvGLUE [1] is designed to evaluate the robustness of language models to noisy and adversarial inputs. It applies typos, word substitutions, syntactic changes, and distracting sentences to five GLUE tasks (SST-2, QQP, QNLI, RTE, and MNLI), and adds human-written adversarial examples. It is widely used (400+ citations), and ReScraper performs the best on it, as shown in our earlier response.

(2) ReScraper also performs the best in question answering over real tweets that may contain natural noise. We further evaluate the 1B-setting models on TweetQA [2], which asks human-written, free-form reading comprehension questions about these tweets. As shown in the table below, ReScraper achieves the best BLEU-1 and METEOR, and raw text, which keeps all of the noise, scores the lowest. As BLEU-1 measures word precision and METEOR also rewards recall, ReScraper's answers are likely more precise while covering the reference answer well.

1B SettingTweetQA (BLEU-1)TweetQA (METEOR)
Raw text50.2130.62
RefinedWeb-rule53.8032.35
UltraX52.6530.72
DataOrchestra51.3931.51
ReScraper57.1032.36

(3) ReScraper also performs the best in reading comprehension over web pages that may contain structural noise. Similarly, we evaluate the 1B-setting models on WebSRC [3], which asks questions over real web pages with key-value, comparison, and table layouts, where answering requires understanding the structure of the page, such as which value belongs to which key or table header. As shown in the table below, ReScraper achieves the best EM and F1, and raw text, which keeps the boilerplate, significantly falls behind it.

1B SettingWebSRC (EM)WebSRC (F1)
Raw text30.5040.48
RefinedWeb-rule31.5042.61
UltraX26.0039.11
DataOrchestra37.2048.86
ReScraper39.0049.63

If the noise in these benchmarks differs from what you have in mind, we are happy to evaluate on benchmarks you specify.

(4) Structural noise is commonly removed before pretraining. Removing boilerplate such as navigation menus, headers, footers, and advertisements from web pages is a standard practice in pretraining pipelines, as mentioned in the technical reports of Llama 3 [4], MAI-Thinking-1 [5], and K2-V2 [6]. Consistent with this practice, removing <extract> in our ablation (Table 3), which keeps the boilerplate in the corpus, lowers every task category in the 1B setting and Core by a relative 16.6%.

Question 1: The 400M reading comprehension result

Response: Reading comprehension in DCLM-Core consists of three tasks: BoolQ, SQuAD, and CoQA. As shown in the table below, we find that the gap to raw text at 400M comes mainly from BoolQ, while ReScraper achieves the best score among all methods on SQuAD and CoQA.

400M SettingBoolQ (Accuracy)SQuAD (EM)CoQA (EM)Reading Comprehension
Raw text59.827.0010.520.03926
C4-rule54.343.6810.61-0.01954
RefinedWeb-rule44.5911.0213.40-0.07126
FineWeb-rule59.799.7314.320.06072
ProX-C57.589.9812.310.03558
UltraX49.0812.6513.82-0.02509
DataOrchestra47.7111.7813.80-0.04011
ReScraper50.1213.2514.94-0.01023

BoolQ is known to be unstable for small models. For example, DataDecide [7] finds that BoolQ is the only one of its ten tasks without non-trivial performance for models from 4M to 1B parameters, and Signal and Noise [8] identifies BoolQ as a benchmark with high noise between checkpoints. We therefore consider the results at larger scales more stable and more meaningful. The standard deviation of Core across seeds drops from up to 0.0040 at 400M to at most 0.0027 at 1B and 3B (Figure 11), and larger settings are closer to the model sizes used in practice. At both 1B and 3B, ReScraper achieves the second-best reading comprehension, close to the best method, and the best Core.

We hope these clarifications address your remaining concerns, and we are happy to discuss further.

AuthorsOpenReview ↗

Reference citations

[1]: Wang, Boxin, et al. "Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models." NeurIPS 2021.

[2]: Xiong, Wenhan, et al. "TWEETQA: A Social Media Focused Question Answering Dataset." ACL 2019.

[3]: Chen, Xingyu, et al. "WebSRC: A Dataset for Web-Based Structural Reading Comprehension." EMNLP 2021.

[4]: Llama Team, AI @ Meta. "The Llama 3 Herd of Models." arXiv 2024.

[5]: Microsoft AI. "MAI-Thinking-1: Building a Hill-Climbing Machine." 2026.

[6]: K2 Team. "K2-V2: A 360-Open, Reasoning-Enhanced LLM." arXiv 2025.

[7]: Magnusson, Ian, et al. "DataDecide: How to Predict Best Pretraining Data with Small Experiments." arXiv 2025.

[8]: Heineman, David, et al. "Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation." arXiv 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.