- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Generative Curation with LLMs (GC-LLM), a pipeline that uses natural-language specifications to select web pages and extract their content as Markdown. The deployed pipeline uses a fine-tuned Qwen3-8B in two stages. It first makes a keep/drop decision from Resiliparse text, then extracts content from script-stripped HTML for the pages it keeps. To reduce the cost of running the 8B model at Common Crawl scale, the authors develop GC-Cascade, which pairs an agent-modified Resiliparse fork with three classifiers trained on the LLM's keep/drop labels. The cascade achieves 0.871 filtering F1 against its teacher at a reported 921× higher throughput. On the DCLM-400M-1x pool, GC-Cascade retains 91.7B tokens, compared with 7.3B for DCLM-baseline and 10.1B for Nemotron-CC. After each corpus is reweighted with Olmix, GC-Cascade achieves 7.5–17% lower best-observed bits per byte (bpb) on the 51-task Olmix suite across three model sizes. The paper also evaluates extraction quality, topic coverage, cross-document repetition, and hallucinations.
Critical strengths
-
The pretraining sweep is substantial. The authors run more than 500 experiments across three model sizes and twelve compute budgets. Using the same AdamH recipe throughout removes per-run optimizer tuning as one explanation for differences between runs. Including unfiltered Resiliparse is also a good choice because it gives readers a direct reference point for assessing whether curation helps.
-
The throughput results make the cascade a practical engineering contribution. I also appreciated the analysis showing that WRAP-style rewriting increases cross-document 4-gram repetition by 1.6–3.3× while spec-based extraction stays close to rule-based extractors. The hallucination audit and the HumanEval fence-stripping caveat make it easier to understand where the pipeline fails.
Critical weaknesses
-
The main claim and the evaluated pipeline are misaligned. The abstract and Sec. 2 describe a generative transformation that takes raw HTML and handles both selection and extraction. In Sec. 3.2, however, the keep/drop decision is made from text already extracted by Resiliparse. Any content lost at that stage is unavailable to the filter. This matters for the example in Figure 2, which motivates the method in part through content that conventional extraction can miss. The deployed extraction spec also asks for faithful transcription into Markdown. The math-page rewrite shown in Figure 2 is evaluated in the continued-pretraining study in App. J, separately from the main pretraining experiments. Finally, the system that produces the strongest results, GC-Cascade, uses a rule-based extractor and classifiers trained on LLM labels. LLMs contribute to building that system offline. The paper needs to identify what this offline contribution adds beyond the LLM-labeling recipes it already discusses in Sec. 2, and tie that contribution to the reported gains.
-
The experiments do not identify which part of the pipeline improves pretraining. With Olmix, GC-LLM and unfiltered Resiliparse reach similar minimum validation loss in Figure 1 and similar bpb at the largest budgets shown for 1B and 2.9B in Figure 4. GC-Cascade, meanwhile, outperforms GC-LLM by 5.0–7.3% in best-observed bpb at those two sizes. The teacher-to-cascade comparison changes both the retained documents and their extracted text. The cascade keeps 91.7B tokens versus 111.3B for its teacher, and the fork has fewer audited hallucinations than the Qwen extractor. Either difference could matter, and the two may interact. The stricter single-spec corpus in App. I is competitive at small and moderate budgets, which gives another reason to investigate the retained subset, although that experiment changes multiple factors. A crossed filter-by-extractor ablation is needed to attribute the cascade's gains. A matched-size comparison against a filter trained on an existing quality signal would also help establish whether the LLM labels themselves add value.
-
The headline comparison combines corpus quality, corpus size, and deduplication. The headline reports each corpus's best observed bpb across budgets. GC-Cascade has far more unique tokens available, so this comparison captures how well a larger corpus can sustain training as the smaller corpora start to repeat. At budgets below 7.3B tokens, the GC-Cascade and DCLM-baseline curves in Figure 4 are close across all three model sizes. The larger gaps appear mainly at higher budgets, where differences in repeated exposure become important. There is also a construction asymmetry in Sec. 5.1.2. The GC corpora and unfiltered Resiliparse are deduplicated inside the experimental WARC pool, while DCLM-baseline and Nemotron-CC are slices of releases deduplicated over a larger pool. A page may be absent from a baseline slice because its retained duplicate lies outside the sampled WARCs. The paper does not quantify this effect. A comparison with the same source pool, deduplication scope, unique-token count, and training budget would make it possible to assess data quality separately from the amount of data available.
-
Olmix makes it hard to tell what the curator contributes and how well the result generalizes. Every corpus is reweighted over topic and external quality bins, and the mixtures are optimized for the same 51-task suite used for the headline comparison. Applying the same procedure to all corpora is reasonable if the goal is to compare the best mixture each pool can produce. It does not establish that permissive curation helps independently of the quality signal used to construct those mixtures. Olmix can upweight high-scoring documents from a lightly filtered pool, so part of the selection is happening at the mixing stage. App. K is relevant here because its experiments without Olmix favor the stricter spec in the fitted compute-optimal regime, although they use older specs and cannot settle the question for the deployed system. Uncheatable Eval is a useful independent outcome, but it still evaluates models trained on the quality-binned mixtures. I would like to see the main comparison on the corpora's natural distributions and an intermediate condition that mixes by topic alone.
-
The extraction evidence is too weak for the claims made about extraction quality. On WebMainBench, the human-annotated benchmark in Table 2, the GC extractors score 0.646–0.708 F1 versus 0.900 for MinerU-HTML. MinerU-HTML's co-development with the benchmark is a reasonable caveat. The gap still needs analysis because it appears in both precision and recall. In particular, the paper should show whether the lower recall comes from different definitions of main content or from missing text that the method also intends to keep. The in-house devset is closer to the authors' intended output, but its gold extractions are LLM-produced and some of its categories explicitly reward content requested by the extraction spec, such as comment attributions. Even there, the deployed Qwen extractor scores 0.805 F1 versus 0.817 for vanilla Resiliparse and 0.815 for MinerU-HTML. The fork reaches 0.828, with no uncertainty intervals reported for these differences. The stronger GLM-5.3 result demonstrates what another extractor can achieve on this devset. Its relevance to the training corpora is limited because it did not build them. Table 1 also excludes a substantial share of examples that exceed 32k tokens, leaving a gap in the model-selection evidence for the long pages that require chunking in deployment.
-
The cost comparison omits work needed to build the deployed pipeline. The reported 5.3e21 FLOPs for GC-Cascade describes the cost of running the cascade. The pipeline also requires labeling with the 8B teacher, training the classifiers, and developing the extractor through 179 agent cycles. The pretraining results rely on an additional mixture search. From the numbers in App. F, the 363 proxy runs at 3e18 FLOPs each total about 1.1e21 FLOPs per corpus, before document-level topic and quality scoring. Reporting build, inference, and mixture-search costs separately would let readers see which costs are incurred once and which recur when the corpus or specification changes. The paper should also reconcile Figure 1's operation order with Sec. 4's description of the text-input FastText classifier, since the order determines how many pages incur extraction cost.
-
The paper has not demonstrated spec-level controllability for the cascade. A natural-language spec is an appealing interface for GC-LLM. GC-Cascade approximates a fixed spec using trained classifiers and a rule-based extractor. A change to the filtering policy may require relabeling and classifier retraining, while a change to extraction may require modifying the fork. The paper does not show how much of that work is required for a realistic spec change or whether the rebuilt cascade follows the new spec. Since the scalable system is the one most users would run, controllability should be evaluated there. One targeted edit, with an evaluation of both the intended behavior change and the effect on unrelated pages, would make this claim much easier to assess.
Questions for authors
-
At 1B, can you cross the Qwen and cascade filters with the Qwen and fork extractors under the same mixing procedure and matched training budgets? For each filter, can the two extractor runs use identical document IDs? Can you also compare the LLM-derived filter against an existing quality-score filter at the same retention rate?
-
Can you rerun DCLM-baseline on the same WARC pool under the same deduplication scope, then compare at matched unique-token counts and training budgets? If rerunning the baseline is infeasible, can you estimate how many documents the release slice loses because the retained copy lies outside the pool and report the actual repeated-token fraction at each budget?
-
How do the main corpora compare under their natural distributions, topic-only mixing, and topic-plus-quality mixing? Can you report all three conditions on an evaluation suite excluded from mixture fitting?
-
Can you break down the WebMainBench precision and recall gaps by error type? Which prompt produced the in-house gold extractions, and why does Sec. 6.1 evaluate 412 pages when Sec. 3.3 and the AI-use statement mention 420?
-
What is included in the 5.3e21 FLOPs estimate? In the deployed system, which text does FastText see, which input does the pooled transformer use, and how many pages pass through the fork?
-
Can you demonstrate one realistic spec change, such as retaining more substantive forum replies, and report the relabeling and retraining required, how often the desired decision changes, and how often unrelated decisions change?
-
Can you make Figure 1 and the abstract explicit about the systems used for the main pretraining results and indicate where the Figure 2 rewrite example was actually evaluated?
4 comments
Response to Reviewer ePF3 (2/2)
W6. The cost comparison omits work needed to build the deployed pipeline. We include a more thorough breakdown of costs in Appendix F, “Compute Cost”; however, it is worth noting that building GC-Cascade costs about 4E+22 FLOPs, and because the fine-tuned Qwen3-8B can be reused, rebuilding it for a new spec costs about 8E+21 FLOPs, comparable to running it on the full Common Crawl. We updated this throughout the draft and the limitations section to clarify the cost.
W6.1. The paper should also reconcile Figure 1's operation order with Sec. 4's description of the text-input FastText classifier, since the order determines how many pages incur extraction cost. The order in Section 4 and Figure 1 has always matched. Text extraction comes before FastText, Pooled Transformer, and ModernBERT. Please clarify if anything remains confusing.
W7. The paper has not demonstrated spec-level controllability for the cascade. Although we don’t rerun distillation for GC-Cascade, we do show several examples of swapping the spec for GC-LLM throughout the Appendix (we offer more than 5 different spec ablations). Even though we haven’t distilled this into GC-Cascade, there is no indication that this wouldn’t work for other specs. We prioritized other experiments because the confidence that this would work is quite high, but can consider running this as other higher-priority experiments conclude.
Q1. At 1B, can you cross the Qwen and cascade filters with the Qwen and fork extractors under the same mixing procedure and matched training budgets? Yes! We ran this experiment and included it in Appendix J. The filtering and extraction both help, but extraction is what sets GC-Cascade and GC-LLM apart.
Q2. Can you rerun DCLM-baseline on the same WARC pool under the same deduplication scope, then compare at matched unique-token counts and training budgets? Done! We address this in the shared response, but the difference in tokens was minimal: 7.3B -> 9.5B tokens, with very minor shifts in eval results. We agree that this is the version that should be reported in the camera-ready version.
Q3. How do the main corpora compare under their natural distributions? We report this in Appendix H. At matched unique tokens and compute budgets under natural mixes, our GC-Cascade is still the best data available, but in this setting it is tied with DCLM.
Q4. Can you break down the WebMainBench precision and recall gaps by error type? Yes, see the above analysis for W5! The 420 vs 412 gap is because we removed documents that Miner-U-HTML produced empty extractions, but we will update our AI-Use statement to address this.
Q5. What is included in the 5.3e21 FLOPs estimate? In the deployed system, which text does FastText see, which input does the pooled transformer use, and how many pages pass through the fork? The 5.3e21 FLOPs estimate is based on running on the Common Crawl; however, we added Appendix F with detailed cost calculations. To your second point, FastText and PooledTransformer both use the extracted text, so every page runs through the fork.
Q6. Can you demonstrate one realistic spec change? We show this in the appendix with our Budget Aware Curation, where we modify the definition of what quality documents look like. We chose not to run this with the cascade to save compute for higher-priority experiments; however, adapting to a new spec is on the order of 8E+21 FLOPs, and training the classifiers alone is under 1E+20 FLOPs.
Q7. Can you make Figure 1 and the abstract explicit about the systems used for the main pretraining results and indicate where the Figure 2 rewrite example was actually evaluated? We redesigned Figure 1 to make it clear that we are using a Harness with the LLM, and we updated Figure 2 with only the main paper examples. Please let us know if anything here is unclear.
Thanks for your helpful comments!
References
[1] Will Held. Curating 25 Trillion Tokens for LLM Pretraining. Open Athena Blog, 2026. https://www.openathena.ai/blog/marin-data-pipeline-overview/
[2] Aaron Grattafiori et al. The Llama 3 Herd of Models. arXiv:2407.21783, 2024.
[3] Christopher Mohri, John Duchi, and Tatsunori Hashimoto. A Bitter Lesson for Data Filtering. arXiv:2605.19407, 2026.
[4] Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, and J. Zico Kolter. Scaling Laws for Data Filtering—Data Curation Cannot be Compute Agnostic. CVPR 2024.
Thank you for the detailed rebuttal and additional experiments. I appreciate that several of my concerns have been addressed. However, the performance advantage remains less convincing under matched data and compute conditions, and the spec-level controllability of GC-Cascade has not been demonstrated. I also remain concerned about the distinct contribution of LLM-based curation over existing approaches. Considering the additional evidence, I have decided to raise my rating to 2, although my main concerns remain.
Thank you for your feedback. We appreciate your acknowledgment that we addressed several of your concerns and your corresponding score adjustment.
We maintain that the data mixed setting is the correct way to compare the datasets (due to matching the deployment setting of these datasets), so we are undeterred that the performance improvements are less pronounced (yet still present!) in the unmixed setting. However, we understand how this can be a point of contention given the departure from traditional dataset evaluation and will keep the unmixed results in the appendix.
We greatly appreciate your time and suggestions that made our paper better!
Response to Reviewer ePF3 (1/2)
Thank you for taking the time to review our paper. We appreciate your recognition of the extensiveness of our experiments and the engineering contribution of our GC-Cascade. We will address your comments in order:
W1. Evaluated pipeline misaligned. We adjusted our Figure 1 to make it clearer that we are discussing an LLM as part of a harness. Figure 2 was an accurate representation of how our system performed on those pages, taken verbatim from the outputs, so we did not run into the same problems as DCLM; however, we recognize that it is confusing that we showed the spec for Appendix J in Figure 2, so we modified the figure to showcase one filtering failure and one extraction failure and to only use our main paper spec. We were very upfront about our system in Section 3.2, and we hope the clearer Figure 1 and Figure 2 address your concern.
W2. A crossed filter-by-extractor ablation is needed. We provide this experiment in Appendix J, “Data Curation Ablations”. Our summarized finding is that both filtering and extraction matter for GC-Cascade outperforming baselines, but GC-Cascade wins over GC-LLM due to extraction. This appears consistent with findings that GC-Cascade avoids extraction errors.
W3. A comparison with the same source pool, deduplication scope, unique-token count, and training budget. We primarily address this in the shared response, but we ran this experiment and included the results in Appendix H, “Impact of Data Mixing.” As we discussed in the shared response, this has not significantly impacted scores, but we agree it is a more sound way to run our evals, so when everything is complete in about a week, we will update it for the camera-ready version.
W4. Olmix makes it hard to tell what the curator contributes. We address this with Appendix H, “Impact of Data Mixing,” and find that without any mixing, our data remains the best even at matched unique tokens and compute. However, as we discussed with reviewer cZRK, we stand by our evaluation setting. Downstream, when training models, it is standard practice to mix data [1][2]. Other papers argue that you should filter your data far less than current pretraining data curation methods dictate [3][4]. For far too long, pretraining data curation filtering has overstepped and tried to support what should be the role of data mixing. It is great for most evals to filter out literature in favor of more code/math data, but this is not a tradeoff you want to make if you want your pretraining data to be useful for both training models good at coding and good at creative writing. We stand by this evaluation setting because it is faithful to how models should be trained and the true purpose of pretraining data curation. Data filtering should not assume the role of data mixing, and we hope more researchers will follow this lead.
W5. Cannot claim better extraction. We ran an error analysis on WebMainBench to break down where the precision and recall gap between our LLM extractors and MinerU-HTML comes from.
We used Gemini 3.8 Flash to label extraction gaps using a 14-category codebook (page furniture, metadata, main text, text not on the page, etc.). We validated Gemini against blind labels from Opus 5.5 with 95.6% agreement on held-out spans. The recall gap is almost entirely formatting. For Qwen3-8B Distill (the GC-LLM extractor), formatting is 16.4 of the 19.2 points, and it is 76–94% of the gap across our extractors. The precision gap is part formatting and part extra text. Formatting is 10.1 of the 27.9 points for Qwen3-8B Distill. Of the extra text, page furniture (menus, footers, related links) is 10.4 points for the 8B model and about 4 for the larger models. Metadata is another 3–5 points, including the "Author -- date" line on every post, which our spec requires and WebMainBench excludes. Text with no basis on the page is under 0.3 points.
If we score on words only (ROUGE-1, dropping whitespace, punctuation and markdown marks), GLM-5.3 and Qwen3-8B Distill match MinerU-HTML on recall:
GLM-5.3's F1 gap shrinks from 19.2 to 4.4 points. The remaining gap is primarily the metadata and spec differences described above.