acceptodds
Under review as a conference paper at ICLR 2027

CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

Abstract

Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance. However, these pipelines typically rely on singular optimization objectives, which inevitably narrows distributional diversity and marginalizes long-tail knowledge, thereby restricting data coverage and underutilizing the vast potential of the open web. To address this limitation, we propose a novel curation paradigm that shifts from linear pruning to the joint optimization of quality, redundancy, and diversity. This framework synergizes dual-track cleaning (rule-based and model-driven) with hybrid deduplication (n-gram and semantic), while employing a multi-objective sampler to balance informational quality with distributional breadth. Applying this framework to Common Crawl, we construct CuraWeb, a 2T-token English corpus. Unlike existing resources, CuraWeb establishes an industrial-grade standard for data curation by recovering a more holistic data distribution with enhanced diversity and minimal redundancy, achieving broader coverage of long-tail knowledge across diverse domains. Experimental evaluations at the 3B scale demonstrate that CuraWeb significantly outperforms state-of-the-art baselines, yielding an average performance gain of 1.8% across a wide range of benchmarks, particularly in knowledge-intensive and reasoning tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, pre-train, data curation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W1LoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CuraWeb, a 2T-token English pretraining corpus constructed from Common Crawl. The main motivation is that highly selective web-data curation pipelines may improve average data quality while reducing distributional diversity and discarding useful long-tail content.

CuraWeb combines several stages in a unified pipeline: multi-dimensional document quality and domain annotation, domain-aware heuristic and model-based filtering, lexical and semantic deduplication, and a final sampling stage that combines content-value and diversity scores. For semantic deduplication, the paper proposes a weighted score-accumulation strategy rather than a single hard similarity threshold. For sampling, document-level quality and diversity scores are combined and transformed into sampling probabilities using a power-based scheme.

The authors train 3B models from scratch under matched model and training settings to compare CuraWeb against DCLM, FineWeb-Edu, Nemotron-CC, and Dolma3. CuraWeb obtains the best average downstream performance at 200B training tokens, with a reported 1.82-point improvement over DCLM, and the advantage persists when training is extended to 1T tokens. The paper also provides component ablations and corpus-level analyses of quality, redundancy, domain coverage, and lexical diversity.

Critical strengths

1. Large-scale and practically meaningful evaluation.

The paper evaluates the proposed curation pipeline by actually training 3B models from scratch under matched architectures, optimization settings, and token budgets. This is substantially more informative than evaluating corpus statistics alone. The reported improvement over strong public corpora is also non-trivial, and the advantage persists when training is extended from 200B to 1T tokens.

2. The paper provides a relatively complete end-to-end analysis of the data-curation pipeline.

The work studies filtering, lexical and semantic deduplication, sampling, domain coverage, lexical diversity, and content-value distributions within one large-scale pipeline. The component ablations and corpus analyses provide useful practical evidence about how different curation stages reshape the final training distribution.

Critical weaknesses

1. The contribution is dominated by engineering integration, while the methodological novelty is not sufficiently established.

The proposed framework combines several practically reasonable components: domain-aware heuristic filtering, model-based multi-dimensional quality scoring, lexical and semantic deduplication, and quality-diversity sampling. However, each of these directions has close precedents in the existing data-curation literature. The paper itself discusses semantic deduplication, DoReMi-style mixture optimization, SampleMix-style quality-diversity sampling, and WebOrganizer-style domain organization, but the experimental section does not provide matched method-level comparisons against these closely related approaches.

As a result, it is difficult to determine which part of CuraWeb constitutes a genuinely new methodological contribution, as opposed to a carefully engineered combination and tuning of existing ideas. I consider the engineering scale valuable, but in its current form I do not find the methodological novelty sufficiently strong for the claims made in the paper.

2. The "joint optimization" framing is stronger than what is actually formulated and validated.

The title presents CuraWeb as jointly optimizing quality, redundancy, and diversity. However, I do not see a unified optimization problem over these three objectives. Instead, they are addressed at different pipeline stages using manually chosen thresholds, rule modifications, empirically calibrated deduplication weights, and a fixed linear combination of quality and diversity for sampling.

For example, the sampling score uses a fixed coefficient of α=0.6\alpha = 0.6, while redundancy is handled separately by the preceding deduplication procedure. There is no demonstrated Pareto analysis, constrained optimization, learned trade-off, or other evidence that the three objectives are jointly optimized rather than sequentially managed.

Relatedly, the end-to-end comparison against other corpora establishes that the full CuraWeb pipeline produces a strong training corpus, but does not establish that the claimed joint-optimization principle is responsible for the improvement. The current ablations mainly remove broad components (semantic deduplication or non-uniform sampling) rather than comparing the proposed implementations against strong alternatives for the same component. Therefore, the causal attribution of the reported 1.82-point gain to the paper's claimed methodological contribution remains unclear.

Questions for authors

1. How does each proposed component compare against the closest existing method under exactly the same input corpus and training budget?

In particular, I would like to see matched comparisons between the proposed semantic deduplication strategy and existing semantic-deduplication methods, and between the proposed quality-diversity sampling strategy and closely related approaches such as SampleMix or other data-mixture/domain-aware sampling methods. Comparing only the final CuraWeb corpus against other final corpora does not isolate the contribution of the proposed methodology. A clear advantage under controlled component-level comparisons would materially strengthen my assessment of novelty.

2. What exactly is being "jointly optimized" in CuraWeb?

Quality, redundancy, and diversity appear to be handled largely through different stages of the pipeline, and I do not see a unified objective or optimization procedure involving all three. Is "joint optimization" intended as a formal methodological claim, or does it mean coordinated use of shared signals across the pipeline? If the former, please clarify the optimization formulation and how the reported method solves it.

3. How sensitive are the final results to the manually selected design choices in the pipeline?

Several important choices appear manually specified or empirically tuned, including filtering thresholds, semantic-deduplication interval weights and target score, the quality-diversity coefficient α=0.6\alpha=0.6, and parameters of the power-sampling function. I would like to know whether the main downstream gains remain stable under reasonable alternatives, particularly for the quality-diversity trade-off. This would help distinguish a robust general principle from a highly tuned engineering pipeline.

3 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you very much for reviewing our paper and for providing so many valuable suggestions. In response to your concerns, we provide the following replies. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to Q1: How does each proposed component compare against the closest existing method under exactly the same input corpus and training budget?

Thank you for this valuable suggestion. Resource constraints have prevented us from conducting the full set of requested controlled comparisons. However, we have earlier experimental records comparing semantic deduplication strategies and sampling strategies on shared candidate corpora. We summarize these existing comparisons below.

Soft-threshold versus hard-threshold semantic deduplication.

We compare our soft-threshold semantic deduplication with hard-threshold deduplication using a fixed threshold of 0.95 on the same input corpus. At the shared 44,000-step checkpoint, our method improves MMLU [1] accuracy by 1.05 percentage points.

Table 1. Semantic deduplication comparison.

Deduplication methodMMLU accuracy (%)
Hard-threshold deduplication (0.95)49.56
Soft-threshold deduplication (ours)50.61

Power sampling versus SampleMix [2].

We compare Power sampling with SampleMix in our experiments. Both methods select training data from the same candidate corpus using the same fused quality scoring scheme. We train 2.4B MoE models on the resulting data and compare their downstream performance at the shared 16,000-step checkpoint. Power sampling improves accuracy by 1.01, 0.85, and 1.89 percentage points on MMLU, CMMLU, and C-Eval, respectively. These results support the sampling strategy under a shared quality scoring scheme.

Table 2. Sampling strategy comparison.

Sampling methodMMLU accuracy (%)CMMLU [3] accuracy (%)C-Eval [4] accuracy (%)
Softmax sampling42.7132.7132.48
Power sampling43.7233.5634.37

These existing experiments provide component-level evidence beyond final-corpus comparisons. We nevertheless acknowledge that they do not replace the full set of strictly matched comparisons against existing methods requested by the reviewer.


Response to Q2: What exactly is being "jointly optimized" in CuraWeb?

Thank you for pointing this out. Joint optimization in CuraWeb refers to the coordinated use of shared signals at each stage of the pipeline to balance quality, redundancy, and diversity. Each stage addresses its primary objective while considering its effects on quality, redundancy, and diversity. We have clarified this meaning in the revised manuscript.

Specifically, for deduplication, DCLM [5] uses a Bloom filter, while FineWeb-Edu [6] inherits FineWeb’s MinHash deduplication. CuraWeb adds semantic deduplication and accumulates similarity evidence from multiple retained documents before removal. This balances redundancy reduction with semantic coverage by limiting false removals.

For quality filtering, DCLM filters by fastText [7] score, while FineWeb-Edu filters by educational value. CuraWeb combines domain labels with multiple quality signals, allowing math and code documents to bypass restrictive heuristics while still undergoing quality checks, which balances quality improvement with domain diversity. For sampling, DCLM adjusts data-source proportions in its mixing experiments, while FineWeb-Edu randomly samples its filtered corpus. CuraWeb combines the content value with the diversity score of each document, allowing quality and semantic coverage to jointly determine sampling probabilities.


AuthorsOpenReview ↗

Author Response (2/3)

Response to Q3: How sensitive are the final results to the manually selected design choices in the pipeline?

Thank you for this question. Below, we explain the hyperparameter choices for three components: data selection, rule-based filtering, and semantic deduplication.

1. Selection hyperparameters

Our sampling procedure is as follows:

si=αqi+(1−α)di,wi=xinσ(k(xi−a)),pi=wi∑jwjs_i=\alpha q_i+(1-\alpha)d_i, \qquad w_i=x_i^n\sigma\left(k(x_i-a)\right), \qquad p_i=\frac{w_i}{\sum_j w_j}

where qiq_i and did_i are the normalized quality and diversity scores, xi∈[0,1]x_i\in[0,1] is the min-max normalized fused score sis_i, and σ\sigma denotes the sigmoid function. We use α=0.6\alpha=0.6, n=3n=3, k=2k=2, and a=0.7a=0.7. The rationale for this configuration considers both the theoretical behavior of the weighting function and the resulting sampling distribution.

Theoretical rationale. The parameters control complementary aspects of sampling. Setting α=0.6\alpha=0.6 gives quality a greater weight while allowing semantic diversity to substantially influence document selection. A document with moderate quality and high semantic distinctiveness can therefore receive a higher score than a higher-quality document from a redundant region. The exponent n=3n=3 provides a superlinear preference for higher-scoring documents. The sigmoid provides a smooth adjustment. With k=2k=2, its value ranges from approximately 0.198 to 0.646 over [0,1][0,1], and a=0.7a=0.7 places the gate’s midpoint at 0.70.7. Documents with positive scores below this midpoint retain nonzero sampling probabilities, allowing graded selection across the score range.

Distributional evidence. Appendix J shows that, with about 681.8M documents sampled by each method, the average score rises from 0.489 before sampling to 0.550 with Softmax and 0.611 with Power. Power sampling selects more high-scoring documents while still keeping documents with medium scores.

2. Rule-filter hyperparameters

The motivations for our rule-filter hyperparameters are summarized in Table 8 of Appendix D. We have further expanded Appendix D in the revised manuscript with the calibration procedure, quantitative threshold comparisons, and quality-stratified retention analyses.

Threshold calibration. We evaluate each rule against model-based reference labels, giving greater importance to avoiding the rejection of higher-quality documents. As shown in Table 1, we select thresholds that improve precision over the default setting and achieve at least 85% weighted precision, while considering the fraction of documents filtered.

Table 1. Rule-filter threshold calibration and filtering performance.

RuleDefault → revised thresholdWeighted precisionFiltered fraction
Minimum word count50 → 2563.93% → 87.26%12.24% → 3.25%
Symbol-to-word ratio0.10 → 0.3859.14% → 87.50%0.24% → 0.01%
Alphabetic ratio0.80 → 0.2646.53% → 86.96%7.79% → 0.01%

Distributional evidence. Reducing the minimum word-count threshold from 50 to 25 increases retention in quality-score segment 4 from 94.4% to 100.0%. The revised rule retains approximately 100% of documents in segments 3–10 on the calibration study’s 0–10 quality scale. These retention results indicate that the adjustment reduces the loss of potentially useful short documents. The analysis is reported in Appendix M.2.

Rule removal and technical-content protection. The five removed rules had weighted precisions ranging from 16.70% to 67.73%. This supports removing constraints that cannot reliably distinguish structural noise from usable content. Similarly, source-specific calibration remained ineffective for Math-, STEM-, and Code-oriented data, motivating the technical-content bypass described in the manuscript. These documents remain subject to subsequent model-based quality checks.

AuthorsOpenReview ↗

Author Response (3/3)

Response to Q3: How sensitive are the final results to the manually selected design choices in the pipeline?

3. Semantic deduplication

We set the target score to T=10T=10 and assign different weights to different cosine-similarity intervals. We evaluate these choices through experimental comparisons and statistical analyses, which are detailed in Appendix E of the revised manuscript.

Experimental validation. We compare Broad matching with our adopted Balanced matching configuration. Table 2 summarizes their interval weights.

Table 2. Similarity-interval weight configurations.

Cosine similarityBroad matchingBalanced matching
[0.85,0.88)[0.85,0.88)1.50
[0.88,0.90)[0.88,0.90)30
[0.90,0.92)[0.90,0.92)52
[0.92,0.94)[0.92,0.94)53
[0.94,0.95)[0.94,0.95)74
[0.95,0.96)[0.95,0.96)75
[0.96,0.97)[0.96,0.97)76
[0.97,0.98)[0.97,0.98)8.56
[0.98,0.99)[0.98,0.99)8.57
[0.99,1.00][0.99,1.00]108

As shown in Table 3, our adopted Balanced matching configuration achieves an MMLU [1] score of 51.11%, outperforming Broad matching by 1.98 percentage points. Compared with Broad matching, it also improves weighted deletion precision by 4.89 percentage points and reduces the deletion rate by 1.58 percentage points. We therefore adopt Balanced matching to reduce erroneous removals while improving downstream performance. Training curves are provided in Appendix E.

Table 3. Filtering performance and downstream improvements.

ConfigurationDeletion rate (%)Weighted deletion precision (%)MMLU (%)
Broad matching12.6482.2049.13
Balanced matching11.0687.0951.11

Statistical analysis. Manual inspection finds that lower-similarity pairs often share topics or terminology but differ in formulas, examples, or factual details. These observations motivate reducing their contributions and requiring more accumulated evidence before removal.

The filtering distributions further show that approximately 69% of clusters have deletion rates below 10%, and approximately 78% have deletion rates below 15%. These findings support concentrating removal in highly redundant regions while limiting its impact on most clusters. These cluster-level statistics and qualitative observations are reported in Appendix M.3.


We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.


References

[1] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding.

[2] Xiangyu Xi et al. (2025). SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity.

[3] Haonan Li et al. (2023). CMMLU: Measuring Massive Multitask Language Understanding in Chinese.

[4] Yuzhen Huang et al. (2023). C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models.

[5] Jeffrey Li et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models.

[6] Guilherme Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.

[7] Armand Joulin et al. (2017). Bag of Tricks for Efficient Text Classification.

Reviewer_PZSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a careful, ground-up data curation approach from CommonCrawl, competing with popular alternatives such as DCLM and FineWeb. It adds additional multi-criterion quality scoring, extra layers of deduplication, and revisits common filtering stages. In empirical ablations at the 3B model parameter scale, the approach outperforms popular baselines for pre-training data curation.

Critical strengths

Good empirical results. Beating DCLM over 200B tokens with a 3B parameter model in an equal optimization setting is a convincing empirical setting for showing the benefits of data quality within an academic setting. The report also is evidence for a strong attention to detail, providing many layers of deduplication, and revisiting popular filtering defaults such as the Gopher heuristic filtering rules.

Critical weaknesses

Lack of novelty: I think all the core ideas in this paper exist in previous literature. However, I think this is still a valuable holistic report, which combines existing ideas with good attention to detail and produces a good cleaned pre-training corpus as a useful artifact.

Some empirical choices remain arbitrary: I appreciate the ablations in Section 4 and the human data inspection which guided several decisions in Section 3. However, some factors like alpha=0.6 seem arbitrary and I could not find details how the power law coefficient n is set. It would be useful to understand how sensitive the proposed framework is to particular values of these selection hyperparameters.

Questions for authors

Q1: In the abstract, you use the phrase "linear pruning" when discussing previous work. Can you explain what it means and what makes the data filtering "linear" in previous work?

Q2: I disagree with the claim that "prevailing data frameworks ... traditionally rely on a singular optimization approach", since DCLM and FineWeb-Edu combine many stages and approaches for data curation, which is also acknowledges in your next paragraph. This multi-stage approach is also shared by CuraWeb, so I think the paper could benefit from some clearer framing of what really sets CuraWeb apart from previous approaches.

Q3: What motivates the use of GPT-4o? There exist both cheaper and better options that have been around for a while now.

Q4: Coulld you provide more detail as to how the L1 and L2 entropy is being computed in Section 5.1? What is the takeaway from this analysis? The finding that FineWeb-Edu filtering drastically reduces domain diversity was already found in previous work like WebOrganizer.

4 comments
AuthorsOpenReview ↗

Author Response (1/4)

Thank you very much for reviewing our paper and for acknowledging the value of our work in the review. Regarding the concerns raised in the review, we provide the following responses. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to Q1: In the abstract, you use the phrase “linear pruning” when discussing previous work. Can you explain what it means and what makes the data filtering “linear” in previous work?

Thank you for pointing out this ambiguity. By “linear pruning,” we intended to describe the tendency of prior baselines to focus primarily on one objective at a given stage. CuraWeb treats quality, redundancy, and diversity as interacting objectives at each stage of the pipeline, with each stage addressing its primary objective while considering its effects on the others. We recognize that the original term was imprecise and have revised the wording in the manuscript. We discuss the baselines’ stage-specific optimization objectives and CuraWeb’s distinctions in detail in our response to Q2.


Response to Q2: DCLM and FineWeb-Edu already many stages and approaches for data curation. What distinguishes CuraWeb, and what did you mean by “a singular optimization approach”?

Thank you for pointing this out. We agree that both frameworks use multiple stages. It should be clarified that our main contribution is a data curation framework that treats quality, redundancy, and diversity as interacting objectives at each stage of the pipeline. Each stage addresses its primary objective while considering its effects on the others. Here, “a singular optimization approach” refers to the tendency of prior baselines to focus on just one objective at a given stage of curation. We recognize that our original wording may have been ambiguous and have clarified it in the revised manuscript.

Specifically, for deduplication, DCLM [1] uses a Bloom filter, while FineWeb-Edu [2] inherits FineWeb’s MinHash deduplication. CuraWeb adds semantic deduplication and accumulates similarity evidence from multiple retained documents before removal. This balances redundancy reduction with semantic coverage by limiting false removals.

For quality filtering, DCLM filters by fastText [3] score, while FineWeb-Edu filters by educational value. CuraWeb combines domain labels with multiple quality signals, allowing math and code documents to bypass restrictive heuristics while still undergoing quality checks, which balances quality improvement with domain diversity.

For sampling, DCLM adjusts data-source proportions in its mixing experiments, while FineWeb-Edu randomly samples its filtered corpus. CuraWeb combines the content value with the diversity score of each document, allowing quality and semantic coverage to jointly determine sampling probabilities.


AuthorsOpenReview ↗

Author Response (2/4)

Response to Q3: What motivates the use of GPT-4o? There exist both cheaper and better options that have been around for a while now.

Thank you for pointing this out. We chose GPT-4o [4] to score document quality and assign domain labels because it was the most capable model we could access in our experimental environment at the time. We believe that using a stronger model available today could improve the quality of these annotations and the effectiveness of the CuraWeb pipeline.


Response to Q4: Could you provide more detail as to how the L1 and L2 entropy is being computed in Section 5.1? What is the takeaway from this analysis?

Thank you for this question. We have added a detailed description of the entropy computation to the Section 5.1 of the revised manuscript.

For the entropy computation, L1 and L2 refer to the 26 coarse-grained domains and 105 fine-grained subdomains in our taxonomy, respectively. At each level, we compute Shannon entropy over the empirical distribution of document labels:

H=−∑c=1Kpclog⁡2pc,pc=ncNH=-\sum_{c=1}^{K}p_c\log_2 p_c, \qquad p_c=\frac{n_c}{N}.

Here, ncn_c is the number of documents assigned to category cc, NN is the total number of documents in the sample, and K=26K=26 for L1 and K=105K=105 for L2. Categories with zero probability contribute zero to the sum. We report entropy in bits and use equal-sized samples of 50M documents per corpus for the corpus comparisons. Higher entropy indicates a less concentrated domain distribution.

For the takeaway, our analysis in Section 5.1 examines whether broad domain coverage is maintained across the stages of CuraWeb’s curation pipeline. Specifically, L1 entropy remains between 4.417 and 4.428 bits through rule filtering, model filtering, and deduplication. It then decreases modestly to 4.330 after value filtering and 4.295 after sampling. These results show that broad domain coverage persists throughout the pipeline. Compared with the baseline corpora, CuraWeb also maintains competitive coverage, with higher L1 and L2 entropies than FineWeb-Edu [2] and Dolma3 [5] and values comparable to those of DCLM [1] and Nemotron-CC [6].


AuthorsOpenReview ↗

Author Response (3/4)

Response to Weakness: Some factors like alpha seem arbitrary, and I could not find details on how the power-law coefficient n is set.

Thank you for pointing this out. Below, we explain the hyperparameter choices for three components: data selection, rule-based filtering, and semantic deduplication.

1. Selection hyperparameters

Our sampling procedure is as follows:

si=αqi+(1−α)di,wi=xinσ(k(xi−a)),pi=wi∑jwjs_i=\alpha q_i+(1-\alpha)d_i, \qquad w_i=x_i^n\sigma\left(k(x_i-a)\right), \qquad p_i=\frac{w_i}{\sum_j w_j}

where qiq_i and did_i are the normalized quality and diversity scores, xi∈[0,1]x_i\in[0,1] is the min-max normalized fused score sis_i, and σ\sigma denotes the sigmoid function. We use α=0.6\alpha=0.6, n=3n=3, k=2k=2, and a=0.7a=0.7. The rationale for this configuration considers both the theoretical behavior of the weighting function and the resulting sampling distribution.

Theoretical rationale. The parameters control complementary aspects of sampling. Setting α=0.6\alpha=0.6 gives quality a greater weight while allowing semantic diversity to substantially influence document selection. A document with moderate quality and high semantic distinctiveness can therefore receive a higher score than a higher-quality document from a redundant region. The exponent n=3n=3 provides a superlinear preference for higher-scoring documents. The sigmoid provides a smooth adjustment. With k=2k=2, its value ranges from approximately 0.198 to 0.646 over [0,1][0,1], and a=0.7a=0.7 places the gate’s midpoint at 0.70.7. Documents with positive scores below this midpoint retain nonzero sampling probabilities, allowing graded selection across the score range.

Distributional evidence. Appendix M.1 shows that, with about 681.8M documents sampled by each method, the average score rises from 0.489 before sampling to 0.550 with Softmax and 0.611 with Power. Power sampling selects more high-scoring documents while still keeping documents with medium scores.

2. Rule-filter hyperparameters

The motivations for our rule-filter hyperparameters are summarized in Table 9 of Appendix D. We have added the calibration procedure, quantitative threshold comparisons, and quality-stratified retention analyses to Appendix M.2 of the revised manuscript.

Threshold calibration. We evaluate each rule against model-based reference labels, giving greater importance to avoiding the rejection of higher-quality documents. As shown in Table 1, we select thresholds that improve precision over the default setting and achieve at least 85% weighted precision, while considering the fraction of documents filtered.

Table 1. Rule-filter threshold calibration and filtering performance.

RuleDefault → revised thresholdWeighted precisionFiltered fraction
Minimum word count50 → 2563.93% → 87.26%12.24% → 3.25%
Symbol-to-word ratio0.10 → 0.3859.14% → 87.50%0.24% → 0.01%
Alphabetic ratio0.80 → 0.2646.53% → 86.96%7.79% → 0.01%

Distributional evidence. Reducing the minimum word-count threshold from 50 to 25 increases retention in quality-score segment 4 from 94.4% to 100.0%. The revised rule retains approximately 100% of documents in segments 3–10 on the calibration study’s 0–10 quality scale. These retention results indicate that the adjustment reduces the loss of potentially useful short documents. The analysis is reported in Appendix M.2.

Rule removal and technical-content protection. The five removed rules had weighted precisions ranging from 16.70% to 67.73%. This supports removing constraints that cannot reliably distinguish structural noise from usable content. Similarly, source-specific calibration remained ineffective for Math-, STEM-, and Code-oriented data, motivating the technical-content bypass described in the manuscript. These documents remain subject to subsequent model-based quality checks.

AuthorsOpenReview ↗

Author Response (4/4)

Response to Weakness: Some factors like alpha seem arbitrary, and I could not find details on how the power-law coefficient n is set.

3. Semantic deduplication

We set the target score to T=10T=10 and assign different weights to different cosine-similarity intervals, as described in Appendix E. We evaluate these choices through experimental comparisons and statistical analyses, which are detailed in Appendix M.3 of the revised manuscript.

Experimental validation. We compare Broad matching with our adopted Balanced matching configuration. Table 2 summarizes their interval weights.

Table 2. Similarity-interval weight configurations. Both configurations use T=10T=10.

Cosine similarityBroad matchingBalanced matching
[0.85,0.88)[0.85,0.88)1.50
[0.88,0.90)[0.88,0.90)30
[0.90,0.92)[0.90,0.92)52
[0.92,0.94)[0.92,0.94)53
[0.94,0.95)[0.94,0.95)74
[0.95,0.96)[0.95,0.96)75
[0.96,0.97)[0.96,0.97)76
[0.97,0.98)[0.97,0.98)8.56
[0.98,0.99)[0.98,0.99)8.57
[0.99,1.00][0.99,1.00]108

As shown in Table 3, our adopted Balanced matching configuration achieves an MMLU score of 51.11%, outperforming Broad matching by 1.98 percentage points. Compared with Broad matching, it also improves weighted deletion precision by 4.89 percentage points and reduces the deletion rate by 1.58 percentage points. We therefore adopt Balanced matching to reduce erroneous removals while improving downstream performance. Training curves are provided in Appendix M.3.

Table 3. Filtering performance and downstream improvements.

ConfigurationDeletion rate (%)Weighted deletion precision (%)MMLU (%)
Broad matching12.6482.2049.13
Balanced matching11.0687.0951.11

Statistical analysis. Manual inspection finds that lower-similarity pairs often share topics or terminology but differ in formulas, examples, or factual details. These observations motivate reducing their contributions and requiring more accumulated evidence before removal.

The filtering distributions further show that approximately 69% of clusters have deletion rates below 10%, and approximately 78% have deletion rates below 15%. These findings support concentrating removal in highly redundant regions while limiting its impact on most clusters. These cluster-level statistics and qualitative observations are reported in Appendix M.3.


We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.


References

[1] Jeffrey Li et al. 2024. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. arXiv preprint arXiv:2406.11794.

[2] Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint arXiv:2406.17557.

[3] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of Tricks for Efficient Text Classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431.

[4] OpenAI. 2024. Hello GPT-4o. OpenAI announcement.

[5] Team Olmo et al. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.

[6] Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2459–2475.


Reviewer_X24KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CuraWeb, a 2T-token English pretraining corpus built from Common Crawl snapshots from 2013–2024, together with the curation pipeline behind it. After text extraction with Resiliparse, a fastText English filter, exact deduplication and URL blacklisting, every document is annotated by a 300M-parameter model distilled from GPT-4o labels on 500K documents, producing nine quality scores (writing, coherence, completeness, logicality, safety, knowledge, education, helpfulness, reasoning) and a two-level domain label from a 26/105-category taxonomy. These annotations drive the remaining stages: a relaxed, domain-aware version of the Gopher heuristics (five rules removed, three thresholds loosened, a bypass for Mathematics/STEM/Code), a model filter that rejects documents failing any of the four presentation dimensions, MinHash-LSH fuzzy deduplication followed by a "soft" semantic deduplication that accumulates similarity-weighted penalties over several edges before declaring a duplicate (embeddinggemma-300m embeddings, weights calibrated on a GPT-4o-labelled benchmark), and a SampleMix-style sampler that combines a content-value score with a K-Means diversity score and applies power sampling with a sigmoid gate. Document counts fall from 184.9B to 6.42B before sampling. A 3B LLaMA-style model trained on 200B tokens reaches 48.07% average accuracy over ten benchmarks against 46.25% for DCLM (with FineWeb-Edu, Nemotron-CC and Dolma3 lower), and the margin over DCLM grows to 1.95 points at 1T tokens. Three ablations (rule-only filtering, no semantic deduplication, uniform sampling) and corpus analyses of domain entropy, lexical diversity, quality scores and STEM retention across pipeline stages and against the baseline corpora are reported.

Critical strengths

The paper addresses a real gap. Single-objective quality filters (FineWeb-Edu, DCLM) narrow domain coverage, and the paper measures this directly with domain entropy and top-3 concentration (26.5% of documents for CuraWeb vs 48.2% for FineWeb-Edu). The pipeline is documented end to end: document counts at every stage, the full heuristic rule table with the rationale for each change (Table 8), annotation prompts and rubrics, the dedup algorithm, and per-stage domain, lexical and quality statistics. This is more transparent than most corpus papers. The filtering ablation draws a useful distinction between yield and accuracy: the model filter doubles the retained high-quality data at equal downstream performance, which is the right way to argue for a recall-oriented filter. Soft semantic deduplication with evidence accumulation is a sensible fix for threshold brittleness, and the paper reports measured false-positive rates per similarity interval instead of asserting a bound. The downstream gain at 200B tokens (+1.82 average over DCLM) persists at 1T (+1.95), and the paper is candid where baselines win (HellaSwag, OpenBookQA, FineWeb-Edu's higher composite quality score).

Critical weaknesses

No benchmark decontamination is described. The largest gains are on MMLU (+6.6 over the best baseline), MMLU-Pro and GSM8K, the benchmarks most sensitive to contamination, and the pipeline deliberately upsamples "Education" and knowledge-intensive web pages, which is where copies of test items live. DCLM, FineWeb-Edu and Nemotron-CC all document decontamination; without it the headline comparison is not interpretable. Every number in Table 1 comes from a single training run, with no seeds, intervals or significance tests, and several per-task differences (RACE, PIQA, SCIQ, Winogrande) are inside run-to-run variance at 3B/200B. The ablations in Figure 3 are curves without numbers, and their model size and token budget are not stated. The baseline comparison is under-specified. The baselines are multi-trillion-token corpora subsampled to 200B tokens, and the paper never states how the subsample was drawn, which Nemotron-CC subset (high-quality, synthetic, or the full mix) or which FineWeb-Edu variant was used. Closely related prior work is missing, which matters for the novelty claim. QuRating (Wettig et al., ICML 2024) annotates a corpus with several quality criteria and samples with a quality–diversity balance; RedPajama-V2 ships dozens of per-document quality signals; Essential-Web v1.0 (arXiv, June 2025) labels 24T tokens with a 12-category taxonomy covering topic, format, complexity and quality using a distilled 0.5B annotator, which is the recipe of the "data understanding system". The sampling stage is SampleMix with the softmax replaced by a power-law gate.

Questions for authors

Was any decontamination against the ten evaluation benchmarks performed, for CuraWeb or for the baseline subsamples? Please report n-gram overlap rates (for example 13-gram) between CuraWeb and each test set, and downstream results after removing overlapping documents, at least for MMLU, MMLU-Pro and GSM8K. For each baseline, which release and subset was used (DCLM-baseline; FineWeb-Edu 1.3T or the score-2 variant; Nemotron-CC high-quality, synthetic or full; which Dolma 3 mix), and how were the 200B and 1T token subsamples drawn? Table 5 lists 122,070 warm-up steps at a global batch of 960 × 8192 tokens, which exceeds the roughly 25k steps of a 200B-token run. What are the actual step counts and warm-up schedule, what are the x-axis units in Figures 2 and 3, and what model size and token budget were used for the ablations in Figure 3? How many training runs underlie Table 1? If one, can a second seed for CuraWeb and DCLM, or evaluation-set bootstrap intervals, be provided?

3 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you very much for reviewing our paper and for providing so many valuable suggestions. In response to your concerns, we provide the following replies. The corresponding revisions are highlighted in blue in the revised manuscript.


Response to Q1: Was any decontamination against the ten evaluation benchmarks performed, for CuraWeb or for the baseline subsamples? Please report n-gram overlap rates (for example 13-gram) between CuraWeb and each test set, and downstream results after removing overlapping documents, at least for MMLU, MMLU-Pro and GSM8K.

Thank you for this meaningful suggestion. Regarding decontamination, we did not apply benchmark decontamination to either CuraWeb or the baseline subsamples.

We would like to clarify that the released versions of DCLM [1], FineWeb-Edu [2], and Nemotron-CC [3] were also not decontaminated. In Section 6 (Limitations) of the Nemotron-CC paper, the authors explicitly state “we did not decontaminate the dataset” and “(FineWeb-Edu, DCLM) were released without decontamination”. Furthermore, Nemotron-CC-v2 [4] states that its English web-crawl dataset extends the original Nemotron-CC using the same pipeline. Thus, CuraWeb and these baselines are compared under the same decontamination setting, ensuring a fair comparison.

To assess contamination empirically, we sampled 1 billion documents from each corpus and conducted a 13-gram overlap analysis against GSM8K, MMLU, and MMLU-Pro. As shown in Table 1, CuraWeb exhibits a similarly low detected contamination rate compared with the baselines, providing additional support for the comparability of our reported results.

Table 1. Document-level contamination detected through 13-gram overlap.

DatasetSampled documentsContaminated documentsContamination rate
DCLM1,000,000,00079,7160.007972%
FineWeb-Edu1,000,000,000135,5420.013554%
Nemotron-CC1,000,000,00050,3100.005031%
Dolma 3 [5]1,000,000,00051,0930.005109%
CuraWeb1,000,000,00057,2110.005721%

We agree that an experiment comparing downstream performance before and after decontamination would be a valuable addition to the paper. Unfortunately, resource constraints have prevented us from completing this experiment at this stage. We commit to including it in the final published version.


Response to Q2: For each baseline, which release and subset was used , and how were the 200B and 1T token subsamples drawn?

Thank you for pointing out the missing details. We use DCLM-baseline-1.0 [6]; FineWeb-Edu score-3 [7], restricted to Common Crawl snapshots from 2013–2024; Nemotron-CC-v2 [8], combining its Medium-Quality, Medium-High-Quality, and High-Quality subsets, excluding the released synthetic subsets; and the Common Crawl component of Dolma 3 [5] (dolma3_mix-6T-1025-7B [9]). For each experiment, we randomly sample data from the corresponding corpus to match the target training budget of 200B or 1T tokens. We have clarified these dataset details and sampling procedures in the Appendix A of the revised manuscript.


AuthorsOpenReview ↗

Author Response (2/3)

Response to Q3: What are the actual step counts and warm-up schedule, what are the x-axis units in Figures 2 and 3, and what model size and token budget were used for the ablations in Figure 3?

Thank you for your careful review. We have corrected these settings in Appendix A of the revised manuscript.

Specifically, for the warm-up schedule, we use approximately 1B tokens with a sequence length of 8,192. Thus, 122,070 in Table 5 refers to training sequences, rather than optimizer steps. We apologize for this unit error and have corrected it in the revised manuscript.

For the training step counts and x-axis units, the x-axes in both Figures 2 and 3 denote optimizer update steps. For the main experiments in Figure 2, we start with a global batch size of 320, increase it to 640 after approximately 50B training tokens, and increase it to 960 after approximately 100B training tokens. The sequence length remains 8,192 throughout training.

For the model size and token budget of the ablations in Figure 3, we train 3B-parameter models for approximately 260B tokens, using a fixed global batch size of 640 and a sequence length of 8,192.


Response to Q4: How many training runs underlie Table 1? If one, can a second seed for CuraWeb and DCLM, or evaluation-set bootstrap intervals, be provided?

Thank you for this suggestion.

Table 1 reports results from one training run per dataset. Unfortunately, computational resource constraints prevent us from completing additional training runs with a second seed for CuraWeb and DCLM [1] at this stage. Nevertheless, the training curves of Figure 2 show that CuraWeb maintains an advantage over DCLM as training progresses, showing that the performance gains are relatively stable throughout training.

For evaluation-set uncertainty, we report 95% bootstrap confidence intervals. As shown in Table 2, CuraWeb achieves the highest average score of 48.07 (95% CI: [47.45, 48.68]), exceeding the strongest baseline DCLM by 1.82 percentage points.

Table 2. Evaluation scores (%) with 95% bootstrap confidence intervals shown in smaller type below each score.

BenchmarkDCLMFineWeb-Edu [2]Nemotron-CC [3]Dolma 3 [5]CuraWeb
GSM8K3.72[2.73,4.80]\underset{[2.73,4.80]}{3.72}3.64[2.65,4.71]\underset{[2.65,4.71]}{3.64}2.56[1.74,3.47]\underset{[1.74,3.47]}{2.56}3.80[2.81,4.88]\underset{[2.81,4.88]}{3.80}8.19[6.70,9.76]\underset{[6.70,9.76]}{8.19}
MathQA24.02[22.51,25.56]\underset{[22.51,25.56]}{24.02}25.53[23.95,27.10]\underset{[23.95,27.10]}{25.53}24.96[23.42,26.53]\underset{[23.42,26.53]}{24.96}24.92[23.38,26.50]\underset{[23.38,26.50]}{24.92}26.00[24.42,27.57]\underset{[24.42,27.57]}{26.00}
MMLU39.15[38.36,39.94]\underset{[38.36,39.94]}{39.15}28.02[27.28,28.76]\underset{[27.28,28.76]}{28.02}36.25[35.48,37.02]\underset{[35.48,37.02]}{36.25}40.56[39.76,41.35]\underset{[39.76,41.35]}{40.56}47.16[46.37,47.95]\underset{[46.37,47.95]}{47.16}
MMLU-Pro11.11[10.56,11.67]\underset{[10.56,11.67]}{11.11}9.20[8.69,9.72]\underset{[8.69,9.72]}{9.20}10.04[9.51,10.58]\underset{[9.51,10.58]}{10.04}12.08[11.50,12.66]\underset{[11.50,12.66]}{12.08}14.90[14.28,15.53]\underset{[14.28,15.53]}{14.90}
RACE36.94[34.07,39.90]\underset{[34.07,39.90]}{36.94}35.02[32.15,37.90]\underset{[32.15,37.90]}{35.02}36.75[33.88,39.71]\underset{[33.88,39.71]}{36.75}36.65[33.78,39.52]\underset{[33.78,39.52]}{36.65}36.84[33.97,39.81]\underset{[33.97,39.81]}{36.84}
HellaSwag71.63[70.75,72.51]\underset{[70.75,72.51]}{71.63}67.16[66.23,68.07]\underset{[66.23,68.07]}{67.16}72.76[71.90,73.63]\underset{[71.90,73.63]}{72.76}66.94[66.01,67.86]\underset{[66.01,67.86]}{66.94}71.09[70.22,71.98]\underset{[70.22,71.98]}{71.09}
PIQA77.37[75.46,79.27]\underset{[75.46,79.27]}{77.37}76.28[74.32,78.24]\underset{[74.32,78.24]}{76.28}78.13[76.22,79.98]\underset{[76.22,79.98]}{78.13}75.84[73.88,77.80]\underset{[73.88,77.80]}{75.84}77.86[75.95,79.71]\underset{[75.95,79.71]}{77.86}
SCIQ90.30[88.40,92.10]\underset{[88.40,92.10]}{90.30}90.20[88.30,92.00]\underset{[88.30,92.00]}{90.20}92.10[90.40,93.70]\underset{[90.40,93.70]}{92.10}88.70[86.70,90.60]\underset{[86.70,90.60]}{88.70}90.90[89.10,92.60]\underset{[89.10,92.60]}{90.90}
Wino.66.46[63.85,69.06]\underset{[63.85,69.06]}{66.46}63.30[60.62,65.90]\underset{[60.62,65.90]}{63.30}63.46[60.77,66.14]\underset{[60.77,66.14]}{63.46}62.90[60.22,65.59]\underset{[60.22,65.59]}{62.90}63.85[61.17,66.46]\underset{[61.17,66.46]}{63.85}
OBQA41.80[38.70,44.90]\underset{[38.70,44.90]}{41.80}44.20[41.10,47.30]\underset{[41.10,47.30]}{44.20}40.30[37.30,43.40]\underset{[37.30,43.40]}{40.30}40.60[37.60,43.70]\underset{[37.60,43.70]}{40.60}43.90[40.80,47.00]\underset{[40.80,47.00]}{43.90}
Avg.46.25[45.64,46.85]\underset{[45.64,46.85]}{46.25}44.26[43.64,44.86]\underset{[43.64,44.86]}{44.26}45.73[45.13,46.33]\underset{[45.13,46.33]}{45.73}45.30[44.68,45.91]\underset{[44.68,45.91]}{45.30}48.07[47.45,48.68]\underset{\mathbf{[47.45,48.68]}}{\mathbf{48.07}}

AuthorsOpenReview ↗

Author Response (3/3)

Response to the concern about related work and novelty

Thank you for highlighting these related studies. Our main contribution is a data curation framework that treats quality, redundancy, and diversity as interacting objectives throughout the pipeline. Each stage addresses its primary objective while considering its effects on the others, aiming to improve data quality and diversity without unnecessarily sacrificing coverage. Our component ablations and corpus analyses provide empirical evidence for the effectiveness of CuraWeb.

Specifically, for deduplication, QuRating [10] operates on the already deduplicated SlimPajama [11] corpus. RedPajama-V2 [12] and Essential-Web [13] both support exact and MinHash-based deduplication. CuraWeb adds semantic deduplication and accumulates similarity evidence from multiple retained documents before removal. This balances redundancy reduction with semantic coverage by limiting false removals.

For quality filtering, RedPajama-V2 supports filtering through quality signals. Essential-Web similarly balances quality and domain coverage during filtering, but does not extend this coordination to other stages. CuraWeb combines domain labels with multiple quality signals, allowing math and code documents to bypass restrictive heuristics while still undergoing quality checks, which balances quality improvement with domain diversity.

For sampling and selection, QuRating balances quality and diversity during data selection but does not extend this coordination to upstream filtering or deduplication. RedPajama-V2 explores selection through quality signals, while Essential-Web uses taxonomy labels to construct targeted subsets. CuraWeb combines the content value with the diversity score of each document, allowing quality and semantic coverage to jointly determine sampling probabilities.


We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.


References

[1] Jeffrey Li et al. 2024. DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. arXiv preprint arXiv:2406.11794.

[2] Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv preprint arXiv:2406.17557.

[3] Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2025. Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2459–2475.

[4] NVIDIA. 2025. NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model. arXiv preprint arXiv:2508.14444.

[5] Team Olmo et al. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.

[6] ML Foundations. n.d. DCLM-baseline-1.0: Official Dataset Card. Hugging Face dataset card.

[7] Hugging Face. n.d. FineWeb-Edu: Official Dataset Card. Hugging Face dataset card.

[8] NVIDIA. n.d. Nemotron-CC-v2: Official Dataset Card. Hugging Face dataset card.

[9] Allen Institute for AI. n.d. Dolma 3 Mix (6T), dolma3_mix-6T-1025-7B: Official Dataset Card. Hugging Face dataset card.

[10] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. QuRating: Selecting High-Quality Data for Training Language Models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52915–52971.

[11] Daria Soboleva. 2023. SlimPajama: A 627B Token, Cleaned and Deduplicated Version of RedPajama. Cerebras technical blog.

[12] Maurice Weber et al. 2024. RedPajama: An Open Dataset for Training Large Language Models. arXiv preprint arXiv:2411.12372.

[13] Andrew Hojel et al. 2025. Essential-Web v1.0: 24T Tokens of Organized Web Data. arXiv preprint arXiv:2506.14111.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.