- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents CuraWeb, a 2T-token English pretraining corpus constructed from Common Crawl. The main motivation is that highly selective web-data curation pipelines may improve average data quality while reducing distributional diversity and discarding useful long-tail content.
CuraWeb combines several stages in a unified pipeline: multi-dimensional document quality and domain annotation, domain-aware heuristic and model-based filtering, lexical and semantic deduplication, and a final sampling stage that combines content-value and diversity scores. For semantic deduplication, the paper proposes a weighted score-accumulation strategy rather than a single hard similarity threshold. For sampling, document-level quality and diversity scores are combined and transformed into sampling probabilities using a power-based scheme.
The authors train 3B models from scratch under matched model and training settings to compare CuraWeb against DCLM, FineWeb-Edu, Nemotron-CC, and Dolma3. CuraWeb obtains the best average downstream performance at 200B training tokens, with a reported 1.82-point improvement over DCLM, and the advantage persists when training is extended to 1T tokens. The paper also provides component ablations and corpus-level analyses of quality, redundancy, domain coverage, and lexical diversity.
Critical strengths
1. Large-scale and practically meaningful evaluation.
The paper evaluates the proposed curation pipeline by actually training 3B models from scratch under matched architectures, optimization settings, and token budgets. This is substantially more informative than evaluating corpus statistics alone. The reported improvement over strong public corpora is also non-trivial, and the advantage persists when training is extended from 200B to 1T tokens.
2. The paper provides a relatively complete end-to-end analysis of the data-curation pipeline.
The work studies filtering, lexical and semantic deduplication, sampling, domain coverage, lexical diversity, and content-value distributions within one large-scale pipeline. The component ablations and corpus analyses provide useful practical evidence about how different curation stages reshape the final training distribution.
Critical weaknesses
1. The contribution is dominated by engineering integration, while the methodological novelty is not sufficiently established.
The proposed framework combines several practically reasonable components: domain-aware heuristic filtering, model-based multi-dimensional quality scoring, lexical and semantic deduplication, and quality-diversity sampling. However, each of these directions has close precedents in the existing data-curation literature. The paper itself discusses semantic deduplication, DoReMi-style mixture optimization, SampleMix-style quality-diversity sampling, and WebOrganizer-style domain organization, but the experimental section does not provide matched method-level comparisons against these closely related approaches.
As a result, it is difficult to determine which part of CuraWeb constitutes a genuinely new methodological contribution, as opposed to a carefully engineered combination and tuning of existing ideas. I consider the engineering scale valuable, but in its current form I do not find the methodological novelty sufficiently strong for the claims made in the paper.
2. The "joint optimization" framing is stronger than what is actually formulated and validated.
The title presents CuraWeb as jointly optimizing quality, redundancy, and diversity. However, I do not see a unified optimization problem over these three objectives. Instead, they are addressed at different pipeline stages using manually chosen thresholds, rule modifications, empirically calibrated deduplication weights, and a fixed linear combination of quality and diversity for sampling.
For example, the sampling score uses a fixed coefficient of , while redundancy is handled separately by the preceding deduplication procedure. There is no demonstrated Pareto analysis, constrained optimization, learned trade-off, or other evidence that the three objectives are jointly optimized rather than sequentially managed.
Relatedly, the end-to-end comparison against other corpora establishes that the full CuraWeb pipeline produces a strong training corpus, but does not establish that the claimed joint-optimization principle is responsible for the improvement. The current ablations mainly remove broad components (semantic deduplication or non-uniform sampling) rather than comparing the proposed implementations against strong alternatives for the same component. Therefore, the causal attribution of the reported 1.82-point gain to the paper's claimed methodological contribution remains unclear.
Questions for authors
1. How does each proposed component compare against the closest existing method under exactly the same input corpus and training budget?
In particular, I would like to see matched comparisons between the proposed semantic deduplication strategy and existing semantic-deduplication methods, and between the proposed quality-diversity sampling strategy and closely related approaches such as SampleMix or other data-mixture/domain-aware sampling methods. Comparing only the final CuraWeb corpus against other final corpora does not isolate the contribution of the proposed methodology. A clear advantage under controlled component-level comparisons would materially strengthen my assessment of novelty.
2. What exactly is being "jointly optimized" in CuraWeb?
Quality, redundancy, and diversity appear to be handled largely through different stages of the pipeline, and I do not see a unified objective or optimization procedure involving all three. Is "joint optimization" intended as a formal methodological claim, or does it mean coordinated use of shared signals across the pipeline? If the former, please clarify the optimization formulation and how the reported method solves it.
3. How sensitive are the final results to the manually selected design choices in the pipeline?
Several important choices appear manually specified or empirically tuned, including filtering thresholds, semantic-deduplication interval weights and target score, the quality-diversity coefficient , and parameters of the power-sampling function. I would like to know whether the main downstream gains remain stable under reasonable alternatives, particularly for the quality-diversity trade-off. This would help distinguish a robust general principle from a highly tuned engineering pipeline.
3 comments
Author Response (2/3)
Response to Q3: How sensitive are the final results to the manually selected design choices in the pipeline?
Thank you for this question. Below, we explain the hyperparameter choices for three components: data selection, rule-based filtering, and semantic deduplication.
1. Selection hyperparameters
Our sampling procedure is as follows:
where and are the normalized quality and diversity scores, is the min-max normalized fused score , and denotes the sigmoid function. We use , , , and . The rationale for this configuration considers both the theoretical behavior of the weighting function and the resulting sampling distribution.
Theoretical rationale. The parameters control complementary aspects of sampling. Setting gives quality a greater weight while allowing semantic diversity to substantially influence document selection. A document with moderate quality and high semantic distinctiveness can therefore receive a higher score than a higher-quality document from a redundant region. The exponent provides a superlinear preference for higher-scoring documents. The sigmoid provides a smooth adjustment. With , its value ranges from approximately 0.198 to 0.646 over , and places the gate’s midpoint at . Documents with positive scores below this midpoint retain nonzero sampling probabilities, allowing graded selection across the score range.
Distributional evidence. Appendix J shows that, with about 681.8M documents sampled by each method, the average score rises from 0.489 before sampling to 0.550 with Softmax and 0.611 with Power. Power sampling selects more high-scoring documents while still keeping documents with medium scores.
2. Rule-filter hyperparameters
The motivations for our rule-filter hyperparameters are summarized in Table 8 of Appendix D. We have further expanded Appendix D in the revised manuscript with the calibration procedure, quantitative threshold comparisons, and quality-stratified retention analyses.
Threshold calibration. We evaluate each rule against model-based reference labels, giving greater importance to avoiding the rejection of higher-quality documents. As shown in Table 1, we select thresholds that improve precision over the default setting and achieve at least 85% weighted precision, while considering the fraction of documents filtered.
Table 1. Rule-filter threshold calibration and filtering performance.
| Rule | Default → revised threshold | Weighted precision | Filtered fraction |
|---|---|---|---|
| Minimum word count | 50 → 25 | 63.93% → 87.26% | 12.24% → 3.25% |
| Symbol-to-word ratio | 0.10 → 0.38 | 59.14% → 87.50% | 0.24% → 0.01% |
| Alphabetic ratio | 0.80 → 0.26 | 46.53% → 86.96% | 7.79% → 0.01% |
Distributional evidence. Reducing the minimum word-count threshold from 50 to 25 increases retention in quality-score segment 4 from 94.4% to 100.0%. The revised rule retains approximately 100% of documents in segments 3–10 on the calibration study’s 0–10 quality scale. These retention results indicate that the adjustment reduces the loss of potentially useful short documents. The analysis is reported in Appendix M.2.
Rule removal and technical-content protection. The five removed rules had weighted precisions ranging from 16.70% to 67.73%. This supports removing constraints that cannot reliably distinguish structural noise from usable content. Similarly, source-specific calibration remained ineffective for Math-, STEM-, and Code-oriented data, motivating the technical-content bypass described in the manuscript. These documents remain subject to subsequent model-based quality checks.
Author Response (3/3)
Response to Q3: How sensitive are the final results to the manually selected design choices in the pipeline?
3. Semantic deduplication
We set the target score to and assign different weights to different cosine-similarity intervals. We evaluate these choices through experimental comparisons and statistical analyses, which are detailed in Appendix E of the revised manuscript.
Experimental validation. We compare Broad matching with our adopted Balanced matching configuration. Table 2 summarizes their interval weights.
Table 2. Similarity-interval weight configurations.
| Cosine similarity | Broad matching | Balanced matching |
|---|---|---|
| 1.5 | 0 | |
| 3 | 0 | |
| 5 | 2 | |
| 5 | 3 | |
| 7 | 4 | |
| 7 | 5 | |
| 7 | 6 | |
| 8.5 | 6 | |
| 8.5 | 7 | |
| 10 | 8 |
As shown in Table 3, our adopted Balanced matching configuration achieves an MMLU [1] score of 51.11%, outperforming Broad matching by 1.98 percentage points. Compared with Broad matching, it also improves weighted deletion precision by 4.89 percentage points and reduces the deletion rate by 1.58 percentage points. We therefore adopt Balanced matching to reduce erroneous removals while improving downstream performance. Training curves are provided in Appendix E.
Table 3. Filtering performance and downstream improvements.
| Configuration | Deletion rate (%) | Weighted deletion precision (%) | MMLU (%) |
|---|---|---|---|
| Broad matching | 12.64 | 82.20 | 49.13 |
| Balanced matching | 11.06 | 87.09 | 51.11 |
Statistical analysis. Manual inspection finds that lower-similarity pairs often share topics or terminology but differ in formulas, examples, or factual details. These observations motivate reducing their contributions and requiring more accumulated evidence before removal.
The filtering distributions further show that approximately 69% of clusters have deletion rates below 10%, and approximately 78% have deletion rates below 15%. These findings support concentrating removal in highly redundant regions while limiting its impact on most clusters. These cluster-level statistics and qualitative observations are reported in Appendix M.3.
We sincerely thank you for the time and valuable suggestions you have provided to help improve the paper. We hope these responses are helpful.
References
[1] Dan Hendrycks et al. (2021). Measuring Massive Multitask Language Understanding.
[2] Xiangyu Xi et al. (2025). SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity.
[3] Haonan Li et al. (2023). CMMLU: Measuring Massive Multitask Language Understanding in Chinese.
[4] Yuzhen Huang et al. (2023). C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models.
[5] Jeffrey Li et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models.
[6] Guilherme Penedo et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.
[7] Armand Joulin et al. (2017). Bag of Tricks for Efficient Text Classification.
Author Response (1/3)
Thank you very much for reviewing our paper and for providing so many valuable suggestions. In response to your concerns, we provide the following replies. The corresponding revisions are highlighted in blue in the revised manuscript.
Response to Q1: How does each proposed component compare against the closest existing method under exactly the same input corpus and training budget?
Thank you for this valuable suggestion. Resource constraints have prevented us from conducting the full set of requested controlled comparisons. However, we have earlier experimental records comparing semantic deduplication strategies and sampling strategies on shared candidate corpora. We summarize these existing comparisons below.
Soft-threshold versus hard-threshold semantic deduplication.
We compare our soft-threshold semantic deduplication with hard-threshold deduplication using a fixed threshold of 0.95 on the same input corpus. At the shared 44,000-step checkpoint, our method improves MMLU [1] accuracy by 1.05 percentage points.
Table 1. Semantic deduplication comparison.
Power sampling versus SampleMix [2].
We compare Power sampling with SampleMix in our experiments. Both methods select training data from the same candidate corpus using the same fused quality scoring scheme. We train 2.4B MoE models on the resulting data and compare their downstream performance at the shared 16,000-step checkpoint. Power sampling improves accuracy by 1.01, 0.85, and 1.89 percentage points on MMLU, CMMLU, and C-Eval, respectively. These results support the sampling strategy under a shared quality scoring scheme.
Table 2. Sampling strategy comparison.
These existing experiments provide component-level evidence beyond final-corpus comparisons. We nevertheless acknowledge that they do not replace the full set of strictly matched comparisons against existing methods requested by the reviewer.
Response to Q2: What exactly is being "jointly optimized" in CuraWeb?
Thank you for pointing this out. Joint optimization in CuraWeb refers to the coordinated use of shared signals at each stage of the pipeline to balance quality, redundancy, and diversity. Each stage addresses its primary objective while considering its effects on quality, redundancy, and diversity. We have clarified this meaning in the revised manuscript.
Specifically, for deduplication, DCLM [5] uses a Bloom filter, while FineWeb-Edu [6] inherits FineWeb’s MinHash deduplication. CuraWeb adds semantic deduplication and accumulates similarity evidence from multiple retained documents before removal. This balances redundancy reduction with semantic coverage by limiting false removals.
For quality filtering, DCLM filters by fastText [7] score, while FineWeb-Edu filters by educational value. CuraWeb combines domain labels with multiple quality signals, allowing math and code documents to bypass restrictive heuristics while still undergoing quality checks, which balances quality improvement with domain diversity. For sampling, DCLM adjusts data-source proportions in its mixing experiments, while FineWeb-Edu randomly samples its filtered corpus. CuraWeb combines the content value with the diversity score of each document, allowing quality and semantic coverage to jointly determine sampling probabilities.