- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses over-removal in DataTrove’s near-duplicate pipeline, which uses MinHash/LSH to find candidate matches, merges them with union-find, and keeps one document per connected component. Because similarity is not transitive, a chain of pairwise matches can merge documents that are not near-duplicates of each other. The authors instead keep at most one document per collision bucket, using band-sharded pruning of redundant buckets, weight-one preclusion, layered greedy selection, and an optional multi-seed extension, with structural upper-bound certificates to assess retention optimality. Theory shows the output never retains fewer documents than unionization, and the gap can be unbounded. On ClueWeb22 and HPLT, the single-seed method retains 32M and 34M more documents than legacy on the same buckets. Four retuned seeds raise this to 64M and 65M and cut Stage 3 peak memory by 48.36% and 5.70%. Greedy reaches at least 99.43% of the strongest per-round bound on active documents. The main finding is higher retention. Whether the extra documents improve training data or model performance, and the runtime cost, are left untested.
Critical strengths
(+) Clearly identifies an important limitation of connected-component-based deduplication.
(+) Introduces a clean bucket-feasible formulation with billion-scale evaluation.
(+) Provides useful theoretical analysis of retention cardinality using upper-bound certificates.
Critical weaknesses
(-) Limited algorithmic novelty. Much of the optimization machinery is inherited from Wu & Cong (2026). The paper’s independent contributions lie primarily in adapting this framework to DataTrove, band-sharded pruning, the unionization comparison, and the multi-seed extension. These are useful systems and formulation contributions, but the paper provides limited evidence isolating how much each new component contributes beyond the inherited solver.
(-) Insufficient validation of deduplication quality and practical utility. The paper focuses on maximizing document retention without demonstrating that the additional documents improve data quality, diversity, or downstream language model performance. If legacy deduplication achieves comparable model performance with fewer documents, it may be more efficient. The paper provides neither an independent evaluation of duplicate quality nor downstream training experiments.
(-) Incomplete evaluation of computational and system costs. The evaluation focuses primarily on Stage 3 peak memory. It does not report end-to-end execution time, stage-level runtime, I/O overhead, or temporary storage requirements, so the overall computational trade-off remains unclear. Moreover, the reported memory reductions combine several configuration changes, making it difficult to isolate the contribution of Stage 2.5 pruning.
(-) Insufficient ablation against simpler greedy selection baselines. The paper does not establish that the layered selection procedure is justified relative to simpler alternatives. Secondary within-layer ordering improves retained-root counts by less than 0.01% over random within-layer ordering, suggesting that the additional ordering heuristics offer only marginal benefit. More importantly, the paper does not provide a full-corpus comparison against a simple non-layered maximal greedy algorithm. It is therefore unclear how much of the retention gain comes from the layered strategy rather than from the basic bucket-feasible formulation itself, and whether the additional algorithmic complexity and computational costs are justified by the benefits.
Questions for authors
-
Please clarify the practical interpretation of the “additional retained documents.” Can the authors report quality, usable-token yield, diversity, and independently measured duplication for the documents retained by their method but discarded by legacy? A controlled training run would directly test whether these documents help.
-
Please separate the retention comparisons in the paper. The <0.01% figure in Section 6.3 / Appendix A.2 is for secondary ordering versus random ordering within the layered method; the same-bucket single-seed comparison in Table 1 is +32M/+34M over legacy. Can the authors add a full-corpus comparison with unlayered maximal greedy, plus runtime and peak-memory measurements, to show whether the method’s added complexity is worthwhile?
2 comments
Thank you for the additional experiments and corrections. The unlayered baselines and stage-level cost measurements provide additional evidence regarding the selection algorithms and computational trade-offs. The results indicate that most retention gains come from the bucket-feasible formulation, while layering provides modest additional retention and some memory benefits.
The data-quality study offers useful diagnostics, but its restricted similarity comparisons and ClueWeb-only coverage leave the practical value of the additional documents uncertain. A controlled training experiment would test the retention policy, not merely the shared ((b,r)) criterion, since the policies produce different training corpora.
While my concerns about the limited algorithmic novelty beyond the prior framework and the lack of downstream validation remain, the additional experiments provide useful evidence. I will therefore raise my rating to 2.
**Response to Reviewer uWPM**
We thank the reviewer and ran the requested experiments; they change how we present the paper.
Unlayered greedy (W4, Q2). Same pruned (14,9;1) buckets, same implementation, one node per run. An independent checker confirmed every output feasible and maximal.
Time is selection after loading; peak is Stage 3 MaxRSS; times vary by up to 30% between runs.
The reviewer is right: the formulation, not layering, drives the gain. The ID-order scan already keeps 26.8M/24.9M more documents than legacy, low-degree-first order adds 6.3M/9.4M, and layering adds only 136,696/58,974 (0.12%/0.008% of Upunct). Layering has the lowest peak; it is the slowest on ClueWeb and the fastest on HPLT. We will present degree-ordered greedy as the baseline, with layering and secondary ordering (<0.01%) as optional refinements, and separate the two comparisons in §6.3.
Costs (W3), with a correction. Table 1's legacy MaxRSS values are not from the same-bucket legacy jobs; the ClueWeb row is from an earlier run on different buckets. Same-bucket jobs, same node count per stage:
Our selection needs 18% more peak memory than union-find on ClueWeb and 33% less on HPLT, where it is also 3× faster. Table 1's figures for ours included the optional certificates. Corrected, the single-seed gains are +33.3M/+34.4M and the four-seed peak reductions are 30%/29% (not 48.36%/5.70%). Stage 2.5, the added cost, reads 0.4/3.6 TB of the pipeline's 48/65 TB and shrinks Stage 3's input 2.1×/11.2×. Without it, ClueWeb Stage 3 needs 279.5 GiB instead of 151.6, takes 5:35 instead of 2:24, and keeps 399,100 fewer documents; extrapolated linearly, HPLT would need about 2.3 TiB, more than our 2.2 TiB nodes.
Novelty (W1). We agree that Wu & Cong (2026) state the formulation, layered greedy and certificates for general hypergraphs. The measurements above isolate this paper's components: bucket-feasible selection in DataTrove (+26.8M/+24.9M even with the simplest scan), Stage 2.5 (needed to fit HPLT on one node; −46% Stage 3 peak on ClueWeb) and optional multi-seed (+32M/+31M at 4.7×/4.0× the signature work). The inherited solver adds 6.5M/9.5M.
The extra documents (W2, Q1). Measured on ClueWeb22 (HPLT's texts were not kept), samples of 10,240 per group, same buckets:
Means; hosts: distinct per sample. The "only" groups exceed the net 33.3M because the methods often keep different members.
A training run would test the criterion (b,r), which binds legacy equally; it is our next step.
Revisions. Corrected Table 1 and every claim based on it; the tables above, the full baseline table and this study in the appendix; a rewritten §6.3.