- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper revisits cluster-based data selection for LM pretraining. Keeping documents near or far from their k-means centroid, or semantic deduplication. It argues that the gains people see from these methods are largely a length artifact. So if you keep a fixed fraction of documents, near-centroid selection wins, but near-centroid documents also tend to be long, so that arm ends up with a much larger token pool and less repetition at fixed compute. Once the arms are matched on tokens instead of documents the advantage disappears. The evidence comes from GPT-2 models at 124M and 350M trained on C4 subsets on 3 seeds each and evaluated on FineWeb-Edu loss; SemDeDup also doesn't beat random on this corpus.
Critical strengths
The experimental hygiene is good. Compute is matched across arms, there are 3 seeds and an explicit random baseline, and the controls build on each other well: document budget, then length-stratified, then token-matched. The numbers are consistent. Complementary arms add up to the full pool, and the gap left after length stratification is roughly what you'd predict from the remaining token difference. I believe the main result as stated: in this setup, near-centroid selection gives no benefit once tokens are matched.
The practical recommendation is simple and worth adopting: report the tokens each arm retains, and the correlation between the selection score and document length.
Critical weaknesses
My main concern is that the confound is demonstrated in a setting that isn't really the one used by the methods the paper pushes back on. All the near/far-centroid arms use an LSA embedding, whereas D4 and SemDeDup use OPT-125M embeddings. For OPT the paper only reports correlation with length. The prototypicality arms are never actually run there. There is also a direction issue. D4 prunes the prototypical documents, which corresponds to this paper's atypical arm, and in this data that arm loses tokens rather than gaining them. So it isn't obvious that this confound explains any published gain; it might even work against D4. I'm not fully sure, since D4's data and pipeline differ, which is why the experiment is needed.
As it stands the evidence supports a narrower claim than the framing around "embedding-based selection" suggests: near-centroid selection in an LSA space, on C4, at 50% retention.
A few more points:
- The abstract and conclusion say the token-matched result holds on two OOD sets. But as far as I can tell only the document-budget checkpoints were evaluated on Wikipedia.
- The repetition mechanism is first presented as a hypothesis and later called "the actual driver".
- At 124M every pool except the short arm is larger than the training budget. The repetition exists only because batches are sampled with replacement. With a normal single pass, only the short arm would repeat.
- Muennighoff et al. (2023) (isn't cited) found that a few epochs of repetition doesn't change loss much. 0.044 nat effect from this little repetition is surprising.
Questions for authors
- Could you run typical/atypical selection in OPT-125M space under both budgets? You already have the embeddings and clusters from the SemDeDup experiment. If the same pattern shows up there, including for the D4 direction, I'd be happy to raise my score. If either arm beats random at matched tokens that would undercut the main claim.
- Were the token-matched models evaluated on Wikipedia? If not, could you add that?
2 comments
5. Muennighoff et al. and the effect size. You were right to find 0.044 nats surprising. A random-subset dose-response, pools of 0.66–2.32B tokens with population composition, i.e. 1.5 down to 0.4 epochs, moves OOD loss by only ≈0.016 nats across the whole range, consistent with Muennighoff et al. Comparing each selection to a random pool of the same size (FineWeb-Edu): far-centroid +0.032 (1.01B), shortest documents +0.046 (≈0.7B), near-centroid −0.007 (1.97B), longest documents −0.008 (2.3B). Token count therefore explains only a small part of the differences. In LSA space, the near-centroid gain survives at equal token count and matches the gain from simply selecting the longest documents: in effect, it is a document-length effect. Token matching hid it because it changes which documents are kept (2.35M vs 2.99M near-centroid documents), the concern hahu and qfqf raised. Matching tokens and length jointly to isolate LSA geometry, near and far differ by 0.000 on FineWeb-Edu and −0.003 on Wikipedia (sign varying across seeds), while far-centroid selection remains 0.008 worse in distribution (consistent across seeds). This also revises a point in your summary: the length-stratified gap (0.016) is not explained by the remaining token difference (≈0.002 predicted), but most likely by residual length differences within the coarse deciles.
6. "Hypothesis" vs "actual driver." Agreed: inconsistent, and given point 5, wrong. That sentence is removed; the revision presents the decomposition above.
7. Scope. Agreed. Claims are now scoped to within-cluster centroid selection on C4 at 50% retention, at 124M/350M, with LSA and OPT embeddings.
Summary. Your review led us to a different conclusion than the submission's. In LSA space, the apparent out-of-distribution effects of centroid-based selection come mainly from the document-length composition of the retained data; retained token count contributes little, and geometry beyond length has no measurable out-of-distribution effect. In OPT space, where selection barely changes length, there is a modest geometric effect on FineWeb-Edu (near-centroid −0.006 to −0.009; D4 direction +0.009) that does not appear on Wikipedia. Document-budget comparisons conflate these factors, and the controls you suggested are how to separate them. Thank you.
Thank you for the most actionable review we received. We ran every experiment you asked for, plus the controls needed to interpret them (all at 124M, 3 seeds each). Two of your methodological objections: (i) repetition arises only from with-replacement sampling, and (ii) that Muennighoff et al. (2023) predict little effect from mild repetition, were correct, and together with your OPT experiment they change our conclusion. Point by point below; numbers are loss differences in nats (negative = better). We will upload a revised PDF reflecting all of this before the discussion closes.
1. Near/far selection in OPT-125M space under both budgets (Q1). In OPT space, the length coupling that drives our LSA results lies mostly between clusters: pooled ρ(length, centroid distance) = −0.72, but within clusters, where selection operates, only −0.16 (LSA: −0.35). Consequently, OPT near/far selection at a 50% document budget keeps 1.50B vs 1.47B tokens with near-identical length distributions (KS = 0.03), whereas LSA keeps 1.97B vs 1.01B. Difference from random at the same ~1.49B tokens (seed s.d. ≈0.001–0.005 on FineWeb-Edu and C4, ≈0.005–0.009 on Wikipedia):
The two budgets agree, as expected when selection does not shift token counts, so the document-budget artifact we reported for LSA does not arise in OPT space. However, near-centroid selection improves FineWeb-Edu by 0.006–0.009, and the D4 direction is worse on FineWeb-Edu and C4. Since length is matched, these differences are geometric. They are modest and evaluation-specific: present on FineWeb-Edu (and, for the D4 direction, on C4), absent on Wikipedia. By your criterion, the near-centroid result undercuts our submission's claim as stated; we have revised the claim accordingly (see the summary).
2. Direction relative to D4. We agree. D4 discards near-centroid points and trains at a fixed token budget on fresh data, so a document-budget token-count bias cannot explain its reported gains; in our data such a bias would, if anything, penalize the D4 direction. (At our scale and on C4, the D4 direction is also not better at matched tokens, point 1, but D4's setting differs in scale, source corpus, and pipeline, so we do not claim this contradicts it.) The revision will say this explicitly, cite D4's own appendix discussion of token-count differences under document-level selection, and narrow our claims to the setting we test.
3. Token-matched models on Wikipedia (Q2). You are right: only document-budget checkpoints had been evaluated on Wikipedia. We have now evaluated every checkpoint on both OOD sets and on in-distribution C4. Token-matched, Wikipedia, vs random: near +0.006, far +0.010, longest documents −0.014. The paper's token-matched result (no near-centroid benefit in LSA space) thus holds on both OOD sets, and the abstract and conclusion will say exactly this. Point 5 explains why token matching alone is not a clean control.
4. Single-pass training. We patched nanoGPT to train single-pass (shuffled non-overlapping blocks; no token repeats until the pool is exhausted) and retrained the document-budget selections. Difference from single-pass random (FineWeb-Edu / Wikipedia / C4), with-replacement values in brackets:
Removing repetition halves the near-centroid advantage on both OOD sets but leaves the far-centroid and short-document penalties essentially unchanged, so those penalties are not caused by repetition. (Single-pass training itself improves random by ≈0.011 nats over nanoGPT's with-replacement window sampling at under one epoch, a practical caveat for small-scale selection studies.)
Continued in next comment.