acceptodds
Under review as a conference paper at ICLR 2027

Token Mass, Not Geometry: A Confound in Embedding-Based Data Selection for Language Model Pretraining

Abstract

Cluster/embedding-based data-selection methods choose pretraining subsets by geometry: representativeness (proximity to a cluster centroid), atypicality, or redundancy. We stress-test this family under matched training compute, three seeds, a common held-out evaluation, and an explicit random baseline. Under the usual *fixed-document-budget* protocol, near-centroid (“representative”) selection beats random on out-of-distribution (OOD) loss by a small, seed-stable margin (0.01 nats on C4). We show this gain is a **retained-token-mass confound**, not a property of embedding geometry. Centroid distance is strongly anti-correlated with document length ( in an LSA space, in the OPT-125M space), so at a fixed document budget, representative selection keeps longer documents and hence *more tokens*, training with less repetition. When we instead match the *token* budget across arms, the benefit vanishes: near-centroid selection ties or slightly underperforms random. This holds across model scales (124M and 350M, three seeds each) and two OOD sets, and a progression of controls shows the near/far-centroid gap shrinking from nats (document budget) to (length-stratified) to (token-matched). Native SemDeDup, evaluated in its own OPT-125M embedding space, also does not beat random on this corpus. We recommend that evaluations of embedding-based selection control for retained token mass and document length, and report against a random baseline at matched compute with multiple seeds: practices that, in our experiments, overturn an apparently positive result.

Primary area:
datasets and benchmarks
Keywords:
data selection, language model pretraining, data-centric ML, document length, empirical analysis of deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_t6gMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits cluster-based data selection for LM pretraining. Keeping documents near or far from their k-means centroid, or semantic deduplication. It argues that the gains people see from these methods are largely a length artifact. So if you keep a fixed fraction of documents, near-centroid selection wins, but near-centroid documents also tend to be long, so that arm ends up with a much larger token pool and less repetition at fixed compute. Once the arms are matched on tokens instead of documents the advantage disappears. The evidence comes from GPT-2 models at 124M and 350M trained on C4 subsets on 3 seeds each and evaluated on FineWeb-Edu loss; SemDeDup also doesn't beat random on this corpus.

Critical strengths

The experimental hygiene is good. Compute is matched across arms, there are 3 seeds and an explicit random baseline, and the controls build on each other well: document budget, then length-stratified, then token-matched. The numbers are consistent. Complementary arms add up to the full pool, and the gap left after length stratification is roughly what you'd predict from the remaining token difference. I believe the main result as stated: in this setup, near-centroid selection gives no benefit once tokens are matched.

The practical recommendation is simple and worth adopting: report the tokens each arm retains, and the correlation between the selection score and document length.

Critical weaknesses

My main concern is that the confound is demonstrated in a setting that isn't really the one used by the methods the paper pushes back on. All the near/far-centroid arms use an LSA embedding, whereas D4 and SemDeDup use OPT-125M embeddings. For OPT the paper only reports correlation with length. The prototypicality arms are never actually run there. There is also a direction issue. D4 prunes the prototypical documents, which corresponds to this paper's atypical arm, and in this data that arm loses tokens rather than gaining them. So it isn't obvious that this confound explains any published gain; it might even work against D4. I'm not fully sure, since D4's data and pipeline differ, which is why the experiment is needed.

As it stands the evidence supports a narrower claim than the framing around "embedding-based selection" suggests: near-centroid selection in an LSA space, on C4, at 50% retention.

A few more points:

  • The abstract and conclusion say the token-matched result holds on two OOD sets. But as far as I can tell only the document-budget checkpoints were evaluated on Wikipedia.
  • The repetition mechanism is first presented as a hypothesis and later called "the actual driver".
  • At 124M every pool except the short arm is larger than the training budget. The repetition exists only because batches are sampled with replacement. With a normal single pass, only the short arm would repeat.
  • Muennighoff et al. (2023) (isn't cited) found that a few epochs of repetition doesn't change loss much. 0.044 nat effect from this little repetition is surprising.
Questions for authors
  1. Could you run typical/atypical selection in OPT-125M space under both budgets? You already have the embeddings and clusters from the SemDeDup experiment. If the same pattern shows up there, including for the D4 direction, I'd be happy to raise my score. If either arm beats random at matched tokens that would undercut the main claim.
  2. Were the token-matched models evaluated on Wikipedia? If not, could you add that?
2 comments

Thank you for the most actionable review we received. We ran every experiment you asked for, plus the controls needed to interpret them (all at 124M, 3 seeds each). Two of your methodological objections: (i) repetition arises only from with-replacement sampling, and (ii) that Muennighoff et al. (2023) predict little effect from mild repetition, were correct, and together with your OPT experiment they change our conclusion. Point by point below; numbers are loss differences in nats (negative = better). We will upload a revised PDF reflecting all of this before the discussion closes.

1. Near/far selection in OPT-125M space under both budgets (Q1). In OPT space, the length coupling that drives our LSA results lies mostly between clusters: pooled ρ(length, centroid distance) = −0.72, but within clusters, where selection operates, only −0.16 (LSA: −0.35). Consequently, OPT near/far selection at a 50% document budget keeps 1.50B vs 1.47B tokens with near-identical length distributions (KS = 0.03), whereas LSA keeps 1.97B vs 1.01B. Difference from random at the same ~1.49B tokens (seed s.d. ≈0.001–0.005 on FineWeb-Edu and C4, ≈0.005–0.009 on Wikipedia):

OPT selectionFineWeb-EduWikipediaC4 (in-dist.)
near-centroid, document budget−0.006+0.005+0.002
near-centroid, token-matched−0.009+0.001−0.001
far-centroid (D4 direction), document budget+0.009+0.004+0.005
far-centroid (D4 direction), token-matched+0.009+0.002+0.004

The two budgets agree, as expected when selection does not shift token counts, so the document-budget artifact we reported for LSA does not arise in OPT space. However, near-centroid selection improves FineWeb-Edu by 0.006–0.009, and the D4 direction is worse on FineWeb-Edu and C4. Since length is matched, these differences are geometric. They are modest and evaluation-specific: present on FineWeb-Edu (and, for the D4 direction, on C4), absent on Wikipedia. By your criterion, the near-centroid result undercuts our submission's claim as stated; we have revised the claim accordingly (see the summary).

2. Direction relative to D4. We agree. D4 discards near-centroid points and trains at a fixed token budget on fresh data, so a document-budget token-count bias cannot explain its reported gains; in our data such a bias would, if anything, penalize the D4 direction. (At our scale and on C4, the D4 direction is also not better at matched tokens, point 1, but D4's setting differs in scale, source corpus, and pipeline, so we do not claim this contradicts it.) The revision will say this explicitly, cite D4's own appendix discussion of token-count differences under document-level selection, and narrow our claims to the setting we test.

3. Token-matched models on Wikipedia (Q2). You are right: only document-budget checkpoints had been evaluated on Wikipedia. We have now evaluated every checkpoint on both OOD sets and on in-distribution C4. Token-matched, Wikipedia, vs random: near +0.006, far +0.010, longest documents −0.014. The paper's token-matched result (no near-centroid benefit in LSA space) thus holds on both OOD sets, and the abstract and conclusion will say exactly this. Point 5 explains why token matching alone is not a clean control.

4. Single-pass training. We patched nanoGPT to train single-pass (shuffled non-overlapping blocks; no token repeats until the pool is exhausted) and retrained the document-budget selections. Difference from single-pass random (FineWeb-Edu / Wikipedia / C4), with-replacement values in brackets:

  • near-centroid: −0.005 / −0.009 / +0.006 [−0.010 / −0.020 / +0.002]
  • far-centroid: +0.033 / +0.046 / +0.013 [+0.034 / +0.036 / +0.016]
  • shortest documents: +0.053 / +0.081 / +0.015 [+0.056 / +0.086 / +0.017]

Removing repetition halves the near-centroid advantage on both OOD sets but leaves the far-centroid and short-document penalties essentially unchanged, so those penalties are not caused by repetition. (Single-pass training itself improves random by ≈0.011 nats over nanoGPT's with-replacement window sampling at under one epoch, a practical caveat for small-scale selection studies.)

Continued in next comment.

5. Muennighoff et al. and the effect size. You were right to find 0.044 nats surprising. A random-subset dose-response, pools of 0.66–2.32B tokens with population composition, i.e. 1.5 down to 0.4 epochs, moves OOD loss by only ≈0.016 nats across the whole range, consistent with Muennighoff et al. Comparing each selection to a random pool of the same size (FineWeb-Edu): far-centroid +0.032 (1.01B), shortest documents +0.046 (≈0.7B), near-centroid −0.007 (1.97B), longest documents −0.008 (2.3B). Token count therefore explains only a small part of the differences. In LSA space, the near-centroid gain survives at equal token count and matches the gain from simply selecting the longest documents: in effect, it is a document-length effect. Token matching hid it because it changes which documents are kept (2.35M vs 2.99M near-centroid documents), the concern hahu and qfqf raised. Matching tokens and length jointly to isolate LSA geometry, near and far differ by 0.000 on FineWeb-Edu and −0.003 on Wikipedia (sign varying across seeds), while far-centroid selection remains 0.008 worse in distribution (consistent across seeds). This also revises a point in your summary: the length-stratified gap (0.016) is not explained by the remaining token difference (≈0.002 predicted), but most likely by residual length differences within the coarse deciles.

6. "Hypothesis" vs "actual driver." Agreed: inconsistent, and given point 5, wrong. That sentence is removed; the revision presents the decomposition above.

7. Scope. Agreed. Claims are now scoped to within-cluster centroid selection on C4 at 50% retention, at 124M/350M, with LSA and OPT embeddings.

Summary. Your review led us to a different conclusion than the submission's. In LSA space, the apparent out-of-distribution effects of centroid-based selection come mainly from the document-length composition of the retained data; retained token count contributes little, and geometry beyond length has no measurable out-of-distribution effect. In OPT space, where selection barely changes length, there is a modest geometric effect on FineWeb-Edu (near-centroid −0.006 to −0.009; D4 direction +0.009) that does not appear on Wikipedia. Document-budget comparisons conflate these factors, and the controls you suggested are how to separate them. Thank you.

Reviewer_hahuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that near-centroid data selection’s advantage (representativeness) over random disappears when retained token budgets are matched. It attributes the original advantage to token mass rather than embedding geometry/representativeness.

Critical strengths
  1. Experiments across two model sizes, multiple seeds, and two evaluation distributions (OOD) strengthen the results, although the two models share the same architecture (GPT-2 backbone).

  2. An interesting and useful finding that challenges how representativeness-based selection is evaluated and identifies a failure mode in near-centroid data selection.

Critical weaknesses
  1. The causal conclusion is overstated. Token matching also changes document counts, length distributions, and potentially the geometric differences between subsets, especially considering that far-from-centroid documents usually consume the token quota faster, thus fewer documents. The experiments do not establish that token mass alone explains the advantage/confound.

  2. Practical guidance is limited. The authors demonstrate that near-centroid selection loses its advantage under a fixed token budget, but do not offer practical guidance beyond that.

A stronger experiment would vary token mass while matching document-length and centroid-distance distributions, allowing document count to change. Population-level geometry measures that do not assume Euclidean geometry (diffusion spectral entropy (Liao et al., 2023), mutual information neural estimation (Belghazi et al., 2018)) could be interesting alternatives to centroid distance for characterizing how the selected subsets differ.

Questions for authors
  1. When token budgets are matched, how do the document counts and length distributions differ between the near-centroid, random, far-centroid subsets? Can the authors report these alongside the main results?

  2. What should a practitioner do differently after reading this?

1 comment

Thank you. Your central objection, that token matching also changes document counts and length distributions, so it cannot isolate token mass, was right, and the experiment you suggested turned out to be decisive. Numbers below are at 124M, 3 seeds, loss differences in nats (negative = better). Full details are in our response to Reviewer t6gM, and the revised PDF will reflect all of it.

Q1. Composition of the token-matched subsets. At the same 1.49B tokens (LSA space); this table will be added to the main results:

token-matched subsetdocumentsmean lengthmedianp10–p90
near-centroid2.35M63430175–1,404
random2.99M49827280–1,046
far-centroid3.63M41026084–852
longest documents1.47M1,011624203–2,049

One correction on direction: in our data, far-from-centroid documents are shorter (within-cluster ρ(length, distance) = −0.35), so they fill the token quota more slowly and the far-centroid subset ends up with more documents, not fewer. Either way, your point stands: near- and far-centroid token-matched subsets differ strongly in length (KS = 0.11).

W1. The stronger experiment you proposed. We ran it. Random subsets of 0.66–2.32B tokens vary token mass while preserving the population's length and centroid-distance distributions (by construction; their length statistics are identical to the full pool), and let the document count vary (1.3M–4.7M). Token mass alone moves out-of-distribution loss by only ≈0.016 nats across this 3.5× range. Against a random subset of the same size (FineWeb-Edu): near-centroid −0.007, longest documents −0.008, far-centroid +0.032, shortest documents +0.046. So our causal conclusion was indeed overstated, and informatively so: token mass explains only a small part of the differences; most of it is document-length composition. Token matching hid the near-centroid length effect precisely because it changed which documents were kept.

We also ran the complementary control: matching tokens and length jointly within (cluster × length-bin) cells and varying only centroid distance (KS(length) = 0.016). Near and far then differ by 0.000 on FineWeb-Edu and −0.003 on Wikipedia, while far-centroid remains 0.008 worse in distribution. Training single-pass (no repetition) leaves the far-centroid and short-document penalties unchanged (+0.033, +0.053). In OPT space, where selection barely changes length, a modest geometric effect remains on FineWeb-Edu (near −0.006 to −0.009; far +0.009), but not on Wikipedia.

Q2 / W2. What a practitioner should do differently.

  1. Report each method's retained token count and document-length distribution, and the within-cluster correlation between the selection score and length. The pooled correlation misleads: in OPT space it is −0.72 overall but −0.16 within clusters.
  2. Compare against random subsets of the same token count — ideally a small random dose-response curve — not a single random baseline at the same document count. A method's effect is its residual from that curve.
  3. Include a length-only baseline. Selecting the longest documents matches the near-centroid gain (−0.008 vs −0.007) at zero embedding cost; a selection method that does not beat it is, in effect, a length filter.
  4. Choose length composition deliberately for the target distribution. Longer documents help long-form out-of-distribution targets (longest documents: −0.014 on Wikipedia at matched tokens) but hurt in-distribution C4 (+0.014); very short documents hurt everywhere.
  5. Train single-pass or report epochs: with-replacement window sampling below one epoch costs ≈0.011 nats and inflates differences between subsets of different sizes.
  6. Test geometric claims with tokens and length matched jointly.

Same architecture. Agreed. Both scales use a GPT-2-style decoder; we will state this as a limitation.

Population-level geometry measures. Thank you for the pointers to diffusion spectral entropy and MINE. We studied centroid distance because it is the criterion the audited methods (SemDeDup, D4/SSL prototypes) actually use, but we agree that a population-level characterization of how the subsets differ would strengthen the analysis, and we note it as future work.

Reviewer_3QUvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper confirms data-constrained scaling laws in the context of data curation by demonstrating that the apparent gains of embedding-based within-cluster selection are driven by a retained token-count confound (less data repetition under fixed compute) rather than document geometry. This claim is substantiated for 124M and 350M parameter decoder-only Transformers trained on a pre-filtered, deduplicated 6M-document subset of C4 and evaluated on FineWeb-Edu and English Wikipedia using OOD loss (which is perplexity, I assume).

Critical strengths
  • Clear and solid contribution based on rigorous methodology (matched compute, multiple random seeds, and an explicit random baseline set a significantly higher empirical standard than prior work relying on single-seed evaluations without repetition controls), clear conceptual discovery (geometric "representativeness" vs long-document selection), graded controls (length-stratified selection narrows the gap and token-matching eliminates it entirely).

  • Transparent limitations: two scales, one main pretraining corpus, no domain-mixing evaluation, and C4 is not redundancy-rich for SemDeDup (Section 5 (Limitations, lines 949–954) and Section 2 (Related Work: Deduplication, lines 244–248)).

Critical weaknesses
  • Limited empirical evaluation: (i) only within-cluster pruning where cluster mixture is held fixed, (ii) on a single, curated web corpus (C4), (iii) at toy/small model scales (124M and 350M), (iv) at a single 50% retention operating point, (v) no downstream benchmark evaluations (perplexity may be sensitive to surface statistics, n-gram frequencies, and document boundary effects).

  • Narrow scope / limited novelty: The scope of the claim seems to be very narrow (see "Limited empirical evaluation" above) and only becomes fully evident after carefully reading the paper. In particular, they do not discuss the deeper implications of using C4 for their broader geometric claim. Similarly, the fundamental mechanism uncovered by the paper reduces to a well-established principle in data-constrained scaling laws (e.g., Muennighoff et al., 2023; Hernandez et al., 2022), which makes the paper's finding an incremental empirical observation rather than a major conceptual breakthrough.

Questions for authors

Questions:

  • Uncurated corpora: How does the length–centroid correlation behave on uncurated corpora? Does high noise or boilerplate alter the correlation? Do you view your conclusions as applying strictly to within-cluster pruning in homogeneous corpora?

  • Connection to scaling laws: Given that data-constrained scaling laws already predict that repetition degrades loss at fixed compute, do you view this work as confirming those scaling laws within data-selection evaluations? Framing the token-mass confound explicitly through this established lens could make the paper's contribution clearer and more unified.

Suggestions:

  • A more informative and scoped title would be stronger and less confusing, like for example "Token Count and Document Length Confounds Centroid-Based Data Selection for Language Models".

  • Terminology choices: Target terminology for a deep learning rather than a statistics/science audience. A few examples: token mass -> token count? arm -> condition or selection variant? data/token repetition -> epoch? etc. Or add plain-language definitions of such terms when they are first used. Of course, this mainly applies to the beginning of the paper (title, abstract, maybe introduction). Once the context is set, it's ok to use the concise, scientific terminology.

  • Complexity vs. core result: The central finding is straightforward (documents near centroids are longer, so document-budget selection retains more total text, which reduces repetition and improves loss under fixed compute). The current exposition introduces heavy notation and terminology that feels disproportionate to this intuitive mechanism.

  • Nit: Add some visual aid to Figure 3 to show the document length / distance to centroid correlation.

4 comments

Thank you for the careful review and constructive suggestions. Since submission we have run the controls requested across the reviews (all at 124M, 3 seeds; full details in our response to Reviewer t6gM). They bear directly on your scaling-law question, and the answer sharpens the contribution rather than reducing it to a known principle.

Q2 / W2. Connection to data-constrained scaling laws. We tested exactly this lens. A random-subset dose-response, random pools of 0.66–2.32B tokens with population composition, i.e. 1.5 down to 0.4 epochs at fixed compute, moves out-of-distribution loss by only ≈0.016 nats across the whole range, as Muennighoff et al. (2023) predict for mild repetition. Retained token count therefore cannot explain the large differences between selection conditions: against a random pool of the same token count, far-centroid selection is still +0.032 worse and the shortest documents +0.046 worse (FineWeb-Edu), and single-pass training with no repetition leaves those penalties unchanged (+0.033, +0.053). What drives them is document-length composition: the near-centroid gain (−0.007 at equal tokens) matches simply selecting the longest documents (−0.008), and matching tokens and length jointly removes the near/far difference out of distribution (0.000 on FineWeb-Edu, −0.003 on Wikipedia). So the token-count mechanism in our submission, and in your summary of it, was wrong, and the scaling-law lens is what shows it: it bounds the token-count component as small, leaving length composition as the main factor. We will frame the paper explicitly through this lens and cite Muennighoff et al. and Hernandez et al. (2022).

Q1. Uncurated corpora. We ran an identical pipeline (the paper's LSA settings, ≈750 documents per cluster) on 500k-document samples of curated C4 and uncleaned C4 (en.noclean). The coupling persists on uncleaned text but is weaker: within-cluster ρ(length, centroid distance) = −0.32 vs −0.58, and at a 50% document budget near-centroid selection keeps 1.63× as many tokens as far-centroid selection, vs 2.63× on curated C4. (Uncleaned documents are also much longer, median 915 vs 272 tokens, since boilerplate is retained. These sample-level values differ from the full-pool numbers in the paper, within-cluster −0.35 with k = 8000 on 6M documents, because LSA and clustering are refit per sample; the comparison uses the identical procedure for both corpora.) So the length bias of centroid selection is not specific to curated text, though it is weaker on uncleaned web data. Note also that the within-cluster correlation is the relevant statistic: in OPT space the pooled correlation is −0.72 but only −0.16 within clusters, so OPT prototypicality barely shifts length. Our conclusions are established only for within-cluster selection on a curated corpus. What transfers is the measurement, not the sign: a selection rule's length bias depends on the corpus and the embedding, near zero for OPT-space prototypicality on C4, strongly toward short documents for faithful SemDeDup (0.69B tokens retained at 50% of documents, vs 1.49B for random), so it must be measured for each method rather than assumed.

W1. Limited evaluation. We agree on each point and will state them prominently: (i) within-cluster selection only (mixing methods also shift length composition and deserve the same audit); (ii) one curated training corpus (Q1 adds the uncurated correlation analysis, not training runs); (iii) 124M and 350M only; (iv) a single 50% retention point; (v) loss-based evaluation only. On (v) your concern is apt: the length effect we identify may partly reflect how well models trained on shorter documents predict tokens deep inside long ones, a document-boundary effect that loss captures and benchmarks may not. We now report in-distribution C4 loss alongside the two OOD sets, but downstream evaluation at larger scale is needed to know whether the effect matters for capabilities.

Clarifications on the setup. We did not deduplicate beyond C4's own cleaning (which removes repeated three-sentence spans), and "loss" is mean per-token cross-entropy in nats (log-perplexity) on fixed, identical held-out batches for every model.

Continued in next comment

Presentation. We will adopt your suggestions:

  • Title. ICLR does not permit title changes during the discussion period; for the next version we propose "Document Length and Token Count Confound Centroid-Based Data Selection for Language Models", close to your suggestion and ordered to match the corrected finding.
  • Terminology. "Token mass" → "retained token count", "arm" → "selection condition", "repetition" → "epochs over the selected pool", each with a plain-language definition on first use.
  • Complexity. The corrected mechanism fits in one sentence — near-centroid documents are longer, and longer documents lower out-of-distribution loss — so we will lead with it and move the formal definitions to the appendix.
  • Figure 3. We will add binned medians of centroid distance by length decile, computed both pooled and within clusters, which also shows the within- vs between-cluster distinction above.

Thank you for the candid rebuttal and for carefully addressing my questions, concerns, and presentation suggestions. I have also read the other reviews and your responses to them.

I commend your scientific transparency and the extensive new experiments, including the uncurated C4 analysis, the faithful SemDeDup implementation, the scaling-law dose-response, the single-pass training, the joint token-and-length control, and the OPT-125M comparison. This new direction sounds very promising, resolves my novelty concern relative to Muennighoff et al. (2023), and meaningfully improves the paper.

These results overturn the original headline claim ("Token Mass, Not Geometry"), replace the SemDeDup baseline, and require a major rewriting of the narrative from the title to the conclusion. Consequently, it is difficult to thoroughly re-evaluate the work without a revised PDF. I will maintain my current evaluation score. However, I strongly encourage the authors to integrate these new findings (and, if possible, expand beyond 124M controls and loss-only evaluations) into a new revised submission for a future conference.

A tentative updated PDF is uploaded. We understand it is already Nov 18th but if the reviewers could take a look, we have more time to finish a camera ready version within the time frame for camera ready version.

Reviewer_qfqfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines whether the apparent benefit of embedding-based data selection for language model pretraining comes from retained token mass rather than embedding geometry. Using C4, the authors compare centroid-based, random, length-based, and semantic-deduplication selection. They find that near-centroid selection favors longer documents and retains more tokens under a fixed document budget. Its advantage over random selection disappears when the retained token budget is matched. The experiments use 124M and 350M models, three seeds, and evaluations on FineWeb-Edu and Wikipedia.

Critical strengths

The paper raises a reasonable concern about retained token counts in document-based data selection. Comparing document-budget and token-matched results is a useful way to examine this issue, and the observed change in performance provides some motivation for further investigation.

Critical weaknesses

1.The writing and presentation do not always reflect what the experiments actually establish.*The title and conclusion attribute the apparent gain to token mass rather than embedding geometry, although the authors acknowledge in Section 5 that neither control fully isolates these factors. The Introduction also broadly characterizes previous evaluations as relying on single model scales, single runs, and single evaluation distributions, although several cited studies already address some of these limitations. D4 has discussed variation in retained token counts under document-level selection, while Sam et al. examine embedding-based data selection under a token budget. The paper should distinguish its specific finding from these earlier observations more carefully. The abstract is crowded with correlations, model sizes, and loss differences, while giving little attention to the limitations of the comparisons. Similar conclusions are repeated throughout the paper without much additional explanation. The main evidence is also split across Table 2 and Figures 4–5, making it harder to compare the controls directly. Together, these presentation choices make it harder to distinguish the specific empirical finding from the broader interpretation the paper draws from it.

  1. Several important steps in the experimental argument remain unresolved. The length-stratified and token-matched controls are useful, but they do not isolate one factor at a time. Length stratification still leaves different retained token counts (1.64B versus 1.35B), while token matching changes both document count and length composition. The reduction in the near/far-centroid gap from 0.044 to 0.016 to 0.002 nats is interesting, but it does not establish that token mass alone explains the original gain. The proposed mechanism is also incomplete: the authors suggest that larger retained pools reduce repetition and that longer documents provide more useful context, yet neither contribution is directly measured or separated. A joint control or a more detailed analysis of actual training exposure would make the argument much stronger. The evaluation is also narrow for the stated conclusion. The main experiments use one corpus, a fixed 50% retention rate, and 124M/350M models, without sensitivity analysis of the retention fraction or clustering setup. The full near/far controls are limited to 124M, the central token-matched comparison is not repeated in the OPT embedding space, and Appendix B reports only document-budget results on Wikipedia. The SemDeDup baseline also needs clarification, since its fixed-budget similarity ranking is not obviously equivalent to the original deduplication procedure. Finally, Appendix A provides basic hyperparameters but little further evidence addressing these missing controls or explaining the mechanism. Taken together, these gaps make the experimental evidence less complete than the paper's main claim requires. Taken together, these gaps leave the paper's central causal explanation insufficiently established, rather than merely limiting its generality.
Questions for authors
  1. In Table 2, length stratification still leaves different token counts, while token matching changes the document count and length distribution. Have the authors considered matching both token count and length distribution? This would help determine whether the observed reversal is attributable to token mass rather than other differences between the selected subsets.

  2. Section 4.2 suggests that larger retained pools reduce repetition and that longer documents provide more useful context. How much data is actually repeated under each selection strategy? A breakdown of these effects would help establish which mechanism accounts for the loss differences, especially given the sampling-with-replacement setup.

  3. Why does Appendix B report only document-budget results on Wikipedia? Are there corresponding token-matched results, or near-centroid selection results in the OPT embedding space? It would also be useful to know whether the finding holds at retention rates other than 50%.

4.The reported baseline ranks documents by maximum within-cluster similarity, which appears different from the original threshold-based deduplication procedure. Is this an equivalent implementation or a simplified variant? A clarification would help assess how the comparison should be interpreted.

2 comments

Thank you. Your review identified the central weakness precisely: our controls did not isolate one factor at a time, and so did not establish the causal claim. We ran the controls you asked for, together with those requested by the other reviewers (all at 124M, 3 seeds), and they show that our causal explanation was wrong in an important respect. Below we report the new evidence and the corrected claim. Our response to Reviewer t6gM gives further detail, and the revised PDF, to be uploaded before the discussion closes, will reflect all of it.

Corrected claim. Under a document budget, LSA near- and far-centroid selection differ strongly in both retained token count and document length. Separating the two shows that document-length composition — not retained token count, and not embedding geometry — accounts for most of the apparent out-of-distribution effects. In OPT space, where selection barely changes length, a modest geometric effect remains on FineWeb-Edu.

Q1 / W2. Controls that vary one factor at a time.

  • Token count only. Random subsets of 0.66–2.32B tokens preserve the population's length and centroid-distance distributions (by construction) while the document count varies (1.3M–4.7M). Out-of-distribution loss moves by only ≈0.016 nats across this range (1.5 down to 0.4 epochs). Against a random subset of the same token count (FineWeb-Edu): near-centroid −0.007, longest documents −0.008, far-centroid +0.032, shortest documents +0.046.
  • Token count and length matched jointly (your Q1). Within each (cluster × length-bin) cell we keep near- or far-centroid documents until half the cell's tokens. The subsets match in tokens (1.517B vs 1.518B) and length distribution (KS = 0.016). Near − far: 0.000 on FineWeb-Edu and −0.003 on Wikipedia (sign varying across seeds); −0.008 on in-distribution C4 (consistent).
  • Exposure only. Single-pass training (no repetition) on the same document-budget pools leaves the far-centroid and shortest-document penalties unchanged (+0.033 and +0.053 vs single-pass random on FineWeb-Edu) and halves the near-centroid advantage (−0.010 → −0.005).

This also explains the 0.044 → 0.016 → 0.002 reduction you questioned. The remaining token difference in the length-stratified control (1.64B vs 1.35B) predicts only ≈0.002 of its 0.016 gap; the rest most likely reflects residual length differences within the coarse deciles. Token matching removed the gap by changing which documents were kept, not by equalizing a token-mass effect.

Q2. Actual training exposure. Under nanoGPT's with-replacement window sampling at 50% document retention:

conditionpool tokensepochsdistinct tokens seenrepeated exposures
near-centroid1.97B0.500.77B21%
random1.49B0.660.72B27%
far-centroid1.01B0.980.63B36%
shortest documents0.71B1.380.53B46%
longest documents2.26B0.440.80B19%

All token-matched conditions have 0.66 epochs and 27% repeated exposures. Single-pass training removes repetition entirely for every pool larger than the training budget, so the differences that survive it are not caused by repetition. Of the two mechanisms you list, repetition contributes little (at most ≈0.016 across the dose-response); document length is the main factor. We do not isolate why length matters (e.g., more within-document context), so we describe it as a length-composition effect.

** Continued in next comment**

Q3. Wikipedia, OPT, retention, and clustering.

  • Wikipedia. Every checkpoint is now evaluated on both OOD sets and on in-distribution C4. Token-matched, Wikipedia, vs random: near +0.006, far +0.010, longest documents −0.014.
  • OPT space, both budgets. OPT selection barely changes length (within-cluster ρ(length, distance) = −0.16 vs −0.35 for LSA; near/far keep 1.50B vs 1.47B tokens), so the two budgets agree: near-centroid −0.006 to −0.009 on FineWeb-Edu and ≈0 on Wikipedia and C4; far-centroid +0.009 on FineWeb-Edu and +0.004 to +0.005 on C4.
  • Retention. We trained only at 50%, but the size of the length shift at other rates can be measured on the paper's clusterings. The LSA near/far token ratio is 1.44× at 75% retention, 1.95× at 50%, and 2.86× at 25% (mean length 803 vs 281 tokens at 25%), while OPT selection stays length-neutral at every rate (≤1.03×). The length bias therefore grows under more aggressive pruning; we have not verified the corresponding loss effects at other rates and state this as a limitation.
  • Clustering setup. On a 500k-document sample, the within-cluster correlation is −0.57 to −0.61 and the near/far token ratio 2.59–2.74× for 167 to 2,666 clusters (a 16× range in cluster size). On uncleaned C4 (en.noclean) the coupling persists but is weaker (−0.32 vs −0.58; token ratio 1.63× vs 2.63×), measured with an identical pipeline.
  • Scale. The new controls are at 124M only; at 350M we have the document-budget and token-matched comparisons. We state this as a limitation.

Q4. SemDeDup. It was a simplified variant, not an equivalent implementation. Ranking by maximum within-cluster similarity at a fixed retention removes both copies of a near-duplicate pair, whereas SemDeDup keeps one representative per duplicate group (on synthetic planted duplicates, the variant dropped both copies of all 40 pairs). We therefore implemented SemDeDup faithfully: the official one-shot procedure in normalized OPT-125M space, keeping the document farthest from the centroid, with a global ε = 0.084 chosen to retain 50% of documents. We trained it with 3 seeds. At this retention it strongly prefers short documents: it keeps 0.69B tokens (median 148 tokens per document) vs 1.49B for random, and is +0.079 worse than random on FineWeb-Edu and +0.069 worse than a random pool of the same token count. Its length composition resembles our shortest-documents condition (mean 231 vs 239 tokens), whose deficit against a same-size random pool is +0.046. Caveat: 50% document retention is far more aggressive than SemDeDup's usual operating point, so this does not evaluate SemDeDup as typically used. It does show that a faithful deduplication rule can shift length composition strongly at a fixed document budget, in the opposite direction from near-centroid selection. The paper's SemDeDup section will be relabeled and revised accordingly.

W1. Presentation and positioning. We agree with each point and will:

  • retitle the paper in the next version (titles cannot change during the discussion period) and remove "token mass, not geometry" from the abstract and conclusion;
  • rewrite the introduction's characterization of prior evaluations, which was too broad;
  • position our result relative to D4's appendix, which already reports token-count differences under document-level selection, and Sam et al., who, as you note, evaluate embedding-based selection under a token budget. Our contribution is the decomposition above — how much of a document-budget comparison is token count, length composition, and geometry — and a protocol for measuring it;
  • shorten the abstract, state its limitations there, and remove repetition;
  • consolidate all controls (document budget, token-matched, length-stratified, joint, single-pass, same-size random) into one table, replacing Table 2 and Figures 4–5;
  • expand Appendix A with the exposure and composition tables above.

We understand that this changes the paper's central claim. We believe the corrected claim is better supported, and we thank you for the review that pushed us to it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.