acceptodds
Under review as a conference paper at ICLR 2027

Hubs or Fringes? Pretraining Data Selection via Web Graph Centrality

Abstract

The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary classifiers for document scoring or mixture optimization, adding computational overhead and dependence on labeled data. We propose WebGraphMix, a lightweight, graph-based data selection framework that computes structural centrality scores over the Common Crawl host-level web graph and uses them to vary the proportion of central versus peripheral documents in the pretraining mixture. We hypothesize that central hosts expose models to reusable abstractions, while peripheral hosts encode specialized, long-tail knowledge. WebGraphMix computes centrality scores efficiently at web scale, requiring no model training, labeled data, or downstream supervision. We integrate WebGraphMix into the DataComp-LM pipeline and train models at 400M and 1B parameter scales with 8B and 28B tokens respectively, evaluating on 23 tasks ranging from factual knowledge to symbolic reasoning. Our experiments show that central and peripheral web regions encode complementary capabilities. Mixture combining both at a ratio of 1:1 achieves 41.4% on average, compared to 39.8% for uniform sampling. Combining structural scores with document-level quality classifier scores further improves performance to 43.8%. These findings demonstrate that web graph topology is a meaningful axis for pretraining data curation, capturing information that is largely orthogonal to existing content-based approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, pretraining, data selection

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fRapOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WebGraphMix, a lightweight pretraining data selection framework based on host-level web graph centrality (mainly betweenness). At 1B scale, a 50/50 mix of the highest- and lowest-betweenness hosts beats random sampling (41.4% vs. 39.8%), and adding a quality score reaches 43.8% vs. 42.3% for quality-only.

Critical strengths
  1. Lightweight framework, efficient to run: one-time centrality computation (<9 hours on 4 H100s), no model training or labeled data.
  2. Better than random and combinable with quality score in terms of performance. At 1B, every centrality + quality run beats quality-only.
  3. A new signal nearly orthogonal to document quality (Spearman ρ = 0.018).
  4. Reasonable controls (Middle-K, in-degree, eigenvector) rule out simple host popularity or link counting.
Critical weaknesses
  1. Limited performance gain. WebGraphMix alone (41.4%) is below the simple quality-filter baseline (42.3%), and WebGraphMix+ beats it by only 1.5 points, around 89% of which comes from four noisy tasks.
  2. Not robust across scale. At 400M, the best 1B mixture is 1.0 point below random, and the best configuration changes between scales.
  3. Weak experimental setting. All configurations are selected on the test benchmark with one seed per 1B run, so several claims (e.g., 43.8 vs. 43.4 against WebOrganizer+) are within noise.
  4. Weak support for the hypothesis. Jeopardy, the most long-tail knowledge task, drops from 15.3 to 13.0, contradicting the claim that peripheral hosts provide long-tail knowledge.
Questions for authors
  1. How you explain the the drop on long-tail knowledge drop ?
  2. How you can predict the performance at scale ? Modern LLMs requires trilliions of tokens, 1b mixture looks limited, and can not demonstrate the approach.
  3. Is it possible to run multiple runs to reduce the noise ?
  4. In weborganizer, domain & topic distribution match is an approach to improve the performance on benchmark, how your sampling approach change the domain / topic ? is this change lead to improvement instead of using your mix approach ?
3 comments

Thank you for the review. Several of your points changed how we present WebGraphMix+.

W1: limited gain, concentrated in noisy tasks

Reviewer: "WebGraphMix alone (41.4%) is below the simple quality-filter baseline (42.3%), and WebGraphMix+ beats it by only 1.5 points, around 89% of which comes from four noisy tasks."

Response: We agree on both points. WebGraphMix alone is below the quality filter (41.42 vs 42.30); we present centrality as a selection signal that needs no labels or classifier, not as a replacement for quality filtering. Your 89% figure for WebGraphMix+ is correct: BoolQ, CommonsenseQA, LSAT-AR and Dyck supply it, and on the 18 tasks where Random is above DCLM's reference baseline the gain over Quality is +0.45. The underlying problem is larger than noisy tasks: we found that WebGraphMix+ selects almost the same documents as the quality filter (general response, error 1), and we withdraw it as evidence that centrality adds to quality. We will post the three-seed comparison with Quality before the discussion closes.

The gain of WebGraphMix over Random is broader: it is ahead on 19 of 23 tasks, gains +1.06 on the 18 tasks above DCLM's reference baseline, and stays positive on held-out tasks (general response).

W2 and Q2: scale

Reviewer: "How you can predict the performance at scale? Modern LLMs requires trillions of tokens, 1b mixture looks limited, and can not demonstrate the approach."

Response: We cannot predict performance at trillion-token scale and make no such claim; the 400M result already shows that the benefit depends on scale. A 1b-1x run takes about 240 H100-hours, and 7b-1x is beyond our budget. Within that budget, we are testing whether the fixed recipe (betweenness, 50/50) holds at half the token budget (14.4B tokens) against Random at the same budget, with results before the discussion closes. The limitations section of the revised paper states the scale limits plainly.

W3 and Q3: single seed, selection on the test benchmark

Reviewer: "All configurations are selected on the test benchmark with one seed per 1B run, so several claims (e.g., 43.8 vs. 43.4 against WebOrganizer+) are within noise." "Is it possible to run multiple runs to reduce the noise?"

Response: Yes. We are retraining Random, Quality, WebGraphMix and WebGraphMix+ with two new seeds each, and will post per-seed scores and paired differences before the discussion closes. To account for selection on the test benchmark, we evaluated the 1B checkpoints on 26 DCLM tasks never used for any decision. Choosing the configuration on those tasks alone selects Katz 50/50, which scores 40.75 on the main suite against 39.84 for Random (details in our reply to Reviewer rs5Y, W1 and Q1). We agree the 0.4-point gap to WebOrganizer+ is too small to call a win, and the revised paper describes the two as comparable.

W4 and Q1: Jeopardy

Response: We agree that Jeopardy does not support the long-tail account. Across our 1B runs, Jeopardy tracks quality filtering: the runs whose training data is essentially the top of the quality ranking score 24.5 to 28.6, while runs with no quality signal score 10.1 to 17.5. Among the latter, the peripheral-only selection (corrected betweenness Bottom-K, 13.1) is also below Random (15.3), so peripheral hosts do not add the trivia Jeopardy tests. The seed runs will show how much of the 2.3-point drop is noise, and we will report that with them.

WebGraphMix's Knowledge gain (34.2 to 35.4) comes from other tasks: MMLU, QA-Wikidata and both ARC sets each rise by 0.7 to 1.3 points, and LSAT-AR, with 230 questions, rises by 5.7. The revised paper presents the long-tail account as a hypothesis consistent with the aggregate results, not as an established mechanism, and discusses the Jeopardy result (§5.1).

Q4: effect on domain and topic

Reviewer: "In WebOrganizer, domain & topic distribution match is an approach to improve the performance on benchmark, how your sampling approach change the domain / topic? Is this change lead to improvement instead of using your mix approach?"

Response: The two halves differ sharply in composition. The central half is weighted toward news articles (9.6% of Random's tokens vs 19.6% of the central half) and politics (8.2% vs 13.2%), while the peripheral half is weighted toward personal blogs (21.5% vs 48.7%). Mixing them largely cancels out: the largest shifts in WebGraphMix relative to Random are personal blogs (+5.6 points) and product pages (−4.1 points).

To test whether this shift causes the gain, we are training a control that reweights Random's selection to WebGraphMix's exact joint topic × format distribution (576 WebOrganizer cells), at the same budget. We will post its score before the discussion closes, against 39.84 for Random and 41.42 for WebGraphMix.

AuthorsOpenReview ↗

Response to Reviewer fRap: seed and control results

Thank you. The seed runs and controls are now posted in our follow-up to all reviewers.

W4 and Q1 (Jeopardy). Across three seeds Random scores 15.3, 14.8 and 13.4 and WebGraphMix 13.0, 14.5 and 16.2 (means 14.5 and 14.6). The drop at seed 124 was seed noise; Jeopardy neither contradicts nor supports the long-tail account, and the revised paper removes that discussion.

W3 and Q3 (seeds). WebGraphMix − Random is +0.51 ± 1.02 on the 23-task average, within noise, and ahead at all three seeds on a precision-weighted average (+0.40; details in follow-up 1/2). WebGraphMix+ − Quality is +1.54 and +0.21 at two seeds.

Q4 (topic and domain). Random reweighted to WebGraphMix's topic × format mix scores 40.62, against 39.84 for Random and 41.42 for WebGraphMix at seed 124 (WebGraphMix's three-seed mean is 40.81). One seed is not enough to separate composition from centrality, and the revised paper flags this as needing multi-seed controls.

W2 and Q2 (scale). At half the token budget, WebGraphMix − Random is +0.79 with one seed, within noise.

W1. The task-level figures in our reply (19 of 23 tasks, +1.06 on 18 tasks) are from seed 124 only; at seeds 125 and 126 the count is 11 and 12 of 23.

W1 - Resolved, the increase is consistent.

W2 - Partially resolved, somehow shows approach is scaling, but require more data point.

W3 - Resolved, gain is small and consistent.

W4 - Partially resolved, but need more seed.

I totally understand the compute concern, and I'd move to accept based on current additional results.

Reviewer_x5RkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WebGraphMix, which uses host-level centrality in the Common Crawl web graph as an unsupervised signal for language-model pretraining data selection. Documents inherit their host's centrality, and training mixtures sample from central and peripheral strata. The method is evaluated within DataComp-LM at 400M and 1B scales using DCLM CORE v2. At 1B, a 50/50 betweenness-centrality mixture reports 41.4% average accuracy versus 39.8% for random sampling; combining centrality with DCLM-fasttext quality scores reports 43.8% versus 42.3% for quality-only selection. Analyses compare middle-centrality sampling, Katz/eigenvector centrality, raw in-degree, and quality combinations. Pure centrality selection does not improve over random at 400M, and the best configuration changes across scales.

Critical strengths
  1. Web topology is a promising and comparatively inexpensive axis for data curation. The method requires neither downstream labels nor proxy-model training, and centrality is computed once and reused. The comparison with raw in-degree and eigenvector centrality is particularly useful: the stronger betweenness result suggests that the effect is not adequately explained by host popularity or link count alone.

  2. The mixture and quality-complementarity analyses go beyond a single leaderboard result. The central/peripheral mixture outperforming either tail and the middle band supports the complementarity hypothesis. Multiple centrality-plus-quality configurations also exceed quality-only selection at 1B. I appreciate the explicit reporting of the negative 400M results and the discussion of scale limitations, crawl-policy bias, and unresolved host/domain confounding.

Critical weaknesses
  1. The headline result is sensitive to evaluation-suite selection and unquantified training variance. Many centrality choices, combination rules, and mixture ratios are evaluated on the same 23 tasks, and the best configuration is then highlighted separately at each scale. There is no held-out task suite or alternative selection protocol showing that the reported configuration would be chosen without consulting this test average. Training-seed variance is estimated only for the 400M uniform baseline, whereas the principal 1B claims rely on single training runs. Bootstrapping evaluation examples does not capture data-order, optimization, or sampling variance. These issues matter because the key Quality-to-WebGraphMix+ gain is 1.5 points and the pure-centrality benefit disappears at 400M.

  2. The combined-score construction is numerically and operationally under-specified. Eq. (4) applies exp(s_i - max_j s_j) directly to centrality and quality scores. Raw betweenness values on a graph of this size are typically large and highly skewed; under the formula as written, many normalized centralities could collapse to zero or numerically underflow. This would change the additive, multiplicative, and division-based rankings and potentially create large tie sets. The exact Top-K/Bottom-K percentiles and tie-breaking rules also need to be stated more clearly. These details are essential for interpreting and reproducing the strongest result.

Questions for authors
  1. What exact numerical scaling is applied before Eq. (4)? Are betweenness scores log-transformed, rank-normalized, or otherwise rescaled? How many documents receive nonzero normalized centrality, how are zero/tied scores handled, and what exact percentiles define Top-K and Bottom-K?

  2. Can you select the configuration on held-out proxy tasks and evaluate it once on DCLM CORE v2, or otherwise account for configuration selection? Can you also replicate the key 1B Random, Quality, WebGraphMix, and WebGraphMix+ runs across multiple training/data-shuffling seeds and report the resulting variance?

3 comments

Thank you for the precise review. You were right to press on Eq. (4), and the answer changes how we describe WebGraphMix+.

W1 and Q2: selection on the test suite and training variance

Reviewer: "Can you select the configuration on held-out proxy tasks and evaluate it once on DCLM CORE v2, or otherwise account for configuration selection? Can you also replicate the key 1B Random, Quality, WebGraphMix, and WebGraphMix+ runs across multiple training/data-shuffling seeds and report the resulting variance?"

Response: Held-out selection. We evaluated existing 1B checkpoints on the 26 DCLM tasks outside our suite (list and table in our reply to Reviewer rs5Y, W1 and Q1). Choosing among the eight pure-centrality configurations by held-out score alone selects Katz 50/50, which we then evaluated once on DCLM CORE v2: 40.75 against 39.84 for Random (+0.91). On the held-out tasks, the four correctly built mixtures stay above Random (+0.44 to +1.51).

These tasks are close to chance at 1B, so we also checked selection within the main suite. Choosing the best of the four correctly built betweenness and Katz mixtures on a random half of the 23 tasks and scoring it on the other half gives an average gain over Random of +1.17, positive in 99% of 20,000 splits. We do not report the same check for the quality-combined configurations, because those runs do not differ meaningfully in their training data (see below).

Training seeds. We are retraining exactly these four runs with two new seeds each, changing initialization and data order. Per-seed scores and paired differences will be posted before the discussion closes.

W2 and Q1: scaling before Eq. (4), zero and tied scores, percentiles

Reviewer: "What exact numerical scaling is applied before Eq. (4)? Are betweenness scores log-transformed, rank-normalized, or otherwise rescaled? How many documents receive nonzero normalized centrality, how are zero/tied scores handled, and what exact percentiles define Top-K and Bottom-K?"

Response: No rescaling is applied: raw scores enter exp(s − max s). Raw betweenness lies between 0 and 0.198, so normalized betweenness lies between 0.820 and 1. Every document from a host in the graph gets a nonzero value, and nothing underflows. Because the max is a constant, the multiplicative score ranks documents by raw c + q and the divisive score by raw c − q, so Eq. (4) adds no ties beyond those in the raw scores.

Your concern about scale is right, though it shows up as compression rather than underflow, and it led us to an error in the paper. Raw betweenness is below 2.34e-5 for 99.97% of hosts, and raw Katz spans only 2.68e-4 to 4.02e-4, while DCLM-fasttext scores span 0 to 1. Outside a few dozen hub hosts (facebook.com at 0.198, en.wikipedia.org at 0.025), centrality differences are far smaller than the quality differences near the selection cutoff, so c + q and c − q select the documents that q alone would. Measured directly, 99.9% of WebGraphMix+'s unique tokens lie inside the Quality selection, and 100% for the Katz + quality mixture. The two halves of WebGraphMix+ also overlap each other by 99.0%, contrary to §3.4, so the model trained on 16.15B unique tokens, each present twice in the training set.

We therefore withdraw WebGraphMix+ as a combination of two signals. It is a stricter quality filter over in-graph hosts with repetition, and its +1.5 over Quality may reflect that stricter threshold, the repetition, or noise; our data cannot separate these. The revised §3.4 states the raw-score scaling and this consequence, and we have removed the claim that the two signals' gains compound. For the camera-ready we will add a rank-normalized combination with disjoint strata, in which both signals carry weight.

Zero and tied scores, percentiles. 17.4% of hosts have betweenness exactly 0. They tie at the bottom of the ranking, ties are broken by document order in the pool, and WebGraphMix's peripheral half includes all of them. Cutoffs are defined over token mass: the central half is the top 8.9% of eligible tokens (top 0.029% of hosts, betweenness ≥ 2.34e-5), the peripheral half is the bottom 8.9% (bottom 28.9% of hosts, betweenness ≤ 1.5e-12), and pure Top-K is the top 17.7% of tokens (top 0.15% of hosts). The full table is in our reply to Reviewer rs5Y (W3 and Q3).

Thank you for the detailed and transparent response. The held-out-task selection results help alleviate my concern about configuration selection, although the additional training-seed results are still needed to assess training variance. The clarification of Eq. (4) is also helpful; however, the finding that WebGraphMix+ largely duplicates a quality-based selection with repetition, together with the corrected Bottom-K results, substantially weakens the paper’s original complementarity claims.

At this stage, I therefore plan to retain my weak-reject score. I encourage the authors to clearly scope the revised claims to the validated pure-centrality results and to report the pending seed experiments.

AuthorsOpenReview ↗

Response to Reviewer x5Rk: seed results and revised scope

Thank you. The seed experiments are now posted in our follow-up to all reviewers.

Training variance. A difference between two single-seed 1B runs has an SD of 0.87. WebGraphMix − Random on the 23-task average is +1.58, −0.45 and +0.40 across seeds (+0.51 ± 1.02), within noise. On a precision-weighted average it is ahead at all three seeds (+0.40, p = 0.03); follow-up 1/2 explains how the weights were set and how sensitive the result is to them. WebGraphMix+ − Quality is +1.54 and +0.21 at two seeds.

Configuration selection. The held-out selection (Katz 50/50, +0.91) and the split-half check (+1.17) in our reply use the seed-124 runs only. At 0.87 SD of a difference, most gaps among centrality variants are within noise, so we no longer draw conclusions from their ordering.

Scope. As you suggest, the revised paper limits its claims to what three seeds support: quality filtering gives a clear gain; WebGraphMix is comparable to uniform sampling on the 23-task average, with a small, consistent gain on the precision-weighted average that remains to be confirmed; and our single-seed controls do not yet isolate a contribution from centrality. The full list of changes is at the end of follow-up 2/2.

Reviewer_vUrYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a method to process common-crawl data for LLM training. It scores each document by how close it is to center in the link graph. Big website are hubs like wikipedia and peripheral sites are small sites with few inbound links. Training data sets the mix of the website based on the distance. At the 1B model scale, 50/50 mix scores 41.4 vs 39.8 for uniform sampling. The standard quality classifier gets 42.3. Adding the quality classifier, combined with the data mix, the model gets 43.8, beats the WebOrganizer+, the prior baseline. Centrality does not help at 400M model size.

Critical strengths
  1. The distance to central is a simple and effective signal. It does not overlap with content filters. The paper showed that adding the distance to quality classifier still gains on model quality.
  2. The paper presents a faire and reproducible set up to rule out confounds in data processing.
Critical weaknesses
  1. The gain is fragile and mainly from two old tasks.
  2. Paper lacks real noise estimate or held-out tests.
Questions for authors
  1. We need more data/runs to show the lift is beyond noise. We need more runs at 1B model on different datasets to validate the set ups.
  2. Every central or peripheral selection takes entire hosts, unlike uniform document sampling. What does a 50/50 mix of two random host subsets score at the same token budget?
2 comments

Thank you for the review and the two concrete requests. The random-host control you asked for is training now.

W1: gain is fragile and mainly from two old tasks

Reviewer: "The gain is fragile and mainly from two old tasks."

Response: This depends on the comparison. For WebGraphMix+ over quality-only selection, we agree: BoolQ (+11.5) and CommonsenseQA (+10.6) supply 63% of the 1.5-point gain, every 1B run scores below DCLM's reference baseline on CommonsenseQA (40.3), and on the 18 tasks where Random is above its reference baseline the gain falls to +0.45. We also found that WebGraphMix+ does not combine the two signals as described (general response, error 1), and we withdraw it as a centrality result.

For WebGraphMix over Random, the gain is broader. WebGraphMix is ahead on 19 of 23 tasks. Dropping the five tasks whose scores vary most across our 1B runs (Jeopardy, CommonsenseQA, BoolQ, ARC-Easy, Dyck) leaves +1.16, against +1.58 on all 23. The four largest gains (CommonsenseQA, Dyck, LSAT-AR, Winogrande) make up 66% of the total; without them the gain is +0.65. On the 18 tasks above reference baseline it is +1.06, and the four correctly built betweenness and Katz mixtures are all positive there (+0.95 to +1.09).

W2: no noise estimate or held-out tests

Reviewer: "Paper lacks real noise estimate or held-out tests."

Response: We added both; details are in the general response and our reply to Reviewer rs5Y (W1 and Q1). Three-seed results for the four headline runs will be posted before the discussion closes. The current noise estimate puts the SD of a difference between two 1B runs at about 0.37, a lower bound. On 26 held-out DCLM tasks never used for any decision, WebGraphMix scores +0.44 over Random (centered accuracy), and selecting the configuration on those tasks alone gives +0.91 over Random on the main suite.

Q1: more runs and other datasets

Reviewer: "We need more data/runs to show the lift is beyond noise. We need more runs at 1B model on different datasets to validate the set ups."

Response: Each headline 1B run will have three training seeds (results before the discussion closes), and we are running the fixed recipe at half the token budget with nothing tuned. On other datasets, we agree a second pool is the stronger test. We could not run it within the discussion period and will add a fixed-recipe run on a second data pool to the camera-ready. Our current pool is DCLM's Common Crawl pool after heuristic filtering and deduplication, with no model-based quality filter.

Q2: random host subsets

Reviewer: "Every central or peripheral selection takes entire hosts, unlike uniform document sampling. What does a 50/50 mix of two random host subsets score at the same token budget?"

Response: We are training this control. Hosts are put in random order and taken whole until each half reaches 16B tokens, with the same eligibility and budget as WebGraphMix. Because the order is random, the two halves together are one sample of whole random hosts. We will post its score before the discussion closes, against 39.84 for document-level Random and 41.42 for WebGraphMix.

AuthorsOpenReview ↗

Response to Reviewer vUrY: seed and control results

Thank you. The results you asked for are now posted in our follow-up to all reviewers.

Q2 (random host subsets). The 50/50 mixture of whole random hosts scores 39.31, against 39.84 for document-level Random and 41.42 for WebGraphMix at the same seed (held-out: 0.99, 2.39 and 2.83). It is the one control clearly below WebGraphMix (2.4 SDs of a difference), so host-level sampling alone does not explain WebGraphMix's scores.

W2 and Q1 (noise, more runs). With three seeds, the SD of a difference between two 1B runs is 0.87, not the 0.37 lower bound in our rebuttal. On the 23-task average WebGraphMix − Random is +0.51 ± 1.02, within noise; on a precision-weighted average it is ahead at all three seeds (+0.40; details in follow-up 1/2). It is also ahead on the held-out tasks at every seed (+0.54), though those tasks are near chance at 1B. The fixed recipe at half the token budget gives +0.79 with one seed, also within noise.

W1 (fragile gain). The task-level figures in our reply (19 of 23 tasks, +1.16 without the five noisiest tasks, +1.06 on 18 tasks) are from seed 124 only. At seeds 125 and 126, WebGraphMix is ahead on 11 and 12 of 23 tasks. On the unweighted average, we agree the gain is fragile, and the revised paper says so.

A run on a second data pool remains planned for the camera-ready.

Reviewer_rs5YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes WEBGRAPHMIX, which uses host-level centrality in the Common Crawl web graph to select pretraining documents. It studies central and peripheral data separately and in mixtures, using betweenness and Katz centrality, and combines these signals with DCLM-fasttext quality scores. Under the DCLM training setup, a balanced betweenness mixture improves the 23-task average from 39.8% to 41.4% at the 1B scale; combining centrality and quality reaches 43.8%, versus 42.3% for quality alone. Pure centrality selection does not yield comparable gains at 400M. The paper also analyzes link counts, host size, quality, and phrase recurrence to investigate what centrality captures.

Critical strengths
  1. The selection signal is simple and practically appealing. Using existing web connectivity complements the usual content-based approach without requiring a new classifier or proxy-model training for the structural scores.

  2. The 1B results provide encouraging evidence of complementarity. The balanced mixture outperforms both extremes and the middle band, while the reported centrality–quality combinations consistently outperform quality-only selection.

  3. Several controls and negative results are informative. The in-degree and eigenvector baselines, middle-band control, full task-level tables, and host/content analyses help interpret the gains. Reporting that pure centrality selection fails to help at 400M is also valuable and makes the scope of the findings clearer.

Critical weaknesses
  1. The robustness of the preferred recipe is not yet established. The 1B-best pure mixture is approximately one point worse than uniform sampling at 400M, and the best quality-combination recipe changes across scales. The main results select the best settings from multiple centralities, mixture ratios, and combination rules without a clearly separate selection/confirmation protocol. Evaluation bootstrapping and three seeds for the 400M uniform baseline are useful, but do not measure training variability for the main 1B comparisons. In particular, the 0.4-point difference from WebOrganizer+ should not be interpreted as established superiority.

  2. The proposed explanation is stronger than the evidence supports. The results suggest that structural sampling is useful, but do not establish that central hosts teach reusable abstractions while peripheral hosts supply complementary long-tail knowledge. Phrase recurrence can reflect boilerplate or duplication, as the authors acknowledge. Likewise, near-zero Spearman correlation with quality does not establish independence, especially given the U-shaped quality profile in Table 22. Topic, format, host concentration, and quality-matched controls would help distinguish the proposed mechanism from ordinary corpus rebalancing.

  3. Important sampling and score-combination details need clarification. Section 3.3 does not clearly specify the exact Top-K/Bottom-K cutoffs used in each experiment or whether percentiles are weighted by hosts, documents, or tokens. Equation (4) maps scores into a bounded range but does not put centrality and quality on comparable scales. The reported centrality values span many orders of magnitude, making any preprocessing before exponentiation important. These details affect reproducibility of the headline combined results, not just implementation convenience.

Questions for authors
  1. Were mixture ratios and combination rules selected using the same 23-task average reported in the main tables? Could you confirm the main 1B comparisons with additional training seeds and provide paired uncertainty estimates for the improvements over uniform and quality-only selection?

  2. How should a practitioner choose a recipe without a downstream benchmark sweep, given the reversal between 400M and 1B? A fixed-recipe transfer experiment on another corpus or training budget would help separate a broadly useful rule from configuration-specific gains.

  3. What are the exact percentile cutoffs for the pure centrality strata? Are they defined over hosts, documents, or token mass, and how are documents sampled within each stratum under the fixed token budget?

  4. What values enter Equation (4): raw centralities, log-transformed values, or rescaled scores? Please report the resulting score ranges and clarify how the two quality-aware selections remain disjoint when both favor high-quality documents.

  5. Does the mixture advantage persist after matching topic/format and quality distributions or controlling host concentration? For the phrase-recurrence analysis, would excluding matches from the same host and common boilerplate preserve the central–peripheral difference?

  6. Since the main graph is undirected, how would a matched mixture based on total undirected degree or Common Crawl's harmonic centrality compare with betweenness? Please also clarify the PageRank comparison: ordinary adjacency eigenvector centrality on a symmetrized graph is not equivalent to PageRank.

5 comments
AuthorsOpenReview ↗

Rebuttal 1/3

Thank you for the careful review. Results shared across reviewers (training seeds, held-out tasks) are summarized in our general response; this reply gives the details.

W1 and Q1: recipe selection, training seeds, paired uncertainty

Reviewer: "Were mixture ratios and combination rules selected using the same 23-task average reported in the main tables? Could you confirm the main 1B comparisons with additional training seeds and provide paired uncertainty estimates for the improvements over uniform and quality-only selection?"

Response: Yes. Mixture ratios and combination rules were chosen on the same 23-task average reported in the main tables. We addressed this in two ways.

Held-out tasks. We evaluated existing 1B checkpoints on the 26 DCLM tasks outside our suite, excluding DCLM's four chain-of-thought tasks: MMLU zero-shot, TriviaQA, SIQA, MathQA, LogiQA, PubMedQA, GPQA (main and diamond), BBQ, AGIEval LSAT-RC, LSAT-LR and SAT-En, Winogender (two variants), simple arithmetic (two variants), enterprise PII classification, and nine BIG-bench tasks (misconceptions, novel concepts, strange stories, strategy QA, conlang translation, conceptual combinations, elementary math QA, logical deduction, understanding fables). None was used for any decision. Scores are DCLM's centered accuracy, (accuracy − chance) / (1 − chance), averaged over the 26 tasks, so 0 is chance.

Run23-task average (main suite)Held-out, centeredHeld-out vs Random
Random39.842.390
Quality42.300.75−1.64
WebGraphMix+43.841.57−0.82
WebGraphMix (betweenness 50/50)41.422.83+0.44
Betweenness 75/25 (peripheral part out-of-graph)41.032.98+0.59
Katz 50/5040.753.90+1.51
Katz 75/2541.043.60+1.21
In-degree 50/5040.212.93+0.54
Middle-K40.311.76−0.62
Betweenness Bottom-K (corrected run)40.042.52+0.13
Katz Bottom-K (corrected run)41.002.12−0.27

The two Bottom-K rows are not the runs in the paper. The paper's 1B Bottom-K selections were built incorrectly (general response, error 2), so we report the correctly built runs here and have updated Table 7. Betweenness 75/25 is also affected: its peripheral quarter comes from out-of-graph hosts, and the table marks it.

Two caveats. These tasks are close to chance at 1B (centered scores of 0.75 to 3.90), so they separate methods less well than the main suite; DCLM's CORE suite was chosen for tasks that show signal at small scale. And MMLU zero-shot uses the same questions as the few-shot MMLU in our main suite. Without it (25 tasks), the differences from Random are: WebGraphMix +0.50, Katz 50/50 +1.51, Katz 75/25 +1.23, in-degree +0.65, Betweenness 75/25 +0.70, Middle-K −0.61, WebGraphMix+ −0.91, Quality −1.75. No conclusion changes.

On these tasks the four correctly built mixtures stay above Random (+0.44 to +1.51), while Middle-K and both quality-filtered runs fall below it, WebGraphMix+ by 0.82 and Quality by 1.64. We do not have an explanation for the quality-filtered runs; the seed replicates will show whether these gaps exceed noise on these tasks.

Selecting on held-out tasks. Choosing among the eight pure-centrality configurations in the table by held-out score alone selects Katz 50/50. On the main suite it scores 40.75 against 39.84 for Random (+0.91). As parameterized in the paper, Katz is close to total degree (reply to Q6), so the configuration that survives held-out selection mixes the most- and least-connected hosts. We take this as evidence for mixing the two ends of the link structure, not for betweenness in particular.

Training seeds. We are retraining Random, Quality, WebGraphMix and WebGraphMix+ with two new seeds each; the seed changes initialization and data order, not the selected data. We will post per-seed scores and paired differences (with SD) for WebGraphMix − Random and WebGraphMix+ − Quality before the discussion closes. The current noise estimate is in the general response. The WebGraphMix+ seeds retrain the selection described in the general response (error 1), so the WebGraphMix+ − Quality difference measures noise on that run, not a centrality effect.

WebOrganizer+. We agree that 43.8 vs 43.4 is within noise, and the revised paper describes the two as comparable.

AuthorsOpenReview ↗

Rebuttal 2/3

Q2: choosing a recipe without a benchmark sweep

Reviewer: "How should a practitioner choose a recipe without a downstream benchmark sweep, given the reversal between 400M and 1B? A fixed-recipe transfer experiment on another corpus or training budget would help separate a broadly useful rule from configuration-specific gains."

Response: Without a sweep, we would recommend a 50/50 mixture of the most and least central hosts, for models of 1B parameters and above. Which centrality to use is not settled at our noise level: betweenness is best on the main suite (41.42), and a total-degree score is best on the held-out tasks (reply to W1 and Q1). We make no recommendation below 1B, given the 400M result.

Fixed-recipe transfer. We fixed the recipe in advance (betweenness, 50/50) and are training the 1B model on 14.4B tokens, half the paper's budget, alongside Random at the same budget, with nothing tuned. Results will be posted before the discussion closes. This tests whether the recipe survives a change of token budget at a fixed model size. It cannot separate model size from token count as the cause of the 400M reversal, because 14.4B tokens is still more than the 8.2B used at 400M.

Another corpus. We agree a second pool is the stronger test, and we could not run it within the discussion period. Our pool is DCLM's 1b-1x Common Crawl pool after RefinedWeb heuristic filtering and deduplication, with no model-based quality filter (WebOrganizer's Corpus-200B). We will add a fixed-recipe run on a second data pool to the camera-ready.

W3 and Q3: exact cutoffs and weighting

Reviewer: "What are the exact percentile cutoffs for the pure centrality strata? Are they defined over hosts, documents, or token mass, and how are documents sampled within each stratum under the fixed token budget?"

Response: Cutoffs are defined over token mass; §3.3 was imprecise on this and describes host percentiles. Every document inherits its host's raw centrality. Documents are sorted by that score and taken in order until the stratum's token budget is full, so whole hosts enter together and nothing is subsampled within a stratum. Ties are broken by document order in the pool. 17.4% of hosts have betweenness exactly 0, and the peripheral half of WebGraphMix includes all of them.

Only documents from hosts in the graph are eligible: 180.4B tokens, after excluding the 12.6B tokens (6.5%) from hosts outside the graph. Random samples from the same eligible pool. The Quality baseline does not: it ranks the full pool, so 7.0% of its tokens come from out-of-graph hosts. This is a difference between Quality and every centrality run, and the revised paper states it (§4.1). Each selection holds 32B tokens, and each 1B run trains on 28.8B tokens drawn from it without replacement.

SelectionTokensShare of eligible tokensShare of hostsBetweenness cutoff
WebGraphMix, central half16Btop 8.9%top 0.029% (about 4,000 hosts)≥ 2.34e-5
WebGraphMix, peripheral half16Bbottom 8.9%bottom 28.9%≤ 1.5e-12
Pure Top-K32Btop 17.7%top 0.15%≥ 5.96e-6
Pure Bottom-K (corrected runs)30Bbottom 16.6%bottom 47.4%≤ 1.2e-10
Middle-K32B17.7%, centered on the median hostpercentiles 29.1 to 70.9n/a

This table is now in Appendix A of the revised paper, and §3.3 describes the selection as implemented. The paper's 1B Bottom-K runs did not use the intended selection (general response, error 2): because the sort direction was not inverted, they consist of the 12.6B tokens from hosts outside the graph plus 19.4B tokens from the most central hosts. The corrected runs in the table come from earlier 30B-token selections. Two mixtures inherit the error in their peripheral part: all 8B peripheral tokens of Betweenness 75/25, and 12.6B of the 24B peripheral tokens of Katz 25/75, come from out-of-graph hosts. §3.3 describes Middle-K as host percentiles 25 to 75; the implemented band is 29.1 to 70.9, and the revised paper corrects it.

AuthorsOpenReview ↗

Rebuttal 3/3

Q4: values entering Eq. (4), score ranges, disjointness

Response: Raw centralities enter Eq. (4), with no log transform or rescaling, and so do raw DCLM-fasttext scores (about −1e-5 to 1.0). After exp(s − max s), normalized betweenness lies between 0.820 and 1, normalized Katz between 0.99987 and 1, and normalized quality between 0.368 and 1.

Because the max is a constant, the multiplicative score ranks documents by raw c + q, and the divisive score by raw c − q. Raw betweenness is below 2.34e-5 for 99.97% of hosts and raw Katz spans 2.68e-4 to 4.02e-4, while quality scores span the full unit interval. At the selection cutoff (a quality score of about 0.04), both rankings therefore select what a quality filter alone would: 99.3% of the multiplicative half and 99.7% of the divisive half are documents that the top 16B tokens by quality over in-graph hosts would contain. Centrality changes the selection only for documents from a few dozen hub hosts.

Disjointness. Your question identified an error in the paper. The two selections are not disjoint. Nothing in the pipeline removes Top-K documents from the Bottom-K portion, and because both rankings follow quality, the two halves of WebGraphMix+ share 99.0% of their tokens. Its training set holds 16.15B unique tokens, 99.9% of them inside the Quality selection. The statement in §3.4 was wrong and the revised paper corrects it; the consequences for WebGraphMix+ are in the general response (error 1). The pure-centrality mixtures are unaffected: the two halves of WebGraphMix share no documents.

W2 and Q5: mechanism, matched controls, phrase recurrence

Response: Two training controls are running, and we checked quality overlap and phrase recurrence.

Topic and format. We reweighted Random's selection so that its joint WebOrganizer topic × format distribution (576 cells) matches WebGraphMix's, at the same budget. Results will be posted before the discussion closes. For reference, WebGraphMix shifts the topic and format mix about as much as the in-degree mixture, which scores 1.2 points lower: total variation from Random is 0.056 vs 0.067 for topic, and 0.108 vs 0.109 for format.

Host concentration. We are training a 50/50 mixture of the largest and smallest hosts by token count, with the same mechanics as WebGraphMix (results before the discussion closes). Betweenness is only weakly related to host size (Spearman 0.25 with tokens per host and 0.31 with documents per host; Table 21).

Quality. 20.2% of WebGraphMix's tokens fall inside the Quality baseline's selection, against 19.8% for the in-degree mixture and 16.5% for Random. Overlap with high-quality documents therefore does not explain the 1.2-point difference between WebGraphMix and the in-degree mixture. We did not have time for a quality-matched training control. We agree that a near-zero Spearman correlation does not imply independence, given the U-shaped profile in Table 22, and the revised paper says "weakly correlated".

Phrase recurrence. With the controls you suggest, the difference does not hold: once central and peripheral hosts are matched on documents per host, the gap reverses, and recurrences in central text come mostly from the same sites. The revised paper presents phrase recurrence as a description of the data, not as evidence for the mechanism. No training result depends on this analysis.

Q6: total degree, harmonic centrality, PageRank wording

Response: Harmonic centrality. We are training a 50/50 mixture ranked by Common Crawl's harmonic centrality, with the same mechanics as WebGraphMix (results before the discussion closes). Harmonic centrality correlates only moderately with betweenness (Spearman 0.449 over 13.9M hosts; Table 21). Common Crawl also prioritizes its crawl by harmonic centrality, so our pool is already denser at high-harmonic hosts than the open web is (§3.1).

Total degree. With the attenuation factor we used (η = 1/dmax = 9.8e-8), the paper's Katz scores are a monotone function of total undirected degree (Spearman 0.9999 with in-links plus out-links). The paper's Katz 50/50 mixture is therefore the total-degree mixture you describe. It scores 40.75, against 41.42 for betweenness and 40.21 for the in-degree mixture. On the held-out tasks the order reverses (over Random: total degree +1.51, in-degree +0.54, betweenness +0.44), so we no longer claim that betweenness beats degree-based mixtures. The revised paper labels these runs as a total-degree measure.

PageRank. We agree, and have removed the description of eigenvector centrality as the undirected analogue of PageRank.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response 1/2

We thank the reviewers for detailed and constructive reviews. All four ask the same two questions: do the 1B gains exceed training noise, and does choosing configurations on the evaluation suite inflate them? We agree both needed checking. This response summarizes what we found, which results are still training, and how the paper has changed. Per-reviewer replies follow.

Corrections to the submitted paper

While answering your questions on Eq. (4) and on the exact cutoffs, we found two errors in the submitted paper. We report them here in full, because they change what the paper can claim.

1. The two halves of WebGraphMix+ are not disjoint. §3.4 states that the two quality-aware selections are disjoint in every configuration. This is wrong. Raw scores enter Eq. (4), and after exp(s − max s) normalized betweenness lies between 0.820 and 1 and normalized Katz between 0.99987 and 1, while normalized quality spans 0.368 to 1. Near the selection cutoff, quality differences are far larger than centrality differences for all but a few dozen hub hosts, so both rankings select almost the same documents as a quality filter alone: 99.3% of the multiplicative half and 99.7% of the divisive half are documents that the top 16B tokens by quality over in-graph hosts would contain. In WebGraphMix+ the two halves share 99.0% of their tokens: the 32B-token training set contains 16.15B unique tokens, 99.9% of which lie inside the Quality baseline's selection, and because both halves were written to the training set, each of those documents appears in it twice. The other multiplicative/divisive and additive/subtractive mixtures overlap by 96 to 100%. WebGraphMix+ is therefore best described as roughly the top half of the quality ranking over in-graph hosts, repeated. Its gain over Quality cannot be attributed to centrality, and we withdraw the claim that the two signals are complementary.

2. The 1B pure Bottom-K selections were built incorrectly. The sort direction was not inverted for these runs, so they contain out-of-graph and highly central hosts rather than the periphery. Correctly built runs score 40.04 (betweenness) and 41.00 (Katz), against 40.0 and 40.5 in the paper; we use them throughout this response, and the revised paper updates Tables 2, 5 and 7, Fig. 2(a) and §5.1. Two mixtures inherit the error in their peripheral part (Betweenness 75/25 and Katz 25/75, Table 8) and are marked wherever they appear. With the corrected runs, mixing beats both pure ends for betweenness but not for Katz. Details are in our reply to Reviewer rs5Y (W3 and Q3).

What is unaffected. WebGraphMix (betweenness 50/50) is correct: its halves are disjoint, contain no out-of-graph documents, and come from the intended ends of the ranking. The same holds for every Top-K run, Betweenness 25/75, Katz 50/50, Katz 75/25, Middle-K, the in-degree mixture and Random. All results below use only these runs unless marked. We audited the 400M selections in the same way: every pure-centrality 400M selection draws from the intended end of the ranking and contains no out-of-graph documents, and the two parts of every 400M mixture are disjoint.

What the new evidence shows

  • Centrality mixing beats uniform sampling under every check so far. WebGraphMix is ahead of Random on 19 of 23 tasks. On the 18 tasks where Random scores above DCLM's reference baseline, it gains +1.06, and the four correctly built betweenness and Katz mixtures gain between +0.95 and +1.09. On 26 held-out DCLM tasks never used for any decision, the four correctly built mixtures with held-out scores (betweenness 50/50, Katz 50/50, Katz 75/25, in-degree 50/50) stay above Random (+0.44 to +1.51). Selecting the configuration on those held-out tasks alone picks Katz 50/50, which scores 40.75 on the main suite against 39.84 for Random.
  • WebGraphMix+ is not evidence for combining centrality with quality. See error 1 above. In addition, BoolQ and CommonsenseQA account for 63% of its 1.5-point gain over Quality, and on the 18 tasks above DCLM's reference baseline the gain is +0.45. The revised paper no longer presents WebGraphMix+ as a centrality result and reports it as what it is.
  • WebGraphMix+ and WebOrganizer+ are comparable. The 0.4-point difference is within noise, and the revised paper says so.
  • The mechanism is a hypothesis. The revised paper presents the account of central and peripheral hosts as a hypothesis the results are consistent with, and withdraws phrase recurrence as evidence for it. With the corrected Bottom-K runs, mixing beats both pure ends for betweenness (41.42 against 39.45 for Top-K and 40.04 for Bottom-K) but not for Katz, where Bottom-K alone (41.00) matches the mixtures (40.75 and 41.04). Katz Bottom-K is also within noise of WebGraphMix (41.42). The evidence therefore supports including peripheral hosts; that mixing them with central hosts adds to this is supported only for betweenness.
AuthorsOpenReview ↗

General Response 2/2

Results available now

  • Held-out evaluation on 26 DCLM tasks, and configuration selection on those tasks (reply to rs5Y, W1 and Q1).
  • Exact cutoffs, weighting and tie handling (reply to rs5Y, W3 and Q3).
  • Eq. (4) score ranges and what they imply for the combined rankings (reply to x5Rk, W2 and Q1).
  • Topic and format composition of each selection, quality overlap, and Katz as a total-degree measure (replies to rs5Y and fRap).
  • Per-task breakdowns of the gains (replies to vUrY and fRap).

Results we will post later

  • Two new training seeds (initialization and data order) for Random, Quality, WebGraphMix and WebGraphMix+ at 1B, with paired differences per seed. The WebGraphMix+ seeds retrain the selection described in error 1, so they measure training noise for that run, not a centrality effect.
  • A 50/50 mixture of two random sets of whole hosts at the same budget (vUrY).
  • Random reweighted to WebGraphMix's joint topic × format distribution (fRap, rs5Y).
  • A 50/50 mixture of the largest and smallest hosts by token count (rs5Y).
  • A 50/50 mixture ranked by Common Crawl's harmonic centrality (rs5Y).
  • The fixed recipe (betweenness, 50/50) at half the token budget, against Random at that budget (rs5Y, vUrY, fRap).

Until the seed runs finish, our best noise estimate comes from existing runs: 19 of our 1B runs fall into 5 groups whose training sets are at least 99% identical. Their pooled per-run SD is 0.26, so a difference between two runs has an SD of about 0.37. This is a lower bound, because those runs share an initialization seed.

Changes in the revised paper

  • Add the held-out evaluation (new appendix). Three-seed means and paired differences for the headline runs will be added when those runs finish.
  • §3.3: define cutoffs over token mass, as implemented, and add a table of exact cutoffs (Appendix A).
  • §3.4: state that raw scores enter Eq. (4), give the score ranges, explain that the product ranks by c + q and the ratio by c − q, and correct the disjointness statement: the two portions overlap by 96 to 100%.
  • Tables 2, 5 and 7, Fig. 2(a) and §5.1: replace the Bottom-K results with the correctly built runs and revise the discussion of pure Bottom-K sampling. Table 8, Fig. 2(a) and §5.2: mark Betweenness 75/25 and Katz 25/75 as affected.
  • Abstract, §4.2, §5.4, the conclusion, Tables 1, 5 and 13–16 and Fig. 2(b): remove WebGraphMix+ as evidence of complementarity, and note that the divisive Bottom-K betweenness and Katz rows of Tables 15–16 were trained on identical data.
  • §3.3: correct the statement that zero-betweenness hosts account for less than 25% of the token budget. They hold 8.4B tokens, which is 26% of a 32B-token Bottom-K selection and 52% of WebGraphMix's peripheral half.
  • §3.4: correct the statement that the classifier is never applied as a threshold. The Top-K combined scores exclude documents with a DCLM-fasttext score at or below −3e-6, about a quarter of the pool.
  • §4.2 and Table 1: describe WebGraphMix+ and WebOrganizer+ as comparable, and remove "more transferable" from the Table 1 caption.
  • §1, §3.3, §5 and §6: present the central and peripheral account as a hypothesis, replace "nearly independent" with "weakly correlated", drop the claim that the two signals' gains compound, and move phrase recurrence to an appendix as a description of the data.
  • §1 and §3.2: remove the description of eigenvector centrality as the undirected analogue of PageRank, and label the Katz runs as a total-degree measure.
  • Limitations: state that configurations were chosen on the evaluation suite.

We have uploaded the revised PDF, with changes marked in blue. It contains all of the changes above; the three-seed results and the training controls will be added when those runs finish. Section, table and figure numbers in this response refer to the submitted version.

AuthorsOpenReview ↗

Follow-up 1/2: training seeds and paired differences

The runs promised in our general response have finished, except the third seed of WebGraphMix+ (81% trained).

Summary. WebGraphMix is ahead of Random at all three seeds on a lower-noise, precision-weighted average (+0.40; the sign holds at every seed under every weighting we tried) and on the 26 held-out tasks (+0.54). With the recipe fixed in advance, it is also ahead at half the token budget (one seed). On the paper's equal-weight 23-task average the gap (+0.51) is within seed noise. Random whole hosts falls clearly below WebGraphMix, so host-level sampling alone does not explain it; the other controls need more seeds to separate. Quality filtering gives the largest and clearest gain.

All results are 1B models evaluated with the paper's pipeline. The seed changes initialization and data order; the selected data is unchanged. The main suite is the paper's 23-task average (accuracy × 100). Held-out is the average over the 26 held-out DCLM tasks in centered accuracy × 100, where 0 is chance.

Per-seed scores, 23-task average

RunSeed 124 (paper)Seed 125Seed 126Mean ± SD
Random39.8441.0540.0240.30 ± 0.65
Quality42.3043.5243.2743.03 ± 0.64
WebGraphMix41.4240.6040.4240.81 ± 0.53
WebGraphMix+43.8443.73training43.79 (2 seeds)

Per-seed scores, held-out (centered)

RunSeed 124Seed 125Seed 126Mean ± SD
Random2.391.412.762.19 ± 0.70
Quality0.752.114.232.36 ± 1.75
WebGraphMix2.832.522.832.73 ± 0.18
WebGraphMix+1.573.20training2.39 (2 seeds)

Paired differences (same seed)

ComparisonSeed 124Seed 125Seed 126Mean ± SD
WebGraphMix − Random, 23-task+1.58−0.45+0.40+0.51 ± 1.02
WebGraphMix − Random, held-out+0.44+1.11+0.07+0.54 ± 0.53
WebGraphMix+ − Quality, 23-task+1.54+0.21training+0.88 (2 seeds)
WebGraphMix+ − Quality, held-out+0.82+1.09training+0.96 (2 seeds)

Seed noise. On the 23-task average a single run has a seed SD of 0.61, so a difference between two single-seed runs has an SD of 0.87. This replaces the 0.37 in our general response, which was too low by more than a factor of two. On the paper's metric, WebGraphMix and Random are within seed noise of each other over three seeds.

A lower-noise comparison. The 23-task average weights every task equally, so its seed noise comes mostly from five tasks with seed SDs of 2.9 to 5.6 points (CommonsenseQA, Dyck, BoolQ, repeat-copy-logic, CS-algorithms); the median task's seed SD is about 1.2. With three seeds this average cannot detect an effect of +0.5: about 26 seed pairs would be needed for 80% power. We therefore also computed a precision-weighted average. Each task is weighted by the inverse of the larger of its seed variance and its sampling variance. The six tasks on which both Random's and Quality's three-seed means are within two sampling SEs of chance (MMLU, BoolQ, CommonsenseQA, LSAT-AR, language identification, repeat-copy-logic) get zero weight. Weights are estimated from the Random and Quality seeds only, never from WebGraphMix. This lowers the seed SD of a single run from 0.61 to about 0.11 (WebGraphMix: 0.13).

Precision-weightedSeed 124Seed 125Seed 126MeanPaired t-test (df 2)
WebGraphMix − Random+0.50+0.26+0.45+0.40p = 0.03
Quality − Random+3.51+3.29+3.45+3.42p < 0.001

We chose this weighting after seeing the seed results, and the size of the gap depends on which near-chance tasks are excluded (+0.25 to +0.43 across the variants we tried; the revised paper reports each). Under every variant, WebGraphMix is ahead of Random at all three seeds.

Held-out tasks. WebGraphMix is ahead of Random at every seed (+0.54 ± 0.53), but only three or four of the 26 tasks are clearly above chance at 1B, and the suite does not separate Quality from Random (+0.18 ± 1.62). Quality's held-out score ranges from 0.75 to 4.23 across seeds, so the Quality and WebGraphMix+ gaps we reported earlier on these tasks were within noise. The held-out result points the same way as the weighted average, but because these tasks are near chance at 1B we give it little weight.

WebGraphMix+ vs Quality. The gain on the 23-task average falls from +1.54 at seed 124 to +0.21 at seed 125. As stated in our general response, WebGraphMix+ is in effect a quality selection, and we make no centrality claim from it.

AuthorsOpenReview ↗

Follow-up 2/2: training controls, half budget, and revised claims

Corrections to earlier replies. The per-task figures in our replies to vUrY (W1), fRap (W1) and x5Rk (W1) are from seed 124 only. At seeds 125 and 126, WebGraphMix is ahead of Random on 11 and 12 of 23 tasks (19 at seed 124). The same applies to the split-half and held-out selection checks. The statement in our general response that centrality mixing beats uniform sampling under every check does not hold on the 23-task average: WebGraphMix is 0.45 below Random at seed 125.

Single-seed comparisons in the paper. Every comparison among centrality variants (Top-K, Bottom-K and mixtures; betweenness, Katz and in-degree; mixing ratios) rests on one seed, and most gaps are below two SDs of a difference (0.87), including the betweenness result cited in our general response. We no longer draw conclusions from their ordering.

Training controls. Each control uses the paper's 32B-token budget and seed 124, one seed per control.

Run23-taskHeld-out
Random whole hosts, 50/50 (vUrY Q2)39.310.99
Random reweighted to WebGraphMix's topic × format mix (fRap Q4, rs5Y Q5)40.622.63
Largest and smallest hosts by tokens, 50/50 (rs5Y Q5)40.273.41
Harmonic centrality, 50/50 (rs5Y Q6)40.312.05
In-degree, 50/50 (paper run)40.212.93
Random, seed 124 / three-seed mean39.84 / 40.302.39 / 2.19
WebGraphMix, seed 124 / three-seed mean41.42 / 40.812.83 / 2.73

The four structured controls (topic × format, host size, harmonic, in-degree) score 40.21 to 40.62. All are 0.80 to 1.21 below WebGraphMix at seed 124 (0.9 to 1.4 SDs of a difference), and all lie between or next to the three-seed means of Random and WebGraphMix. One seed per control is not enough to separate them from either, so we do not yet claim that betweenness adds to what topic and format composition or host size would give.

Random whole hosts is the one control clearly below WebGraphMix: 2.11 below at seed 124 (2.4 SDs of a difference) and 1.50 below its three-seed mean, and the lowest run in this table on both metrics. Host-level sampling alone therefore does not explain WebGraphMix's scores (vUrY Q2).

Half the token budget. The recipe was fixed in advance (betweenness, 50/50) and trained alongside Random on 14.4B tokens at seed 124, with nothing tuned.

Run23-taskHeld-out
Random, 14.4B tokens37.510.99
WebGraphMix, 14.4B tokens38.302.92
Difference+0.79+1.93

The difference has the same sign as at the full budget, but with one seed it is within noise (0.9 and 1.2 SDs of a difference). We do not present it as evidence that the recipe transfers.

What the paper will claim. Quality filtering gives a clear gain at 1B. WebGraphMix, which uses no quality classifier, is comparable to uniform sampling on the 23-task average and ahead at every seed on the precision-weighted average and on the held-out tasks, but the effect is small and our single-seed controls do not yet isolate a contribution from centrality. The revision will:

  • Abstract, §1, §4.2 and conclusion: replace the single-seed +1.6 with three-seed means and paired differences, and describe WebGraphMix as comparable to uniform sampling on the 23-task average, with a small, consistent gain on the precision-weighted average that remains to be confirmed.
  • Table 1: three-seed means ± SD for Random, Quality, WebGraphMix and WebGraphMix+ (third seed added when it finishes), on both averages. A new appendix gives the weights, how they were set, and the sensitivity above.
  • §5: mark all centrality-variant comparisons as single-seed, give the 0.87 noise estimate beside them, and remove conclusions drawn from their ordering, including the Jeopardy discussion.
  • New appendix: the controls and half-budget results above.
  • Limitations: the evidence for a centrality effect at 1B is not conclusive, and separating it from topic, format and host-size composition needs multi-seed controls.

We thank the reviewers for pressing on training variance. These runs were prompted by your questions, and they sharpen what the paper can claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.