- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes WebGraphMix, a lightweight pretraining data selection framework based on host-level web graph centrality (mainly betweenness). At 1B scale, a 50/50 mix of the highest- and lowest-betweenness hosts beats random sampling (41.4% vs. 39.8%), and adding a quality score reaches 43.8% vs. 42.3% for quality-only.
Critical strengths
- Lightweight framework, efficient to run: one-time centrality computation (<9 hours on 4 H100s), no model training or labeled data.
- Better than random and combinable with quality score in terms of performance. At 1B, every centrality + quality run beats quality-only.
- A new signal nearly orthogonal to document quality (Spearman ρ = 0.018).
- Reasonable controls (Middle-K, in-degree, eigenvector) rule out simple host popularity or link counting.
Critical weaknesses
- Limited performance gain. WebGraphMix alone (41.4%) is below the simple quality-filter baseline (42.3%), and WebGraphMix+ beats it by only 1.5 points, around 89% of which comes from four noisy tasks.
- Not robust across scale. At 400M, the best 1B mixture is 1.0 point below random, and the best configuration changes between scales.
- Weak experimental setting. All configurations are selected on the test benchmark with one seed per 1B run, so several claims (e.g., 43.8 vs. 43.4 against WebOrganizer+) are within noise.
- Weak support for the hypothesis. Jeopardy, the most long-tail knowledge task, drops from 15.3 to 13.0, contradicting the claim that peripheral hosts provide long-tail knowledge.
Questions for authors
- How you explain the the drop on long-tail knowledge drop ?
- How you can predict the performance at scale ? Modern LLMs requires trilliions of tokens, 1b mixture looks limited, and can not demonstrate the approach.
- Is it possible to run multiple runs to reduce the noise ?
- In weborganizer, domain & topic distribution match is an approach to improve the performance on benchmark, how your sampling approach change the domain / topic ? is this change lead to improvement instead of using your mix approach ?
3 comments
Response to Reviewer fRap: seed and control results
Thank you. The seed runs and controls are now posted in our follow-up to all reviewers.
W4 and Q1 (Jeopardy). Across three seeds Random scores 15.3, 14.8 and 13.4 and WebGraphMix 13.0, 14.5 and 16.2 (means 14.5 and 14.6). The drop at seed 124 was seed noise; Jeopardy neither contradicts nor supports the long-tail account, and the revised paper removes that discussion.
W3 and Q3 (seeds). WebGraphMix − Random is +0.51 ± 1.02 on the 23-task average, within noise, and ahead at all three seeds on a precision-weighted average (+0.40; details in follow-up 1/2). WebGraphMix+ − Quality is +1.54 and +0.21 at two seeds.
Q4 (topic and domain). Random reweighted to WebGraphMix's topic × format mix scores 40.62, against 39.84 for Random and 41.42 for WebGraphMix at seed 124 (WebGraphMix's three-seed mean is 40.81). One seed is not enough to separate composition from centrality, and the revised paper flags this as needing multi-seed controls.
W2 and Q2 (scale). At half the token budget, WebGraphMix − Random is +0.79 with one seed, within noise.
W1. The task-level figures in our reply (19 of 23 tasks, +1.06 on 18 tasks) are from seed 124 only; at seeds 125 and 126 the count is 11 and 12 of 23.
W1 - Resolved, the increase is consistent.
W2 - Partially resolved, somehow shows approach is scaling, but require more data point.
W3 - Resolved, gain is small and consistent.
W4 - Partially resolved, but need more seed.
I totally understand the compute concern, and I'd move to accept based on current additional results.
Rebuttal
Thank you for the review. Several of your points changed how we present WebGraphMix+.
W1: limited gain, concentrated in noisy tasks
Reviewer: "WebGraphMix alone (41.4%) is below the simple quality-filter baseline (42.3%), and WebGraphMix+ beats it by only 1.5 points, around 89% of which comes from four noisy tasks."
Response: We agree on both points. WebGraphMix alone is below the quality filter (41.42 vs 42.30); we present centrality as a selection signal that needs no labels or classifier, not as a replacement for quality filtering. Your 89% figure for WebGraphMix+ is correct: BoolQ, CommonsenseQA, LSAT-AR and Dyck supply it, and on the 18 tasks where Random is above DCLM's reference baseline the gain over Quality is +0.45. The underlying problem is larger than noisy tasks: we found that WebGraphMix+ selects almost the same documents as the quality filter (general response, error 1), and we withdraw it as evidence that centrality adds to quality. We will post the three-seed comparison with Quality before the discussion closes.
The gain of WebGraphMix over Random is broader: it is ahead on 19 of 23 tasks, gains +1.06 on the 18 tasks above DCLM's reference baseline, and stays positive on held-out tasks (general response).
W2 and Q2: scale
Reviewer: "How you can predict the performance at scale? Modern LLMs requires trillions of tokens, 1b mixture looks limited, and can not demonstrate the approach."
Response: We cannot predict performance at trillion-token scale and make no such claim; the 400M result already shows that the benefit depends on scale. A 1b-1x run takes about 240 H100-hours, and 7b-1x is beyond our budget. Within that budget, we are testing whether the fixed recipe (betweenness, 50/50) holds at half the token budget (14.4B tokens) against Random at the same budget, with results before the discussion closes. The limitations section of the revised paper states the scale limits plainly.
W3 and Q3: single seed, selection on the test benchmark
Reviewer: "All configurations are selected on the test benchmark with one seed per 1B run, so several claims (e.g., 43.8 vs. 43.4 against WebOrganizer+) are within noise." "Is it possible to run multiple runs to reduce the noise?"
Response: Yes. We are retraining Random, Quality, WebGraphMix and WebGraphMix+ with two new seeds each, and will post per-seed scores and paired differences before the discussion closes. To account for selection on the test benchmark, we evaluated the 1B checkpoints on 26 DCLM tasks never used for any decision. Choosing the configuration on those tasks alone selects Katz 50/50, which scores 40.75 on the main suite against 39.84 for Random (details in our reply to Reviewer rs5Y, W1 and Q1). We agree the 0.4-point gap to WebOrganizer+ is too small to call a win, and the revised paper describes the two as comparable.
W4 and Q1: Jeopardy
Response: We agree that Jeopardy does not support the long-tail account. Across our 1B runs, Jeopardy tracks quality filtering: the runs whose training data is essentially the top of the quality ranking score 24.5 to 28.6, while runs with no quality signal score 10.1 to 17.5. Among the latter, the peripheral-only selection (corrected betweenness Bottom-K, 13.1) is also below Random (15.3), so peripheral hosts do not add the trivia Jeopardy tests. The seed runs will show how much of the 2.3-point drop is noise, and we will report that with them.
WebGraphMix's Knowledge gain (34.2 to 35.4) comes from other tasks: MMLU, QA-Wikidata and both ARC sets each rise by 0.7 to 1.3 points, and LSAT-AR, with 230 questions, rises by 5.7. The revised paper presents the long-tail account as a hypothesis consistent with the aggregate results, not as an established mechanism, and discusses the Jeopardy result (§5.1).
Q4: effect on domain and topic
Reviewer: "In WebOrganizer, domain & topic distribution match is an approach to improve the performance on benchmark, how your sampling approach change the domain / topic? Is this change lead to improvement instead of using your mix approach?"
Response: The two halves differ sharply in composition. The central half is weighted toward news articles (9.6% of Random's tokens vs 19.6% of the central half) and politics (8.2% vs 13.2%), while the peripheral half is weighted toward personal blogs (21.5% vs 48.7%). Mixing them largely cancels out: the largest shifts in WebGraphMix relative to Random are personal blogs (+5.6 points) and product pages (−4.1 points).
To test whether this shift causes the gain, we are training a control that reweights Random's selection to WebGraphMix's exact joint topic × format distribution (576 WebOrganizer cells), at the same budget. We will post its score before the discussion closes, against 39.84 for Random and 41.42 for WebGraphMix.