- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a controlled empirical study of how natural long documents should be selected for continued pretraining aimed at context extension. Using Qwen2.5-3B extended to 128K with matched 6B-token budgets, the authors compare scoring-free constructions (repacking, random sampling), generic-quality selectors (perplexity, gzip), an LLM-judge pipeline, and likelihood-based long-dependency selectors (LongFilter, Prospector, LongPPL key-token density). HELMET is used for all selection decisions and RULER is held out. The main findings are: random sampling of natural long documents beats repacking at long evaluation lengths; generic-quality and LLM-judge pipelines do not exceed random sampling; LongPPL key-token density gives the best downstream scores; and a cheaper variant (truncLongPPL, scoring only front-64K target positions) retains most of the benefit with a 2.28× scoring speedup. A fixed 66/34 short/long mixture experiment and post-SFT evaluation on LongBench v2 and MRCR show the benefit persists in a more realistic recipe.
Critical strengths
S1: A clean, controlled, apples-to-apples comparison of complete data-selection pipelines under matched budgets.
S2: Sound evaluation protocol — HELMET is used for all tuning decisions while RULER remains genuinely held out, and seven short-context tasks measure retention.
S3: Informative capability-level analysis: gains concentrate on retrieval-oriented tasks at 64K–128K, while aggregation and variable tracking show no consistent benefit; the paper is refreshingly honest about where selection does not help.
S4: The front-64K scoring variant offers a useful cost–quality operating point (2.28× measured speedup with near-equal downstream scores), which is practically relevant since full-128K scoring costs 10^4–10^5 H20 GPU-hours.
Critical weaknesses
W1: The paper is mainly an empirical comparison and does not propose a new data-selection method. All evaluated selectors are taken from prior work, and the contribution is limited to benchmarking them under a shared protocol.
W2: truncLongPPL, the paper's main methodological contribution, is a straightforward truncation of the existing LongPPL heuristic (restrict scoring to the first 64K target positions). While it improves scoring efficiency, the modification is simple and has limited novelty, and the reported front-to-full selected-set agreement is imperfect (76–92% containment), so the two pipelines are not equivalent even at the document level.
W3: The conclusions are drawn exclusively on natural long documents. Given that recent long-context recipes increasingly rely on synthetic long documents (e.g., synthetic agent/repository data in GLM-5, retrieval-based constructions like NExtLong), the paper neither analyzes nor discusses whether its findings — in particular the superiority of LongPPL-density selection and the failure of generic-quality signals — transfer to selecting or mixing synthetic long data. This limits the practical guidance the study can offer for modern training recipes.
W4: Generality is limited: a single 3B backbone, a single 6B-token budget, one internal corpus (which impedes exact replication), and a scorer-dependent density threshold whose cutoff (τ = 0.05) is admittedly specific to the scorer, corpus, and configuration.
Questions for authors
Q1: Why do you think generic-quality and LLM-judge signals fail to beat random sampling within the fixed pool? Is it because the pool is already length-filtered and homogeneous, leaving little quality variance for these signals to exploit? Did you examine what properties the LongPPL-selected documents have beyond key-token density (e.g., genre, structure)?
Q2: Do you have any evidence — even preliminary — on whether LongPPL-density selection behaves sensibly on synthetically constructed long documents, or on mixtures of natural and synthetic long data?
Q3: How sensitive are the conclusions to the scorer model? LongPPL density is computed with a fixed external scorer; would a scorer more closely matched to the trained backbone change the ranking of selectors?
Q4: Do you expect the density threshold τ and the front-64K window to transfer to larger backbones, longer target contexts (e.g., 256K+), or larger token budgets, or would they need to be re-tuned per setting?
5 comments
Responses to the Critical Weaknesses (part2)
W4. How well do the findings generalize beyond the original experimental setting?
Answer W4:
Additional experiments show that truncLongPPL’s selection gains extend to 256K contexts, different model families at 7B/8B scale, and 20B-token mixed-training recipes, with the advantage retained after SFT.
(1) Natural-long-only training: Qwen2.5-3B, 6B tokens, 128K/256K.
All training tokens come from natural long documents, without short-data mixing.
| Metric | random-long | LongFilter | truncLongPPL |
|---|---|---|---|
| 128K training: RULER average, 8K–128K | 78.03 | 79.68 | 81.31 |
| 256K training: RULER average, 8K–256K | 75.81 | 77.25 | 78.63 |
| 256K training: 256K endpoint | 62.91 | 65.75 | 67.05 |
The 128K values are means over three training seeds; the 256K values are point estimates. truncLongPPL outperforms random-long and LongFilter at both target lengths. At 256K, its average/endpoint gains over random-long are 2.82/4.14 points. These length-extension results have been incorporated into Section 7 and Table 2 of the revised manuscript.
(2) Short/long mixed training: three backbones, short:long = 66:34.
Within each backbone, the short-data component is fixed, and only long-data selection changes. “Random” and “selected” denote randomly sampled and truncLongPPL-selected long documents, respectively.
| Setting | Qwen2.5-3B | Qwen3-8B | OLMo 3-7B |
|---|---|---|---|
| Training length | 128K | 256K | 256K |
| Total budget: short + long | 10B = 6.6B + 3.4B | 20B = 13.2B + 6.8B | 20B = 13.2B + 6.8B |
| RULER average: random → selected | 78.61 → 81.94 | 85.61 → 87.08 | 65.68 → 72.99 |
| 256K endpoint: random → selected | — | 73.12 → 77.20 | 53.06 → 55.94 |
RULER averages cover 8K–128K for the 3B setting and 8K–256K for the 7B/8B settings. The 3B results are means over three training seeds; the 7B/8B results are point estimates. Average RULER improves by 3.33/1.47/7.31 points, respectively. The mixed-training results are now presented in Sections 8.1 and 8.2, with the Qwen3-8B and OLMo 3-7B comparisons reported in Table 3.
(3) Subsequent SFT of the mixed-training models: Qwen3-8B and OLMo 3-7B.
We start from the 256K, 20B-token, 66:34 mixed-training checkpoints above and apply the same LongMagpie SFT recipe with 64K packing. “Random” and “selected” refer to long-data selection during the preceding continued pretraining.
| Overall score: random → selected | Qwen3-8B | OLMo 3-7B |
|---|---|---|
| LongBench v2 | 33.87 ± 0.31 → 35.00 ± 0.20 | 27.67 ± 0.42 → 29.33 ± 0.31 |
| MRCR | 24.99 ± 0.33 → 25.61 ± 0.29 | 13.73 ± 0.27 → 15.16 ± 0.26 |
Values are mean ± standard deviation over three evaluation seeds for each fixed SFT model, rather than repeated training runs. Both models retain the selection advantage in overall scores on both benchmarks. These SFT findings have been incorporated into Section 8.3, with complete results and evaluation details provided in Appendices P and Q.
Responses to the Reviewer’s Questions (part1)
Q1(a). Why do you think generic-quality and LLM-judge signals fail to beat random sampling within the fixed pool? Is it because the pool is already length-filtered and homogeneous, leaving little quality variance for these signals to exploit?
Answer Q1(a):
No. Our interpretation is that generic-quality proxies do not directly measure long-context training utility. PPL and gzip capture overall predictability and compressibility, respectively, without directly quantifying the predictive value of additional context. Fang et al. show that averaging perplexity over all tokens can obscure performance on long-context key tokens [1]. Gao et al. also find that downstream long-context performance can decline even as perplexity improves [2]. Our judge assesses dependencies using limited fragment pairs and may be sensitive to sampling coverage, judging biases, and reasoning limitations [3]. Recognizing semantic dependencies does not directly quantify training utility. Consequently, better generic-quality or judge scores do not necessarily identify more useful long-context training data.
Length filtering alone does not establish that the candidate pool is highly homogeneous.
Q1(b). Did you examine what properties the LongPPL-selected documents have beyond key-token density (e.g., genre, structure)?
Answer Q1(b):
To characterize the selected content, we analyzed 1,000 documents each from truncLongPPL and random sampling. Topics were automatically annotated based on the WebOrganizer taxonomy [4], languages were identified using fastText [5], and structure was measured using predefined excerpt markers, informed by WIMBD’s descriptive corpus-auditing approach [6].
| Dimension | Attribute | truncLongPPL (front) | Random |
|---|---|---|---|
| Language | English | 44.0% | 40.8% |
| Language | Non-English/mixed | 56.0% | 59.2% |
| Topic | Science/technology | 18.6% | 27.2% |
| Topic | Finance/business | 11.4% | 7.1% |
| Topic | Religion | 11.3% | 7.5% |
| Structure | Body text | 96.2% | 98.0% |
| Structure | Headings | 91.3% | 92.3% |
| Structure | Lists | 84.6% | 77.6% |
The samples share common structural features but show differences in language and topic composition. In our coarse-domain reweighting control, restoring the candidate pool’s proportions of books, webpages, and papers changes HELMET/RULER by at most 0.3/0.2 points, while the main gains remain. Thus, coarse source composition alone does not explain the advantage, although this control does not individually match the topic and structural attributes reported above.
Q2. Is there preliminary evidence on synthetic or mixed natural/synthetic long-context data?
Answer Q2:
Please see W3 for our experiment combining LongPPL-based root-document selection with NExtLong synthesis.
Q3. Would a scorer closer to the training backbone change the selector ranking?
Answer Q3:
With Gemma held fixed as the scorer, the selected data improves performance across the evaluated model families and scales. As reported in W4, the average RULER gains at 256K are +2.82 for Qwen2.5-3B in the 6B-token natural-long-only setting, and +1.47/+7.31 for Qwen3-8B/OLMo 3-7B in their respective 20B-token, 66:34 short/long mixtures. Thus, an effective scorer need not belong to the same model family as the training backbone in these settings.
Due to the limited rebuttal period, we have not completed a direct comparison with alternative scorers. We are considering using Qwen or OLMo models as scorers in future work, holding the candidate pool, retained pool size, and training budget fixed. This would directly test whether closer alignment with the training backbone improves selection quality or changes selector rankings. The current evidence supports transfer across training backbones, rather than robustness to scorer replacement.
[1] Fang L, Wang Y, Liu Z, et al. What is Wrong with Perplexity for Long-context Language Modeling? ICLR, 2025.
[2] Gao T, Wettig A, Yen H, Chen D. How to Train Long-Context Language Models (Effectively). ACL, 2025.
[3] Zheng L, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023.
[4] Wettig A, et al. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. ICML, 2025.
[5] Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of Tricks for Efficient Text Classification. EACL, 2017.
[6] Elazar Y, et al. What’s In My Big Data? ICLR, 2024.
Responses to the Critical Weaknesses (part1)
We thank the reviewer for the careful assessment, especially the constructive suggestions on generalizability and synthetic data.
W1. What does the paper contribute beyond benchmarking existing selectors?
Answer W1:
Determining which natural-long-document selection pipelines improve context extension under a shared protocol remains an important question that prior work has not fully answered. Differences in corpora, models, and training budgets make existing results difficult to compare directly. Comparable downstream training evidence can guide data selection and compute allocation; providing this evidence is our core contribution.
We build on Fang et al.’s LongPPL key-token signal and explicitly acknowledge OLMo 3’s exploration of its use for document selection. We position truncLongPPL as an efficient prefix-scoring variant, emphasizing three empirical findings:
Selection rules affect a signal’s practical utility. Under the same training setup, LongPPL-density performs similarly to gzip with the middle-60% rule, whereas selecting the high-density tail yields better results. A signal therefore needs to be evaluated together with its selection rule.
Prefix scoring offers a practical cost–quality trade-off. Front-64K scoring achieves downstream performance close to full scoring in reference experiments while providing a 2.28× scoring speedup.
The gains are capability-specific. Improvements are most consistent on long-range retrieval, while aggregation and variable tracking show no consistent benefit.
These controlled experiments clarify the utility, operating conditions, and capability limits of the evaluated selection pipelines.
W2. What is the value of simple prefix truncation when its selected documents differ from full scoring?
Answer W2:
The value of truncLongPPL lies in reducing computational cost through a simple scoring modification, with its effectiveness verified through downstream training. Scoring millions of long documents at full length requires substantial computation. Finding a simplification that preserves training benefits therefore has practical value.
Our goal is to retain training utility at lower cost; reproducing the full-scoring selected set document by document is not required. The reported 76–92% one-way containment confirms that the sets differ. The relevant test is whether these different sets produce comparable downstream performance under the same training budget.
The reference experiments use the same backbone, 128K training length, 6B-token budget, and τ = 0.05:
HELMET is identical, and RULER differs by only 0.11 points, while prefix scoring provides a 2.28× complete-scoring speedup. These results support comparable training utility in the evaluated setting despite the different selected sets.
W3. Can the selection approach also benefit synthetic long-context training?
Answer W3:
We have added an experiment combining our selection approach with NExtLong, providing preliminary evidence that natural-document selection can improve the training utility of synthetic long-context data. Given the limited rebuttal period and substantial training cost, we prioritized a Qwen3-8B-Base, 256K, 2B-token comparison. We use either randomly sampled or LongPPL-selected natural documents as root documents, construct synthetic long documents through NExtLong, and then perform continued pretraining.
LongPPL-selected roots achieve higher RULER scores at all six evaluation lengths from 8K to 256K. The average increases from 83.15 to 83.72, and the 256K score increases from 65.65 to 66.42:
Thus, even under a limited training budget, using natural-document selection before synthesis yields downstream improvements. Our core question concerns which signals are useful for selecting long documents; systematically optimizing short/long or natural/synthetic mixture ratios is a broader training-recipe question. This experiment demonstrates the compatibility of the two approaches.