acceptodds
Under review as a conference paper at ICLR 2027

Selecting Natural Long Documents for Context Extension

Abstract

Natural long documents are widely used to extend language models to longer contexts, yet length alone does not reveal whether they provide useful long-range supervision. We present a controlled study of how such documents should be selected, and scored. Using Qwen2.5-3B, 128K-token continued training, and matched 6B-token budgets, we compare repacking shorter documents from the preceding training stage into long sequences, random sampling of natural long documents, and generic-quality, LLM-judge, and likelihood-based selection; HELMET tasks are used for model selection and RULER for held-out evaluation. Random sampling is only slightly better than repacking at shorter evaluation lengths, but leads by approximately 2.5 HELMET points and 3.3 RULER points at 128K. Within a fixed pool of natural long documents, the evaluated generic-quality and LLM-judge pipelines do not exceed random sampling, whereas the likelihood-based long-dependency pipelines achieve higher observed scores. Selecting documents by their density of LongPPL key tokens—tokens whose prediction improves markedly when the long context is available—achieves the highest observed scores among the evaluated pipelines. Its most consistent cross-benchmark gains are retrieval-oriented and widen toward 64K–128K, whereas QA effects are benchmark-dependent and aggregation and variable tracking show no consistent gain. Because scoring every document over the full 128K span is expensive, we introduce an efficient variant, , that restricts scoring to the first 64K target positions and yields downstream scores close to those of the full-128K pipeline while providing a measured end-to-end scoring speedup. Finally, in one evaluated 66/34 short/long mixture, replacing random long documents with selected ones improves macro-averaged HELMET by 4.6 points and RULER by 3.3 points without a material additional observed reduction on seven short-context tasks, and this advantage is further corroborated after supervised fine-tuning on LongBench v2 and MRCR. Overall, we hope the controlled and comprehensive comparisons will provide more practical references for long-context training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ruWmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a controlled empirical study of how natural long documents should be selected for continued pretraining aimed at context extension. Using Qwen2.5-3B extended to 128K with matched 6B-token budgets, the authors compare scoring-free constructions (repacking, random sampling), generic-quality selectors (perplexity, gzip), an LLM-judge pipeline, and likelihood-based long-dependency selectors (LongFilter, Prospector, LongPPL key-token density). HELMET is used for all selection decisions and RULER is held out. The main findings are: random sampling of natural long documents beats repacking at long evaluation lengths; generic-quality and LLM-judge pipelines do not exceed random sampling; LongPPL key-token density gives the best downstream scores; and a cheaper variant (truncLongPPL, scoring only front-64K target positions) retains most of the benefit with a 2.28× scoring speedup. A fixed 66/34 short/long mixture experiment and post-SFT evaluation on LongBench v2 and MRCR show the benefit persists in a more realistic recipe.

Critical strengths

S1: A clean, controlled, apples-to-apples comparison of complete data-selection pipelines under matched budgets.

S2: Sound evaluation protocol — HELMET is used for all tuning decisions while RULER remains genuinely held out, and seven short-context tasks measure retention.

S3: Informative capability-level analysis: gains concentrate on retrieval-oriented tasks at 64K–128K, while aggregation and variable tracking show no consistent benefit; the paper is refreshingly honest about where selection does not help.

S4: The front-64K scoring variant offers a useful cost–quality operating point (2.28× measured speedup with near-equal downstream scores), which is practically relevant since full-128K scoring costs 10^4–10^5 H20 GPU-hours.

Critical weaknesses

W1: The paper is mainly an empirical comparison and does not propose a new data-selection method. All evaluated selectors are taken from prior work, and the contribution is limited to benchmarking them under a shared protocol.

W2: truncLongPPL, the paper's main methodological contribution, is a straightforward truncation of the existing LongPPL heuristic (restrict scoring to the first 64K target positions). While it improves scoring efficiency, the modification is simple and has limited novelty, and the reported front-to-full selected-set agreement is imperfect (76–92% containment), so the two pipelines are not equivalent even at the document level.

W3: The conclusions are drawn exclusively on natural long documents. Given that recent long-context recipes increasingly rely on synthetic long documents (e.g., synthetic agent/repository data in GLM-5, retrieval-based constructions like NExtLong), the paper neither analyzes nor discusses whether its findings — in particular the superiority of LongPPL-density selection and the failure of generic-quality signals — transfer to selecting or mixing synthetic long data. This limits the practical guidance the study can offer for modern training recipes.

W4: Generality is limited: a single 3B backbone, a single 6B-token budget, one internal corpus (which impedes exact replication), and a scorer-dependent density threshold whose cutoff (τ = 0.05) is admittedly specific to the scorer, corpus, and configuration.

Questions for authors

Q1: Why do you think generic-quality and LLM-judge signals fail to beat random sampling within the fixed pool? Is it because the pool is already length-filtered and homogeneous, leaving little quality variance for these signals to exploit? Did you examine what properties the LongPPL-selected documents have beyond key-token density (e.g., genre, structure)?

Q2: Do you have any evidence — even preliminary — on whether LongPPL-density selection behaves sensibly on synthetically constructed long documents, or on mixtures of natural and synthetic long data?

Q3: How sensitive are the conclusions to the scorer model? LongPPL density is computed with a fixed external scorer; would a scorer more closely matched to the trained backbone change the ranking of selectors?

Q4: Do you expect the density threshold τ and the front-64K window to transfer to larger backbones, longer target contexts (e.g., 256K+), or larger token budgets, or would they need to be re-tuned per setting?

5 comments
AuthorsOpenReview ↗

Responses to the Critical Weaknesses (part1)

We thank the reviewer for the careful assessment, especially the constructive suggestions on generalizability and synthetic data.

W1. What does the paper contribute beyond benchmarking existing selectors?

Answer W1:

Determining which natural-long-document selection pipelines improve context extension under a shared protocol remains an important question that prior work has not fully answered. Differences in corpora, models, and training budgets make existing results difficult to compare directly. Comparable downstream training evidence can guide data selection and compute allocation; providing this evidence is our core contribution.

We build on Fang et al.’s LongPPL key-token signal and explicitly acknowledge OLMo 3’s exploration of its use for document selection. We position truncLongPPL as an efficient prefix-scoring variant, emphasizing three empirical findings:

  1. Selection rules affect a signal’s practical utility. Under the same training setup, LongPPL-density performs similarly to gzip with the middle-60% rule, whereas selecting the high-density tail yields better results. A signal therefore needs to be evaluated together with its selection rule.

  2. Prefix scoring offers a practical cost–quality trade-off. Front-64K scoring achieves downstream performance close to full scoring in reference experiments while providing a 2.28× scoring speedup.

  3. The gains are capability-specific. Improvements are most consistent on long-range retrieval, while aggregation and variable tracking show no consistent benefit.

These controlled experiments clarify the utility, operating conditions, and capability limits of the evaluated selection pipelines.

W2. What is the value of simple prefix truncation when its selected documents differ from full scoring?

Answer W2:

The value of truncLongPPL lies in reducing computational cost through a simple scoring modification, with its effectiveness verified through downstream training. Scoring millions of long documents at full length requires substantial computation. Finding a simplification that preserves training benefits therefore has practical value.

Our goal is to retain training utility at lower cost; reproducing the full-scoring selected set document by document is not required. The reported 76–92% one-way containment confirms that the sets differ. The relevant test is whether these different sets produce comparable downstream performance under the same training budget.

The reference experiments use the same backbone, 128K training length, 6B-token budget, and τ = 0.05:

MetricFull-128K LongPPL-densityFront-64K truncLongPPL
HELMET56.2356.23
RULER81.3481.23
Complete scoring time16.55 s/document7.25 s/document

HELMET is identical, and RULER differs by only 0.11 points, while prefix scoring provides a 2.28× complete-scoring speedup. These results support comparable training utility in the evaluated setting despite the different selected sets.

W3. Can the selection approach also benefit synthetic long-context training?

Answer W3:

We have added an experiment combining our selection approach with NExtLong, providing preliminary evidence that natural-document selection can improve the training utility of synthetic long-context data. Given the limited rebuttal period and substantial training cost, we prioritized a Qwen3-8B-Base, 256K, 2B-token comparison. We use either randomly sampled or LongPPL-selected natural documents as root documents, construct synthetic long documents through NExtLong, and then perform continued pretraining.

LongPPL-selected roots achieve higher RULER scores at all six evaluation lengths from 8K to 256K. The average increases from 83.15 to 83.72, and the 256K score increases from 65.65 to 66.42:

Evaluation length8K16K32K64K128K256KAverage
Random roots + NExtLong93.9292.7388.6783.7474.1865.6583.15
LongPPL-selected roots + NExtLong93.9893.2489.0084.2275.4466.4283.72
Gain+0.06+0.51+0.33+0.48+1.26+0.77+0.57

Thus, even under a limited training budget, using natural-document selection before synthesis yields downstream improvements. Our core question concerns which signals are useful for selecting long documents; systematically optimizing short/long or natural/synthetic mixture ratios is a broader training-recipe question. This experiment demonstrates the compatibility of the two approaches.

AuthorsOpenReview ↗

Responses to the Critical Weaknesses (part2)

W4. How well do the findings generalize beyond the original experimental setting?

Answer W4:

Additional experiments show that truncLongPPL’s selection gains extend to 256K contexts, different model families at 7B/8B scale, and 20B-token mixed-training recipes, with the advantage retained after SFT.

(1) Natural-long-only training: Qwen2.5-3B, 6B tokens, 128K/256K.

All training tokens come from natural long documents, without short-data mixing.

Metricrandom-longLongFiltertruncLongPPL
128K training: RULER average, 8K–128K78.0379.6881.31
256K training: RULER average, 8K–256K75.8177.2578.63
256K training: 256K endpoint62.9165.7567.05

The 128K values are means over three training seeds; the 256K values are point estimates. truncLongPPL outperforms random-long and LongFilter at both target lengths. At 256K, its average/endpoint gains over random-long are 2.82/4.14 points. These length-extension results have been incorporated into Section 7 and Table 2 of the revised manuscript.

(2) Short/long mixed training: three backbones, short:long = 66:34.

Within each backbone, the short-data component is fixed, and only long-data selection changes. “Random” and “selected” denote randomly sampled and truncLongPPL-selected long documents, respectively.

SettingQwen2.5-3BQwen3-8BOLMo 3-7B
Training length128K256K256K
Total budget: short + long10B = 6.6B + 3.4B20B = 13.2B + 6.8B20B = 13.2B + 6.8B
RULER average: random → selected78.61 → 81.9485.61 → 87.0865.68 → 72.99
256K endpoint: random → selected—73.12 → 77.2053.06 → 55.94

RULER averages cover 8K–128K for the 3B setting and 8K–256K for the 7B/8B settings. The 3B results are means over three training seeds; the 7B/8B results are point estimates. Average RULER improves by 3.33/1.47/7.31 points, respectively. The mixed-training results are now presented in Sections 8.1 and 8.2, with the Qwen3-8B and OLMo 3-7B comparisons reported in Table 3.

(3) Subsequent SFT of the mixed-training models: Qwen3-8B and OLMo 3-7B.

We start from the 256K, 20B-token, 66:34 mixed-training checkpoints above and apply the same LongMagpie SFT recipe with 64K packing. “Random” and “selected” refer to long-data selection during the preceding continued pretraining.

Overall score: random → selectedQwen3-8BOLMo 3-7B
LongBench v233.87 ± 0.31 → 35.00 ± 0.2027.67 ± 0.42 → 29.33 ± 0.31
MRCR24.99 ± 0.33 → 25.61 ± 0.2913.73 ± 0.27 → 15.16 ± 0.26

Values are mean ± standard deviation over three evaluation seeds for each fixed SFT model, rather than repeated training runs. Both models retain the selection advantage in overall scores on both benchmarks. These SFT findings have been incorporated into Section 8.3, with complete results and evaluation details provided in Appendices P and Q.

AuthorsOpenReview ↗

Responses to the Reviewer’s Questions (part1)

Q1(a). Why do you think generic-quality and LLM-judge signals fail to beat random sampling within the fixed pool? Is it because the pool is already length-filtered and homogeneous, leaving little quality variance for these signals to exploit?

Answer Q1(a):

No. Our interpretation is that generic-quality proxies do not directly measure long-context training utility. PPL and gzip capture overall predictability and compressibility, respectively, without directly quantifying the predictive value of additional context. Fang et al. show that averaging perplexity over all tokens can obscure performance on long-context key tokens [1]. Gao et al. also find that downstream long-context performance can decline even as perplexity improves [2]. Our judge assesses dependencies using limited fragment pairs and may be sensitive to sampling coverage, judging biases, and reasoning limitations [3]. Recognizing semantic dependencies does not directly quantify training utility. Consequently, better generic-quality or judge scores do not necessarily identify more useful long-context training data.

Length filtering alone does not establish that the candidate pool is highly homogeneous.

Q1(b). Did you examine what properties the LongPPL-selected documents have beyond key-token density (e.g., genre, structure)?

Answer Q1(b):

To characterize the selected content, we analyzed 1,000 documents each from truncLongPPL and random sampling. Topics were automatically annotated based on the WebOrganizer taxonomy [4], languages were identified using fastText [5], and structure was measured using predefined excerpt markers, informed by WIMBD’s descriptive corpus-auditing approach [6].

DimensionAttributetruncLongPPL (front)Random
LanguageEnglish44.0%40.8%
LanguageNon-English/mixed56.0%59.2%
TopicScience/technology18.6%27.2%
TopicFinance/business11.4%7.1%
TopicReligion11.3%7.5%
StructureBody text96.2%98.0%
StructureHeadings91.3%92.3%
StructureLists84.6%77.6%

The samples share common structural features but show differences in language and topic composition. In our coarse-domain reweighting control, restoring the candidate pool’s proportions of books, webpages, and papers changes HELMET/RULER by at most 0.3/0.2 points, while the main gains remain. Thus, coarse source composition alone does not explain the advantage, although this control does not individually match the topic and structural attributes reported above.

Q2. Is there preliminary evidence on synthetic or mixed natural/synthetic long-context data?

Answer Q2:

Please see W3 for our experiment combining LongPPL-based root-document selection with NExtLong synthesis.

Q3. Would a scorer closer to the training backbone change the selector ranking?

Answer Q3:

With Gemma held fixed as the scorer, the selected data improves performance across the evaluated model families and scales. As reported in W4, the average RULER gains at 256K are +2.82 for Qwen2.5-3B in the 6B-token natural-long-only setting, and +1.47/+7.31 for Qwen3-8B/OLMo 3-7B in their respective 20B-token, 66:34 short/long mixtures. Thus, an effective scorer need not belong to the same model family as the training backbone in these settings.

Due to the limited rebuttal period, we have not completed a direct comparison with alternative scorers. We are considering using Qwen or OLMo models as scorers in future work, holding the candidate pool, retained pool size, and training budget fixed. This would directly test whether closer alignment with the training backbone improves selection quality or changes selector rankings. The current evidence supports transfer across training backbones, rather than robustness to scorer replacement.

[1] Fang L, Wang Y, Liu Z, et al. What is Wrong with Perplexity for Long-context Language Modeling? ICLR, 2025.

[2] Gao T, Wettig A, Yen H, Chen D. How to Train Long-Context Language Models (Effectively). ACL, 2025.

[3] Zheng L, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS, 2023.

[4] Wettig A, et al. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. ICML, 2025.

[5] Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of Tricks for Efficient Text Classification. EACL, 2017.

[6] Elazar Y, et al. What’s In My Big Data? ICLR, 2024.

Reviewer_sHtTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a controlled study of document scoring and selection methods. The paper begins by comparing data construction and selection pipelines from related work, showing that the proposed modification to the LongPPL method, truncLongPPL, achieves the best HELMET and RULER scores. From a methodological perspective, the results show clear insight into the landscape of different pipelines.

The paper is hard to read in general. The motivation for the study is unclear in the introduction. The general idea of the study and the truncLongPPL variant is interesting. However, the execution of the ideas lacks the fundamentals to be considered acceptable.

Critical strengths

S1: Introduction of the truncLongPPL variant. The authors modify the LongPPL method by adding a key-token density term that restricts scoring to the first 64K target positions and yields results similar to the full 128K scoring pipeline.

S2: Extensive study of data construction and selection pipelines. The evaluation of different data construction and selection methods brings more light to the problem and can be useful to a broader audience.

Critical weaknesses

W1: Missing background and methodology. The name of the proposed method is mentioned only vaguely in the paper. It is not clear which method the results are presented (e.g., Figures 3 and 4).

W2: Weak baselines. In many of the results in Sections 5, 6, and 7, the authors mostly compare truncLongPPL with its other variants or random. It would be more interesting and beneficial for the paper's results to include a comparison across different methods (the most obvious candidate is the LongPPL method).

W3: Paper presentation and readability. The paper is generally hard to read and often relies on additional context. The introduction is confusing, often introducing terms that were not defined earlier or are difficult to understand from context. It is hard to understand the study's motivation from the introduction and related work. The HELMET and RULER benchmarks are never properly explained (e.g., packing decreases the metrics, but the data selection methods increase them). The τ\tau values are never mentioned in previous equations or how they contribute to the compared methods. Also, some of the important main-text results are referenced in the appendix, which should be treated as additional information rather than as the basis for the authors' claims.

Here are also some of my comments and suggestions to improve the paper:

C1: I would suggest removing the contribution iteration on line 67 and keeping only the blue box, since it conveys the same information.

C2: It is really evident that the paper was generated using an LLM. The authors should consider restructuring and rewriting some parts of the paper (e.g., the abstract, the introduction, and the word “precluding” on line 111).

Questions for authors

Q1: From which data/dataset is the set of documents C sourced? Is it possible to provide the sources in the paper?

3 comments
AuthorsOpenReview ↗

Responses to the Critical Weaknesses

We thank the reviewer for the detailed feedback, especially the specific suggestions on method clarity, baseline comparisons, and readability.

W1. The name of the proposed method is mentioned only vaguely in the paper. It is not clear which method the results are presented (e.g., Figures 3 and 4).

Answer W1:

Appendix A provides the complete pipeline constructions, scoring signals, and selection rules. Figures 3 and 4 both analyze LongPPL-density selection, varying the density threshold and scoring window, respectively, as discussed in Sections 5.1 and 5.2. Both use Qwen2.5-3B, 128K training sequences, and a 6B-token budget.

SettingFigure 3Figure 4
Fixed configurationFull-128K scoringτ = 0.05
Varied parameterDocument-density threshold, τ = 0.01–0.08Scoring-input span: front-16K/32K/64K and full-128K

We hope these clarifications address your concerns about the method definitions and their correspondence to the figures.

W2.Why do later experiments focus on variants and random sampling, and where is the full LongPPL baseline?

Answer W2:

Section 4 compares the complete selection pipelines under controlled conditions. Sections 5, 6, and 8 then examine selection settings, capability-level effects, and training utility within a fixed short/long mixture.

The full LongPPL-density baseline is included in Figure 4 as full-128K, originally labeled “whole 128K.” Its direct comparison with prefix scoring is:

MetricFull-128K LongPPL-densitytruncLongPPL, front-64K
HELMET56.2356.23
RULER81.3481.23
Complete scoring time16.55 s/document7.25 s/document

These are single reference runs from the window ablation, using the same model, 128K training length, 6B tokens, and τ = 0.05. Prefix scoring achieves downstream performance close to full scoring while providing a 2.28× complete-scoring speedup.

Our core contribution is the controlled comparison and analysis of selection utility; truncLongPPL is an efficient prefix-scoring variant of the existing LongPPL-density pipeline. We hope this clarifies the inclusion of the full-scoring baseline and the purpose of the later analyses.

We have also revised the organization and exposition of the updated manuscript. Your valuable feedback has helped us make the presentation clearer, and we hope these clarifications and revisions address your questions.

W3. How will the paper clarify its motivation, benchmarks, parameters, and presentation of the main evidence?

Answer W3:

Our motivation is to determine which natural-long-document selection pipelines provide training value under a fixed long-context continued-pretraining budget, and how to reduce their scoring cost. Document length alone does not guarantee that its information requires long-range context. Differences in models, data mixtures, and budgets also make comparisons across papers difficult. We therefore fix the candidate pool, backbone, target length, training objective, and token budget, first comparing pipelines and then analyzing thresholds, windows, and capability gains.

HELMET [1] and RULER [2] evaluate long-context capabilities; ProLong [3], NExtLong [4], and OLMo 3 [5] also report results on both benchmarks. Their roles in our study are:

BenchmarkCapabilities evaluated in our studyExperimental role
HELMET [1]Recall, ICL, re-ranking, long-document QA, RAGSelecting thresholds and scoring windows
RULER [2]Retrieval, aggregation, variable tracking, QAHeld-out evaluation after configurations are fixed

For the 128K comparison, we report results at 8K, 16K, 32K, 64K, and 128K, together with macro-averages. Higher scores indicate better task performance. Differences between packing and selection pipelines reflect the resulting models’ performance under the same evaluation protocol; no direction of improvement is assumed in advance.

Regarding parameters, α = 2 and β = −2 are fixed token-level criteria, τ is the document-density threshold, and r is the scoring-prefix length divided by the target context length. The revised main text defines these parameters, and the captions of Figures 3 and 4 identify the fixed and varied configurations. Key comparisons remain in the main text, while the appendix provides complete breakdowns and implementation details.

We hope these additions clarify the motivation, evaluation, and parameter settings and address your concerns about readability.

[1] Yen H, Gao T, et al. HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly. ICLR, 2025.

[2] Hsieh C-P, Sun S, Kriman S, et al. RULER: What’s the Real Context Size of Your Long-Context Language Models? COLM, 2024.

[3] Gao T, Wettig A, Yen H, Chen D. How to Train Long-Context Language Models (Effectively). ACL, 2025: 7376–7399.

[4] Gao C, Wu X, Lin Z, Zhang D, Hu S. NExtLong: Toward Effective Long-Context Training without Long Documents. ICML, 2025: 18566–18584.

[5] Team OLMo, et al. Olmo 3. arXiv:2512.13961, 2025.

AuthorsOpenReview ↗

Responses to the Reviewer’s Suggestions and Questions

C1. The contribution list at line 67 duplicates the blue summary box.

Answer C1:

Thank you for this specific suggestion. We have removed the duplicate contribution list and retained the summary box as a concise overview of the main findings, reducing repetition in the introduction.

C2. The abstract, introduction, and wording need revision.

Answer C2:

We have revised the abstract and introduction, clarified terminology, shortened long sentences, and replaced “precluding” with more direct wording. Thank you for these constructive suggestions, which have helped improve the paper’s clarity and readability.

Q1: From which data/dataset is the set of documents C sourced? Is it possible to provide the sources in the paper?

Answer Q2:

C is drawn from our internal pretraining corpus, primarily derived from Common Crawl with additional cleaning. Section 2.1 of the revised manuscript describes the construction procedure: we sample 25% of the internal corpus by document, then retain documents containing at least 128K tokens, yielding approximately 3M candidate documents.

The pool includes books, scanned documents, manuals, academic papers, and webpages, spanning English, Chinese, and mixed/other languages.

Reviewer_sHtTOpenReview ↗

Response to the authors

I thank the authors for their well-written answers to my stated weaknesses and questions. I have decided to increase the rating from 2 to 3.

Reviewer_3LXgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how to select natural long documents for 128K context extension. Using Qwen2.5-3B and 6B-token budgets, with HELMET for validation and RULER held out, it compares repacking, random sampling, perplexity, gzip, an LLM judge, LongFilter, Prospector, and LongPPL key-token density. Random long documents beat repacking at 128K; LongPPL density with an absolute threshold performs best; scoring only the first 64K tokens (truncLongPPL) matches full-128K scoring at 2.28x lower cost; and the advantage holds in a 66/34 short/long mixture and after SFT.

Critical strengths
  • A controlled comparison of selection pipelines under one protocol, with a clean validation/held-out split.
  • The reproduction and explanation of OLMo 3's negative result (the middle-60% rule is too permissive to reach the high-density tail) is a useful finding.
  • The fixed-mixture experiment (three seeds, std at most 0.27, gaps of 4.6 and 3.3 points) and the SFT follow-up are convincing.
  • Thoughtful controls and candid limitations.
Critical weaknesses
  1. Positioning. The signal comes from Fang et al. (2025) and its use as a selector from OLMo 3; the additions here are an absolute tail threshold and prefix-only scoring. Calling truncLongPPL a new selector slightly overstates this. Related work could also cover the long-context data-engineering line (Fu et al. 2024, Xiong et al. 2023, LongWanjuan) and discuss LongCE, the loss-reweighting alternative from the same LongPPL paper, which seems the most natural competitor to discarding most of the pool.

  2. Selection rate is not controlled. LongPPL keeps about 1.7%, LongFilter about 1.6%, the judge 5%, PPL 40%, Prospector 50%, gzip 60%. The two best pipelines are also the two most selective. Section 4.2 hedges appropriately, but the abstract presents this as a conclusion about signal types. Matched retention rates would make the comparison much stronger.

  3. Sections 4 to 6 are single-seed. The ordering among random, repack, gzip, judge, and mid-PPL (all within 0.6 points), the threshold plateau, and the two-decimal agreement in Section 5.2 are within plausible noise. Seeds for truncLongPPL, LongFilter, and random would help, as would reporting documents, tokens, and epochs per configuration, since at tau of 0.05 or above the retained pool is close to one epoch at 6B tokens.

  4. Window results deserve discussion. Front, tail, and scattered windows all match full-128K scoring, suggesting the density is a document-level property rather than a measure of specific long-range dependencies. A short qualitative look at what gets selected (language, genre, repetition) would clarify this.

Questions for authors
  1. How many documents, tokens, and epochs does each pipeline train on?
  2. Do PPL, gzip, and the judge still fall below random at matched retention rates?
  3. What is the total scoring cost relative to training cost?
4 comments
AuthorsOpenReview ↗

Responses to the Critical Weaknesses (part1)

We thank the reviewer for the detailed feedback, especially the constructive suggestions on experimental robustness.

W1(a). The additions are an absolute threshold targeting the high-density tail and prefix-only scoring. Calling truncLongPPL a new selector slightly overstates its novelty.

Answer W1(a):

Our core contribution is a controlled comparison of natural-long-document selection pipelines under a shared candidate pool, backbone, and training budget, validated through downstream training. Across three independent training seeds, truncLongPPL improves over random sampling by 4.54 HELMET / 3.28 RULER points, with the most consistent gains on long-range retrieval.

truncLongPPL provides a simple, easy-to-implement scoring modification that reduces computation while preserving training utility. Scoring millions of long documents at full length is expensive. In the reference experiments, prefix scoring provides a 2.28× speedup of the complete scoring pipeline while retaining downstream performance close to full scoring. Therefore, identifying a simple modification that preserves training gains has practical value in its own right.

W1(b). How does the study relate to LongCE and prior long-context data engineering?

Answer W1(b):

LongCE, introduced alongside LongPPL by Fang et al. [1], reweights token losses. Our study instead examines document selection while keeping the standard cross-entropy objective fixed, allowing the comparison to focus on different selection pipelines.

We will expand the discussion of LongCE and the related data-engineering work highlighted by the reviewer. A direct comparison or combination with LongCE requires separate controlled training experiments and remains a direction for future work.

W2. How does the study relate to LongCE and prior long-context data engineering?

Answer W2:

No. We compare complete pipelines with explicitly defined selection rules; retention rate is part of each pipeline’s configuration. gzip follows OLMo 3 in removing the bottom and top 20%, retaining the middle 60% [2]. Prospector uses the top-50% setting based on the original paper [3]. LongFilter originally retains the top 20% [4]; we use its scoring signal and select the highest-scoring documents up to the shared 6B-token budget. PPL’s top/middle 40% and the judge’s top 5% are baseline configurations specified in our study. Accordingly, our comparison evaluates these complete configurations under a common training budget.

W3. Are the reported performance gains robust across training seeds?

Answer W3:

The main performance advantage of truncLongPPL persists across three independent training seeds. We have extended all nine main pipelines to three training seeds under the same Qwen2.5-3B, 128K, 6B-token setting. Results below are reported as mean ± standard deviation. truncLongPPL improves over random-long by 4.54 / 3.28 points and over LongFilter by 2.22 / 1.63 points on HELMET / RULER, respectively. These three-seed results have been incorporated into Figure 2 and Section 4.2 of the revised manuscript, with the complete numerical results reported in Appendix M (Table 11).

MetrictruncLongPPLLongFilterProspectorrandom-longREPACKgzip (mid60)LLM-judge (top5)PPL (mid40)PPL (top40)
HELMET55.59 ± 0.5553.37 ± 0.3051.98 ± 0.2851.05 ± 0.5650.31 ± 0.4550.08 ± 0.7950.18 ± 0.4149.62 ± 0.5847.41 ± 0.41
RULER81.31 ± 0.1479.68 ± 0.1278.41 ± 0.0578.03 ± 0.0876.81 ± 0.2877.68 ± 0.2677.46 ± 0.0277.25 ± 0.1275.43 ± 0.12

[1] Fang L, Wang Y, Liu Z, et al. What is Wrong with Perplexity for Long-context Language Modeling? ICLR, 2025.

[2] OLMo Team, et al. Olmo 3. arXiv:2512.13961, 2025.

[3] Chen L, et al. Long Context Is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models. ACL, 2024: 8222–8234.

[4] Deng H, et al. Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data. arXiv:2510.25804, 2025.

AuthorsOpenReview ↗

Responses to the Critical Weaknesses (part2)

W4. Do similar results across scoring windows indicate document-level attributes rather than specific long-range dependencies?

Answer W4:

No. Our current analyses do not establish that density reflects document-level attributes. The window experiments show that selection based on different regions can achieve downstream performance close to full scoring. To characterize the selected content, we sampled 1,000 documents each from front, tail, scattered, full, and random, and compared language, topic, and structure.

We used fastText for language identification [1], automatic topic annotation based on the WebOrganizer taxonomy [2], and predefined excerpt markers for structure, informed by WIMBD’s descriptive corpus-auditing approach [3].

DimensionAttributeFour window-selected sets (range)Random
LanguageEnglish44.0–51.6%40.8%
LanguageNon-English/mixed48.4–56.0%59.2%
TopicScience/technology18.6–24.5%27.2%
TopicFinance/business11.3–13.6%7.1%
TopicReligion6.3–11.3%7.5%
StructureBody text95.9–96.2%98.0%
StructureHeadings86.8–91.3%92.3%
StructureLists84.6–87.2%77.6%

Ranges report the minimum and maximum percentages across the four window-selected sets. Body text, headings, and lists are common across all groups. Relative to random, the selected samples contain higher proportions of English, finance/business content, and lists, and a lower proportion of science/technology content.

Our coarse-domain control restores the candidate pool’s proportions of books, webpages, and papers within the selected data. HELMET/RULER changes by at most 0.3/0.2 points, while the main gains remain. Thus, coarse-domain composition alone does not explain the main gains; this control does not individually match the language, topic, or structure attributes above.

They therefore cannot determine whether similar window performance arises from stable document attributes or contextual dependencies distributed across positions.

[1] Joulin A, Grave E, Bojanowski P, Mikolov T. Bag of Tricks for Efficient Text Classification. EACL, 2017.

[2] Wettig A, et al. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation. ICML, 2025.

[3] Elazar Y, et al. What’s In My Big Data? ICLR, 2024.

AuthorsOpenReview ↗

Responses to the Reviewer’s Questions

Q1. How many documents, tokens, and epochs are used by each pipeline?

Answer Q1:

(1) Main 6B-token comparison. All pipelines use the same 6B-token training budget. The table distinguishes the retained pool size from the number of training samples used.

PipelineRetained countShare of reference poolTraining samples usedTraining tokensReference epochs
truncLongPPL53,7801.67%45,7776B0.8512
LongFilter45,7771.42%45,7776B1
Prospector1,611,09250%45,7776B0.0284
Random3,222,185100%45,7776B0.0142
REPACK45,777N/A45,7776B1
gzip (mid60)1,933,31160%45,7776B0.0237
LLM judge (top5)161,1095%45,7776B0.2841
PPL (mid40)1,288,87440%45,7776B0.0355
PPL (top40)1,288,87440%45,7776B0.0355

(2) 10B-token mixed training. The short data comes from OLMo 3-7B’s midtraining data. All three configurations use Qwen2.5-3B, 128K, and a 10B-token budget. The two mixed configurations share a short:long token ratio of 66:34 and an identical short-data component, differing only in long-document selection. Short documents are packed into 131,072-token sequences; long documents are sampled from the corresponding previously selected pools. The sample counts below are calculated from the paper’s nominal token budgets.

ConfigurationTotal training tokensShort-data tokensLong-data tokensShort-data 128K sequencesLong-data 128K sequences
dolma3-short10B10B076,2930
dolma3-short + random-long10B6.6B3.4B50,35425,939
dolma3-short + truncLongPPL10B6.6B3.4B50,35425,939

(3) SFT. We use LongMagpie packed into 64K sequences, yielding 37,633 packed samples. Training uses 32,000 samples with a nominal 2B-token budget, corresponding to 0.8503 epochs.

Q2. Do PPL, gzip, and the judge still underperform random sampling at matched retention rates?

Answer Q2:

Under the selection configurations evaluated in our study, PPL, gzip, and the judge all have lower three-seed mean scores than random sampling. Each pipeline’s retention rate follows its selection rule and forms part of its configuration. All three approaches retain enough eligible documents to complete the same 6B-token training budget without reusing documents. Different retention rates therefore do not imply different training volumes. Results below are mean ± standard deviation over three independent training seeds, using Qwen2.5-3B, 128K, and 6B tokens:

Metricrandom-longPPL (top40)PPL (mid40)gzip (mid60)LLM-judge (top5)
HELMET51.05 ± 0.5647.41 ± 0.4149.62 ± 0.5850.08 ± 0.7950.18 ± 0.41
RULER78.03 ± 0.0875.43 ± 0.1277.25 ± 0.1277.68 ± 0.2677.46 ± 0.02

Q3. What is the total scoring cost relative to training?

Answer Q3:

The estimated full-128K and front-64K scoring costs are approximately 55.0× and 24.1× the cost of a 6B-token training run, respectively. Prefix scoring reduces scoring cost by approximately 56.2%.

Training cost is calculated from a reference training log with the same model and target length: Qwen2.5-3B, 128K, 128 H20 GPUs, and sequence parallelism of 2. Training on 6B tokens requires 2,156 H20 GPU-hours. The estimated full-pool scoring costs are 118,505 H20 GPU-hours for full-128K and 51,913 H20 GPU-hours for front-64K.

Dear Reviewer 3LXg,

We sincerely appreciate the insightful feedback you provided, which has been instrumental in enhancing the quality of our work. We have prepared detailed responses for your review.

Please let us know if we need to provide any further clarifications or feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

Dear Reviewers, ACs, and SACs,

We sincerely thank all reviewers for their careful assessments and constructive suggestions. Your feedback has helped us clarify the contribution, strengthen the empirical evidence, and improve the paper’s organization and presentation. We appreciate the reviewers’ recognition of the following aspects:

1. Controlled comparisons and evaluation protocol. Reviewers recognized the study’s comparison of complete data-selection pipelines under matched training budgets, with HELMET used for configuration selection and RULER reserved for held-out evaluation. (Reviewers 3LXg and ruWm)

2. Informative empirical findings. Reviewers noted that the systematic study of data construction and selection pipelines improves our understanding of natural-long-document selection, and that the analysis of selection rules provides useful insights. (Reviewers 3LXg and sHtT)

3. Practical value of prefix scoring. Reviewers recognized that truncLongPPL reduces scoring costs while achieving downstream performance close to full scoring, providing a useful cost–quality trade-off for large-scale document selection. (Reviewers sHtT and ruWm)

4. Training validation and capability-level analysis. Reviewer 3LXg appreciated the fixed short/long mixture and subsequent SFT experiments. Reviewer ruWm valued the capability-level analysis, particularly the finding that gains concentrate on retrieval rather than uniformly improving all long-context capabilities.

In response to the reviewers’ main concerns, we have added the following experiments, analyses, and revisions:

1. Additional training seeds to assess robustness. We extended all nine main pipelines to three independent training seeds. Under the Qwen2.5-3B, 128K, 6B-token setting, truncLongPPL improves the mean HELMET/RULER scores over random-long by 4.54/3.28 points, and over LongFilter by 2.22/1.63 points. These results are included in Section 4.2, Figure 2, and Appendix M.

2. Validation at longer contexts, larger model scales, and larger training budgets. In the Qwen2.5-3B, 256K, 6B-token long-document training setting, truncLongPPL improves average RULER and the 256K endpoint over random-long by 2.82/4.14 points. In the Qwen3-8B and OLMo 3-7B, 256K, 20B-token, 66:34 short/long mixed-training settings, average RULER improves by 1.47 and 7.31 points, respectively. These experiments retain the same scorer, τ = 0.05, and front-half scoring configuration. Results appear in Section 7 and Table 2, and Section 8.2 and Table 3.

3. Post-SFT evaluation across model families. After applying the same LongMagpie SFT recipe to the Qwen3-8B and OLMo 3-7B mixed-training models, the selected-data configurations retain their advantage in overall scores on both LongBench v2 and MRCR. We also explicitly distinguish training seeds from evaluation seeds for fixed models. Results appear in Section 8.3 and Appendices P and Q.

4. Combination with synthetic long-document construction. We compared randomly sampled and LongPPL-selected natural documents as root documents for NExtLong under a Qwen3-8B-Base, 256K, 2B-token setting. Root selection increases average RULER across six evaluation lengths from 83.15 to 83.72, and the 256K score from 65.65 to 66.42. This preliminary result supports combining natural-document selection with synthetic construction. See Appendix R.

5. Analysis of selected-document attributes. We sampled 1,000 documents from each of the front, tail, scattered, full, and random groups, comparing language, topic, and structure. These results are reported in our responses to Reviewers 3LXg and ruWm. We distinguish descriptive differences from causal explanations and avoid attributing similar window performance directly to particular document attributes.

6. Data-volume and computational-cost accounting. We report retained-pool sizes, training usage, reference epochs, and sample accounting for mixed training and SFT. We also compare scoring costs with training costs, distinguishing direct timing measurements from scale extrapolations. These details are provided in Appendix S.

Our core contribution is to establish the practical utility of natural-long-document selection pipelines through controlled downstream training, and to validate a simple modification that reduces scoring costs. After actively engaging in discussions with the reviewers and incorporating the improvements listed above, reviewer sHtT raised their rating from 2 to 3, and reviewer ruWm maintained their positive rating.

We sincerely thank all Reviewers, ACs, and SACs for their time and constructive feedback, which have helped make the paper better supported and more clearly presented.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.