acceptodds
Under review as a conference paper at ICLR 2027

ComposeLong: Self-Supervised Composition of Long-Context Training Data from Short Texts

Abstract

Effective long-context training for large language models (LLMs) requires long training sequences with meaningful dependencies across distant parts of the context. Constructing such data presents two fundamental challenges: naturally occurring long documents are scarce, and document length alone does not guarantee useful long-range dependencies. Existing approaches address these challenges either by synthesizing long sequences from short texts or through quality-based filtering of existing long documents. However, both directions rely on hand-designed rules or criteria: synthesis methods do not learn from the structure of real long documents, while filtering methods retain only a fraction of the already limited supply of naturally occurring long documents. We introduce ComposeLong, a self-supervised framework that learns how to construct long-context training data using real long documents with strong long-range dependencies as supervision. Once trained, ComposeLong applies the learned patterns of document organization to large collections of short documents, synthesizing long-context training data without generating new text. ComposeLong outperforms prior synthesis methods on most reported metrics across RULER and LongBench v2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context LLM, Training data synthesis, Document composition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1DEiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ComposeLong, a self-supervised framework for constructing long-context pretraining data from short texts. It trains an embedding-based LLM composer to recover the correct continuation and ordering of chunks from real long documents, mixed with distractors. The learned composer then selects and orders chunks retrieved from short-document corpora, assembling approximately 128K-token training sequences while preserving the original text. Its central contribution is using real document organization as reusable supervision for data composition.

Validation uses continued pretraining of Qwen2.5-3B under matched downstream training budgets, followed by RULER evaluation and shared instruction tuning for LongBench v2. ComposeLong improves average RULER performance from the strongest baseline’s 71.91 to 73.31, with gains at every evaluated context length. LongBench v2 overall accuracy increases from 28.23% to 29.42%. Ablations examine learned ordering, history corruption, supervision filtering, retrieval, and composition granularity; the results particularly support the importance of learned ordering and robustness to imperfect selection histories.

Critical strengths

The framework turns scarce long documents into reusable supervision for data construction. ComposeLong learns chunk selection and ordering from existing document structure, then applies that knowledge to abundant short-text corpora. It requires no manual composition annotations and preserves the original text during synthesis. This provides a practical way to expand long-context training data while moving beyond fixed retrieval and concatenation rules.

Critical weaknesses

W1. The reported improvements are modest, and their stability is not established. ComposeLong improves average RULER performance by 1.40 points over LiteLong and LongBench v2 accuracy by 1.19 percentage points, approximately six additional correct answers among 503 examples. The reported results provide neither repeated training runs nor uncertainty estimates, and downstream evaluation uses only Qwen2.5-3B. Consequently, the evidence does not establish whether these advantages reliably exceed training variability and evaluation uncertainty. This is consequential because improved downstream performance is the principal justification for the additional composer-training pipeline.

W2. The claimed transfer of useful long-range structure remains insufficiently validated. The composer learns to reconstruct document membership and ordering, which may capture topical consistency, style, or local coherence without establishing dependencies useful to the target model. The paper does not directly connect this reconstruction objective to target-model-specific long-context learning benefits. Moreover, removing LongFilter-selected supervision changes average RULER performance only from 73.31 to 73.28, providing little evidence that selecting documents with stronger measured long-range dependencies drives the gains. The results support learned ordering, but leave the central explanation and its transferability across target models under-supported.

Questions for authors

Q1. Were the results repeated across random seeds for data synthesis and downstream training? What are the resulting means and variances, and do paired tests on LongBench v2 support a reliable improvement given the approximately six additional correct answers?

Q2. Do the benefits extend across different capabilities, such as information retrieval, cross-document reasoning, and code understanding, or are the aggregate gains concentrated in particular tasks? Which claims of broader task transfer are supported by the current evaluation?

Q3. Which model computes the LongFilter scores? What evidence establishes that dependencies identified by this scorer, and ordering learned by the Qwen3-4B composer, provide useful training signals for Qwen2.5-3B? Is there evidence supporting reuse across target models beyond this single configuration?

Q4. What is the total cost of embedding construction, composer training, and synthesis relative to the baseline pipelines? Can the trained composer and synthesized corpus be reused unchanged across target models and tasks, or does adaptation introduce additional costs that affect the practical significance of the modest performance gains?

Reviewer_Zo9WOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ComposeLong, a data synthesis method that synthesizes training datasets for long-context training from short-context datasets. The paper employs a self-supervised learning approach to construct a model that learns contextual dependencies within high-quality long-context data, with the aim of synthesizing short texts into long-text data. ComposeLong was used to synthesize approximately 4 billion tokens of training data and outperformed other baselines for long-text synthesis on two downstream benchmarks, RULER and LongBench V2.

Critical strengths
  1. The paper explores an important and practical issue: the training data required for pre-training on long texts.

  2. The techniques used in the paper are novel to me.

  3. The paper is fairly easy to follow.

Critical weaknesses
  1. The necessity of using synthetic data in long-context training is unclear. Although high-quality long-context data is difficult to obtain, it is unclear whether ComposeLong data or low-quality data that is simply longer in length would be more suitable for long-context pretraining.

  2. The paper utilized composed data exclusively for all experiments on long-context pretraining; however, this experimental setup does not correspond to the conditions encountered in actual pretraining. High-quality long-context data remains the best data for this stage, whilst it is unclear how composed data should be utilized in practical long-context training scenarios. This renders the paper's contribution to solving practical problems unclear.

Questions for authors
  1. How should I use Composed data in actual long-context pre-training?

  2. Why was the Qwen2.5 series model used as the baseline? It does not seem to be a very new model.

  3. As long-context pretraining typically requires a large amount of data, I look forward to seeing the authors discuss the computational efficiency of this method. Approximately how much computational resources are required to get 1 billion tokens?

  4. Throughout the paper, I have come across different terms such as "short-context" and "short-text" to describe what appears to me to be the same thing. Do they mean the same thing? If so, I suggest using a consistent term.

Reviewer_i7hXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses genuine long documents curated by LongFilter for self-supervision, training a PE-Rank–style composer (Qwen3-Embedding-0.6B → MLP projector → Qwen3-4B, with a ListMLE objective plus history corruption) to learn "chunk selection + ordering." At inference time, for short-text corpora it retrieves candidates with document-level BM25 and composes ~128K-token documents via per-chunk anchors. In downstream Qwen2.5-3B CPT for 500 steps: RULER avg 73.31 vs. LiteLong 71.91 / EntropyLong 70.52 / NExtLong 66.73; LongBench v2 overall 29.42.

Critical strengths

The overall related-work survey in the paper is fairly comprehensive, and the relevant baseline methods have all been reproduced.

Critical weaknesses
  1. The paper evaluates on very few benchmarks, and the ablation study uses only RULER as the observation metric. Even in the main experiments, it can be seen that the trends of RULER and LongBench v2 are not consistent.
  2. The base model used in this paper is single and limited in size, and the reported results show a considerable gap compared with those reported in the NextLong and EntropyLong papers. We suggest scaling to larger models to observe the effectiveness of the proposed method.
Questions for authors
  1. I have a strong concern that the authors did not actually reproduce the relevant baseline methods. In this paper, the retrieved document pool is very large and is shuffled, whereas NextLong already revealed that retrieval with more fine-grained (more detailed) chunks yields better performance.
  2. The authors assume that there exists an ordering among the retrieved chunks; however, this does not seem to make sense, since similar retrieved documents are inherently unordered. This is also suggested by the fact that LongFilter does not take effect.
  3. I would like to know how many additional GPU hours the proposed method consumes compared with the other baseline methods.
  4. How should the authors interpret the inconsistent trends between RULER and LongBench v2? How does the proposed method perform on more benchmarks, such as HELMET, MRCR, or LongBench-Pro?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.