- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces ComposeLong, a self-supervised framework for constructing long-context pretraining data from short texts. It trains an embedding-based LLM composer to recover the correct continuation and ordering of chunks from real long documents, mixed with distractors. The learned composer then selects and orders chunks retrieved from short-document corpora, assembling approximately 128K-token training sequences while preserving the original text. Its central contribution is using real document organization as reusable supervision for data composition.
Validation uses continued pretraining of Qwen2.5-3B under matched downstream training budgets, followed by RULER evaluation and shared instruction tuning for LongBench v2. ComposeLong improves average RULER performance from the strongest baseline’s 71.91 to 73.31, with gains at every evaluated context length. LongBench v2 overall accuracy increases from 28.23% to 29.42%. Ablations examine learned ordering, history corruption, supervision filtering, retrieval, and composition granularity; the results particularly support the importance of learned ordering and robustness to imperfect selection histories.
Critical strengths
The framework turns scarce long documents into reusable supervision for data construction. ComposeLong learns chunk selection and ordering from existing document structure, then applies that knowledge to abundant short-text corpora. It requires no manual composition annotations and preserves the original text during synthesis. This provides a practical way to expand long-context training data while moving beyond fixed retrieval and concatenation rules.
Critical weaknesses
W1. The reported improvements are modest, and their stability is not established. ComposeLong improves average RULER performance by 1.40 points over LiteLong and LongBench v2 accuracy by 1.19 percentage points, approximately six additional correct answers among 503 examples. The reported results provide neither repeated training runs nor uncertainty estimates, and downstream evaluation uses only Qwen2.5-3B. Consequently, the evidence does not establish whether these advantages reliably exceed training variability and evaluation uncertainty. This is consequential because improved downstream performance is the principal justification for the additional composer-training pipeline.
W2. The claimed transfer of useful long-range structure remains insufficiently validated. The composer learns to reconstruct document membership and ordering, which may capture topical consistency, style, or local coherence without establishing dependencies useful to the target model. The paper does not directly connect this reconstruction objective to target-model-specific long-context learning benefits. Moreover, removing LongFilter-selected supervision changes average RULER performance only from 73.31 to 73.28, providing little evidence that selecting documents with stronger measured long-range dependencies drives the gains. The results support learned ordering, but leave the central explanation and its transferability across target models under-supported.
Questions for authors
Q1. Were the results repeated across random seeds for data synthesis and downstream training? What are the resulting means and variances, and do paired tests on LongBench v2 support a reliable improvement given the approximately six additional correct answers?
Q2. Do the benefits extend across different capabilities, such as information retrieval, cross-document reasoning, and code understanding, or are the aggregate gains concentrated in particular tasks? Which claims of broader task transfer are supported by the current evaluation?
Q3. Which model computes the LongFilter scores? What evidence establishes that dependencies identified by this scorer, and ordering learned by the Qwen3-4B composer, provide useful training signals for Qwen2.5-3B? Is there evidence supporting reuse across target models beyond this single configuration?
Q4. What is the total cost of embedding construction, composer training, and synthesis relative to the baseline pipelines? Can the trained composer and synthesized corpus be reused unchanged across target models and tasks, or does adaptation introduce additional costs that affect the practical significance of the modest performance gains?