acceptodds
Under review as a conference paper at ICLR 2027

LDIS: Efficient Long-Context Training Data Synthesis via Interleaved-Study

Abstract

Long-context modeling enables language models to process complete documents, code repositories, and extended interaction histories. However, natural text provides sparse supervision for long-range dependencies, while recent hybrid models further face additional constraints from the finite memory capacity of the recurrent blocks. We propose Long-context training Data synthesis via Interleaved Study (LDIS). A teacher generates question–answer (QA) pairs from growing document prefixes, which we then interleave with source text at the chunk boundaries. This changes the training prefix distribution across the sequence, adding targets that encourage finite-state blocks to retain and use earlier information. We normalize document and answer losses separately to prevent document tokens from diluting answer supervision. Under matched student training token budgets, LDIS improves long-context performance across model sizes and teacher sources when adapting full attention to sliding-window attention and extending context length, while largely maintaining general capability. Ablations support interleaving and the loss design. We also explore self-improvement without an external teacher, using an internally trained model to supervise a new student initialized from the same base checkpoint.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context data synthesis, long-context extension, fine-tuning, continual pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k7fVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LDIS, a long-context training method that interleaves teacher-generated question–answer pairs with source documents and separately normalizes document and answer losses. Experiments show improvements in long-context performance under matched student training token budgets, across architectural adaptation and context extension settings.

Critical strengths
  1. The motivation is clear, and the method is simple and intuitive. It converts implicit, sparse learning signals in natural documents into explicit answer-prediction targets.

  2. The ablation studies are well designed, comparing interleaved versus end-of-document QA placement, global versus separate answer-loss normalization, and different learning rates. These experiments support the proposed design choices and suggest that the gains are not merely attributable to a favorable learning-rate setting.

Critical weaknesses
  1. The paper does not sufficiently establish that the synthesized QA pairs require long-range information. If many questions can be answered using local context or general knowledge, the gains may primarily reflect additional high-quality QA supervision rather than stronger learning of long-range dependencies.

  2. The empirical advantage over existing data synthesis approaches remains unclear. Comparisons primarily involve unaugmented ProLong data and internal ablations, without direct evaluation against established synthetic-data baselines.

  3. Substantial capability gaps remain after adaptation. On the paper’s 256K evaluation suite, the original Qwen3.5-4B achieves an average score of 59.9, whereas the best LDIS-adapted variant reaches 39.7. Thus, LDIS only partially recovers the lost capability, leaving a considerable efficiency–quality trade-off.

  4. The methodological novelty appears limited. Interleaving auxiliary supervision with source text builds on established ideas, and the paper could more clearly distinguish its technical contribution from prior instruction-augmented training and self-study approaches.

Questions for authors
  1. Evaluate larger models and include direct comparisons with existing data synthesis methods under matched training budgets.

  2. Extend the experiments beyond ProLong to determine whether the benefits generalize across source corpora. The exclusive reliance on ProLong currently limits the demonstrated scope of the contribution.

2 comments
AuthorsOpenReview ↗

Response to Reviewer k7fV

We appreciate the thorough assessment. We ran eight new runs on the mechanism, the missing baselines and the dependence on ProLong. All use the Table 1 recipe (Qwen3.5-4B-SWA-2FA, 64K stage, 7.0B tokens, same hyper-parameters). All QAs come from the Qwen3.5-4B teacher, with the ProLong-IS-Qwen loss. Tables 9 and 10 and Appendix C of the revised PDF report them.

All table, figure and section numbers below refer to the revised PDF.

Training dataRULER@32KRULER@64KHELMETMRCR-2LB-V2-SAA-LCRAvg
ProLong (Table 3)75.860.726.832.833.620.534.9
ProLong + LDIS (ProLong-IS-Qwen, Table 3)89.174.834.534.340.024.041.5
Distant-only QAs81.764.631.032.837.923.037.9
Local-only QAs68.855.422.128.730.718.831.1
Untie the Knots76.461.027.831.031.719.834.2
Untie the Knots + LDIS86.378.333.435.841.521.842.2
NExtLong74.658.924.928.434.316.332.6
NExtLong + LDIS84.971.033.136.043.822.541.3
Longmino77.561.427.727.333.516.333.2
Longmino + LDIS83.167.330.235.037.920.038.1

W1. QA supervision or long-range memory?

  • What the QAs depend on (new Appendix C of the revised PDF). For 3,165 inserted QAs we measured how much of the teacher's answer likelihood is recovered from only the last W tokens. 64% need more than the 4K attention window to recover half of the gain (median distance 8K tokens), and 2.5% need no context.
  • Controlled runs. We train on 7% of the IS-Qwen QAs that are local (answerable from the last 4K tokens) versus 7% that are distant, matched in QA count, answer tokens and insertion positions (details in Appendix C.2).
  • Result. Distant-only QAs reach 37.9 Avg; local-only QAs reach 31.1 and do not improve on ProLong, so QA supervision by itself is not the source of the gain. The RULER gap is +0.7 / +3.9 / +8.5 / +12.9 / +9.3 at 4K / 8K / 16K / 32K / 64K.

W2 / Q1. Synthetic-data baselines

  • Untie the Knots and NExtLong, on the same ProLong data and budget, do not improve the ProLong average (34.2 and 32.6 vs 34.9).
  • With LDIS added they reach 42.2 and 41.3 (+8.0 and +8.7), with no new QA generation. LDIS is complementary to them.

W3. Gap to the original Qwen3.5

We agree that a gap remains. A sweep over the number of full-attention layers would give a fuller view of the efficiency–quality trade-off, but it is not the primary target of this study: we fix an efficient architecture (2–3 of 8 full-attention layers) and ask how much of its lost long-context ability the training data can recover. Share of the gap from the unadapted efficient model to the original that each recipe closes:

  • 3FA, 256K suite: ProLong 37%, IS-GPT 49%.
  • 3FA, 64K suite (256K stage): ProLong 43%, IS-Qwen 54%.

LDIS is a data method, complementary to more full-attention layers or tokens. In exchange, the efficient model decodes 1.8x faster at 64K and keeps 3.2x less cached state per sequence (Appendix I).


W4. Novelty relative to instruction-augmented training and self-study

  • Instruction Pre-Training generates instruction–response pairs from a whole text and appends them after it, as IN2 also places the question after the context. LDIS differs in what each QA sees and where it goes: it is generated from the prefix alone and inserted at that point, at many depths of one long document, so its answer depends on content read earlier in the sequence, often beyond the attention window (W1).
  • Cartridges uses self-study to train a compact KV cache for one corpus, with the model frozen. LDIS brings the idea to training the model's own parameters, with answer supervision normalised separately from document next-token prediction.
  • Ablations of the two LDIS choices: moving the same QAs to the end of the row, as in Instruction Pre-Training and IN2, costs 10.3 points at RULER@32K (Table 7), and averaging the loss globally costs 9.2 (Table 8).

Q1. Larger models, and comparisons under matched budgets

  • Larger models: we could not train a model above 4B within the rebuttal period. Over the sizes we have, the gain does not shrink: +5.9 / +5.4 / +6.6 Avg at 0.8B / 2B / 4B (64K stage, Fig. 4), and it holds for Nemotron-H (Table 5).
  • Matched-budget comparisons: the Untie the Knots and NExtLong runs start from the same ProLong data and use the same recipe and 7.0B-token budget as ProLong and ProLong-IS-Qwen.

Q2. Other data sources

On Longmino (Dolma 3), LDIS gives +4.9 Avg and +5.9 RULER@64K. The gain is smaller than on ProLong because fewer rows carry QAs. In Longmino, only rows built around a document of at least 32K tokens qualify: 20% of rows, against 63% in ProLong.


We hope these results address your concerns and are happy to discuss further.

Thank you to the authors for the detailed response and clarifications. In light of the rebuttal, I have raised my Informativeness score, while keeping my Rating unchanged.

Reviewer_YntCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LDIS, which inserts teacher-generated question–answer pairs throughout long documents to improve long-context training. It separately normalizes document and answer losses to give more weight to answer supervision. Experiments on Qwen3.5 and Nemotron show improvements over training on the original ProLong data.

Critical strengths
  1. The paper addresses an interesting problem. Long documents alone may not provide enough supervision for learning long-range dependencies, and inserting questions throughout the document is a reasonable way to address this.

  2. The experiments cover different model sizes, teachers, and long-context benchmarks. The ablations also show benefits from interleaving QAs and separately weighting the answer loss.

Critical weaknesses
  • It is unclear whether the gains come from better long-range memory or additional QA supervision. The paper does not measure how often the generated questions require distant information. Earlier inserted answers may also bring relevant information closer to later questions. Although the placement ablation is useful, more controlled analysis is needed to support the proposed explanation.

  • The total cost is not fully evaluated. The experiments match student training tokens, but generating and filtering QAs requires additional teacher computation. The reported inference speedups come from changing the attention architecture. They do not establish the efficiency of the data-generation method, which is a main claim of the paper.

Questions for authors

How many generated questions require information outside the sliding window, and how far is that information from the answer position?

In the end-only QA ablation, are the retained QA pairs and the amount of original document text matched after truncation?

What is the total cost of generating and filtering the QAs, and how does the method compare with additional ProLong training under the same overall compute budget?

3 comments
AuthorsOpenReview ↗

Response to Reviewer YntC

Thank you for the precise questions. To test whether the gain comes from long-range dependencies or from QA supervision as such, we added a dependency analysis and two controlled training runs. All new runs use the Table 1 recipe (Qwen3.5-4B-SWA-2FA, 64K stage, 7.0B training tokens, same hyper-parameters, seed and evaluation harness).

All table, figure and section numbers below refer to the revised PDF.


W1 / Q1. Long-range dependence of the QAs

Dependency analysis (new Appendix C of the revised PDF). For 3,165 inserted QAs we measured how much of the teacher's answer likelihood is recovered when only the last W tokens before the question are visible. Earlier inserted QAs inside that window are included. Concretely, the gain of a QA is how much the teacher's answer NLL drops when the full prefix is visible instead of no context, i.e. NLL0−NLLfull\mathrm{NLL}_0 - \mathrm{NLL}_{\text{full}}. The fraction of this gain recovered by the last W tokens is g(W)=(NLL0−NLLW)/(NLL0−NLLfull)g(W) = (\mathrm{NLL}_0 - \mathrm{NLL}_W)/(\mathrm{NLL}_0 - \mathrm{NLL}_{\text{full}}).

Share of QAs (%) whose answer gain is recovered from the last W tokens:

RecoveredClosed-book≤1K≤2K≤4K≤8K≤16K≤32KMedian W
half of the gain2.5%17.625.633.653.471.283.08K
90% of the gain2.5%5.58.711.325.742.757.932K
  • The "Recovered" column gives the threshold on g(W)g(W): a QA counts as recovered within W once g(W)≥0.5g(W) \ge 0.5 (half of the gain) or g(W)≥0.9g(W) \ge 0.9 (90% of the gain).
  • 64% of QAs need more than the 4K attention window to recover half of the gain (86% for 90% of it), and only 2.5% can be answered with no context.
  • Earlier inserted QAs shorten the dependency for only 11% of QAs. They do not turn distant questions into local ones.

Controlled runs (Table 9 and Appendix C.2 of the revised PDF). We train on 7% of the IS-Qwen QAs that are local (answerable from the last 4K tokens) versus 7% that are distant, matched in QA count, answer tokens and insertion positions.

Training dataRULER@32KRULER@64KHELMETMRCR-2LB-V2-SAA-LCRAvg
ProLong (Table 3)75.860.726.832.833.620.534.9
ProLong + LDIS, all QAs (Table 3)89.174.834.534.340.024.041.5
Distant-only QAs81.764.631.032.837.923.037.9
Local-only QAs68.855.422.128.730.718.831.1
RULER4K8K16K32K64K
Distant-only97.495.488.481.764.6
Local-only96.791.579.968.855.4
Gap+0.7+3.9+8.5+12.9+9.3
  • Distant-only QAs beat local-only QAs by 6.8 Avg, and the RULER gap widens from 4K to 32K.
  • Local-only QAs do not improve on ProLong. QA supervision by itself is not the source of the gain.
  • Distant-only QAs, at 7% of the QA volume, recover 45% of the full IS-Qwen gain.

Q2. Is the End ablation matched?

Yes. The End variant is produced by reordering each interleaved training row: the same QAs and the same document tokens, with the QAs moved to the end of the row. The transform checks that the token multiset, the number of answer tokens and the row length are unchanged. Section 6 and Appendix E of the revised PDF now state this and say "end of the training row".

AuthorsOpenReview ↗

Response to Reviewer YntC (Continued)

W2 / Q3. Teacher cost

QA generation is a one-time, offline cost. It is paid once per corpus, not once per training run, and the resulting data is reused across runs and across models, which is how synthetic training data is generally used. In this paper alone:

  • Across model sizes: one QA set from the Qwen3.5-4B teacher trained the 0.8B, 2B and 4B students.
  • Across architectures: one QA set trained both the 2FA and 3FA students, and four further runs in this rebuttal, with no new generation.
  • Across model families: the GPT-teacher QAs generated for Qwen were re-tokenized on CPU and reused to train Nemotron-H.

For completeness, the measured cost, now reported in Appendix C.5 of the revised PDF (same GPU type for teacher and student; generation includes filtering):

ItemGPU-hours
QA generation for the rows one 7B-token run trains on (67K rows), once258
Student training, 7B tokens, per run292

We generated QAs for the whole eligible pool (165K rows, 633 GPU-hours), more than one run needs.

  • Worst case, no reuse: suppose a single run pays for generating every QA it trains on. LDIS then costs 550 GPU-hours in total: 258 for generation plus 292 for training on 7B tokens. Spending that compute on plain data instead means training ProLong on twice the tokens (7B at 64K, then 7B at 256K), which costs about 584 GPU-hours or more, yet it still scores lower on the 64K suite: 44.7 vs 47.5 for LDIS (3FA). Including generation, LDIS is ahead at lower total compute.
  • With reuse: the generation cost is shared by every run that uses the set, so its share per run shrinks with each additional model.

We hope these analyses answer your questions, and we would be glad to clarify anything further.

Thanks for the response. I will keep my score.

Reviewer_fMijOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper addresses long-context training for hybrid language models, which combine memory-efficient recurrent layers with a few attention layers and must compress long inputs into a fixed-size memory. The motivation is that ordinary text provides little training signal for using distant information, since most tokens can be predicted from nearby context. The proposed method, LDIS, uses a teacher model that reads each training document in growing chunks and, after each chunk, writes a question whose answer requires recalling earlier content. These question–answer pairs are inserted into the document where they were generated, and the model is trained on the augmented text. The answer loss is weighted separately, so the relatively few answer tokens are not drowned out by the document tokens. The method is evaluated in two settings: converting Qwen3.5 models to a faster variant in which most full-attention layers are replaced with sliding-window attention, and extending the context length of Nemotron-H-4B from 8K to 64K tokens.

Critical strengths

The proposed approach is simple and practical: it plugs into an existing long-context training pipeline, keeps the original data mixture, and requires no architectural changes, The experiments are comprehensive. They cover two model families, two training regimes (adapting a model to a cheaper attention design and extending its context length), three model sizes, and several teachers, including a closed model, a model from the same family, and a self-trained model. Evaluation spans five long-context benchmarks and seven general-capability benchmarks. The ablations are also informative.

Critical weaknesses
  1. The paper is hard to follow and poorly organized.
  • The abstract relies on terms it never explains, such as “chunk boundaries,” “recurrent blocks,” and “training prefix distribution,” and it assumes familiarity with hybrid models. The introduction uses “prefix-grounded QAs” before the concept is defined. Most importantly, the paper’s central claim is that LDIS changes the “prefix distribution” seen during training. That phrase recurs in the abstract, introduction, the interleaved study section, and conclusion, but it is never defined. -The method and the experimental setups both sit under “Preliminaries,” and setup details reappear in Sections 3.1 and 3.2, so readers must piece the experiments together from several places. Section 2.4 is a single sentence pointing to a figure.
  • The paper discusses related work in only a four-line paragraph in the introduction, without contrasting its novelty with respect to other methods.
  • The notation is heavy and overloaded. The formalism makes a simple method look more complex than it is.
  • The KL distillation variant is defined at the end of Section 2.2 but never appears in any result. The paper needs a major revision in terms of presentation.
  1. Missing baselines. ProLong is the only data baseline. The paper should compare against the synthetic long-context data methods it cites in the appendix, for example. Otherwise it is unclear whether LDIS improves on existing synthetic-supervision approaches or only on plain long-document training.

  2. Modest novelty. Inserting model-generated question–answer pairs into pretraining documents is mainly an empirical recipe. That raises the bar for baseline coverage, which the current experiments do not meet.

  3. Limited context lengths. Experiments are run only at 64K and 256K tokens, while current long-context models routinely target 1M tokens or more. It is therefore unclear whether LDIS’s gains persist at longer contexts. This matters because the cost of generating questions over ever-growing prefixes rises quickly with document length.

Questions for authors

Please address the weaknesses listed above

1 comment
AuthorsOpenReview ↗

Response to Reviewer fMij

We thank the reviewer for the careful reading and concrete suggestions. We revised the presentation substantially and added two published synthetic-data baselines. All new runs use the Table 1 recipe (Qwen3.5-4B-SWA-2FA, 64K stage, 7.0B training tokens, same hyper-parameters, seed and evaluation harness).

All table, figure and section numbers below refer to the revised PDF.


W1. Presentation

We agree, and have uploaded a revised PDF. All changed or new text is marked in blue. In detail:

  • "Prefix distribution" is now gone from the abstract, introduction and conclusion, which now say directly what interleaving changes: it adds prediction targets throughout the document that can only be predicted from earlier content. Section 3 keeps one sentence on it, with the definition (the contexts that precede supervised targets).
  • Abstract and introduction: "chunk boundaries" and "recurrent blocks" are now replaced by a plain description, one sentence says what a hybrid model is, and "prefix-grounded QAs" is defined before it is used.
  • Structure: now nothing sits under "Preliminaries" any more. The sections are now Introduction, Background and Related Work, Method, Experimental Setup, Results, Analysis, Conclusion, so the method, the setup and the results are each described in one place, and the one-sentence Section 2.4 is folded into the method.
  • Related work: now it is in the main text (Section 2), contrasting LDIS with prior methods (see W3).
  • Notation: we still keep the precise loss definition, which Ablation 2 builds on, but the abstract and introduction now describe the method without notation.
  • KL variant: the paper was previously unclear here. We explain as follows. The KL loss to the teacher's top-20 token distribution is used at the 64K stage with an open-weight teacher (ProLong-IS-Qwen, ProLong-IS-Nemotron). Every other run uses the hard-label answer loss. Both variants therefore appear in the main results, and both work. Section 3 now says so.

W2. Synthetic-data baselines

We further trained on two of the synthetic long-context data methods cited in our appendix, Untie the Knots and NExtLong, on the same ProLong raw data and the same training budget, each with and without LDIS augmentation (Table 10 and Appendix C of the revised PDF).

Training dataRULER@32KRULER@64KHELMETMRCR-2LB-V2-SAA-LCRAvg
ProLong (Table 3)75.860.726.832.833.620.534.9
ProLong + LDIS (ProLong-IS-Qwen, Table 3)89.174.834.534.340.024.041.5
Untie the Knots76.461.027.831.031.719.834.2
Untie the Knots + LDIS86.378.333.435.841.521.842.2
NExtLong74.658.924.928.434.316.332.6
NExtLong + LDIS84.971.033.136.043.822.541.3
  • Neither method improves the ProLong average in this setting (34.2 and 32.6 vs 34.9 Avg).
  • Adding LDIS gives +8.0 and +8.7 Avg (RULER@64K +17.3 and +12.1). No new QAs were generated: the IS-Qwen QAs stay attached to their 4K chunks.
  • LDIS is therefore complementary to these methods, not a variant of them.

W3. Novelty

The synthetic-data methods we cite form document-level training instances. NExtLong interleaves retrieved hard-negative chunks and trains with plain next-token prediction. Untie the Knots shuffles chunks and adds a task of recovering their order. Neither adds supervision about the document's content. LDIS instead keeps the document in order and inserts prefix-grounded QAs at many intermediate positions, with the answer loss normalised separately. Both choices have a measured positive effect: moving the same QAs all to the end of the row costs 10.3 points at RULER@32K (Table 7), and averaging the loss globally costs 9.2 (Table 8). Section 2 of the revised paper gives this contrast in more detail, method by method.


W4. Limited context lengths

  • Gains at longer contexts: the gain grows with evaluation length up to the longest we train (Fig. 3, Table 7; 256K results in Table 4).
  • Generation cost vs length: it grew about linearly in our runs, not faster: 13.8 GPU-seconds per 64K-token row and about 50 per 256K-token row (4× the length, 3.6× the cost). All insertion points of a row share one document prefix, which is cached rather than re-read, and our 256K recipe uses 8K instead of 4K chunks, so the number of generations grows more slowly than the document. The cost is paid once per corpus and reused across training runs and models.
  • 1M: we did not train at 1M. The practical limit is the teacher's own context length (256K for Qwen3.5), which is why we include the self-teacher variant (Table 6).

We hope the revision and the new baselines address your concerns, and we are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.