- Rating
- 2
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LDIS, a long-context training method that interleaves teacher-generated question–answer pairs with source documents and separately normalizes document and answer losses. Experiments show improvements in long-context performance under matched student training token budgets, across architectural adaptation and context extension settings.
Critical strengths
-
The motivation is clear, and the method is simple and intuitive. It converts implicit, sparse learning signals in natural documents into explicit answer-prediction targets.
-
The ablation studies are well designed, comparing interleaved versus end-of-document QA placement, global versus separate answer-loss normalization, and different learning rates. These experiments support the proposed design choices and suggest that the gains are not merely attributable to a favorable learning-rate setting.
Critical weaknesses
-
The paper does not sufficiently establish that the synthesized QA pairs require long-range information. If many questions can be answered using local context or general knowledge, the gains may primarily reflect additional high-quality QA supervision rather than stronger learning of long-range dependencies.
-
The empirical advantage over existing data synthesis approaches remains unclear. Comparisons primarily involve unaugmented ProLong data and internal ablations, without direct evaluation against established synthetic-data baselines.
-
Substantial capability gaps remain after adaptation. On the paper’s 256K evaluation suite, the original Qwen3.5-4B achieves an average score of 59.9, whereas the best LDIS-adapted variant reaches 39.7. Thus, LDIS only partially recovers the lost capability, leaving a considerable efficiency–quality trade-off.
-
The methodological novelty appears limited. Interleaving auxiliary supervision with source text builds on established ideas, and the paper could more clearly distinguish its technical contribution from prior instruction-augmented training and self-study approaches.
Questions for authors
-
Evaluate larger models and include direct comparisons with existing data synthesis methods under matched training budgets.
-
Extend the experiments beyond ProLong to determine whether the benefits generalize across source corpora. The exclusive reliance on ProLong currently limits the demonstrated scope of the contribution.
2 comments
Thank you to the authors for the detailed response and clarifications. In light of the rebuttal, I have raised my Informativeness score, while keeping my Rating unchanged.
Response to Reviewer k7fV
We appreciate the thorough assessment. We ran eight new runs on the mechanism, the missing baselines and the dependence on ProLong. All use the Table 1 recipe (Qwen3.5-4B-SWA-2FA, 64K stage, 7.0B tokens, same hyper-parameters). All QAs come from the Qwen3.5-4B teacher, with the ProLong-IS-Qwen loss. Tables 9 and 10 and Appendix C of the revised PDF report them.
All table, figure and section numbers below refer to the revised PDF.
W1. QA supervision or long-range memory?
W2 / Q1. Synthetic-data baselines
W3. Gap to the original Qwen3.5
We agree that a gap remains. A sweep over the number of full-attention layers would give a fuller view of the efficiency–quality trade-off, but it is not the primary target of this study: we fix an efficient architecture (2–3 of 8 full-attention layers) and ask how much of its lost long-context ability the training data can recover. Share of the gap from the unadapted efficient model to the original that each recipe closes:
LDIS is a data method, complementary to more full-attention layers or tokens. In exchange, the efficient model decodes 1.8x faster at 64K and keeps 3.2x less cached state per sequence (Appendix I).
W4. Novelty relative to instruction-augmented training and self-study
Q1. Larger models, and comparisons under matched budgets
Q2. Other data sources
On Longmino (Dolma 3), LDIS gives +4.9 Avg and +5.9 RULER@64K. The gain is smaller than on ProLong because fewer rows carry QAs. In Longmino, only rows built around a document of at least 32K tokens qualify: 20% of rows, against 63% in ProLong.
We hope these results address your concerns and are happy to discuss further.