- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduce a 3 stage framework OPAL, to distill the full-attention teacher to linearization student. OPAL improve the OPD stage with data and LR strategies, including difficulty curriculum, length curriculum, and WSD lr schedule. OPAL reaches the SOTA performance on NIAH and mathematical reasoning compared with prior linearization methods.
Critical strengths
The comprehensive experiment on both NIAH and math reasoning, reaching high recovery rate from 83.4% to 100.7%, and outperforming prior works with a large margin.
Critical weaknesses
- Ablation study is not enough. The paper does not include ablation on length curriculum, difficulty curriculum and WSD schedule to isolate the effect of their contribution.
- Contribution need clarify. From my perspective, the length curriculum[1], difficulty curriculum[2,3] and WSD schedule have been proposed in privous works. The authors should clarify their contribution over previous works.
[1]Zhang Y, Chai J, Fu Y, et al. Are Full Rollouts Necessary for On-Policy Distillation?[J]. arXiv preprint arXiv:2605.31490, 2026.
[2]Jia Y, Zhang C, Diao X, et al. What makes a good curriculum? disentangling the effects of data ordering on llm mathematical reasoning[C]//Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026: 34472-34488.
[3] Gao Z, Kim J, Sun W, et al. Prompt curriculum learning for efficient llm post-training[C]//International Conference on Learning Representations. 2026, 2026: 58614-58652.
Questions for authors
- Are the baselines trained on the same dataset with OPAL? It seems that the authors use the publicly release model without further training, and if so, it will make it hard to compare methods on the math reasoning task. I suggest the authors compare methods with matched training data.
- The stage 3, Prolonged On-Policy Distillation, perform OPD over long sequences, however, the context length in NIAH is up to 32K, while HypeNet reports results on 256K context length. Could the author further provide the result on longer context rather than 32k?
- Regarding the question3, actually I think applying OPAL on long context task and large model is hard, since we always need a full-attention model as a teacher to compute the logits over long context, especially in OPD, you need to compute it for each rollout, making it inefficient. Could the author further discuss this question?
3 comments
Thank you for the detailed response and the additional experiments. The new results address most of my concerns and demonstrate strong performance. I’m raising my rating from 2 to 3 and lowering my confidence score.
Thank you so much for your response! Please let us know if there are any other concerns you would like us to address.
Ablation Depth and Explanation of Novel Components
Thank you for your constructive comments. We aim to address your feedback.
W1. Ablation Depth
W2. Contribution Over Previous Works
Thank you for posing this question. We highlight how our method differs from prior work.
We agree aspects of the recipe exist in other forms, but these elements are new and, based on our results, quite effective when applied to the problem of linearization.
Q1. Baseline Comparison
The baselines are not trained on the same dataset as OPAL. However, we have two responses.
Q2. Longer Context Evaluation
Our experiments were initially structured around Qwen3-4B which has a context length of 32K. However, we've since added additional experiments that demonstrate that the method generalizes even in regimes where the model is forced to locate information outside of its native context and training generation length using YaRN when necessary.
Q3. Efficiency Analysis
First, we would like to clarify that using a teacher model to compute logits over long context is necessary for any distillation pipline, including off-policy KD. The additional cost of OPD is specifically rollout generation, which we provide a wall-clock control for (see our response to Reviewer jW6Q for additional details).