- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes On-Policy Attention Self-Distillation (OPASD), which extends OPSD by introducing attention-level distillation. The method uses attention distributions from a teacher conditioned on the reference solution to provide additional supervision for the student.
Experiments on Qwen3-1.7B, 4B, and 8B across AIME 2024, 2025, 2026, and HMMT 2025 show that OPASD improves mathematical reasoning accuracy over the original OPSD, while producing shorter responses, reducing training compute, and achieving more stable training.
Critical strengths
1. Good experimental results and ablation studies.
OPASD consistently outperforms the original OPSD across three Qwen3 model sizes. The paper also includes ablation studies on attention loss functions and distillation depth to examine the effects of different design choices.
2. Improved training stability and efficiency.
Compared with the original OPSD, OPASD alleviates the increase in reasoning length and the decline in accuracy during training, while also reducing training compute costs.
Critical weaknesses
- Insufficient comparisons with related methods.
The paper mainly compares OPASD with the original OPSD. Although the authors provide several ablation studies, comparisons with existing on-policy representation or attention distillation methods, such as OPRD and OISD, are missing. This makes it difficult to assess the advantages of the proposed attention distillation approach over related methods.
Questions for authors
Q1. Comparison with related methods.
Could the authors compare OPASD with existing on-policy representation or attention distillation methods, such as OPRD and OISD? If a direct comparison is not feasible due to differences in training setups, could the authors clarify how OPASD differs from these methods and what advantages it offers?
Q2. Generalizability of training efficiency.
The reported training speedup is interesting. Since the experiments use OpenThoughts-Math-30K for training, could the authors discuss whether the reduction in response length and training compute would also hold on other training datasets?
7 comments
Response on Comparisons with Related Distillation Methods (Part 2/3)
Comparison with related methods. Could the authors compare OPASD with existing on-policy representation or attention distillation methods, such as OPRD and OISD? If a direct comparison is not feasible due to differences in training setups, could the authors clarify how OPASD differs from these methods and what advantages it offers?
We have expanded the evaluation to include SDFT [1], OPRD [2], and OISD [3]. We compare them with OPASD using Qwen3-4B on AIME24, AIME25, AIME26, and HMMT25 under comparable training and evaluation settings. As shown below, OPASD achieves the highest accuracy on all four benchmarks. On average, it improves over OPSD by 8.40 percentage points, SDFT by 2.29 points, OPRD by 3.05 points, and OISD by 4.96 points.
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | Avg. |
|---|---|---|---|---|---|
| OPSD | 51.11 | 43.88 | 46.11 | 28.33 | 42.36 |
| OPRD | 58.61 | 47.78 | 52.50 | 31.94 | 47.71 |
| SDFT | 60.83 | 48.89 | 53.05 | 31.11 | 48.47 |
| OISD | 58.06 | 46.68 | 50.27 | 28.16 | 45.80 |
| OPASD | 63.33 | 50.27 | 55.56 | 33.89 | 50.76 |
OPSD and SDFT distill next-token predictions from teachers conditioned on a reference solution and an expert demonstration, respectively. OPRD matches teacher and student hidden states along student-generated trajectories. The teacher’s attention provides direct guidance on where to attend, a signal that complements the prediction or representation targets used by these methods. OISD uses the model’s final layer to supervise next-token predictions and attention at an intermediate layer, using the problem and generated response as input. OPASD conditions the teacher on the privileged solution, projects its attention onto student-visible tokens, and distills that attention alongside next-token predictions.
References
[1] Shenfeld, Idan, et al. "Self-distillation enables continual learning." ICML 2026
[2] Yang, Shenzhi, et al. “OPRD: On-Policy Representation Distillation.” arXiv preprint arXiv:2606.06021 (2026).
[3] Liu, Xinyu, et al. “OISD: On-Policy Internal Self-Distillation of Language Models.” Findings of EMNLP 2026.
Response on Training Efficiency Gains Beyond OpenThoughts-Math-30K (Part 3/3)
Generalizability of training efficiency. The reported training speedup is interesting. Since the experiments use OpenThoughts-Math-30K for training, could the authors discuss whether the reduction in response length and training compute would also hold on other training datasets?
To assess whether the efficiency gains extend beyond OpenThoughts-Math-30K, we conducted additional experiments with Qwen3-4B using DeepMath-103K [1] as the training dataset. We compared OPSD and OPASD in response length, estimated model compute, wall-clock time, and reasoning accuracy.
On DeepMath-103K, OPASD maintains short responses and stable rollout accuracy throughout training, while OPSD’s responses grow longer and its rollout accuracy declines. OPASD guides the student’s attention toward the context emphasized by the solution-conditioned teacher, helping it preserve relevant information and avoid repeating earlier reasoning steps. OPASD generates 61.7% fewer response tokens than OPSD, reducing estimated model compute by 59.9% and cutting wall-clock training time from 5.20 to 2.83 hours, a 1.84× speedup. Peak allocated GPU memory also falls by 56.9%. OPASD improves accuracy on all four benchmarks, with an average gain of 5.56 percentage points over OPSD. These results show that OPASD’s training efficiency gains extend beyond OpenThoughts-Math-30K to DeepMath-103K.
| Method | AIME24 | AIME25 | AIME26 | HMMT25 | Avg. |
|---|---|---|---|---|---|
| OPSD | 56.38 | 47.50 | 49.44 | 29.16 | 45.62 |
| OPASD | 61.94 | 55.00 | 53.89 | 33.89 | 51.18 |
References
[1] He, Zhiwei, et al. "DeepMath-103K: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning." ICLR 2026
Summary of Responses to Reviewer wxJ2's Comments (Part 1/3)
We thank the reviewer for the thoughtful and encouraging feedback. We appreciate their recognition of OPASD’s consistent accuracy gains over OPSD across three Qwen3 model sizes, the ablations on attention loss and distillation depth, and the improvements in training stability and efficiency through shorter responses, more stable accuracy, and lower training compute.
Summary of response