- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Layer Agreement for Target Reformulation (LaTR), a method for improving on-policy distillation (OPD) by reformulating the teacher's target distribution using intermediate-layer information. Rather than directly matching the teacher's final-layer predictions, LaTR measures whether the teacher's and student's intermediate-to-final probability changes agree with the correction direction indicated by their final-layer probability gap. This layer-agreement signal is used to adjust teacher logits, construct a modified Top-K target distribution, and perform confidence-weighted reverse-KL distillation. Experiments on three Qwen student models across six mathematical reasoning benchmarks and LiveCodeBench-v6 show consistent improvements over Top-K OPD.
Critical strengths
-
Reformulating the distillation target using teacher-student layer agreement is an interesting direction beyond conventional token selection or loss weighting.
-
LaTR consistently outperforms the baselines across three student models on mathematical reasoning and code-generation benchmarks.
Critical weaknesses
-
The motivation for using intermediate-to-final probability changes as indicators of supervision reliability could be further clarified. While the paper provides empirical evidence linking layer agreement to lower teacher-student divergence, it remains unclear why a student's internal probability revision in the teacher-preferred direction should imply that stronger supervision is beneficial. Conversely, a student whose internal revision opposes the teacher's correction may require stronger, rather than weaker, supervision. Further discussion or experiments connecting these revision patterns to actual learning outcomes would strengthen the motivation.
-
The proposed reliability score and target construction require further justification. The asymmetric formulation in Eq. (3), particularly the different treatment of teacher-only and student-only support, appears heuristic and lacks sufficient justification.
-
The OPSD baseline raises concerns about implementation fidelity and fairness. OPSD performs substantially worse than Top-K OPD on LiveCodeBench-v6. Given the performance reported in prior SDPO studies, the current performance is questionable.
Questions for authors
Please check the weaknesses above
Minor:
- The term "Uniform" in Figure 6(e) and Appendix K may be misleading, as this baseline replaces intermediate-layer predictions with uniform distributions to retain the final-layer predictions. Consider renaming it to "Final-Layer Only" to better clarify the setting.
4 comments
I appreciate the rebuttal. Although empirically observed, I am still not very convinced why and how does direction-aware fusion architecture work. Do the authors have any insight into this?
Response to Reviewer igae
Thank you for the follow-up. Our reasoning proceeds from what intermediate layers reveal, to why teacher–student compatibility matters, and finally to how these insights motivate LaTR’s correction rule. Intermediate predictions expose how token preferences develop, which we use to assess whether teacher guidance is supported by both models’ current predictive behavior.
1. Intermediate supervision constrains how predictions are formed. Patient Knowledge Distillation transfers information from multiple hidden layers, while OPRD aligns teacher and student representations on student-generated trajectories. OISD further supervises intermediate predictions and attention using the final layer. These studies report learning benefits from supervising internal representations or predictive behavior beyond final outputs ([1], [2], [3]). They motivate using intermediate information, without implying that layer alignment recovers a complete reasoning process.
2. Layer revisions reveal how token preferences develop. DoLa and SLED show that differences between intermediate and final predictions can improve output selection ([4], [5]). This motivates interpreting a probability revision as evidence of whether subsequent computation strengthens or suppresses a candidate. LaTR asks whether that revision supports the correction required by the final teacher–student gap. Agreement between the models alone is insufficient, since both could revise a token against the required correction.
3. Knowledge transfer also depends on compatibility. Rethinking OPD reports that successful transfer depends on both new teacher capabilities and compatible teacher–student generation patterns ([6]). Inspired by this distinction, we retain the teacher as the knowledge source and use the student’s revision as a local compatibility cue.
4. LaTR combines teacher knowledge with evidence of current compatibility. The final teacher–student gap specifies the required correction, while both models’ layer revisions indicate whether their predictive refinement supports that direction. When both support it, LaTR adds reinforcement because the teacher’s guidance is corroborated by both its own refinement and the student’s current predictive behavior. When only the teacher supports it, LaTR preserves the original teacher score because student opposition may reflect knowledge not yet acquired, rather than unsuitable guidance. When the teacher’s own revision opposes the correction, LaTR attenuates it because the teacher’s final preference relative to the student is not corroborated by its own refinement. The resulting rule preserves opportunities to learn new teacher knowledge while using compatibility evidence to determine additional reinforcement or attenuation. These bounded adjustments reshape candidate selection and relative target probabilities. The direction and asymmetry interventions in Section 4.4 and Appendix K support this interpretation in the evaluated settings.
[1] Sun et al. Patient Knowledge Distillation for BERT Model Compression. EMNLP-IJCNLP, 2019.
[2] Yang et al. OPRD: On-Policy Representation Distillation. arXiv:2606.06021, 2026.
[3] Liu et al. OISD: On-Policy Internal Self-Distillation of Language Models. arXiv:2605.29089, 2026.
[4] Chuang et al. DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models. ICLR, 2024.
[5] Zhang et al. SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Models. NeurIPS, 2024.
[6] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
If any aspect remains unclear, we would be happy to provide further clarification.
Response to Reviewer igae
Dear Reviewer igae,
Thank you again for the time and care you have put into reviewing our submission. Your constructive feedback has helped us strengthen the paper.
With less than 8 hours left in the discussion period, we wanted to send a brief follow-up. For your question on the mechanism of our direction-aware fusion architecture, we explained it via intermediate-layer behaviors, teacher-student compatibility and LaTR’s correction rule, with supporting ablation results in Section 4.4 and Appendix K. We also grounded our reasoning in related works on intermediate supervision and layer-wise logit revision.
We would be grateful if you could let us know whether our responses have addressed your concerns. If any questions remain, we would be happy to discuss them.
Best regards,
The Authors
Response to Reviewer igae
Thank you for the constructive feedback. We have addressed all comments in the revised manuscript and responses below.
All revisions have been incorporated into the updated PDF, and all responses and references below refer to this revised version.
To test whether layer-revision patterns help determine beneficial target-correction directions, we added controlled interventions in Section 4.4, Table 3, with full settings and benchmark results in Appendix K, Table 19.
The decreases connect correction direction to downstream learning outcomes. They support reinforcing jointly supported corrections and show that uniformly strengthening corrections under student opposition is less effectivethan LaTR's evidence-dependent treatment. LaTR adjusts candidate-level target probabilities, preserving teacher guidance when the teacher supports the correction and attenuating it when the teacher's own revision opposes it. These results support the proposed rule under the tested settings.
To test whether teacher-only and student-only support warrant different correction rules, we added two targeted controls in Section 4.4, Table 3, with full settings and results in Appendix K, Table 19. Each changes only the corresponding one-sided treatment.
The decreases support preserving teacher guidance when only the student disagrees and attenuating corrections opposed by the teacher's own layer evidence. These controlled comparisons provide empirical justification for distinguishing the source of disagreement in Eq. (3)'s asymmetric target construction.
The asymmetry reflects the teacher’s role as the supervision anchor. Student opposition may reflect knowledge not yet acquired, whereas teacher-side opposition indicates that the teacher’s own revision does not support the required correction.
Thank you for raising this concern. We identified an error in our previous code evaluator: correctness was determined using a single public example rather than the complete test suite, inflating scores and compromising the comparison.
We corrected the evaluator and re-evaluated all methods under a common protocol. Appendix R.2 specifies the tool-assisted generation with 32 responses per problem. A response now counts as correct only if it passes all public and hidden tests using the official execution checker.
The corrected Table 1 reports:
We adopted this suggestion and renamed the control "Final-Layer Only" in the revised Figure 7(e), formerly Figure 6(e), and the corresponding tables and discussion in Section 4.4 and Appendix K. The description explicitly states that final predictions are retained while intermediate readouts are replaced with uniform distributions.