- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of relying solely on output-space reverse KL divergence in on-policy distillation (OPD). Existing OPD methods typically use a stopped-gradient student snapshot for rollouts and compute output reverse KL divergence against the teacher's next-token distribution on student-generated response prefixes. The authors argue that such output-level supervision primarily constrains token distributions but does not explicitly transfer structured relationships among response states, particularly the attention and representation geometry encoded by QKV relations. To address this limitation, the paper proposes Mini-OPD, which introduces a lightweight relational transfer term in addition to output KL. Specifically, the method aligns teacher and student QKV relations at a selected response position within a selected pair of response layers. The overall objective can be summarized as a combination of output-space distillation and process-space relational transfer, while restricting the additional supervision to a single response-layer pair and a single response position to reduce training overhead. The experimental evaluation compares Mini-OPD with relevant baselines, including OPD, OPRD-Bridge, and feature-projector-based methods, on mathematical reasoning tasks. The paper also employs common-response diagnostics and a full layer sweep to analyze cross-layer alignment patterns and sensitivity to layer selection.
Critical strengths
The paper accurately identifies a limitation of output-level OPD: it supervises the teacher's next-token distribution without explicitly modeling the relational structure among response states. Extending supervision from the output token space to QKV relations and the process space is a natural and meaningful direction. Rather than adopting full-layer projection or comprehensive OPRD-style alignment, Mini-OPD performs relational transfer using only a single response-layer pair and a single response position. This design is more lightweight than dense feature alignment and is better suited to practical LLM post-training scenarios.
Critical weaknesses
The core of Mini-OPD can be summarized as "output-level OPD + response-level QKV relational transfer." However, substantial prior work already exists on internal representation distillation, attention/relation distillation, and sequence-level distillation. The causal attribution of the reported performance gains remains insufficiently established. The observed improvements could potentially arise from several factors, including additional response-level supervision, teacher noise filtering, longer training time, position matching and layer-level regularization. End-to-end performance improvements alone are insufficient to demonstrate that the gains primarily originate from QKV relational transfer itself. The paper would benefit from more explicit relation-specific ablation studies, such as shuffled relations, random layer pairs, shuffled token positions, and a projector-only baseline.
Questions for authors
Are the performance improvements of Mini-OPD over OPD and OPRD-Bridge still statistically significant across multiple random seeds? Are confidence intervals or variance estimates reported? Were OPD, OPRD-Bridge, and feature-projector-based baselines compared under matched computational budgets?