- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies on-policy distillation and argues that matching individual hidden-state representations between the teacher and student may not be sufficient. To address this, it proposes Tangent On-Policy Distillation (TOPD), which additionally aligns the directions of representation changes between adjacent tokens and layers. The experimental results show improvements over existing distillation baselines on mathematical reasoning benchmarks.
While the proposed directional matching can be viewed as a useful complement to pointwise state matching, I find its underlying motivation insufficiently established. The paper would benefit from stronger empirical analysis explaining why local representation changes matter for distillation, a theoretical justification for the proposed objectives, or a more rigorous discussion of relevant prior work to better establish the significance of the approach.
Critical strengths
- Clarity: The paper is clearly written and easy to follow.
- Relevance: The paper applies its proposed method to on-policy distillation, an emerging and important research setting.
Critical weaknesses
1. Insufficient motivation for local-direction matching. The central motivation for aligning local representation changes remains unclear. The authors argue that models can exhibit different local directions despite having well-aligned hidden states. However, this observation alone does not justify introducing two additional direction-matching objectives. The paper lacks a convincing theoretical or empirical explanation of why token-wise and layer-wise direction alignment is important, beyond serving as complementary regularization to pointwise state matching.
Although the trajectory-divergence analysis provides some empirical support for local-direction matching, I believe the authors need to provide stronger motivation for the proposed approach, either theoretically or empirically, by addressing two key questions: 1. Why is it necessary to match directional changes across consecutive tokens and layers? 2. What important information do these changes capture that is not already captured by pointwise state matching? Do these objectives improve optimization, transfer additional knowledge, or simply act as regularization? The current paper does a poor job of providing a strong foundation and justification for the proposed method.
2. Limited novelty: Given the lack of strong motivation and analysis, I find the proposed directional matching rather incremental and limited in novelty. My concern is not its simplicity, but the lack of convincing justification for why such a simple approach is meaningful and necessary. Beyond its application to on-policy distillation, transferring temporal relationships between representations and inter-layer representation changes has already been explored in knowledge distillation (Xiao et al., 2021; Udayangani et al., 2025 [MemKD]). Although the exact formulations differ, the underlying idea is not novel.
3. Computational overhead. Although the authors state that the proposed directional objectives introduce only lightweight operations, the paper lacks a quantitative analysis of the additional training cost compared with pointwise state matching. Reporting training time, memory usage, and computational overhead would help assess whether the performance improvements justify the additional cost.
4 .Minor: Some terms in Equation (3) are only defined later in Equation (5). The notation is also used for both the teacher and the token dimension, which may cause confusion. Additionally, the figures (1 and 2) are dense and could be simplified into clearer, more abstract illustrations.
References:
- Zeyu Xiao, Xueyang Fu, Jie Huang, Zhen Cheng, Zhiwei Xiong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2113-2122
- N. Udayangani, K. Nandakishor and M. Palaniswami, "MemKD: Memory-Discrepancy Knowledge Distillation for Efficient Time Series Classification," ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Questions for authors
-
Have the authors considered what local representation changes actually mean in sequential models such as RNNs and transformers? Since hidden states naturally accumulate information as a sequence is processed, what additional information do their local differences capture? Could this understanding lead to a more meaningful teacher–student alignment strategy than directly matching first-order differences?
-
Choice of adjacent-state differences. What motivates restricting the proposed alignment to first-order differences between neighboring tokens and layers? Could differences between more distant hidden states capture more meaningful information? An ablation varying the distance between the compared states would help justify this design choice.