acceptodds
Withdrawn from ICLR 2027

Tangent On-Policy Distillation: Local-Direction Supervision Beyond Outputs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) reduces the train–test mismatch by training on student- generated trajectories, but most existing methods supervise output distributions and do not directly constrain how intermediate representations evolve along those trajectories. This raises the question: can local changes in hidden representations provide an effective supervision signal beyond output matching? We introduce Tangent On-Policy Distillation (TOPD), which constructs teacher and student token-by-layer hidden-state grids on the same student-generated rollout and aligns first finite-difference directions along adjacent response tokens and aligned network layers. TOPD combines this local-direction supervision with pointwise state matching, using cosine distance and detached, capped loss-ratio balancing to calibrate their scales. Distilling JustRL-DeepSeek-1.5B into DeepSeek-R1-Distill-Qwen- 1.5B on DAPO-Math-17K, TOPD achieves 58.66 Macro Avg@16 across AIME24, AIME25, and AIMO, outperforming the strongest prior distillation baseline by 4.16 percentage points (7.6% relative). Ablations support this design: removing temporal or depth supervision lowers Macro Avg@16 by 1.70 and 1.00 points, respectively, while the proposed cosine objective with detached, capped balancing outperforms fixed weighting, magnitude matching, differentiable balancing, and uncapped balancing. These results show that local representation directions provide an effective complement to output-level supervision in on-policy distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
on policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_vAYtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies on-policy distillation and argues that matching individual hidden-state representations between the teacher and student may not be sufficient. To address this, it proposes Tangent On-Policy Distillation (TOPD), which additionally aligns the directions of representation changes between adjacent tokens and layers. The experimental results show improvements over existing distillation baselines on mathematical reasoning benchmarks.

While the proposed directional matching can be viewed as a useful complement to pointwise state matching, I find its underlying motivation insufficiently established. The paper would benefit from stronger empirical analysis explaining why local representation changes matter for distillation, a theoretical justification for the proposed objectives, or a more rigorous discussion of relevant prior work to better establish the significance of the approach.

Critical strengths
  1. Clarity: The paper is clearly written and easy to follow.
  2. Relevance: The paper applies its proposed method to on-policy distillation, an emerging and important research setting.
Critical weaknesses

1. Insufficient motivation for local-direction matching. The central motivation for aligning local representation changes remains unclear. The authors argue that models can exhibit different local directions despite having well-aligned hidden states. However, this observation alone does not justify introducing two additional direction-matching objectives. The paper lacks a convincing theoretical or empirical explanation of why token-wise and layer-wise direction alignment is important, beyond serving as complementary regularization to pointwise state matching.

Although the trajectory-divergence analysis provides some empirical support for local-direction matching, I believe the authors need to provide stronger motivation for the proposed approach, either theoretically or empirically, by addressing two key questions: 1. Why is it necessary to match directional changes across consecutive tokens and layers? 2. What important information do these changes capture that is not already captured by pointwise state matching? Do these objectives improve optimization, transfer additional knowledge, or simply act as regularization? The current paper does a poor job of providing a strong foundation and justification for the proposed method.

2. Limited novelty: Given the lack of strong motivation and analysis, I find the proposed directional matching rather incremental and limited in novelty. My concern is not its simplicity, but the lack of convincing justification for why such a simple approach is meaningful and necessary. Beyond its application to on-policy distillation, transferring temporal relationships between representations and inter-layer representation changes has already been explored in knowledge distillation (Xiao et al., 2021; Udayangani et al., 2025 [MemKD]). Although the exact formulations differ, the underlying idea is not novel.

3. Computational overhead. Although the authors state that the proposed directional objectives introduce only lightweight operations, the paper lacks a quantitative analysis of the additional training cost compared with pointwise state matching. Reporting training time, memory usage, and computational overhead would help assess whether the performance improvements justify the additional cost.

4 .Minor: Some terms in Equation (3) are only defined later in Equation (5). The notation TT is also used for both the teacher and the token dimension, which may cause confusion. Additionally, the figures (1 and 2) are dense and could be simplified into clearer, more abstract illustrations.

References:

  1. Zeyu Xiao, Xueyang Fu, Jie Huang, Zhen Cheng, Zhiwei Xiong; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2113-2122
  2. N. Udayangani, K. Nandakishor and M. Palaniswami, "MemKD: Memory-Discrepancy Knowledge Distillation for Efficient Time Series Classification," ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Questions for authors
  1. Have the authors considered what local representation changes actually mean in sequential models such as RNNs and transformers? Since hidden states naturally accumulate information as a sequence is processed, what additional information do their local differences capture? Could this understanding lead to a more meaningful teacher–student alignment strategy than directly matching first-order differences?

  2. Choice of adjacent-state differences. What motivates restricting the proposed alignment to first-order differences between neighboring tokens and layers? Could differences between more distant hidden states capture more meaningful information? An ablation varying the distance between the compared states would help justify this design choice.

Reviewer_RbDoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, the authors proposed a new variant of on-policy distillation method, called TOPD, which built on top of a related work OPRD. In additional to adding hidden-state matching on student’s roll-outs, TOPD also matches the finite-difference directions along the token and layer axes. They use cosine distance to measure the direction difference, and use a specific set of co-efficient for combining three different loss terms: state, token and layer as shown in Eq. (11). They evaluated TOPD in a same architecture distillation setting, in particular, distilling JustRL-DeepSeek-1.5B into DeepSeek-R1-Distill-Qwen-1.5B on DAPO-Math-17K, evaluated the distilled model’s performances on three mathematical reasoning benchmarks. Moreover, they also performed additional experiments to examine the effect of newly added loss terms and also the cross-architecture distillation setting.

Critical strengths
  1. Comparing to OPRD, the newly added two loss terms introduced little computational overhead.
  2. The authors performed multiple ablation studies to examine the proposed method.
  3. The outline of the paper is clear and easy to follow.
Critical weaknesses

I list the main concerns here and minor issues in the Questions for Author box.

  1. Considering the related work OPRD, the novelty of the proposed method here is very incremental, the motivation of adding the token and layer loss terms is not very clear and do not have enough evidence to support it both theoretically and empirically. Eq. (4) did not really support that state matching will fail to control local differences, and also detailed analysis on Section 4.5 did not really support the motivation claim that two models may be well aligned at individual states while changing in different directions, as cosine similarity is already very high at the beginning (0.9534).
  2. The main contribution of this paper seems to be the empirical improvements over OPRD, however, main results presented in Section 4.2 seems to be just single run without variance analysis. Moreover, baselines from Table 1 are not re-runs with the same setting as TOPD, so the comparison is not reliable. Another issue from Table 1 is, TOPD is the only method that produced a student model whose performance exceeds teacher model for all three benchmarks by large margins. This seems too good to be true! This suggested there is possibly an evaluation issue in all benchmarks.
  3. The hyperparameters of TOPD are fixed across all experiments and Figure 4 shows that the results are very sensitive to those hyperparameter selection, therefore, it is not clear if similar improvements observed from Table 1 will apply to new models and new datasets.
  4. Most of the analysis from Section 4.3 are just stating the results in text from Figure 4, but there is no insights or explanation on why do we observe those results. For example, why using last-4000 token did not give more performance boost as last-2000? Is this model specific result or data specific result?
  5. More ablation studies can be performed including examining more baselines cited in the related work, more methods to measure the direction differences other than cosine similarity, etc.
  6. On the cross-architecture experiments, there is no explanation on how the representation projection is learnt and performed. Table 4 reported the distillation on Qwen3 models, but no results for Phi-4-mini.
Questions for authors
  1. Table 1, Figure 3 are showing the same results in different format, why do we need it?
  2. Why there is more baselines in Table 4, and what is OPRD-Bridge different from OPRD?
  3. What is the training-time and memory overhead? How is rr distributed over training, and how often does the cap reach?
  4. Does the method help at larger scales, longer training, other domains or other teacher?
Reviewer_zPJrOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TOPD, which supplements on-policy hidden-state matching with directional alignment across neighboring tokens and layers. Detached, capped weighting balances these losses. Mathematical reasoning experiments report 58.66 macro Avg@16 against OPRD’s 54.50, alongside ablations and transfer across model architectures and tokenizers.

Critical strengths

The approach is straightforward and plausibly useful: matching local changes can complement imperfect state alignment. It uses representations already computed by the teacher and leaves student inference unchanged, offering a practical extension to existing distillation methods.

Critical weaknesses
  1. The reported accuracy requires clarification. The stated AIME evaluation comprises 480 binary outcomes per benchmark, yet scores such as 56.039% and 35.389% cannot result from that denominator, even after rounding. Undocumented aggregation, borrowed baseline results, and absent uncertainty estimates undermine confidence in the improvement.

  2. Important experiments are underspecified. Cross-tokenizer distillation lacks a token-correspondence rule, which dimensional projection cannot replace. The failure-prediction analysis does not explain its labels or failure-onset definition. These omissions prevent verification of the transfer and early-warning claims.

Questions for authors
  1. What evaluation counts and aggregation procedure produced Table 1? Were results averaged across runs, or were any problems or responses excluded?

  2. How are text positions and adjacent differences matched across tokenizers? Do TOPD and the state-only comparison use identical checkpoints and projection interfaces?

  3. How is differentiable balancing implemented? With neither capping nor the denominator floor active, fully differentiating the ratios reduces Eq. 11 to state-only training. Why does this ablation perform differently?

  4. How were errors and their onset positions labeled independently of the detection signals? What false-positive rate accompanies the reported warning lead time?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.