acceptodds
Under review as a conference paper at ICLR 2027

OPTD: On-Policy Truncated Distillation

Abstract

On-policy representation distillation (OPRD) trains a student language model on its own samples to match its teacher’s hidden states at every layer, giving every layer’s squared error the same weight. Hidden states grow in scale with depth, so this equal weighting sends most of the gradient on the hidden states to the last few layers. Splicing the student’s hidden state into the teacher one layer at a time shows that errors in these layers damage the teacher’s prediction far less than the loss implies. Weighting layers by the measured damage improves accuracy, yet a shuffled placebo that assigns the same weights to random layers improves it even more. Every reweighting scheme we test, this placebo and a uniform one on normalised errors included, starves the last layers of gradient. On-policy truncated distillation (OPTD) keeps only that shared property by dropping the last four layers from OPRD’s sum. With a public teacher trained from its student by reinforcement learning, OPTD improves accuracy over OPRD by 3.97 points on problems from four mathematics and science benchmarks, beats every reweighting scheme, and outscores the teacher on all four.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, representation distillation, knowledge distillation, hidden-state matching, reasoning models, activation patching, layer truncation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mTb4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a well-written empirical paper. it starts with a useful observation that raw hidden-state MST error in OPRD is dominated by late-layer representation scale, while holding a much smaller version of the damage. The paper conducts comprehensive experiments, including rewriting and placebos, and proposes a simple method to truncate the last four layers. OPTD-4 improves pooled accuracy by 3.97 points over the reproduced OPRD baseline.

Critical strengths
  • The paper is empirically sound with a clean solution. It motivates the problem well from observations, and the proposed algorithm is straightforward and easy to implement.
  • The experimentation is comprehensive. It compares along with several placebos and straightforward weighting.
Critical weaknesses
  • The paper is limited to one highly specific model pair, despite covering multiple datasets.
  • The splice measurement is not a depth-neutral measure of layer importance. Naturally error will propagate, so under this measurement earlier layers contribute more.
  • Based on the previous points, it does not mean that later layers are not important due to the cascading factor. Especially during early learning, the signals from later layers may not be stable or accurate due to compound error. A gradual learning from early to late layers might be more effective.
  • In practice, it's hard to determine how many layers to truncate. This seems to be more of an empirical choice.
Questions for authors

See weaknesses. I suggest the author to focus on 1) expanding to another model pair. 2) study learning schedule: fit earlier layers first then late layers 3) how to select N empirically without choosing all.

Reviewer_pSSeOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces On-Policy Truncated Distillation (OPTD), proposing that dropping the last N layers' loss terms from on-policy representation distillation (OPRD) improves performance. The authors use a layerwise activation splice diagnostic to demonstrate that standard OPRD allocates most of the gradient to the deepest layers due to growing residual-stream norms, even though late-layer errors have minimal downstream effects. After showing that a shuffled placebo weighting outperforms measured importance weighting, they propose OPTD-4. The method is evaluated using a single 1.5B parameter model pair (DeepSeek-R1-Distill-Qwen-1.5B and JustRL-DeepSeek-1.5B) trained for 500 steps.

Critical strengths
  • The layerwise activation splice diagnostic offers a training-free, causal method to measure downstream sensitivity to representation errors.

  • The paper clearly articulates how equal MSE coefficients implicitly allocate disproportionate gradient energy to deep layers simply because residual-stream norms grow with depth.

Critical weaknesses
  • The validation is highly specific to one base model, relying on a single 1.5B parameter student-teacher pair trained for a very brief 500 steps. This severely limits the generalizability of the findings and reads more like an internal experiment report than a broad methodological advancement.

  • OPTD is evaluated strictly in a same-architecture setting with 28 layers. The paper fails to test whether truncation remains effective or how NN should be chosen in cross-architecture scenarios.

  • Several of the core OPTD-N ablation arms (N=2, 6, 8, and 12) rely on only a single training seed, weakening the statistical reliability of the truncation plateau claims.

Questions for authors
  • Can you provide empirical evidence that OPTD's benefits generalize beyond the single 1.5B DeepSeek model pair to larger scales (e.g., 30B+) and longer training horizons?

  • How does OPTD perform in cross-architecture distillation scenarios where the student and teacher do not share an identical layer structure?

  • Why were only single seeds used for the majority of the OPTD-N ablation runs (N=2, 6, 8, 12), and how does this lack of variance measurement impact the confidence of your plateau analysis?

Reviewer_9Jf3OpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1(−1)
Presentation
1
Soundness
2
Read review
Paper summary

This paper finds that OPRD’s equally weighted losses concentrate direct hidden state gradient energy in the final layers, although representation discrepancies in these layers have a relatively limited effect on the teacher’s outputs. Based on activation patching and reweighting experiments, the authors propose On Policy Truncated Distillation (OPTD), which removes supervision from the final layers while preserving the full architecture. Experiments using one teacher and one student show an improvement of 3.97 percentage points over OPRD on the pooled evaluation across four benchmarks.

Critical strengths

The causal probing experiments provide evidence that representation discrepancies in the final layers have a relatively limited effect on the teacher’s outputs.

Critical weaknesses
  1. Although Section 3 shows that discrepancies in the final layers have limited effects on the teacher’s outputs, it does not sufficiently explain why removing their supervision improves performance.
  2. Distillation experiments use only one teacher and one student. More models should be evaluated.
  3. Evaluation focuses on mathematics and science. A broader range of benchmarks would strengthen the results.
Questions for authors
  1. Why does the distilled student outperform its teacher? Further analysis and verification would be helpful.
  2. Please also address the concerns listed under Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.