acceptodds
Under review as a conference paper at ICLR 2027

LastOPD: Taming Collapse in Latent On-Policy Distillation

12 views

Abstract

On-policy distillation (OPD) corrects a student on the responses it writes, but its signal is the teacher's next-token distribution: it tells the student *what* the teacher says but misses *how* it thinks. Latent supervision promises the missing part by aligning the student's latent states to the teacher's. Recent methods such as OPRD bring this signal into on-policy distillation. However, we observe two failures of this recipe when distilling Qwen3-4B and Qwen3-8B into Qwen3-1.7B-Base. **Early gain, late collapse**: latent supervision alone lifts MATH-500 accuracy from 25 to 46 in 10 steps, but subsequent training degrades performance down to 11 with no recovery. **Better alignment, worse behavior**: although the alignment metric steadily improves throughout this collapse, the most aligned model turns out to be the worst performing. Further analysis suggests a mismatch in how the latent signal is applied: layers paired by depth play different roles in the two models, so continued alignment may pull the student toward teacher states it cannot understand. To address this, we propose LastOPD, which applies the latent signal only at the last-layer state, the common interface both LM heads read, and only during a 10-step crossfade into token-level OPD. This keeps the useful part of the latent signal and hands the student to token-level supervision before the collapse sets in. Extensive experiments show that LastOPD improves MATH-500 over token-only OPD by 5.55 and 4.02 points with the 4B and 8B teachers, leads on most held-out datasets, and reaches the final score of token-only OPD in about half the steps. Code is available at https://anonymous.4open.science/r/LastOPD-C321.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zcb3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the collapse of latent-level on-policy distillation and proposes LastOPD, which applies latent supervision only to the final hidden layer during an initial crossfade period before transitioning to token-level OPD. The experiments demonstrate that latent alignment can provide substantial early improvements but may eventually cause severe performance degradation despite continuously improving alignment metrics.

Critical strengths
  1. The observation that latent-level alignment can lead to severe performance collapse, even as the alignment objective continues to improve, is particularly interesting.

  2. The J-Lens analysis is also insightful, revealing differences in how teacher and student models represent and expose intermediate reasoning across layers.

These findings provide valuable empirical evidence that representation similarity does not necessarily imply functional alignment.

Critical weaknesses

1. Why does last-layer latent alignment provide such substantial benefits during early training?

The paper demonstrates that applying latent supervision during the initial training steps significantly accelerates learning compared to token-level OPD. However, the underlying mechanism remains unclear. The authors argue that the final hidden layer provides a shared interface for next-token prediction and suggest that latent supervision strengthens the student's existing reasoning process. While the J-Lens analysis supports this interpretation to some extent, it does not fully explain why matching the teacher's final-layer representation provides a stronger learning signal than directly matching its output distribution. For example, does latent supervision transfer additional information about the teacher's reasoning process that is not captured by token-level distillation, or does it primarily improve the optimization dynamics during the initial training phase? A more intuitive explanation or additional analysis distinguishing these possibilities would strengthen the paper.

2. Does projected latent alignment imply functional alignment?

LastOPD employs a trainable projection network gψg_\psi to align the student's final hidden representation zSz_S with the teacher's representation zTz_T:

gψ(zS)≈zT.g_\psi(z_S) \approx z_T.

However, since the teacher and student use different LM heads, WTW_T and WSW_S, respectively, alignment in the projected latent space does not necessarily imply alignment in the actual output space:

\;\not\Rightarrow\; W_S z_S \approx W_T z_T.$$ In particular, even if the projected student representation produces teacher-consistent logits, $$W_T g_\psi(z_S) \approx W_T z_T,$$ this does not guarantee that the student's own LM head produces corresponding predictions: $$W_S z_S \approx W_T g_\psi(z_S).$$ Although the paper analyzes representation similarity using CKA and projected cosine similarity, it remains unclear whether the learned projection leads to functional alignment between the teacher and student. Could the authors evaluate whether the projected student representations produce teacher-consistent output distributions under $W_T$, and whether these distributions agree with the predictions generated by the student's actual LM head $W_S$? Such an analysis would help clarify whether latent supervision transfers functionally meaningful information rather than merely improving similarity in the projected representation space.
Questions for authors

See weakness

Reviewer_XiYhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates training degradation in cross-size on-policy hidden-state distillation (OPRD): student accuracy improves early in training but subsequently drops sharply, even as the representation similarity being optimized continues to increase. Through multiple analyses, the authors find that sustained hidden-state supervision may degrade performance because teacher and student layers process information differently. Based on these findings, they propose LastOPD, which aligns only the final-layer hidden states during early training and gradually shifts to token-level supervision. Across two cross-size Qwen3 configurations, LastOPD improves MATH-500 accuracy over token-only OPD by 5.55 and 4.02 percentage points, respectively, while also improving performance on most other mathematical benchmarks. Overall, the paper translates an observation of training degradation into a simple and effective design, providing valuable empirical insights into how internal representation supervision should be used.

Critical strengths

The paper presents an insightful empirical finding and builds a clear, coherent research narrative around it. The authors observe that students’ problem-solving performance initially improves and then deteriorates, even as hidden-state alignment continues to improve. This exposes a discrepancy between representation matching and actual capability. The subsequent analyses naturally motivate using hidden-state supervision only during early training. The method is simple, and the empirical comparisons demonstrate its practical value.

Critical weaknesses
  1. The analysis of the collapse mechanism remains limited. The paper establishes that representation alignment can improve while task performance deteriorates, and the layerwise analyses provide useful explanatory clues. However, the hypothesis that the student first absorbs usable information and is subsequently affected by teacher representations it cannot effectively use lacks an operational way to identify these two types of information. The evidence convincingly shows that direct alignment can fail, but the explanation of how degradation occurs remains largely indirect, limiting the mechanistic contribution.

  2. The independent contribution of final-layer supervision is not sufficiently established. Since the final layers of both models support next-token prediction, choosing this location is reasonably motivated. However, compared with OPRD, LastOPD changes both the supervision location and how the projector is trained, so the existing comparisons primarily establish the effectiveness of the combined recipe.

  3. Validation beyond mathematical reasoning is limited. Although the paper evaluates on eight mathematical benchmarks, it does not examine other reasoning domains, such as commonsense, logical, or code reasoning. The results therefore demonstrate breadth within mathematics but do not establish whether the observed collapse and the benefits of LastOPD generalize to other forms of reasoning. Evaluating representative non-mathematical tasks would help clarify the scope of both the empirical findings and the proposed method.

Questions for authors

Please see the weakness section.

Reviewer_k9qzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study latent hidden-state supervision in on-policy distillation (OPD) and propose LastOPD, which applies a feature-alignment loss only at the last-layer state and linearly crossfades it into token-level reverse top-k OPD. Specifically, the paper shows that latent-only OPD (OPRD) on cross-size Qwen3 pairs exhibits an “early gain, late collapse” pattern, rising from 25.4 to 46.2 on MATH-500 within 10 steps and then falling to about 11. It also shows that the optimized alignment proxy keeps improving during this collapse (“better alignment, worse behavior”). The authors attribute this to depth-paired layers playing different readout roles (the teacher “knows early, tells late”) and to CKA/cosine similarity being inflated by a few massive-activation dimensions. Additionally, on Qwen3-4B/8B → Qwen3-1.7B-Base, LastOPD improves MATH-500 over token-only OPD by 5.55/4.02 points and the eight-dataset mean by 3.93/2.10, and reaches token-only OPD’s final score in roughly half the steps. On a same-lineage pair (JustRL-1.5B → R1-Distill-1.5B), however, it underperforms both latent-only OPRD-Vanilla and token-only OPD on MATH-500.

Critical strengths

1. Clear and practically relevant empirical phenomena: The paper documents a reproducible failure mode of layerwise latent OPD (early peak followed by collapse below the starting point), together with a proxy/behavior decoupling. Both are useful cautionary findings for practitioners adopting OPRD-style recipes.

2. Rich diagnostic analysis: The combination of J-Lens readouts, linear value probes, layer-to-layer CKA, massive-activation masking, and rollout-type breakdowns gives a multi-angle picture of where teacher and student representations do and do not correspond.

3. Simple, cheap method: LastOPD requires no layer pairing or pre-trained frozen projector, and needs teacher latents only for 10 steps. It is therefore easy to adopt on top of standard OPD at negligible extra cost.

Critical weaknesses

1.Why Last layer?: The only rationale for the last layer is that it is a “common interface to next-token prediction”. Yet the post-norm last-layer state is one linear map away from the logits, so this term is largely redundant with the token-level KL, and the trainable MLP can absorb much of the alignment pressure. The ablations never test a middle/single intermediate layer with crossfade, or all-layer OPRD with the same crossfade, so the paper cannot separate the effect of where from when.

2. The Insight Looks good, but is that true or can it generalize? The paper actually has narrow experimental scope and weak generality of the diagnosis. Specifically, all models are in the Qwen family, there is only one cross-size student (i.e., Qwen3-1.7B-Base), teachers are at most 8B, training with limit steps, and only math is studied. Additionally, the diagnostic claims (e.g., Insight in Section 4 with Only 8 samples) is not well supported. Notably, the GRPO control (App. C.3) also peaks at step 10 and then decays, which suggests that part of the “early gain, late collapse” may be a property of this student and training regime rather than of latent supervision specifically.

3. What is the difference between LastOPD and an auxiliary regularizer: Eq. 2 is standard normalized feature regression, and the contribution reduces to placing it at the final layer plus a linear 10-step loss schedule. The paper lacks controls showing that the teacher’s latent content matters, such as a random/shuffled target, a forward-KL or full-vocabulary warm-up for 10 steps, or an SFT warm-up. Without them, LastOPD is hard to distinguish from a generic early-stage auxiliary loss.

4. Insufficient baselines: Comparisons are limited to token-only reverse top-16 OPD, OPRD (Bridge/Vanilla, ± token), and the authors’ own always-on variant. Author need to include more recent OPD or latent regularized OPD to demonstrate the advantages of this method.

Questions for authors

Please check my Weaknesses Section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.