- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the collapse of latent-level on-policy distillation and proposes LastOPD, which applies latent supervision only to the final hidden layer during an initial crossfade period before transitioning to token-level OPD. The experiments demonstrate that latent alignment can provide substantial early improvements but may eventually cause severe performance degradation despite continuously improving alignment metrics.
Critical strengths
-
The observation that latent-level alignment can lead to severe performance collapse, even as the alignment objective continues to improve, is particularly interesting.
-
The J-Lens analysis is also insightful, revealing differences in how teacher and student models represent and expose intermediate reasoning across layers.
These findings provide valuable empirical evidence that representation similarity does not necessarily imply functional alignment.
Critical weaknesses
1. Why does last-layer latent alignment provide such substantial benefits during early training?
The paper demonstrates that applying latent supervision during the initial training steps significantly accelerates learning compared to token-level OPD. However, the underlying mechanism remains unclear. The authors argue that the final hidden layer provides a shared interface for next-token prediction and suggest that latent supervision strengthens the student's existing reasoning process. While the J-Lens analysis supports this interpretation to some extent, it does not fully explain why matching the teacher's final-layer representation provides a stronger learning signal than directly matching its output distribution. For example, does latent supervision transfer additional information about the teacher's reasoning process that is not captured by token-level distillation, or does it primarily improve the optimization dynamics during the initial training phase? A more intuitive explanation or additional analysis distinguishing these possibilities would strengthen the paper.
2. Does projected latent alignment imply functional alignment?
LastOPD employs a trainable projection network to align the student's final hidden representation with the teacher's representation :
However, since the teacher and student use different LM heads, and , respectively, alignment in the projected latent space does not necessarily imply alignment in the actual output space:
\;\not\Rightarrow\; W_S z_S \approx W_T z_T.$$ In particular, even if the projected student representation produces teacher-consistent logits, $$W_T g_\psi(z_S) \approx W_T z_T,$$ this does not guarantee that the student's own LM head produces corresponding predictions: $$W_S z_S \approx W_T g_\psi(z_S).$$ Although the paper analyzes representation similarity using CKA and projected cosine similarity, it remains unclear whether the learned projection leads to functional alignment between the teacher and student. Could the authors evaluate whether the projected student representations produce teacher-consistent output distributions under $W_T$, and whether these distributions agree with the predictions generated by the student's actual LM head $W_S$? Such an analysis would help clarify whether latent supervision transfers functionally meaningful information rather than merely improving similarity in the projected representation space.Questions for authors
See weakness