- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
Contribution: The authors aimed to answer the question of whether training an intermediate-layer decoder on a teacher model's full probability distribution produces better observed-token NLL than training on its argmax, even with temperature calibration. Based on my reading, they did indeed answer the question, with the result being positive, across several models and decoder parameterizations. In particular, the authors fit multiple readout families at different transformer depths, keeping the backbone and output head frozen, and compared hard versus soft targets under matched fitting procedures. They tested two Qwen models and a Llama model, evaluated on held-out text, and investigated calibration, readout capacity, and shuffled teacher distributions.
The primary Qwen experiments report NLL improvements of 0.124 and 0.224 nats after calibration. The Llama replication reports approximately 0.27 nats.
The paper's more distinctive finding is that reassigning or swapping non-argmax teacher probabilities reduces the benefit, suggesting that position-specific distributional information matters under the tested fitting procedures.
Critical strengths
- I believe that the claim was soundly proven, and there are sufficient ablations. The authors conducted controlled comparisons among different architectures, depths, and readouts, and used independent held-out evaluations and many calibration and target controls.
Critical weaknesses
- Writing quality is very low. Introduction in particular is written in a very confusing and circular way, with sentences winding around. The premise of the paper is not well-articulated, and the core experiments and findings are not motivated strongly.
- In a sense, I think the outcome of this paper is not one that is particularly strong. The main argument is that soft-target supervision should be expected to outperform argmax supervision on a distribution-sensitive objective (after scalar calibration); the paper does indeed establish this, but fails to articulate why the result is an important advance beyond existing knowledge-distillation literature.
Questions for authors
- What is the main scientific insight beyond the established advantages of soft-target distillation? In particular, what conclusion about intermediate representations follows from these experiments that would not already be anticipated from the distinction between hard and soft supervision?
- What practical or scientific decision changes because of these findings? What evidence supports that broader implication?