- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes O³PD, which replaces distribution matching in on-policy distillation with ordinal supervision: the student is trained to reproduce the teacher's top-K token ranking using adjacent pairwise Bradley-Terry losses, without fitting the teacher's probability magnitudes. This makes the objective invariant to teacher temperature and frees the student from reproducing the teacher's confidence profile. Across six Qwen3 teacher–student configurations on three math competition benchmarks, O³PD outperforms OPD on the three-benchmark average in every configuration, by up to 2.9 points, and also exceeds SFT and GRPO. Analysis shows O³PD departing further from the teacher's probabilities than OPD while reproducing its ranking more faithfully and maintaining higher entropy.
Critical strengths
- The separation of teacher ordering from magnitude is conceptually clear and leads to a method with desirable theoretical properties.
- O³PD improves the three-benchmark average in all 6 configurations, improves AIME25 in every configuration, and exceeds both SFT and GRPO at every student scale.
Critical weaknesses
All experiments use Qwen3. The insight that "ordering matters more than magnitude" should be tested across families. Similarly, math reasoning is a narrow domain where top-K tokens may be unusually informative. In open-ended generation or instruction following, the distribution tail may carry more information that ordinal supervision discards. The paper also uses LoRA exclusively. Full fine-tuning experiments would clarify whether the gains persist when the student has more capacity to potentially match teacher magnitudes.
Questions for authors
Clarification
The temperature sweep (Figure 7) uses only 3 training steps. Do the relative rankings of temperatures persist after full training (100 steps)? If the optimal temperature changes during training, per-pair temperature tuning might be insufficient.
Future work
Have you considered a hybrid approach where ordinal supervision is combined with a mild distribution-matching term (e.g., KL with small weight)? This might retain the ranking benefit while providing some signal about the tail distribution.
Thank you for the rebuttal and additional experiments. While most of concerns are addressed, my W2 is still partially resolved, which is important since it is directly related to the soundness of proposed method.
First, as shown in Figure 1, this paper is positioned between Logit-based KD and Distribution-based KD. Furthermore, since many previous works fall into distribution-based KD, I think it must be clearly demonstrated whether order-based KD, which appears to be a generalized framework for distribution-based KD, is superior.
However, I cannot find any performance comparisons with recent distribution-based KD (beyond the JSD) and logit-based KD baselines. Furthermore, the authors did not sufficiently provide the theoretical relationship between distribution-based KD and order-based KD in their responses. While I understand that it is hard to conduct many experiments under the limited resources and short discussion period, it is difficult to be fully convinced of the validity of the proposed method now