acceptodds
Under review as a conference paper at ICLR 2027

Order-Only On-Policy Distillation

19 views

Abstract

On-policy distillation (OPD) trains a student on its own generations by matching the teacher's token probabilities. These probabilities entangle two kinds of information: which tokens the teacher prefers, and how confident it is. The latter reflects the teacher's own capacity and calibration, which a smaller student may be unable to reproduce. Because the teacher temperature changes these values without changing the ordering, OPD is also sensitive to the temperature choice. We ask whether the teacher's ordering of candidate tokens is enough, and propose Order-Only On-Policy Distillation (OPD). At each student-generated prefix, OPD trains the student to reproduce the order of the teacher's top- tokens, without fitting their probability values. The objective is therefore invariant to the teacher temperature and needs no temperature tuning. Across six teacher–student pairs, OPD outperforms OPD on the average over HMMT25, AIME24, and AIME25 in every pair, by up to 2.9 points, with gains of up to 5.6 points on AIME25. Other ranking losses yield similar gains, so the benefit comes from ordinal supervision itself. Analysis shows that OPD departs further from the teacher's probabilities than OPD, yet reproduces its ranking more faithfully and yields a stronger student. The ranking, not the probability values, thus carries the knowledge that on-policy distillation needs to transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge Distillation, On-Policy Distillation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gfGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes O³PD, which replaces distribution matching in on-policy distillation with ordinal supervision: the student is trained to reproduce the teacher's top-K token ranking using adjacent pairwise Bradley-Terry losses, without fitting the teacher's probability magnitudes. This makes the objective invariant to teacher temperature and frees the student from reproducing the teacher's confidence profile. Across six Qwen3 teacher–student configurations on three math competition benchmarks, O³PD outperforms OPD on the three-benchmark average in every configuration, by up to 2.9 points, and also exceeds SFT and GRPO. Analysis shows O³PD departing further from the teacher's probabilities than OPD while reproducing its ranking more faithfully and maintaining higher entropy.

Critical strengths
  1. The separation of teacher ordering from magnitude is conceptually clear and leads to a method with desirable theoretical properties.
  2. O³PD improves the three-benchmark average in all 6 configurations, improves AIME25 in every configuration, and exceeds both SFT and GRPO at every student scale.
Critical weaknesses

All experiments use Qwen3. The insight that "ordering matters more than magnitude" should be tested across families. Similarly, math reasoning is a narrow domain where top-K tokens may be unusually informative. In open-ended generation or instruction following, the distribution tail may carry more information that ordinal supervision discards. The paper also uses LoRA exclusively. Full fine-tuning experiments would clarify whether the gains persist when the student has more capacity to potentially match teacher magnitudes.

Questions for authors

Clarification

The temperature sweep (Figure 7) uses only 3 training steps. Do the relative rankings of temperatures persist after full training (100 steps)? If the optimal temperature changes during training, per-pair temperature tuning might be insufficient.

Future work

Have you considered a hybrid approach where ordinal supervision is combined with a mild distribution-matching term (e.g., KL with small weight)? This might retain the ranking benefit while providing some signal about the tail distribution.

Reviewer_XAjsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Order-Only On-Policy Distillation O3PD\mathrm{O}^{3}\mathrm{PD}, which trains a student model using only the ranking of the teacher's top-K tokens rather than their exact probabilities. At each student-generated prefix, the teacher provides its top-K tokens and their ordering. The student then learns this ordering through an adjacent logistic ranking loss. Since temperature scaling does not change token rankings, the supervision is invariant to the teacher's temperature.

The experiments cover six teacher-student pairs from the Qwen3 family. Students are trained on the mathematical reasoning subset of OpenThoughts and evaluated on HMMT25, AIME24, and AIME25. O3PD\mathrm{O}^{3}\mathrm{PD} outperforms the top-8 JSD OPD baseline on the three-benchmark average across all six pairs. The paper also compares different ranking losses and K values and analyzes token-level agreement with the teacher during training.

Critical strengths
  1. The main idea is simple and intuitive. O3PD\mathrm{O}^{3}\mathrm{PD} separates the teacher's token preferences from its exact probability values. The temperature-invariance property follows naturally from using token rankings, and the proposed loss is straightforward to implement.

  2. The experimental results show a fairly consistent improvement. O3PD\mathrm{O}^{3}\mathrm{PD} achieves a higher average score across the three benchmarks for all six teacher-student pairs and improves AIME25 in every setting. The ablations on ranking losses and K values also suggest that the improvements are not limited to one particular configuration.

Critical weaknesses
  1. The paper uses the evaluation benchmarks for checkpoint selection. Specifically, checkpoints are evaluated every 20 steps, and the best average across HMMT25, AIME24, and AIME25 is reported. This introduces potential test-set selection bias, especially when the differences between methods are small. Table 5 reports only one training seed, with no confidence intervals. Some improvements are just 0.3 or 0.6 points, making it difficult to determine whether they are statistically meaningful.

  2. The experiments do not fully establish the claim that token ordering alone is sufficient and probability matching is unnecessary. The main OPD baseline uses top-8 JSD with T=1 and the same learning rate as O3PD\mathrm{O}^{3}\mathrm{PD}. Although Figure 7 shows temperature sensitivity, those experiments only run for three steps on HMMT25, which is not enough to identify the best OPD configuration over full training.

Questions for authors
  1. Could the authors report results from the final training checkpoints and checkpoints selected using a held-out validation set for all six pairs? Multiple training seeds and problem-level confidence intervals would also help assess the reliability of the reported improvements.

  2. Does O3PD\mathrm{O}^{3}\mathrm{PD} still outperform OPD when the baseline is tuned over full training for teacher temperature, learning rate, K, and divergence choice, with hyperparameters selected on held-out validation data?

  3. For Figure 5, are the metrics computed using the same prompts and prefixes for OPD and O3PD\mathrm{O}^{3}\mathrm{PD}, or are they based on separately sampled on-policy responses? Also, is the student evaluated in thinking or non-thinking mode?

Reviewer_qkQvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript claims that teacher’s token probabilities include both relative preferences between tokens and the confidence, and matching their specific values may impose unnecessary constraints when the teacher and student differ in reasoning behavior and capacity. To mitigate this issue, this work proposes order-only on-policy distillation (O^3PD), which transfers the teacher’s token ordering without matching its probability magnitudes. Specifically, at each student-generated prefix, O^3PD selects the teacher’s top-K candidates and trains the student using an adjacent pairwise logistic ordering objective. This study demonstrates the effectiveness of O^3PD through several experiment results.

Critical strengths
  1. The proposed method is intuitive and straightforward to incorporate into OPD. It utilizes the teacher’s ordered top-K candidates as supervision. The ablation study of several order-only objectives provides experimental evidence that the performance improvement is not specific to a single loss formulation, supporting the flexibility of this method.

  2. Teacher-temperature invariance is a practical advantage of the method. Since supervision depends only on candidate identities and their ordering, positive temperature scaling leaves the objective unchanged, which removes a teacher temperature selection.

Critical weaknesses
  1. A fundamental limitation of the proposed method is that it does not directly encourage the student to prefer the teacher’s top-K candidates over the rest of the tokens. Since the ordering loss is a function of pairwise logit differences, it is invariant to shifting the student logits by a common constant: the ordering loss remains unchanged while the total probability mass the student assigns to the teacher’s top-K tokens can be very small. Consequently, a low ordering loss does not imply that these candidates are actually selected at decoding time. However, the manuscript provides neither a theoretical analysis nor empirical evidence. For example, JSD@Top-8 in Figure 5 is computed on the renormalized distributions, so it is hard to check how much mass leaves the top-K set; a direct comparison between the student’s total probability mass on the teacher’s top-K tokens and the mass on the remaining vocabulary cannot be found in the current manuscript.

  2. The current baselines are not sufficient to verify the efficacy of O^3PD. Since the core idea lies in discarding the teacher’s probability value matching rather than proposing a new form of the ordering loss, the performance comparison against previous methods that relax exact distribution matching or utilize ordering information is essential. Specifically, key baselines are discussed in the related work but absent from the experiments; RKKD [1] and PLD [2] should be included at a minimum. Moreover, the limitation of distribution matching is experimentally verified through a JSD-based OPD. I think that OPD variants with alternative divergences, recent distribution matching distillation methods, and logit-based distillation methods should be also compared under the same setting.

Minor

  • In Table 4, several best-performing entries are not highlighted in bold.

[1] Enhancing Logits Distillation with Plug&Play Kendall's τ\tau Ranking Loss

[2] PLD: A Choice-Theoretic List-Wise Knowledge Distillation

Questions for authors
  1. I am curious whether the order-only objectives listed in Table 2 differ in computational cost. For instance, the logistic loss compares all K(K-1)/2 pairs, whereas the adjacent logistic loss compares K-1 adjacent pairs. Since the choice bewteen these objectives is presented as a design decision, a comparative analysis of computational cost, such as GPU memory and training time, would help readers to check the performance and efficiency trade-off.

  2. The current experiment results are restricted to the Qwen3 family and mathematical reasoning tasks. Therefore, it remains unclear whether the performance improvements are also shown in other settings. Specifically, I would be interested in seeing whether O^3PD remains effective across different model architectures and other tasks beyond mathematical reasoning, such as instruction following and code generation. Snice this study frames the ordering-based objective as an alternative to distribution matching, these experimental results would strengthen the claim.

1 comment

Thank you for the rebuttal and additional experiments. While most of concerns are addressed, my W2 is still partially resolved, which is important since it is directly related to the soundness of proposed method.

First, as shown in Figure 1, this paper is positioned between Logit-based KD and Distribution-based KD. Furthermore, since many previous works fall into distribution-based KD, I think it must be clearly demonstrated whether order-based KD, which appears to be a generalized framework for distribution-based KD, is superior.

However, I cannot find any performance comparisons with recent distribution-based KD (beyond the JSD) and logit-based KD baselines. Furthermore, the authors did not sufficiently provide the theoretical relationship between distribution-based KD and order-based KD in their responses. While I understand that it is hard to conduct many experiments under the limited resources and short discussion period, it is difficult to be fully convinced of the validity of the proposed method now

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.