acceptodds
Under review as a conference paper at ICLR 2027

LadderOPD: Logit-Free On-Policy Distillation from the Teacher’s Top-k Order

Abstract

On-policy distillation (OPD) trains a small model on its own samples while a teacher scores every token with its full next-token distribution. Many strong teachers are served through interfaces that return only sampled text or a short list of top candidates, so this distribution is often unavailable. We study how much of it OPD actually needs. We first introduce GroundKL, a measurement that takes an open teacher’s logits as ground truth and tests how well each signal a closed interface could return recovers the exact per-token OPD signal. The signal is concentrated at high-entropy forking tokens, and there the order of the teacher’s top-k candidates recovers it nearly as well as sixteen samples from the teacher. Guided by this finding, we propose LadderOPD, which trains on the teacher’s ranked top-k token IDs alone. Across five teacher–student settings in two model families, LadderOPD is statistically equivalent to OPD with top-k probabilities or full logits within three GSM8K points in all ten comparisons, and it matches them on code as well. Shuffling the order at a growing share of forking tokens degrades the student step by step. For verifiable reasoning with a shared tokenizer, a teacher’s ranked top-k list is therefore enough for on-policy distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, logit-free distillation, knowledge distillation, large language models, top-k ranking, black-box distillation, reasoning models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yk6mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether on-policy distillation can be performed effectively using only the teacher’s top-k token candidates and their ordering, rather than exact probabilities. Through GroundKL analysis, the authors measure correlations between limited teacher information and the exact per-token OPD signal, observing that the signal is concentrated at positions with high student entropy. Based on these findings, LadderOPD combines a coverage loss that concentrates the student’s probability mass within the teacher’s top-k candidate set with an order loss that matches the student’s distribution within that set to a fixed target distribution constructed from the ranks. The authors compare GSM8K performance across five teacher–student settings and report performance comparable to probability-based OPD through equivalence tests with a margin of 3 percentage points.

Critical strengths
  1. The paper distinguishes different levels of teacher information available for on-policy distillation and systematically investigates the effects of removing exact probabilities. The research question concerning the relationship between teacher information and distillation performance is interesting.

  2. The main method comparisons use multiple shared random seeds and equivalence tests, rather than relying solely on the absence of statistically significant differences. This is an appropriate statistical approach for comparing distillation with limited teacher information against probability-based OPD.

  3. The evaluation covers multiple teacher–student configurations across two model families and extends to code generation. The paper also includes progressive ranking corruption experiments and component ablations. Furthermore, it reports limitations on open-ended tasks and across different tokenizers, clarifying the scope of its findings.

Critical weaknesses

1. The practical need for the assumed teacher-access scenario is unclear.

The main weakness concerns the assumed teacher-access scenario. The problem formulation of distinguishing levels of teacher information and investigating the minimum information needed for distillation is interesting, and considering an interface that provides only rankings offers some novelty. However, the paper does not sufficiently explain how often scenarios arise in which top-k candidates and their ordering are available but probabilities are not. Text-only interfaces cannot provide the required rankings, while interfaces that provide top-k probabilities already offer information that the proposed method deliberately discards. Concrete use cases for ranking-only interfaces and reasons for revealing rankings instead of probabilities are needed.

2. The core method is simple and heuristic, and the validity of the chosen target distribution is insufficiently established.

The core of the proposed method is to assign fixed probabilities according to the teacher’s ranking and use them as a target distribution to train the student. However, the rationale for choosing an exponential rank distribution, and why this form is more appropriate than other probability assignment schemes, is not sufficiently explained. Simplicity itself is not a problem, but the reasons why this heuristic works and which components are necessary require clearer validation.

3. The sensitivity analysis of the temperature parameter across model sizes is insufficient.

The temperature parameter τ\tau is a central design element for converting rankings into target probabilities, but the sensitivity experiments are limited to the Qwen 7B→1.5B setting. Smaller students may benefit more from a target distribution concentrated on the teacher’s top candidate, whereas students with greater capacity and stronger initial performance may make more effective use of a less concentrated distribution that allocates probability to alternative candidates. This is a hypothesis that requires validation, and the current results do not establish how much the optimal temperature depends on student size or the teacher–student gap. Temperature sweeps under consistent conditions across multiple model configurations are needed, for example by fixing the teacher and varying the student size.

4. Comparisons across training budgets are insufficient.

The current results are based on limited training budgets of 150 updates for the main settings and 40 updates for the 32B→7B setting. Similar performance between rank-based and probability-based OPD under these conditions does not establish that exact probabilities offer no additional benefit with more training. Rankings may suffice at small budgets, while actual probabilities could become more useful as training progresses. Performance curves over training updates or compute budgets are needed to determine whether the observed equivalence holds throughout training and under sufficiently trained conditions.

5. Unequal numbers of runs across experiments raise concerns about the interpretation and robustness of some comparisons.

The temperature experiments in Table 13 report results from one, three, or six runs depending on the setting. A single run cannot characterize variability across seeds, and directly comparing means computed from different numbers of runs and different seed sets makes it difficult to distinguish effects from random variation. In Table 2, the main access-level comparisons use six runs, while some baselines and controls use three. Although the authors disclose the run counts and conduct the main statistical tests on shared seeds, the means presented in the tables may not correspond to the samples used in the paired comparisons.

6. The connection between GroundKL analysis and the actual training objective is insufficiently validated.

The ranking analysis in Table 1 measures the Spearman correlation between the teacher’s negative rank and the exact OPD signal. In contrast, LadderOPD converts rankings into a fixed probability distribution and uses a KL loss that assigns approximately 86.5% probability to the top-ranked candidate under the default setting. Therefore, a correlation between ranking and the OPD signal does not directly establish that training with this target distribution provides correction directions similar to those of the original OPD.

7. The experimental scope is limited for assessing generalization.

Although the paper includes two model families and several evaluation tasks, the main experiments focus on small Qwen students and GSM8K. Student sizes are limited to 7B, and code-generation validation is restricted to MBPP. Some transfer evaluations use a single seed and small evaluation sets, making the robustness of these results difficult to assess.

Questions for authors

See the weaknesses above. My primary concern is that the assumed teacher-access scenario is not sufficiently justified.

Reviewer_edVnOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LadderOPD, an on-policy distillation method that operates when the teacher's full probability distribution is unavailable. Instead of relying on full logits or top-k probabilities, the method utilizes only the teacher's ranked top-k token IDs. It constructs a rank-based target distribution and combines a coverage loss with a ranking-based KL objective.

Critical strengths
  1. The paper addresses an interesting and practical problem: how to perform OPD when only partial information from the teacher is available.

  2. The core idea of extracting as much useful supervision as possible from limited teacher signals is intuitive and promising.

  3. The experimental results are encouraging, showing that the proposed method achieves performance comparable to full-logit OPD across several teacher–student settings.

Critical weaknesses

1. Insufficient explanation for why partial teacher information can outperform full logits.

LadderOPD occasionally outperforms conventional OPD with full teacher probabilities, despite using strictly less information. However, the paper does not sufficiently explain why this happens. Is the improvement due to implicit regularization, reduced sensitivity to probability calibration, or differences in the optimization objectives? A more detailed analysis would help clarify why discarding information can sometimes improve downstream performance.

2. Lack of theoretical justification for the exponential rank-based target distribution.

The proposed method constructs a target distribution using an exponential decay over teacher ranks: $ q_\tau(r) \propto \exp(-r/\tau). $

Although this is a reasonable heuristic, the paper does not provide a clear mathematical justification for this particular functional form. Since rankings do not preserve the probability gaps between tokens, it is unclear why an exponential distribution should appropriately approximate the teacher's supervision signal. Comparisons with alternative rank-to-probability mappings or a stronger theoretical motivation would improve the paper.

3. Unclear connection between the GroundKL analysis and the proposed training objective.

The GroundKL analysis, particularly Table 1, is difficult to interpret without carefully reconstructing the evaluation procedure. The table reports Spearman correlations between the teacher-derived signals and the exact per-token OPD signal across student-generated token positions. However, this interpretation is not sufficiently clear from the presentation.

More importantly, it remains unclear how these correlations justify the proposed training objective. While the results suggest that teacher rankings contain useful information about the original OPD signal, they do not directly establish that the proposed rank-based KL loss produces gradients aligned with those of conventional OPD.

It would be valuable to compare the gradients induced by LadderOPD and full-logit OPD, for example through gradient cosine similarity or other measures of gradient agreement. This would provide more direct evidence that the proposed objective preserves the optimization signal of conventional OPD.

4. Clarity and presentation.

Although the overall motivation is intuitive, the paper is somewhat difficult to follow, especially in the transition from the GroundKL analysis to the construction of the LadderOPD loss. The relationship among the measured rank correlations, the exponential rank-based target distribution, and the resulting optimization gradients is not clearly explained.

A more intuitive and unified explanation of how the empirical findings motivate the specific loss formulation would significantly improve the readability and technical clarity of the paper.

Questions for authors

see weakness

Reviewer_rVTBOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how much of teacher information required for on-policy distillation. It finds that the signals for OPD are primarily concentrated at high-entropy tokens. At these tokens, the rank of a sampled token among the teacher’s top-k predictions serves as a strong proxy for the exact signal. The authors propose LadderOPD which restricts training to the teacher’s top-k token IDs. It incorporates a coverage loss over this top-k set with a coverage loss on the top-k set and a KL to a fixed rank-based target at high-entropy tokens. Experiments show that LadderOPD achieves performance within ±3 percentage points of standard OPD, regardless of whether top-k or full-distribution probabilities are used.

Critical strengths
  1. The paper is well-motivated, easy to follow. The studied problem is important and might attract many audiences’ attention.
  2. The experiments are rigorous and solid, claims and hypothesises are sufficiently supported by empirical evidence. The conclusion of the rank can replace the logits in OPD is non-trivial.
  3. Limitations are honestly discussed in the paper. The authors also provided failure cases which help readers judge the scope it can apply.
Critical weaknesses
  1. The experiments are quite narrow. experiments are conducted on GSM8K, MATH500 and MBPP, and all students are 7B or smaller. Broader settings, such as harder reasoning benchmarks, agentic or multi-turn tasks, tool calling, are not covered.
  2. The Full logits setting does not beat LadderOPD on GSM8K, the reason and discussion are not very clear. I suggest the authors could dive deeper into the issue.
  3. The layout leaves noticeable white space, especially around page 7-8.
Questions for authors

See weakness above.

1 comment

Thanks for the responses, I will keep my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.