- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether on-policy distillation can be performed effectively using only the teacher’s top-k token candidates and their ordering, rather than exact probabilities. Through GroundKL analysis, the authors measure correlations between limited teacher information and the exact per-token OPD signal, observing that the signal is concentrated at positions with high student entropy. Based on these findings, LadderOPD combines a coverage loss that concentrates the student’s probability mass within the teacher’s top-k candidate set with an order loss that matches the student’s distribution within that set to a fixed target distribution constructed from the ranks. The authors compare GSM8K performance across five teacher–student settings and report performance comparable to probability-based OPD through equivalence tests with a margin of 3 percentage points.
Critical strengths
-
The paper distinguishes different levels of teacher information available for on-policy distillation and systematically investigates the effects of removing exact probabilities. The research question concerning the relationship between teacher information and distillation performance is interesting.
-
The main method comparisons use multiple shared random seeds and equivalence tests, rather than relying solely on the absence of statistically significant differences. This is an appropriate statistical approach for comparing distillation with limited teacher information against probability-based OPD.
-
The evaluation covers multiple teacher–student configurations across two model families and extends to code generation. The paper also includes progressive ranking corruption experiments and component ablations. Furthermore, it reports limitations on open-ended tasks and across different tokenizers, clarifying the scope of its findings.
Critical weaknesses
1. The practical need for the assumed teacher-access scenario is unclear.
The main weakness concerns the assumed teacher-access scenario. The problem formulation of distinguishing levels of teacher information and investigating the minimum information needed for distillation is interesting, and considering an interface that provides only rankings offers some novelty. However, the paper does not sufficiently explain how often scenarios arise in which top-k candidates and their ordering are available but probabilities are not. Text-only interfaces cannot provide the required rankings, while interfaces that provide top-k probabilities already offer information that the proposed method deliberately discards. Concrete use cases for ranking-only interfaces and reasons for revealing rankings instead of probabilities are needed.
2. The core method is simple and heuristic, and the validity of the chosen target distribution is insufficiently established.
The core of the proposed method is to assign fixed probabilities according to the teacher’s ranking and use them as a target distribution to train the student. However, the rationale for choosing an exponential rank distribution, and why this form is more appropriate than other probability assignment schemes, is not sufficiently explained. Simplicity itself is not a problem, but the reasons why this heuristic works and which components are necessary require clearer validation.
3. The sensitivity analysis of the temperature parameter across model sizes is insufficient.
The temperature parameter is a central design element for converting rankings into target probabilities, but the sensitivity experiments are limited to the Qwen 7B→1.5B setting. Smaller students may benefit more from a target distribution concentrated on the teacher’s top candidate, whereas students with greater capacity and stronger initial performance may make more effective use of a less concentrated distribution that allocates probability to alternative candidates. This is a hypothesis that requires validation, and the current results do not establish how much the optimal temperature depends on student size or the teacher–student gap. Temperature sweeps under consistent conditions across multiple model configurations are needed, for example by fixing the teacher and varying the student size.
4. Comparisons across training budgets are insufficient.
The current results are based on limited training budgets of 150 updates for the main settings and 40 updates for the 32B→7B setting. Similar performance between rank-based and probability-based OPD under these conditions does not establish that exact probabilities offer no additional benefit with more training. Rankings may suffice at small budgets, while actual probabilities could become more useful as training progresses. Performance curves over training updates or compute budgets are needed to determine whether the observed equivalence holds throughout training and under sufficiently trained conditions.
5. Unequal numbers of runs across experiments raise concerns about the interpretation and robustness of some comparisons.
The temperature experiments in Table 13 report results from one, three, or six runs depending on the setting. A single run cannot characterize variability across seeds, and directly comparing means computed from different numbers of runs and different seed sets makes it difficult to distinguish effects from random variation. In Table 2, the main access-level comparisons use six runs, while some baselines and controls use three. Although the authors disclose the run counts and conduct the main statistical tests on shared seeds, the means presented in the tables may not correspond to the samples used in the paired comparisons.
6. The connection between GroundKL analysis and the actual training objective is insufficiently validated.
The ranking analysis in Table 1 measures the Spearman correlation between the teacher’s negative rank and the exact OPD signal. In contrast, LadderOPD converts rankings into a fixed probability distribution and uses a KL loss that assigns approximately 86.5% probability to the top-ranked candidate under the default setting. Therefore, a correlation between ranking and the OPD signal does not directly establish that training with this target distribution provides correction directions similar to those of the original OPD.
7. The experimental scope is limited for assessing generalization.
Although the paper includes two model families and several evaluation tasks, the main experiments focus on small Qwen students and GSM8K. Student sizes are limited to 7B, and code-generation validation is restricted to MBPP. Some transfer evaluations use a single seed and small evaluation sets, making the robustness of these results difficult to assess.
Questions for authors
See the weaknesses above. My primary concern is that the assumed teacher-access scenario is not sufficiently justified.
Thanks for the responses, I will keep my positive rating.