- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper does on-policy distillation on prefixes sampled by the student, but replaces numerical next-token matching with local ranking of the teacher's Top-K. Rank-OPD keeps only head–head and head–tail pairs, sets the margin from the teacher's preference strength, and adds a candidate-set mass constraint. Rank-OPD-T turns under-margin pairs into a bounded probability transport and matches the resulting dense target. Experiments distill Qwen3-4B-Instruct into Qwen3-1.7B-Non-Thinking on DAPO-Math-17K and evaluate eight math benchmarks.
Critical strengths
1.Rank-OPD-T has a clear mean gain, and not all of it is dense matching. Table 1 raises the eight-benchmark average from 41.19 (OPD) to 43.69 (Rank-OPD-T), best on seven of eight sets. On the four-benchmark subset, ρ=0 scores 38.86 and ρ=0.05 scores 40.15. The teacher stays frozen and inference uses only the student. This is consequential because it is the main empirical support for replacing OPD's numerical next-token target with a ranking-corrected one.
2.The Rank-OPD pieces can be checked separately. Mass-only 34.70, HH+HT without mass 36.66, and the full objective 38.06 on the same subset; a large fixed margin is worse than the adaptive one. That supports the design as a set of choices rather than a single inseparable recipe.
Critical weaknesses
1.The main-table gain stacks two changes. Reported OPD is sampled-token k1 reverse KL; Rank-OPD-T is dense candidate-set matching plus transport. The +2.50 therefore mixes sparse-versus-dense matching with ranking correction. Standalone Rank-OPD is only +0.28 over EOPD and is lower on AIME24, AIME25, MATH-500, and HMMT Nov. This is consequential because the claim that local ranking is useful supervision is much stronger for Rank-OPD-T than for Rank-OPD, and the headline comparison does not isolate the ranking part.
2.Rank-OPD-T is not shown as a plug-and-play component of existing OPD objectives. The text describes it as a drop-in correction for distribution-matching OPD, but it is trained as a standalone target and is never attached to EOPD or AOPD. All reported results also stay on one Qwen3 4B→1.7B pairing and math contest sets. This is consequential because the evidence then supports a recipe in this setting more directly than a general change to OPD supervision.
Questions for authors
1.Does Rank-OPD constrain the order of tokens in the generated response, or only pairwise next-token scores within the teacher's Top-K at the current prefix? Concretely, is a Rank-OPD-T violation only a pair whose student logit gap is still below the adaptive margin?
2.Table 1's reported OPD run is sampled-token k1 reverse KL, while Rank-OPD-T is dense matching on the candidate set plus transport. Can the main table include a matched FKL / ρ=0 row? If the claim that local ranking is useful supervision rests mainly on Rank-OPD-T, how much margin remains for standalone Rank-OPD against EOPD?
3.Rank-OPD-T is described as a plug-and-play component for distribution-matching OPD. Is there a result inserting the same transport into EOPD or AOPD under the same budget? If not, should the claim be restricted to this standalone target?
7 comments
Matched No-Transport Baseline and Standalone Rank-OPD vs EOPD (Question 2 & Weakness 1)
Thank you for the suggestion. We provide the comparisons between the forward-KL baseline and our methods in Table 1. The results show that Rank-OPD-T outperforms it by 3.16% in macro-average accuracy. This comparison supports the contribution of ranking-guided transport within the same dense-matching setup. We will include this baseline in the main table of the revised manuscript.
Table 1. Additional comparison with the FKL baseline on eight mathematical reasoning benchmarks.
| Method | AIME 2024 | AIME 2025 | AIME 2026 | MATH-500 | AMC 2023 | OlympiadBench | HMMT Feb 2026 | HMMT Nov 2025 | Macro Avg |
|---|---|---|---|---|---|---|---|---|---|
| FKL (no transport) | 33.33 | 27.71 | 23.75 | 82.45 | 74.53 | 51.11 | 19.89 | 11.45 | 40.53 |
| Rank-OPD | 35.42 | 23.75 | 25.00 | 86.60 | 72.03 | 57.49 | 21.02 | 14.79 | 42.01 |
| Rank-OPD-T | 36.04 | 27.08 | 26.46 | 87.45 | 76.25 | 58.12 | 21.21 | 16.88 | 43.69 |
Regarding the comparison between Rank-OPD and EOPD, we'd like to clarify that Rank-OPD was initially designed as a baseline to explore whether the teacher’s local ranking structure could provide useful supervision for OPD. Surprisingly, we found that this simple design is effective: without dense distribution matching, this ranking baseline achieves performance comparable to EOPD, a strong OPD variant that additionally requires full-vocabulary teacher entropy computation at every token position, with significantly higher training efficiency, as shown in Table 2. Building upon these findings, we developed Rank-OPD-T, which further strengthens distribution matching with a ranking-refined teacher target , achieving the highest macro-average accuracy among all evaluated OPD variants.
Table 2: Training time, peak GPU memory, and macro-average accuracy for Rank-OPD and EOPD.
| Method | Training Time | Peak GPU Memory (GiB) | Macro Avg. Accuracy (%) |
|---|---|---|---|
| Rank-OPD | 6h 04m 59s | 21.40 | 42.01 |
| EOPD | 6h 30m 00s | 27.92 | 41.73 |
Applicability to Existing OPD Methods (Question 3)
Table 3. Accuracy (%) of EOPD and EOPD-T on eight mathematical reasoning benchmarks.
| Method | AIME 2024 | AIME 2025 | AIME 2026 | MATH-500 | AMC 2023 | OlympiadBench | HMMT Feb 2026 | HMMT Nov 2025 | Macro Avg |
|---|---|---|---|---|---|---|---|---|---|
| EOPD | 35.83 | 24.17 | 22.71 | 86.85 | 71.72 | 57.42 | 19.32 | 15.83 | 41.73 |
| EOPD-T | 34.17 | 24.79 | 23.54 | 86.90 | 74.06 | 58.20 | 21.40 | 16.04 | 42.39 |
Following your suggestion, we apply our ranking-guided transport mechanism to EOPD, yielding EOPD-T. As shown in Table 3, EOPD-T outperforms EOPD on seven of the eight benchmarks and improves the macro-average accuracy from 41.73% to 42.39% (+0.66 pp), with only a slight drop on AIME 2024. This suggests that our ranking-guided transport mechanism can generalize beyond vanilla OPD.
We sincerely thank you for the thoughtful review and for recognizing the empirical improvements and component analyses of our methods. We appreciate the constructive comments and address each question below.
Clarification of Local Ranking Supervision (Question 1)
Thank you for raising this concern. Rank-OPD constrains the relative scores of alternative next tokens for the same position. At each student-generated prefix, both candidates in a pair come from the teacher’s top-K set. As described in Section 3.1, we retain head–head (HH) and head–tail (HT) pairs.
Rank-OPD-T uses the same violation criterion as Rank-OPD. For a retained pair (i,j) where the teacher prefers i to j, a violation occurs precisely when the student’s logit gap is below the teacher-gap-adaptive margin, gt,ijS<mt,ij. This includes both reversed preferences and correctly ordered pairs with insufficient separation. A gap equal to the margin satisfies the constraint.
A violated pair contributes to the pairwise hinge loss in Rank-OPD and triggers bounded probability transport in Rank-OPD-T. A pair that meets the margin contributes zero hinge loss and triggers zero transport of its own.