- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates sampled-token on-policy distillation (OPD) when teacher and student models differ in scale or reasoning behavior. The authors argue that the sampled token’s log-ratio can miss unfavorable changes in its preference relative to other tokens, which they call “sampled-token tunnel vision.” They also give an example showing that aggregating local reverse-KL contributions can lose information about the desired update direction. Their method, NC-OPD, subtracts a weighted average of neighboring log-ratios from the sampled token’s log-ratio. The neighbors include tokens ranked above the sampled token by the teacher and tokens ranked below it by the student. Experiments on Qwen3 models report improvements over the baselines tested in mathematical reasoning, code generation, multi-teacher distillation, and cross-domain supervision.
Critical strengths
-
A clear rationale for anchor-relative advantages. The swap example in Section 4 shows that the same local reverse-KL aggregate can correspond to opposite relative update directions. The pairwise contrast underlying Eq.(2) distinguishes these cases while retaining the sampled-token policy-gradient interface. This helps explain why relative-preference information is useful for advantage estimation.
-
Consistent improvements over several baselines. The experiments compare NC-OPD with standard OPD, Trust Region OPD, Student Top-K, Teacher Top-K, and Hybrid OPD. NC-OPD has the highest reported score on all three mathematical benchmarks. Its average gains over the strongest baselines are 3.00 and 3.17 percentage points in the two settings. It also improves on the strongest baselines by 2.25 points in code generation and 2.15 points in multi-teacher distillation, suggesting that the benefits extend beyond a single task. The ablations in Fig.6 and the contribution analysis in Fig.7 offer explanations for which neighbors are useful that can be tested further.
Critical weaknesses
-
The evaluated algorithm is not uniquely specified. The main-text weight in Eq.(3) is proportional to π_{\theta}(j)^α, whereas Appendix B, Eq.(5), additionally multiplies it by |r_i − r_a|. This factor reweights neighbors and can change the aggregate advantage’s sign when individual contrasts disagree. Section 6.1 caps both neighbor subsets at k = 8, whereas Appendix B retains eight student-below candidates but all teacher-above candidates within a top-60 pool. Furthermore, the probability inequalities defining the categories on lines 322–325 are reversed relative to the verbal descriptions on lines 326, 330, and 340 and the neighborhood described in the abstract. The reversed log-ratio convention is not itself an error: substituting Eq.(4) into Eq.(6) recovers the main-text contrast when weights and neighborhoods agree. However, they do not agree as written. These discrepancies prevent the reader from uniquely identifying the evaluated variant and assessing which components account for the reported gains.
-
The evidence does not fully support the proposed mechanism. Figures 2 and 5 compare cross-origin advantages with a same-origin reference. However, different students may need different updates on the same prefix, so disagreement with the reference does not by itself mean that the guidance is wrong. It is also unclear whether the comparisons use matched prefixes and sampled tokens. Under on-policy sampling and the usual differentiation assumptions, the sequence-level REINFORCE estimator in Section 2 is unbiased for the stated reverse-KL objective. NC-OPD subtracts a neighborhood baseline that depends on the sampled token, which generally changes the expected update. The paper should therefore clarify whether “tunnel vision” refers to unfavorable individual updates or bias in the expected gradient, and whether NC-OPD preserves the original objective or deliberately changes the update rule. Teacher Top-K and Hybrid OPD already use information from multiple tokens. NC-OPD’s gains over these methods therefore need more explanation than the inclusion of neighboring candidates. Comparisons that separately vary the contrast, candidate selection, and weighting would help identify what drives the improvement. In Fig.6, removing a category also removes tokens shared with other categories and changes the total number of neighbors. The resulting performance differences cannot be attributed to category membership alone.
-
Appendix B describes validation with eight responses per problem on AIME and AMC every two training steps, while Section 6.1 reports avg@32 results. These can be separate stages of evaluation, but the checkpoint-selection rule and the relationship between the two evaluation protocols are not clearly explained. Fig.1(b) and Table 2 also report different gains for the same transfer setting without explaining whether they use different checkpoints or evaluation settings. In the multi-teacher experiment, Sections 6.1 and 6.3 identify the math and code teachers but do not explain how each training sample is assigned to a teacher or how their signals enter the NC-OPD advantage. These details are needed to reproduce the experiments and interpret the reported comparisons.
-
The caption of Fig.1(c) says GPT-5.6 Terra was used for the analysis, whereas Appendix C.2 explicitly describes a fixed lexical matcher and states that no LLM-as-a-judge prompt was used. The figure reports percentages, while Eq.(9) defines occurrences per 1,000 words. The paper needs to explain which procedure produced the plotted values and how they were calculated. Appendix C.2 already notes that these lexical measurements describe observable response behavior rather than semantic correctness. The interpretation of Fig. 1(c) should reflect this limitation, once the analysis method and units are clarified.
Questions for authors
-
Which version of NC-OPD was used for Table 2 and Fig.8? Eq.(3) and Appendix B give different weighting and neighborhood rules. Please specify whether the additional gap factor in Eq.(5) was used, whether the teacher-side neighborhood was capped at eight tokens or included all eligible candidates within the top-60 pool, and which ranking conditions were implemented given the conflict between lines 322–325 and the prose. Appendix B states that confidence calibration was disabled. Does this apply to all reported experiments?
-
What does “directional unreliability” mean here: unfavorable individual updates or bias in the expected gradient? Please explain how the token-level implementation relates to the sequence-level objective in Section 2. For Figs.2 and 5, are the signals compared on the same prefixes and sampled tokens, and why is the same-origin signal a suitable reference when the students have different distributions? Since the neighborhood baseline depends on the sampled token, does NC-OPD preserve the original objective or intentionally change the expected update?
-
How were the reported checkpoints selected, and were the AIME/AMC validation scores used for this purpose?
-
Does “Qwen3-8B-TIR-DAPO” in Appendix B refer to the Qwen3-8B thinking teacher trained for 400 RL steps on DAPO in Section 6.1? Also, which teacher and student checkpoints were used for the same-origin reference in Fig.2? Section 3 specifies the cross-origin pair, but the corresponding same-origin pair is not clearly identified.
-
For Fig.6, which benchmark does “Performance” refer to, and at what training step was it measured?
Response to the authors
Thank you to the authors for their response.
The additional ablation study addresses one of my most critical concerns about the original submission.
However, I maintain my rating for the following reason.
A clear motivation and an effective methodology alone do not establish the soundness of a paper. The theoretical derivation connecting them also matters. I remain unconvinced by the original theoretical justification, and I consider the new explanation in the rebuttal to substantially change the interpretation of the proposed method.
In particular, the authors now clarify that NC-OPD is not an unbiased estimator of the original reverse-KL gradient, but instead introduces a different expected update through an adaptive surrogate. I consider this a substantive change in the theoretical interpretation rather than a minor clarification.
Therefore, while I appreciate the additional experiments and clarification, I believe the theoretical foundation requires more substantial revision than can be adequately assessed during the rebuttal period.
Accordingly, I maintain my rating.