acceptodds
Under review as a conference paper at ICLR 2027

Breaking Sampled-Token Tunnel Vision Enables Robust Capability Transfer

Abstract

On-policy distillation (OPD) efficiently transfers capabilities acquired through costly post-training by providing dense teacher supervision along student-generated trajectories. Yet sampled-token OPD can become unreliable in cross-origin teacher–student settings. Through controlled comparisons between successful and failed training regimes, we identify sampled-token tunnel vision: analogous to human tunnel vision, the advantage estimator focuses on a single sampled token's teacher–student probability ratio while overlooking the update's effects on surrounding alternatives. This narrow view can obscure the suppression of teacher-preferred candidates or the amplification of less-preferred ones, undermining capability transfer. Simply aggregating reverse-KL contributions over nearby tokens can also lose directional information: the same aggregate may arise when the sampled token should be promoted in one case but suppressed in another, relative to its neighbors. We therefore propose **Neighborhood Contrastive On-Policy Distillation** (NC-OPD), which reconstructs the sampled-token advantage through pairwise comparisons within a lightweight neighborhood of tokens ranked above the sampled token by the teacher or below it by the student. This provides explicit relative-preference guidance while preserving the original sampled-token policy-gradient interface. Across cross-scale and cross-reasoning-mode transfer settings, **NC-OPD** consistently improves mathematical reasoning, code generation, and multi-domain performance. These results highlight neighborhood-relative advantage estimation as an effective research line to robust on-policy capability transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Post-training, capability transfer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_B8BBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates sampled-token on-policy distillation (OPD) when teacher and student models differ in scale or reasoning behavior. The authors argue that the sampled token’s log-ratio can miss unfavorable changes in its preference relative to other tokens, which they call “sampled-token tunnel vision.” They also give an example showing that aggregating local reverse-KL contributions can lose information about the desired update direction. Their method, NC-OPD, subtracts a weighted average of neighboring log-ratios from the sampled token’s log-ratio. The neighbors include tokens ranked above the sampled token by the teacher and tokens ranked below it by the student. Experiments on Qwen3 models report improvements over the baselines tested in mathematical reasoning, code generation, multi-teacher distillation, and cross-domain supervision.

Critical strengths
  1. A clear rationale for anchor-relative advantages. The swap example in Section 4 shows that the same local reverse-KL aggregate can correspond to opposite relative update directions. The pairwise contrast underlying Eq.(2) distinguishes these cases while retaining the sampled-token policy-gradient interface. This helps explain why relative-preference information is useful for advantage estimation.

  2. Consistent improvements over several baselines. The experiments compare NC-OPD with standard OPD, Trust Region OPD, Student Top-K, Teacher Top-K, and Hybrid OPD. NC-OPD has the highest reported score on all three mathematical benchmarks. Its average gains over the strongest baselines are 3.00 and 3.17 percentage points in the two settings. It also improves on the strongest baselines by 2.25 points in code generation and 2.15 points in multi-teacher distillation, suggesting that the benefits extend beyond a single task. The ablations in Fig.6 and the contribution analysis in Fig.7 offer explanations for which neighbors are useful that can be tested further.

Critical weaknesses
  1. The evaluated algorithm is not uniquely specified. The main-text weight in Eq.(3) is proportional to π_{\theta}(j)^α, whereas Appendix B, Eq.(5), additionally multiplies it by |r_i − r_a|. This factor reweights neighbors and can change the aggregate advantage’s sign when individual contrasts disagree. Section 6.1 caps both neighbor subsets at k = 8, whereas Appendix B retains eight student-below candidates but all teacher-above candidates within a top-60 pool. Furthermore, the probability inequalities defining the categories on lines 322–325 are reversed relative to the verbal descriptions on lines 326, 330, and 340 and the neighborhood described in the abstract. The reversed log-ratio convention is not itself an error: substituting Eq.(4) into Eq.(6) recovers the main-text contrast when weights and neighborhoods agree. However, they do not agree as written. These discrepancies prevent the reader from uniquely identifying the evaluated variant and assessing which components account for the reported gains.

  2. The evidence does not fully support the proposed mechanism. Figures 2 and 5 compare cross-origin advantages with a same-origin reference. However, different students may need different updates on the same prefix, so disagreement with the reference does not by itself mean that the guidance is wrong. It is also unclear whether the comparisons use matched prefixes and sampled tokens. Under on-policy sampling and the usual differentiation assumptions, the sequence-level REINFORCE estimator in Section 2 is unbiased for the stated reverse-KL objective. NC-OPD subtracts a neighborhood baseline that depends on the sampled token, which generally changes the expected update. The paper should therefore clarify whether “tunnel vision” refers to unfavorable individual updates or bias in the expected gradient, and whether NC-OPD preserves the original objective or deliberately changes the update rule. Teacher Top-K and Hybrid OPD already use information from multiple tokens. NC-OPD’s gains over these methods therefore need more explanation than the inclusion of neighboring candidates. Comparisons that separately vary the contrast, candidate selection, and weighting would help identify what drives the improvement. In Fig.6, removing a category also removes tokens shared with other categories and changes the total number of neighbors. The resulting performance differences cannot be attributed to category membership alone.

  3. Appendix B describes validation with eight responses per problem on AIME and AMC every two training steps, while Section 6.1 reports avg@32 results. These can be separate stages of evaluation, but the checkpoint-selection rule and the relationship between the two evaluation protocols are not clearly explained. Fig.1(b) and Table 2 also report different gains for the same transfer setting without explaining whether they use different checkpoints or evaluation settings. In the multi-teacher experiment, Sections 6.1 and 6.3 identify the math and code teachers but do not explain how each training sample is assigned to a teacher or how their signals enter the NC-OPD advantage. These details are needed to reproduce the experiments and interpret the reported comparisons.

  4. The caption of Fig.1(c) says GPT-5.6 Terra was used for the analysis, whereas Appendix C.2 explicitly describes a fixed lexical matcher and states that no LLM-as-a-judge prompt was used. The figure reports percentages, while Eq.(9) defines occurrences per 1,000 words. The paper needs to explain which procedure produced the plotted values and how they were calculated. Appendix C.2 already notes that these lexical measurements describe observable response behavior rather than semantic correctness. The interpretation of Fig. 1(c) should reflect this limitation, once the analysis method and units are clarified.

Questions for authors
  1. Which version of NC-OPD was used for Table 2 and Fig.8? Eq.(3) and Appendix B give different weighting and neighborhood rules. Please specify whether the additional gap factor in Eq.(5) was used, whether the teacher-side neighborhood was capped at eight tokens or included all eligible candidates within the top-60 pool, and which ranking conditions were implemented given the conflict between lines 322–325 and the prose. Appendix B states that confidence calibration was disabled. Does this apply to all reported experiments?

  2. What does “directional unreliability” mean here: unfavorable individual updates or bias in the expected gradient? Please explain how the token-level implementation relates to the sequence-level objective in Section 2. For Figs.2 and 5, are the signals compared on the same prefixes and sampled tokens, and why is the same-origin signal a suitable reference when the students have different distributions? Since the neighborhood baseline depends on the sampled token, does NC-OPD preserve the original objective or intentionally change the expected update?

  3. How were the reported checkpoints selected, and were the AIME/AMC validation scores used for this purpose?

  4. Does “Qwen3-8B-TIR-DAPO” in Appendix B refer to the Qwen3-8B thinking teacher trained for 400 RL steps on DAPO in Section 6.1? Also, which teacher and student checkpoints were used for the same-origin reference in Fig.2? Section 3 specifies the cross-origin pair, but the corresponding same-origin pair is not clearly identified.

  5. For Fig.6, which benchmark does “Performance” refer to, and at what training step was it measured?

Reviewer_fXujOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles a distribution matching problem in on-policy distillation of language models. It is motivated from observation of student mimics teacher distribution, ignoring neighborhood token in selection space. NC-OPD reduces this problem by distilling token preferences including not only the sampled token, but also its neighbors. Substantial performance gain was reported and generalizability as well.

Critical strengths
  1. The motivation is clear and it raises up an important problem in on-policy distillation.
  2. The objective formulation is straightforward and easy to understand.
  3. Reports remarkable performance gains in three benchmark datasets: AMC, AIME 24, and AIME 25.
Critical weaknesses

The provided performance gain is notable and beneficial in on-policy distillation. However, there are three major concerns.

  1. Insufficient analysis on performance gains. NC-OPD shows remarkable performance gains, but the paper does not provide sufficient component analysis to support and interpret benefits from it.
  2. Lack of novelty. The concept of utilizing relationships among different tokens is very similar to logit distillation in typical knowledge distillation. NC-OPD can be viewed as an implementation or application of this concept in on-policy distillation. This perspective weakens the novelty. I have one novelty in my mind, but I am not sure about it. I will leave it in my question #1.
  3. Weak linkage between problem and results. The paper argues that mitigating sampled-token tunnel vision improves performance. This statement requires two evidences: the benchmark performance is superior to baselines; the performance gain comes from mitigating sampled-token tunnel vision. The latter evidence is missing as of now.
Questions for authors
  1. What is the authors' intension of proposingd neighborhood correction? I am confused between improving existing reverse KD gradient and deriving a novel optimization objective?
  2. Although the authors has acknowledged the weakness in theoretical analysis, I would like to ask for a brief sketch of why neighborhood correction leads to performance gains over existing OPD.
1 comment
Reviewer_fXujOpenReview ↗

Response to the authors

Thank you to the authors for their response.

The additional ablation study addresses one of my most critical concerns about the original submission.

However, I maintain my rating for the following reason.

A clear motivation and an effective methodology alone do not establish the soundness of a paper. The theoretical derivation connecting them also matters. I remain unconvinced by the original theoretical justification, and I consider the new explanation in the rebuttal to substantially change the interpretation of the proposed method.

In particular, the authors now clarify that NC-OPD is not an unbiased estimator of the original reverse-KL gradient, but instead introduces a different expected update through an adaptive surrogate. I consider this a substantive change in the theoretical interpretation rather than a minor clarification.

Therefore, while I appreciate the additional experiments and clarification, I believe the theoretical foundation requires more substantial revision than can be adequately assessed during the rebuttal period.

Accordingly, I maintain my rating.

Reviewer_NfQiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the failure of sampled-token on-policy distillation (OPD) when the teacher and student model differ substantially in their generation distributions from the practical problem the authors faced. The authors categorized such problem as “sampled-token tunnel vision”: a standard sampled-token advantage is determined by the teacher–student probability discrepancy of the sampled token only, and lead to improper suppressing or promoting this token which may not be aligned with the teacher model. To address this problem, the paper proposes Neighborhood Contrastive On-Policy Distillation (NC-OPD). Instead of using the sampled token’s pointwise log-ratio alone, NC-OPD compares the sampled token against a local set of alternatives and constructs an anchor-relative advantage. The method is evaluated on mathematical reasoning, code generation, multi-teacher distillation, and cross-domain expert supervision which all achieve significant improvement compared with other methods.

Critical strengths
  1. This work identifies a useful failure mode of OPD in practical training process and make further analysis about it to promote a possible solution which I really appreciate.

  2. The main results are strong and do demonstrate the effectiveness of NC-OPD. It is consistently better than sampled-token OPD and several distribution-alignment baselines in both main transfer settings, and the paper further tests code generation, multi-teacher distillation, and cross-domain expert supervision.

  3. The method preserves the sampled-token policy-gradient interface while using only a relatively small neighborhood. This is practically appealing compared with full-vocabulary alignment and is well aligned with the paper’s stated goal of robust capability transfer under distribution mismatch.

Critical weaknesses
  1. In main content, there is a proportion imbalance problem between analysis and experiment results. Specifically, the experiment results appeal in page 8(total 9 pages) and before this there are lots of functional analysis experiments which are too many for a work. Actually, many functional analysis can be put in the discussion after the results instead of before main experiments. The tunnel vision problem uncovering progress can be simplified and specific content can be presented in appendix. No matter how many analytical results there are, they are not as powerful as experimental results.

  2. Following up the weakness 1, this work's experiment is insufficient, though existing results could demonstrate the effectiveness of NC-OPD. The model selection is only one kind of model(Qwen3) and main benchmark selected is also insufficient which is only 2 kinds of benchmark(AIME 24 and AIME25 is the same series.).

  3. The figures could be optimized again. For example, figure 1 and figure 3 are all describing the backward of tunnel vision which is repetitive in function. Figure 6 should also be optimized. Maybe histogram is better than box plots for this figure.

Questions for authors
  1. The algorithm described in the main text section 5 does not appear to exactly match the implementation described in Appendix B, and the differences are important enough to affect reproducibility and interpretation. Which exact NC-OPD formulation and top-k selection produced the headline results in Table 2 and Figure 8?

  2. How does NC-OPD behave for teacher–student pairs from genuinely different model families?

  3. Is there any risk that NC-OPD will degenerate to SFT-like log probs alignment because NC-OPD actually takes more candidate tokens in consideration for model update? Moreover, is the top-k selection sensitive to specific training process or downstream tasks?

Comments on the paper

1 comment

Thank you for the detailed responses and clarifications. After carefully reading the rebuttal, I find that it has addressed my main concerns regarding the implementation, the interpretation of the update rule, and the evaluation protocol. I have updated my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.