acceptodds
Desk rejected at ICLR 2027

BCOD: Relaxing Suppressive Token Feedback in On-Policy Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) provides dense teacher supervision on student-generated reasoning trajectories, but its sampled-token reverse-KL update can suppress student-sampled alternatives that the teacher assigns lower probability. We propose Balanced-Clip On-policy Distillation (BCOD), which preserves positive teacher–behavior gaps and scales non-positive gaps by . At the on-policy point, its expected token update descends a convex asymmetric -divergence with the same teacher-matching optimum as reverse KL. The operator needs only one rollout. In a matched ablation isolating the operator, attenuation improves Avg@32/Pass@32 by over OPD; in the full evaluated system, which inherits SCOPE's correctness routing, BCOD reaches Avg@32 and Pass@32, versus and for standard OPD (five datasets; SkyWork-OR1-7B as teacher, R1-Distill-Qwen-1.5B as student). On fixed sets of 32 rollouts per prompt, BCOD also retains more observed base-model solutions. These results support attenuating suppressive token feedback in the evaluated OPD setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Distillation, Reasoning Language Models, Asymmetric Divergences

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2rW2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Balanced-Clip On-policy Distillation (BCOD), a modification of sampled-token on-policy distillation that leaves positive teacher–behavior log-probability gaps unchanged while multiplying non-positive gaps by a fixed scalar λ\lambda. The full training system additionally inherits SCOPE’s correctness-based routing, where correct trajectories receive weighted SFT and incorrect trajectories receive the modified distillation loss. The paper provides a local asymmetric ff-divergence interpretation of the resulting on-policy token update and evaluates the method on mathematical reasoning benchmarks using Avg@32 and Pass@32.

Critical strengths

The paper identifies a potentially relevant failure mode of sampled-token OPD: negative teacher–student log-probability gaps may suppress student-generated alternatives even when those alternatives are not necessarily invalid. The proposed modification is simple and inexpensive, and the paper includes a useful 2×22\times2 ablation separating suppression-branch rescaling from correctness routing. Under this matched protocol, the clip-only variant raises Avg@32/Pass@32 from 49.0/69.6 to 51.1/71.7 relative to OPD.

Critical weaknesses
  1. Limited methodological novelty and justification. BCOD mainly applies a fixed sign-dependent rescaling to the sampled-token OPD coefficient, while correctness routing and trajectory weighting are inherited from SCOPE. The asymmetric ff-divergence analysis only characterizes the local unclipped on-policy update, not the full training procedure. More importantly, the paper does not establish why this specific rescaling is preferable to other simple ways of weakening negative OPD updates, making the contribution feel incremental.

  2. The empirical evidence is difficult to verify under a consistent protocol. The reported Pass@32 values are not fully reconciled with the stated per-prompt definition and benchmark sizes, and the support-transition statistics do not transparently reproduce the headline Pass@32 results. In addition, key comparisons use different checkpoints or data budgets. The matched ablation uses λ=0.5\lambda=0.5, while the primary result uses λ=0.3\lambda=0.3 under a different protocol. These differences make it difficult to attribute the reported gains specifically to BCOD.

Questions for authors
  1. Can the authors provide a fully matched comparison isolating the BCOD operator? In particular, comparing λ=1.0,0.5,0.3\lambda=1.0,0.5,0.3 under the same data, training horizon, checkpoint selection, and evaluation protocol would clarify whether the gain is consistently attributable to the proposed rescaling.
  2. Can the authors compare BCOD against alternative ways of limiting negative OPD feedback under the same budget? For example, clipping or flooring large negative teacher–student gaps would directly test whether scaling the entire negative branch is necessary.
  3. Can the authors clarify the exact Pass@32 evaluation procedure? Given the stated binary per-prompt definition, the reported AIME24/AIME25 values do not appear to follow directly from benchmark sizes of 30 and 15 problems. Are these numbers averaged over multiple evaluation rollout sets or produced by a resampling estimator?
Reviewer_67wqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes BOCD (Balanced-Clip On-policy Distillation) which selectively relaxes non-positive token probability gaps in OPD, and reports both avg@32 and pass@32 with 2 models (Qwen2.5-based & Qwen3) on 5 math benchmarks across 7 baseline methods.

Critical strengths

None

Critical weaknesses
  • Authors provide Proposition 1 but this is just the mathematical expression of the part of the method. I think, in general, a proposition should be a noteworthy result, though not quite at the level of a theorem. In addition, bolding the proposed method in Table 1 rather than the best result is inappropriate.
  • The motivation of BOCD is unclear, and the algorithmic improvement is heuristic. Diversity collapse is a well known issue in RLVR but why did this paper address OPD algorithms? λ\lambda is very heuristic and I see no sensitivity analysis on λ\lambda. I think the paper should start by clearly motivating existing OPD issues with prior work or analysis to justify the proposed method.
Questions for authors
  • Why does a suppression branch that can reduce the probability of student-sampled alternatives improve accuracy and diversity?
  • In Table.1, BCOD achieves clear improvements in both pass@1 and pass@32 compared to other baseline methods across benchmarks. Could the authors provide their rationales behind this?
Reviewer_NfZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the sign structure of the sampled-token reverse-KL signal used in on-policy distillation (OPD). For a teacher-student log probability gap, the method distinguishes a positive “boost” branch from a non-positive “suppression” branch. The authors argue that suppressing a student-sampled token merely because the teacher assigns it lower probability can harm solution coverage when multiple valid reasoning paths exist. To solve this problem, authors proposed Balanced-Clip On-Policy Distillation (BCOD) leaves positive gaps unchanged and scales non-positive gaps by a factor. It is worth noting that the evaluated full BCOD system inherits SCOPE method architecture, and the truly method design is only the decoupled clip operation with a factor. On the primary results, BCOD achieved some extent of improvement compared with standard OPD.

Critical strengths
  1. This work presents a simple but useful modification based on SCOPE. Separating positive and non-positive teacher–student gaps makes the mechanism easy to reason about, and the asymmetric f-divergence derivation clarifies that attenuation can preserve the teacher-matching optimum while changing the local geometry of the update.

  2. The method has a practical sampling-cost advantage which is defined with only one rollout per prompt.

Critical weaknesses
  1. The misalignment between BCOD and actual full system. In abstract, BCOD stems from the standard OPD, however in main text BCOD system actually inherits SCOPE method architecture. Moreover, the main results in table 1 is also misalignment. BCOD is based on SCOPE, so the baseline should also be the SCOPE instead of current base. In current table 1, a large part of the improvement with BCOD comes from the SCOPE instead of BCOD method the authors added(SCOPE-N4 improves nearly 5 points, and BCOD only improves about 2 points compared with SCOPE-N4 in average, in table 1 upper part.). The ablation study in table 2 would reach same conclusion. So if this method is improving SCOPE, at least, it should be called BC-SCOPE for example, instead of BCOD, because the SCOPE part in your full system is not your contribution or work.

  2. Following weakness 1, the novel algorithm design or improvement is only a factor which is too small to be regarded as a work innovation. Of course, it's not the case that the more complex the algorithm design is, the better. But such adding a heuristic hyperparameter is more likely a training trick.

Questions for authors
  1. Can the authors run a controlled λ sweep (for example 0.1, 0.3, 0.5, 0.7, 1.0) under the same routing, data, and optimization protocol? The numerical selection is heuristic and need according ablation study.

  2. Can the authors provide the standard OPD and BCOD based on OPD instead of SCOPE comparison at an identical optimization horizon for better comparing the actual contribution of BCOD isolating the influence of SCOPE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.