- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Balanced-Clip On-policy Distillation (BCOD), a modification of sampled-token on-policy distillation that leaves positive teacher–behavior log-probability gaps unchanged while multiplying non-positive gaps by a fixed scalar . The full training system additionally inherits SCOPE’s correctness-based routing, where correct trajectories receive weighted SFT and incorrect trajectories receive the modified distillation loss. The paper provides a local asymmetric -divergence interpretation of the resulting on-policy token update and evaluates the method on mathematical reasoning benchmarks using Avg@32 and Pass@32.
Critical strengths
The paper identifies a potentially relevant failure mode of sampled-token OPD: negative teacher–student log-probability gaps may suppress student-generated alternatives even when those alternatives are not necessarily invalid. The proposed modification is simple and inexpensive, and the paper includes a useful ablation separating suppression-branch rescaling from correctness routing. Under this matched protocol, the clip-only variant raises Avg@32/Pass@32 from 49.0/69.6 to 51.1/71.7 relative to OPD.
Critical weaknesses
-
Limited methodological novelty and justification. BCOD mainly applies a fixed sign-dependent rescaling to the sampled-token OPD coefficient, while correctness routing and trajectory weighting are inherited from SCOPE. The asymmetric -divergence analysis only characterizes the local unclipped on-policy update, not the full training procedure. More importantly, the paper does not establish why this specific rescaling is preferable to other simple ways of weakening negative OPD updates, making the contribution feel incremental.
-
The empirical evidence is difficult to verify under a consistent protocol. The reported Pass@32 values are not fully reconciled with the stated per-prompt definition and benchmark sizes, and the support-transition statistics do not transparently reproduce the headline Pass@32 results. In addition, key comparisons use different checkpoints or data budgets. The matched ablation uses , while the primary result uses under a different protocol. These differences make it difficult to attribute the reported gains specifically to BCOD.
Questions for authors
- Can the authors provide a fully matched comparison isolating the BCOD operator? In particular, comparing under the same data, training horizon, checkpoint selection, and evaluation protocol would clarify whether the gain is consistently attributable to the proposed rescaling.
- Can the authors compare BCOD against alternative ways of limiting negative OPD feedback under the same budget? For example, clipping or flooring large negative teacher–student gaps would directly test whether scaling the entire negative branch is necessary.
- Can the authors clarify the exact Pass@32 evaluation procedure? Given the stated binary per-prompt definition, the reported AIME24/AIME25 values do not appear to follow directly from benchmark sizes of 30 and 15 problems. Are these numbers averaged over multiple evaluation rollout sets or produced by a resampling estimator?