acceptodds
Withdrawn from ICLR 2027

KB-OPD: KL-Bridge On-Policy Distillation with Behavior-Corrected Prefix Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) provides dense token-level supervision on student-generated trajectories, while generalized OPD (G-OPD) extends this framework through reward extrapolation. However, both depend on the prefixes visited by the student, which can limit access to productive reasoning trajectories early in training. Teacher guidance can improve these prefixes, but incorporating teacher-guided tokens into policy updates requires accounting for the mismatch between the sampling and student distributions. We propose KB-OPD, which combines constrained prefix sampling with reward extrapolation to learn from both guided prefixes and student-generated continuations. During a short warm-up phase, KB-OPD geometrically blends student and teacher distributions, selecting the largest teacher influence permitted by local and cumulative KL budgets. These constraints control both token-level and sequence-level deviation from the student, while a stopping rule hands generation back to the student for the remaining response. Clipped behavior importance weights and a weighted prefix–suffix objective incorporate bridge tokens into policy updates alongside student-generated tokens. After warm-up, sampling returns to pure student rollouts while retaining the G-OPD supervision signal. Experiments on mathematical reasoning and code generation show improved average performance over OPD and ExOPD across single-teacher, multi-teacher, and strong-to-weak distillation settings.

Primary area:
reinforcement learning
Keywords:
On-Policy Distillation, Large Language Models, Trust-Region Exploration, Importance Sampling, Reward Extrapolation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wis2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

KB-OPD extends extrapolated on-policy distillation with teacher-guided prefix generation during a short warm-up. It geometrically blends teacher and student distributions, choosing the largest mixing coefficient permitted by local and cumulative KL budgets before handing generation back to the student. Guided tokens contribute through clipped action-level importance weights and a separately normalized prefix loss. The paper derives a sequence-level KL bound and evaluates the method on mathematical reasoning and code generation, including single-teacher, multi-teacher, and strong-to-weak settings. The reported domain-average improvements over ExOPD range from 0.4 to 1.2 percentage points.

Critical strengths
  1. The paper addresses a relevant problem: poor early student prefixes can limit effective teacher supervision. The geometric bridge and cumulative budget provide a coherent way to control teacher intervention, and the sequence-level KL derivation is sound under the stated assumptions (Section 3.2; Appendix A.2–A.3). Appendix A.4 also appropriately acknowledges the limits of action-level correction and the bias introduced by clipping.

  2. The complete procedure shows encouraging consistency across the tested settings, improving all six reported domain/setting averages over ExOPD (Tables 1–2). The inclusion of both mathematics and code, together with component and warm-up ablations, provides useful initial evidence beyond a single benchmark result.

Critical weaknesses
  1. The significance and independent contribution of the proposed combination remain insufficiently established. TRB (Plyusov et al., 2026) already combines geometric behavior blending, local KL-constrained teacher influence, and warm-up guidance, while MiniLLM (Gu et al., ICLR 2024, Section 2.2) already uses teacher-mixed sampling with approximate per-token importance correction. Reward extrapolation is inherited from G-OPD/ExOPD (Yang et al., 2026), as the manuscript acknowledges. KB-OPD adds cumulative budgeting and a particular prefix/suffix update design, but compares mainly with unguided OPD and ExOPD (Section 4.1, lines 299-303). A comparison with the closest guided method on the same ExOPD backbone is needed to establish the value of these additions. Moreover, the experiments do not directly connect the gains to improved access to productive prefixes or establish whether the benefit persists with stronger student initialization. Results on Qwen3 1.7B/4B students over 50 steps provide limited support for a broadly reusable contribution, and do not establish the asymptotic-performance claim in lines 317-319.

  2. The evaluation does not establish the robustness, attribution, or resource value of the modest gains. Appendix B.2 explicitly selects checkpoints by the highest evaluation-suite mean (lines 859-860), so the reported benchmarks also guide model selection. The 0.4-1.2-point gains over ExOPD are reported without independent training repeats or uncertainty estimates; multiple evaluation completions do not address training variability. Some ablation interpretations also remain ambiguous: retaining KL constraints in the pure-teacher variant can change guidance feasibility and prefix length (Table 3), while the claimed wider extrapolation window lacks ExOPD comparisons at matched lambda values (Section 5.4, lines 460-472). Finally, online teacher decoding and longer training responses introduce additional computation, but actual token costs and GPU-hours are not reported. These gaps make it difficult to determine whether the gains reliably justify the added mechanisms.

Questions for authors
  1. Please clarify the checkpoint-selection procedure in Appendix B.2, lines 859–860, including whether multi-teacher math and code results come from the same checkpoint. Can you provide prespecified final-step or independently validation-selected results, together with variability across independent training runs?

  2. How does KB-OPD compare with TRB-style guidance on the same ExOPD backbone under comparable tuning and compute budgets? This targeted comparison would help establish the incremental contribution beyond existing guided distillation (Section 4.1, lines 299–303).

  3. What evidence connects the gains to the proposed early-prefix bottleneck? A diagnostic comparing student continuation success from matched-length prefixes, or evidence across stronger student initializations, would help establish when the intervention remains useful (Section 3.2; Section 4.1, lines 305–309).

  4. Which KL constraints remain active in the pure-teacher ablation, and how do its guidance feasibility and prefix lengths compare with the default? These details are needed to interpret the decrease in Table 3.

  5. Is bridge guidance active at the λ=1\lambda=1 point labeled OPD in Figure 5? A matched ExOPD λ\lambda sweep would help determine whether guidance specifically broadens the useful extrapolation range (Section 5.4, lines 460–472).

  6. What are the additional teacher-decoding tokens, total generated tokens, and GPU-hours? An equal-compute comparison with ExOPD or the closest guided baseline would help determine whether the reported accuracy improvements justify their cost (Section 3.4; Figure 3).

Reviewer_KVhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes KL-Bridge On-Policy Distillation (KB-OPD), an extension of ExOPD designed to improve the quality of prefixes in sampled trajectories. While the proposed method shows empirical improvements, it introduces substantial complexity relative to the modest gains reported. Moreover, the limited baseline comparisons and ablation studies do not sufficiently establish the benefits of the proposed components.

Critical strengths
  1. The paper is logically organized and easy to follow.
  2. The methodology is clearly presented.
  3. The experimental results show improvements over the evaluated baselines.
Critical weaknesses
  1. KB-OPD requires maintaining both models and computing their mixed policy during sampling. This appears to introduce substantial implementation complexity and may complicate integration with widely used inference engines, such as vLLM and SGLang. The computational overhead and practical implications are insufficiently explored.
  2. The experiments include only OPD and ExOPD as baselines, omitting relevant teacher-guided sampling methods, such as TRB and SKD. Since the KL-Bridge sampling strategy is presented as a central contribution, comparisons with these methods are necessary to establish its relative effectiveness.
  3. The method introduces numerous configurable elements and hyperparameters, including A+,αmin⁡,αmax⁡,η,C,Lmax⁡,w,γB,wmin⁡,\mathcal{A}{+}, \alpha{\min}, \alpha_{\max}, \eta, C, L_{\max}, w, \gamma_B, w_{\min}, and wmax⁡w_{\max}. The current ablation studies do not adequately isolate their contributions or assess sensitivity to their settings. Given the modest performance gains, the paper does not convincingly demonstrate that the added complexity is justified.
  4. The authors save checkpoints every five steps and report results from the checkpoint with the highest evaluation-suite score. If the same suite is used for both checkpoint selection and final reporting, this procedure risks biasing the reported performance upward.
Questions for authors
  1. Could the authors clarify how mixed-policy sampling is implemented? What additional computational and memory costs does it incur, and what modifications are required to the underlying inference engine? Does the implementation scale to larger models with TP/PP/EP?
  2. Could the authors include comparisons with relevant teacher-guided sampling methods, particularly TRB and SKD, under comparable training and evaluation conditions?
  3. Could the authors provide more comprehensive component ablations and hyperparameter sensitivity analyses, together with a clear account of how the settings were selected? Given the number of interacting components, evidence of robustness across reasonable configurations would help address concerns about reproducibility and tuning burden.
  4. Could the authors report results across multiple independent runs, including means and standard deviations, and clarify whether checkpoint selection uses a separate validation set? Reporting only the best checkpoint from a single run does not adequately characterize variability and may undermine the fairness of the comparison, particularly if checkpoint-selection procedures differ across methods.
Reviewer_Ve5EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses the early-training weakness of on-policy distillation: poor student-generated prefixes can limit access to productive reasoning trajectories. During a short warm-up, it samples prefixes from a geometric interpolation of teacher and student distributions, choosing the largest teacher influence permitted by local and cumulative KL budgets before handing generation back to the student. The paper combines a lot of prior work. It learns from both segments using behavior-corrected updates and ExOPD-style reward extrapolation. Geometric interpolation, local KL-constrained teacher guidance, and temporary warm-up already appear in TRB, while reward extrapolation comes from G-OPD/ExOPD. The paper’s specific contribution is their combination with an explicit cumulative prefix budget, adaptive handover, and separately weighted prefix–suffix learning, although it does not directly compare against TRB.

Critical strengths

The paper presents a coherent implementation of constrained teacher guidance, behavior-corrected updates, and reward extrapolation. Although the components largely build on prior work, their integration is clearly specified and accompanied by mathematical analysis of the sampling constraints.

The experiments show consistent average improvements over OPD and ExOPD across math and coding, including single-teacher, multi-teacher, and strong-to-weak settings. Component ablations provide useful evidence about the roles of prefix learning, behavior correction, and the two KL budgets, although they do not substitute for comparisons with the closest guided-sampling methods.

Critical weaknesses

The central sampling idea substantially overlaps with TRB, which already motivates early teacher guidance, geometrically interpolates teacher and student policies under a local KL constraint, and returns to student-only sampling after warm-up. KB-OPD adds a cumulative budget and prefix-specific stopping rule, but does not compare against TRB to establish the value of these extensions.

[1] Plyusov et al. (2026), “Trust-Region Behavior Blending for On-Policy Distillation,” arXiv:2605.31159.

The broader idea of improving student trajectories through teacher intervention during generation is also established by SKD. Although its token-replacement mechanism differs, it addresses the same tradeoff between trajectory quality and proximity to student behavior. A direct comparison is needed to justify KB-OPD’s additional budgeting machinery.

[2] Xu et al. (ICLR 2025), “Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling,” arXiv:2410.11325.

Reward extrapolation is inherited from G-OPD/ExOPD. The comparison against ExOPD demonstrates the benefit of adding guided sampling, but does not distinguish the proposed method from simply combining TRB with the same extrapolated objective. This is a crucial missing baseline.

[3] Yang et al. (2026), “Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation,” arXiv:2602.12125.

The remaining components—behavior importance weighting and separate prefix/suffix normalization—are reasonable adaptations of standard optimization techniques. Their ablations establish usefulness within this implementation, but do not by themselves demonstrate substantial methodological novelty.

[4] Precup et al. (2000), “Eligibility Traces for Off-Policy Policy Evaluation”; Munos et al. (2016), “Safe and Efficient Off-Policy Reinforcement Learning.”

Overall, the reported gains over ExOPD are modest, and the method appears largely to assemble established ideas. Even small improvements over the missing baselines would not alone resolve this concern of lack of novelty; the paper needs evidence that its additions deliver a robust, practically meaningful advantage under matched computational budgets.

Questions for authors

Why was the method not compared against TRB, which seems to contain the core idea ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.