acceptodds
Under review as a conference paper at ICLR 2027

Weak-to-Strong Generalization via Direct On-Policy Distillation

Abstract

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model’s RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
weak-to-strong generalization, on-policy distillation, reinforcement learning, knowledge distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DKLuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Direct On-Policy Distillation (Direct-OPD), a weak-to-strong post-training method designed to transfer the policy change induced by reinforcement learning on a small model to a stronger student without running expensive RL directly on the student. Instead of matching the weak teacher's final post-RL distribution πT\pi _ T, Direct-OPD defines the teacher policy shift as the log-probability ratio between the post-RL teacher πT\pi _ T and its pre-RL reference checkpoint πTref\pi _ {T _ {\mathrm{ref}}}, ΔT(y∣x)=log⁡πT(y∣x)−log⁡πTref(y∣x)\Delta _ T(y \mid x) = \log \pi _ T(y \mid x) - \log \pi _ {T _ {\mathrm{ref}}}(y \mid x) (Section 1 Eq. 1; Section 2.2 Eq. 5), interprets it as a dense implicit reward under KL-regularized RL (Eqs. 6–7), and applies a zero-discount, immediate-token local surrogate on the student's top-kk (k=16k=16) candidate set StS _ t along student-sampled rollouts while regularizing the student πθ\pi _ \theta toward its own initialization πS\pi _ S (Section 2.3, Eqs. 8–15) with a sign-based adaptive KL controller (Section 2.4, Eq. 16).

The method is evaluated using two 1.5B teacher checkpoint pairs (R1-Distill-1.5B →\to JustRL-1.5B and Nemotron-1.5B →\to QuestA-Nemotron-1.5B) and three students (Qwen3-1.7B, Qwen3-4B, and R1-Distill-7B) on AIME 2024 and AIME 2025 (ave@32, Section 3, Table 1, and Appendix B Table 5). In the strict weak-to-strong setting (JustRL-1.5B, which scores 51.3% / 37.5%), Direct-OPD improves stronger students (Qwen3-4B from 72.5% / 65.6% to 77.6% / 68.8% peak, or +3.87 / +1.38 points at step 300 in Table 5; R1-Distill-7B from 56.7% / 40.5% to 63.1% / 48.8% peak, or +4.37 / +6.24 points at step 300). Section 3.2 (Figure 2) further shows that 1,500 steps of 1.5B RL plus 300 steps of Direct-OPD (5,152 A100 GPU-hours) outperforms 1,500 steps of direct 7B RL (10,240 A100 GPU-hours) on R1-Distill-7B, and Section 3.3 (Appendix Figure 7) demonstrates sequential composition of the JustRL and QuestA shifts on Qwen3-1.7B. While the core transfer idea is clean and practically appealing, significant gaps between the idealized sequence objective and the implemented local top-kk surrogate, the absence of closely related baselines (WSPO, G-OPD, Proxy-Tuning), and single-seed evaluation restricted solely to AIME prevent me from recommending acceptance in the current version.

Critical strengths

1. Clean conceptual decoupling of the RL policy shift from the weak teacher's absolute capability ceiling.
Standard on-policy distillation forces the student toward πT\pi _ T, which blends the desirable improvements learned during RL with the weak base capacity of πTref\pi _ {T _ {\mathrm{ref}}}. By extracting the log-ratio shift log⁡(πT/πTref)\log(\pi _ T / \pi _ {T _ {\mathrm{ref}}}) on student-visited states while anchoring the KL penalty to the student's own initialization πS\pi _ S (Eq. 8), Direct-OPD avoids pulling a stronger student down toward a weaker teacher—as vividly demonstrated in Figure 1(a) on R1-Distill-7B (+6.4 points with Direct-OPD vs. roughly -6.7 points with vanilla OPD).

2. Encouraging weak-to-strong empirical gains and concrete compute savings over direct 7B RL.
In the strict weak-to-strong JustRL-1.5B setting (51.3% / 37.5% on AIME 24/25), Direct-OPD improves both Qwen3-4B (72.5% / 65.6% →\to 77.6% / 68.8% peak in Table 1; 76.37% / 66.98% at step 300 in Table 5) and R1-Distill-7B (56.7% / 40.5% →\to 63.1% / 48.8% peak; 61.04% / 46.77% at step 300). Furthermore, Section 3.2 (Figure 2) shows that training the 1.5B teacher for 1,500 RL steps plus 300 Direct-OPD steps on R1-Distill-7B takes roughly half the compute (5,152 vs. 10,240 A100 GPU-hours) while achieving higher AIME 2025 accuracy than direct 7B GRPO.

Critical weaknesses

1. Missing closest baselines (WSPO, G-OPD, Proxy-Tuning, and exact local target distillation) and single-seed evaluation confined solely to AIME.
Although Section 5 discusses closely related methods that also transfer pre/post-RL log-probability ratios or reference-anchored shifts—most notably WSPO (offline weak-to-strong ratio matching), G-OPD (generalized OPD with a pre-RL reference), and inference-time Proxy-Tuning—none of them are evaluated in the experiments (even vanilla OPD appears only in a single curve in Figure 1a). In addition, every teacher–student configuration is run for only a single training trajectory with no training-seed error bars or per-problem bootstrap confidence intervals (Appendix B, p. 18); Table 1 reports benchmark-specific peak checkpoints across steps 0–300; and AIME 2024/2025 (only 30 problems each) are simultaneously used for hyperparameter sweeps (horizon in Section 4.2, KL in Section 4.3), checkpoint selection, and final reporting, with no independent held-out evaluation benchmark (e.g., MATH-500, OlympiadBench, or HMMT) and no decontamination audit against Skywork-OR1-RL-Data. Note also that at the common step-300 endpoint (Table 5), the gain on Qwen3-4B AIME 2025 is only +1.38 percentage points (65.60% →\to 66.98%), while baseline measurement variations across sweeps in Appendix B already reach 0.8–1.6 percentage points (48.3% vs. 49.10% on QuestA–Qwen3-1.7B; 42.55% vs. 44.18% on JustRL–Qwen3-1.7B). This directly drives my weak-reject rating because the empirical evidence is neither statistically isolated from noise on the strongest student nor compared against the closest existing ratio-transfer methods.

2. Unresolved theoretical and empirical gap between the idealized sequence objective (Eqs. 8–9) and the uncentered, zero-discount top-kk local update (Eqs. 13–15, 18–20).
The paper motivates Direct-OPD via the sequence-level objective in Eq. (8), yet implements a myopic per-token update (Eq. 15) that discards future return-to-go and includes a non-invariant top-kk support-mass gradient Uθ∇θlog⁡mθU _ \theta \nabla _ \theta \log m _ \theta (Appendix C, Eq. 18). The paper provides no ablation testing whether the empirical gains come from (i) immediate token shift rt(v)r _ t(v) vs. sequence return-to-go, (ii) the uncentered mass-shifting term Uθ∇θlog⁡mθU _ \theta \nabla _ \theta \log m _ \theta vs. a centered top-kk update ∇θUθ=∑v∈Spˉθ(v)(r(v)−Uθ)∇θlog⁡pθ(v)\nabla _ \theta U _ \theta = \sum _ {v \in S}\bar{p} _ \theta(v)(r(v) - U _ \theta)\nabla _ \theta \log p _ \theta(v), or (iii) direct token-level KL minimization toward the local multiplicative target π∗(v∣st)∝πS(v∣st)(πT(v∣st)/πTref(v∣st))1/α\pi^*(v \mid s _ t) \propto \pi _ S(v \mid s _ t)(\pi _ T(v \mid s _ t)/\pi _ {T _ {\mathrm{ref}}}(v \mid s _ t))^{1/\alpha} implied by Eq. (9). Similarly, the sequential composition experiment in Section 3.3 (JustRL →\to QuestA for 300+300300 + 300 steps, Figure 7) lacks a 600-step single-shift control (JustRL →\to JustRL) or a reversed-order control (QuestA →\to JustRL) to separate true policy-shift composability from simply training for 300 additional steps.

Questions for authors
  1. Assessment-critical (Comparisons with closest ratio-transfer baselines): Under matched student rollout budgets on Qwen3-4B and R1-Distill-7B with the JustRL-1.5B teacher pair, how does Direct-OPD compare against (a) WSPO, (b) G-OPD, (c) inference-time Proxy-Tuning, and (d) directly minimizing the token-level KL divergence toward the normalized local target ∝πS(v∣st)(πT(v∣st)/πTref(v∣st))1/α\propto \pi _ S(v \mid s _ t)(\pi _ T(v \mid s _ t)/\pi _ {T _ {\mathrm{ref}}}(v \mid s _ t))^{1/\alpha} on StS _ t?

  2. Assessment-critical (Ablating the local top-kk surrogate in Eq. 15 & Appendix C Eq. 18): Why was the uncentered update gθ(s)=∇θUθ+Uθ∇θlog⁡mθg _ \theta(s) = \nabla _ \theta U _ \theta + U _ \theta \nabla _ \theta \log m _ \theta chosen despite its sensitivity to additive reward constants? How does performance change if you (a) center rewards within the top-kk support (rt(v)−Uθr _ t(v) - U _ \theta, which removes the ∇θlog⁡mθ\nabla _ \theta \log m _ \theta term), (b) use discounted return-to-go instead of immediate rt(v)r _ t(v), or (c) vary k∈{8,16,64,full vocab}k \in \lbrace 8, 16, 64, \text{full vocab} \rbrace? Also, how are tokenizers aligned across R1-Distill/Nemotron teachers and Qwen3 students, and is rt(v)r _ t(v) clipped when πT(v∣st)\pi _ T(v \mid s _ t) or πTref(v∣st)\pi _ {T _ {\mathrm{ref}}}(v \mid s _ t) is near zero?

  3. Assessment-critical (Multi-seed reliability, held-out benchmarks, and sequential controls):

    • Can you report at least 3 independent training seeds (with fixed step-300 evaluation and paired bootstrap confidence intervals) for the strict weak-to-strong JustRL →\to Qwen3-4B and R1-Distill-7B runs, as well as results on an untouched held-out math benchmark (e.g., MATH-500 or OlympiadBench) that was not used for KL or checkpoint selection?
    • Which specific KL configuration (fixed value vs. adaptive KL with initial α0\alpha _ 0) was used for each row of Table 1 and Table 5?
    • In Section 3.3 (Figure 7), what happens if Qwen3-1.7B is trained for 600 steps using JustRL alone, or in the reversed order QuestA →\to JustRL?
Reviewer_zmxqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Direct-OPD uses the log-probability difference between a small teacher’s pre- and post-RL checkpoints as a dense reward for a stronger student. Training uses student-generated trajectories and a KL constraint toward the student’s initialization. Experiments demonstrate improvements on mathematical reasoning, including cases where the student already outperforms the teacher, and compare the cost of this approach with direct RL on the larger model.

Critical strengths

The paper provides evidence that a weaker teacher can improve an already stronger student, with gains also present at the final checkpoint. The cost comparison includes teacher RL training, making the proposed training route practically relevant.

Critical weaknesses
  1. The advantage over continued training of the larger model remains unclear. The current comparisons do not sufficiently establish a better cost–performance tradeoff against well-tuned direct RL with a sufficiently long training budget.

  2. Weak-teacher supervision carries a risk of negative transfer. Changes in teacher probabilities do not necessarily indicate improved correctness, and the KL constraint alone does not ensure preservation of the student’s existing capabilities. The current evaluation provides limited coverage of this risk.

  3. A simpler SFT warm-start alternative has not been ruled out. If the gains mainly reflect the transfer of reasoning patterns, SFT on high-quality responses from the small teacher, followed by RL, might achieve comparable or better results. A direct comparison is missing.

Questions for authors
  1. With sufficient training for both approaches, can Direct-OPD at least match the final performance of continued RL on the larger model? Is the observed advantage primarily faster progress, better eventual performance, or both?

  2. Does Direct-OPD outperform SFT on high-quality responses from the same teacher followed by RL? How can the contribution of transferring reasoning patterns be distinguished from that of the teacher’s probability shift?

  3. What practical setting is this method best suited to? OPD has clear uses in consolidating capabilities into a target model. For an already capable student that can undergo further training, what specific constraint motivates choosing Direct-OPD, and do the experiments demonstrate its value under that constraint?

Reviewer_tLxjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a weak teacher with reinforcement learning, then trains a stronger student using the teacher’s improvement, measured by the log-probability difference between its post-RL and pre-RL checkpoints. The authors claim that a weaker teacher’s improvement can improve an already stronger student, and that this approach can achieve better accuracy at lower training cost than directly training the stronger student with RL. They also claim that independently learned policy shifts can be composed sequentially. Experiments evaluate transfer from two teacher checkpoint pairs to several students on AIME 2024 and AIME 2025, compare against direct student RL, and demonstrate sequential transfer from two teachers.

Critical strengths

An appealing method with evidence of weak-to-strong improvement. Transferring the teacher’s improvement rather than imitating its final policy is a neat idea. The strongest result is that the JustRL teacher’s policy shift improves students that already substantially outperform the teacher, particularly Qwen3-4B. These gains also persist at the common final checkpoint, rather than appearing only at separately selected benchmark peaks. This provides a credible demonstration that a weaker model’s training-induced change can supply useful supervision to a stronger model in the studied settings.

Critical weaknesses
  1. The evaluation is too small and narrow to establish robustness. Evaluation is limited to AIME 2024 and AIME 2025, two small sets from the same competition-math distribution. Averaging 32 samples per problem reduces generation noise but does not address the limited coverage of distinct problems. The paper also reports only one training trajectory per teacher–student pair, without independent-seed uncertainty estimates or per-problem confidence intervals. Although the common-endpoint results are helpful, the headline results select the best checkpoint separately for each benchmark. Together, these limitations make it difficult to assess the reliability of smaller gains or how representative the findings are. A larger evaluation set would address coverage, while independent training runs would help establish stability.

  2. The comparisons do not sufficiently isolate the claimed advantages over direct RL or the benefits of sequential composition. For the direct-RL comparison, the local teacher is trained on DAPO, while student transfer uses Skywork-OR1-RL-Data; the direct-RL baseline’s dataset is not separately specified clearly enough to establish matched data exposure. If the baseline uses DAPO alone, the transfer route receives additional training on a different dataset, creating a substantial confound. The recipes also use different maximum response lengths -- 16k for direct RL versus 2k for transfer -- and different KL regularization. Short-prefix training may be a legitimate advantage of the method, but without comparable tuning of the RL baseline, it is unclear how much of the reported cost advantage depends on these choices. Furthermore, the experiments do not establish direct RL’s attainable peak performance: the displayed run is not shown to have reached a performance plateau. The sequential-composition experiment similarly lacks controls for additional training. Applying JustRL followed by QuestA improves performance, but the paper does not compare against continuing JustRL for the same total 600 steps, using QuestA alone for 600 steps, reversing the order, or mixing the signals. Consequently, the experiment demonstrates that a second transfer stage can help, but does not establish that composing independently learned shifts provides a benefit beyond additional training or a different teacher.

Questions for authors
  1. Clarification -- training data: What exact datasets and subsets are used for teacher RL, student transfer, and direct student RL in Figure 2? Does the direct-RL baseline receive the same prompt data as the two-stage route? If not, is there a matched-data comparison showing that the accuracy and compute advantages persist?

  2. Clarification -- rollout lengths and baseline tuning: Why is the maximum response length set to 16k for direct RL and 2k for transfer? What response-length and KL tuning was performed for the direct-RL baseline? Please report actual generated-token counts alongside GPU-hours, since maximum lengths alone do not establish the difference in generation cost.

  3. Clarification -- peak performance: Was the direct-RL run still improving at the end of the reported budget? What are the best observed direct-RL and Direct-OPD scores, and the compute required to reach them? Evidence that direct RL has plateaued, or results from a longer run, would materially affect my interpretation of the performance advantage.

  4. Additional evidence for robustness: Are per-problem results or independent training runs available to quantify uncertainty in the gains, particularly for initially stronger students and the direct-RL comparison? Evaluation on a larger math benchmark would also help determine whether the improvements extend beyond the small AIME sets.

  5. Additional evidence for sequential composition: At the same total training budget, how does JustRL -> QuestA compare with 600 steps of JustRL alone or QuestA alone? Results with reversed order or mixed signals would further clarify whether the gain comes specifically from composing distinct shifts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.