acceptodds
Under review as a conference paper at ICLR 2027

Step Ahead, Distill Back: One-Step RL-Guided On-Policy Distillation

Abstract

On-policy distillation (OPD) offers dense supervision for reasoning models, but its use with reward-trained experts commonly follows a train-then-distill workflow. We show that effective guidance can instead come from a teacher only one reward-gradient step ahead of the student. Our method, On-Policy Gradient Distillation (OPGD), rebuilds this teacher from the current actor at every iteration, allowing it to inherit accumulated learning while adapting to fresh verified outcomes. The teacher–actor log ratio isolates the preferences induced by this adaptation and supplies signed token advantages for OPD on the same rollouts. A first-order analysis connects these preferences to interactions between token score gradients, explaining how response-level feedback is redistributed within and across responses. Distillation reweights the outcome gradient through the policy's local geometry while preserving nonnegative first-order alignment. On Qwen3-4B, OPGD improves mean Pass@1 over GRPO by percentage points across four mathematical benchmarks at a common -update budget. Separate training on coding prompts also yields faster learning and higher accuracy across three code-generation benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On policy distillation, reinforcement learning with verifiable rewards (RLVR)

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g2WFOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes OPGD: at each iteration, it takes a temporary reward-gradient step from the current model, uses the resulting token-level log-probability changes as advantages, then restores the original model for distillation, requiring no external teacher. Its contributions are the one-step teacher construction and a first-order explanation of feedback transfer across tokens and responses, together with geometric reweighting of the reward gradient through distillation. On Qwen3-4B, the best checkpoints within 100 updates improve mean Pass@1 across four mathematical benchmarks from GRPO’s 37.55% to 46.77%, reaching 48.91% when retaining the strongest 10% of token targets. Separate coding runs also show gains, but training costs are higher, and no direct comparisons with RISE or OPD² are reported.

Critical strengths
  1. OPGD provides a conceptually simple way to incorporate outcome-level rewards into OPD through a one-step teacher. The method is clearly presented, with first-order analysis explaining contextual feedback and nonnegative alignment with the reward gradient.

  2. OPGD achieves strong results on Qwen3-4B across mathematical reasoning and code generation, improving mean mathematical Pass@1 over GRPO by 9.2 percentage points within the same 100-update budget.

Critical weaknesses

The experimental comparison with closely related methods combining RLVR and OPD, such as RISE, is insufficient, leaving OPGD’s relative effectiveness unclear.

Questions for authors

OPGD distills into the original student after reverting the temporary reward update, whereas RISE distills into the student after its RLVR update. What is the underlying significance of this distinction, and how does it affect the resulting optimization dynamics?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the positive assessment and constructive feedback.

The comparison with methods combining RLVR and OPD, such as RISE, is insufficient.

We implemented a RISE-style baseline with OPGD's model, data, rollout protocol, batch size and learning rate. The student is retained after the RLVR update, OPD uses the post-RLVR student's top-100 vocabulary support plus a tail-probability bucket, the extrapolation coefficient β\beta is linearly decayed from 1.21.2 to 1.01.0 and the OPD objective is JSD. We omit RISE's beta decay and anchor/EMA update, so neither method uses additional stabilization.

Each RISE outer step performs one RLVR update and one OPD update, whereas OPGD performs one update. The two methods learn at similar rates over the first 30 outer steps, after which RISE collapses (Figure 12). At outer step 50 (eval reward is the mean accuracy on AIME'24 and AIME'25):

MethodOuter StepActor real updatesEval reward (%)Response length (tokens)
RISE (logit-space)306054.6about 7.4k
OPGD303055.5about 6.5k
RISE (logit-space)501000about 15.4k
OPGD505055.7about 8.4k

We note that our experimental setting differs in several respects from the original RISE paper, such as training dataset, batch size and max response length. More extensive, method-specific hyperparameters tuning could potentially yield better performance. However, due to the limited time and computational resources available, we were only able to conduct this initial comparison and did not perform an exhaustive tuning process.

OPD² is complementary, since it applies OPD after RL training is complete, whereas OPGD interleaves teacher construction and distillation with RLVR.

OPGD distills into the original student after reverting the temporary reward update, whereas RISE distills into the student after its RLVR update. What is the significance of this distinction?

Comparison DimensionRISEOPGD
RLVR Update1 actual update1 virtual update
OPD Update11
Actual Actor Updates21
Actor RollbackNoYes
Reuse ResponseTrueTrue
Final UpdateRLVR + OPDOPD only

The distinction is the parameter state at which the distillation target is computed and applied. In OPGD, the reward step is temporary: we construct the virtual teacher, compute the teacher–actor log ratio, restore the actor, and distill into the original actor. Because teacher and actor share all earlier training, the log ratio is exactly the preference change from the latest reward step (Eq. 6). The actor's only update is the distillation step, which keeps nonnegative first-order alignment with the outcome gradient (Eq. 11).

In RISE, the post-RLVR student is retained and OPD is applied at that state. Each iteration therefore moves the actor twice, and the teacher–student discrepancy reflects both the dense preference signal and the state change from the RLVR update. Figure 12 shows similar early learning but stronger later stability for OPGD. We will add a schematic and pseudocode that distinguish the virtual teacher, the restored actor in OPGD, the post-RLVR student in RISE, and the final actor update.

We hypothesize that directly combining RLVR and OPD may introduce gradient interference due to their distinct optimization signals, particularly in sequential-update settings like RISE, where both gradients independently update the actor. Such interactions may destabilize training, leading to excessively long generations and eventual collapse. Moreover, both methods reuse rollouts, making the training process not strictly on-policy. In RISE, computing likelihoods using the post-RLVR policy on rollouts generated by an earlier policy introduces a policy mismatch that may bias the resulting gradient estimates.

Thanks for your reply. I'll keep my score.

Reviewer_h2MfOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

OPGD computes an outcome gradient from verified rollouts of the current actor, performs one virtual SGD step to obtain a virtual teacher, and then constructs a signed token advantage from the teacher–actor log-ratio for distillation back into the actor. The authors use NTK/score-gradient approximations to explain credit redistribution within and across responses, and report early-learning advantages on Qwen3-4B mathematical and code training curves.

Critical strengths
  • It attempts to obtain teacher-like token preferences within the same RLVR training loop, avoiding separate training of an external expert.
  • Score-gradient interactions and positive-semidefinite geometry provide a testable explanation for within-response credit redistribution and are worth preserving in future work.
Critical weaknesses
  • Equations (9–11) do not directly cover finite-step SGD with α=0.1, same-batch target construction, top-10% selection, and the subsequent clipped AdamW update; the paper provides neither an error bound nor real finite-step alignment statistics.
  • OPGD’s single step is slower, yet it is compared with GRPO mainly by update count and best checkpoint; there is no independent-batch control for same-batch bias. The provided version omits multiple cited appendices, so key experiments and reproduction settings cannot be verified.
Questions for authors
  1. Under independent teacher/rollout batches, different α values, and different group sizes, does the cosine alignment of the actual finite-step update remain consistent with the linear theory? Please provide the linearization error.
  2. Does OPGD still outperform GRPO/DAPO under matched wall-clock time, GPU-hours, teacher backward passes, and preregistered checkpoint selection?
  3. Please provide the missing Appendices A/C/D/E/H, complete numerical code results, all training seeds, checkpoint-selection rules, and contamination checks. Without this information, it should not treat the paper’s best result as robust evidence.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful review and constructive feedback.

The practical procedure uses finite-step SGD with α=0.1, same-batch target construction, top-10% selection, and a clipped AdamW update, but the paper provides neither an error bound nor finite-step alignment statistics.

The finite-step update stays closely aligned with the first-order theory. We compare the actual distillation direction with its linear prediction,

e_\theta=\frac{\|\Delta\theta_{\mathrm{actual}}-\Delta\theta_{\mathrm{linear}}\|_2}{\|\Delta\theta_{\mathrm{actual}}\|_2},$$ where $g_T$ is the outcome gradient on the teacher batch, $F_R$ is the score second moment on the rollout batch, and $\Delta\theta_{\mathrm{actual}}$ is the distillation direction obtained with the finite-step teacher. We report the same statistics in logit space, over three step sizes $\alpha$ and four group sizes. Ranges span the four group sizes: | $\alpha$ | Parameter cosine | Logit cosine | Logit relative error | | --- | --- | --- | --- | | 0.025 | 0.9956–0.9998 | 0.9977–0.9983 | 0.059–0.068 | | **0.1 (paper)** | **0.9950–0.9992** | **0.9826–0.9851** | 0.173–0.188 | | 0.4 | 0.9925–0.9968 | 0.8315–0.8970 | 0.443–0.562 | Parameter directions stay aligned over all step sizes, while the logit-space error grows with $\alpha$, consistent with the second-order error term in Eq. (15). The expansion in Appendix B is derived on a fixed batch, so it applies directly to same-batch construction and includes top-10% selection through the retained-position mask $V_S$ (see Eqs. 15–18). It describes the gradient that the clipped AdamW actor update receives. We have added this diagnostic to Appendix A.3 of the revised paper and present Eqs. (9–11) in Section 4 as a leading-order explanation of the one-step update. > Under independent teacher/rollout batches, different α values, and different group sizes, does the cosine alignment of the actual update remain consistent with the linear theory? > **Yes.** The diagnostic above uses independent teacher and rollout batches, and the alignment stays consistent across all tested step sizes and group sizes (Tables 7–8, Appendix A.3). **Step-size sweep.** Each range spans group sizes $G\in\{2,4,8,16\}$: | Virtual step size $\alpha$ | Parameter cosine | Parameter relative error | Logit cosine | Logit relative error | | --- | --- | --- | --- | --- | | 0.025 | 0.9956–0.9998 | 0.020–0.094 | 0.9977–0.9983 | 0.059–0.068 | | **0.1 (paper)** | **0.9950–0.9992** | 0.045–0.099 | **0.9826–0.9851** | 0.173–0.188 | | 0.4 | 0.9925–0.9968 | 0.117–0.151 | 0.8315–0.8970 | 0.443–0.562 | **Group-size sweep at $\alpha=0.1$:** | Group size $G$ | Parameter cosine | Parameter relative error | Logit cosine | Logit relative error | | --- | --- | --- | --- | --- | | 2 | 0.9988 | 0.0567 | 0.9851 | 0.1729 | | 4 | 0.9985 | 0.0605 | 0.9826 | 0.1876 | | 8 | 0.9992 | 0.0447 | 0.9833 | 0.1827 | | 16 | 0.9950 | 0.0995 | 0.9839 | 0.1782 | The parameter-space cosine remains above 0.995 for every tested group size at the paper's step size, and the logit-space cosine remains above 0.982. The alignment is therefore consistent across the tested step sizes and group sizes, while the increasing logit-space error at larger $\alpha$ is consistent with the expected finite-step nonlinear deviation.
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

OPGD is slower, yet it is compared with GRPO mainly by update count and best checkpoint. Does it still outperform under matched wall-clock time and GPU-hours?

Yes. With 100 updates in 31.3 hours, OPGD outperforms GRPO with 200 updates (56.6 hours) and DAPO (40.0 hours) on Pass@1, Pass@4 and Pass@8 (Table 4). Times are rescaled to 8×A800, so GPU-hours are eight times the wall-clock hours:

MethodUpdatesWall clock (h)GPU-hoursPass@1 (%)Pass@8 (%)
GRPO10015.912737.5560.56
GRPO20056.645343.4963.04
DAPO10040.032042.9261.54
OPGD10031.325046.7765.05

OPGD adds a virtual reward backward pass and teacher scoring to each update (Figure 5), yet GRPO does not reach its accuracy within 200 updates, at 1.8 times the wall-clock time. All runs use the same checkpoint rule (Appendix D). GPU-hours are now reported in Table 4 alongside the runtime breakdown.

The current version omits multiple appendices, making key experiments and reproduction settings difficult to verify.

We apologize that this material was hard to find. It is in the appendix of the revised paper:

  • Complete results: all configurations, benchmarks and Pass@k metrics in Table 9 (Appendix C).
  • Training settings: mathematical and code configurations in Tables 10–11 (Appendix D).
  • Checkpoint rule: each run is validated every 10 updates on AIME'24 and AIME'25, and the checkpoint with the highest mean Pass@1 within its budget is reported (Appendix D). HMMT-Feb'25 and HMMT-Nov'25 are never used for selection, and OPGD outperforms every in-house baseline on both (Table 1).
  • Seeds: three seeds for OPGD and GRPO (Appendix A.1, Figure 8).
  • Contamination: DeepMath-103K was decontaminated against standard mathematical benchmarks by its authors, and HMMT-Nov'25 took place after the release of both DeepMath-103K and Qwen3. All methods share the same training data.

The revised paper also includes an appendix guide and collects the remaining details in one place, including model and tokenizer versions, rollout limits, sampling settings and evaluation protocols.

Reviewer_q8qPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether a teacher obtained by taking a single reward gradient step from the current actor can provide effective distillation supervision. They propose On-Policy Gradient Distillation (OPGD), which constructs a teacher at every iteration by taking a single gradient step. They test OPGD on Qwen3-4B across mathematical reasoning benchmarks and show improvements over methods such as GRPO.

Critical strengths
  1. The paper proposes an approach to incorporating on-policy distillation into RLVR without requiring a separately trained teacher. This provides a practically appealing solution to obtain more fine-grained training signals without the need of a separately trained teacher.

  2. The proposed method outperforms baselines such as GRPO and DAPO on mathematical reasoning, with improvements extending to code-generation tasks. These results suggest that OPGD can improve reasoning performance substantially over current methods.

Critical weaknesses
  1. The experiments don't seem to fully isolate the mechanisms responsible for the improvements. From my understanding, the virtual teacher is constructed using only a single SGD step along the initial GRPO outcome gradient, where the importance-sampling ratio is 1 and PPO clipping is inactive. The actor is then updated using AdamW on the distillation objective. As the theoretical analysis shows, this approximately transforms the original outcome gradient through FkgkF_kg_k. It is then unclear how much of the improvement comes from the proposed distillation mechanism with cross-response contributions, the resulting gradient transformation, or the choice of optimizer for the virtual teacher. Although the appendix discusses how different virtual optimizers affect the update, it does not provide empirical comparisons. This is consequential to the assessment because these distinctions are important to understanding why OPGD outperforms methods such as GRPO, and what the core mechanisms are that directly contributes to its gains.

  2. The main experiments focus on just a single model (Qwen3-4B), and performance variability across independent training runs is not reported. Although the improvements are substantial and extend to code-generation tasks, evaluations across additional base models along with multi-seed results would help strengthen the robustness and generality of the proposed method.

Questions for authors
  1. Could the authors better isolate the mechanisms responsible for OPGD's improvements? In particular, given the first-order interpretation of OPGD as applying the gradient transformation (FkgkF_k g_k), it would be interesting to understand how much of the benefit arises from this transformation versus other aspects of the distillation procedure. Additionally, how do the results depend on the choice of virtual teacher optimizer (SGD versus AdamW), especially given that the teacher one-step and the actor are trained with different optimizers. Could the authors investigate how much of OPGD's improvement comes from using reward information from other responses in the batch to provide token-level supervision, eg. for responses that would otherwise receive zero advantage under GRPO?

  2. How robust are the reported improvements across independent training runs? Could the authors provide multi-seed evaluations and additional experiments on other base models to show whether the gains generalize beyond Qwen3-4B?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the careful review and constructive feedback.

The improvement may come from cross-response distillation, the resulting gradient transformation, or the choice of optimizer for the virtual teacher.

We agree that these effects should be disentangled. Most of the gain comes from the distillation transformation of the outcome gradient, and cross-response supervision mainly speeds up early learning. We added two controlled comparisons (eval reward is the mean accuracy on AIME'24 and AIME'25):

  • Cross-response supervision. We mask the distillation loss on zero-advantage responses while constructing the virtual teacher from the full rollout batch. This removes only the loss terms assigned to these responses; cross-response interactions among retained positions remain unchanged. Their unmasked targets are induced by score interactions with responses carrying nonzero advantages (Eq. 9). Masking slows early learning but reaches a similar final Eval reward (Figure 10).
  • Gradient transformation. GRPO applies the same outcome gradient without distillation. Masked OPGD still clearly outperforms GRPO and passes GRPO's update-100 accuracy by update 40 (Figure 10).
  • Virtual-teacher optimizer. An AdamW virtual step that temporarily copies the actor's optimizer state collapses after about 20 updates, whereas the SGD step stays stable (Figure 9). This is likely because the copied moments accumulate gradients of the distillation objective (Section 4.1).

We have added both comparisons to Appendix A.2 of the revised paper.

Does reward information from other responses help responses that would otherwise have zero advantage under GRPO?

Yes. A response with zero group advantage still receives token-level feedback, because its sampled-token scores interact with reward-bearing tokens elsewhere in the batch (Eq. 9). In the first diagnostic batch, nearly half of the responses have zero advantage, yet most of their tokens receive nonzero targets of both signs (Section 5.3). Removing these targets slows early progress (Figure 10), so this feedback mainly accelerates learning.

The main experiments use only Qwen3-4B, and variability across independent training runs is not reported.

OPGD's advantage holds across seeds, model sizes and model families (values in %):

SettingBenchmarksGRPOOPGD
Qwen3-4B, 3 seeds, update 504 math, mean accuracy24.9 ± 2.043.4 ± 2.2
Qwen3-8B, update 1004 math, mean Pass@139.248.6
Qwen3-8B, update 1003 code, mean Pass@156.257.9
OLMo 3 7B Instruct, update 1004 math, mean Pass@138.4942.14

Across the three seeds, OPGD stays ahead at every evaluated update (Figure 8). The Qwen3-8B model is trained on mathematics, so the code benchmarks test out-of-domain transfer (Table 2). On OLMo 3 7B, OPGD also raises training reward faster (Figure 4) and reaches the highest mean Pass@4 (Table 3). We have added these results to Section 5.2 and Appendix A.1 of the revised paper.

Thanks for the additional results and discussion; my main concerns have been addressed. I have thus increased my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We have uploaded a revised paper with new text in blue, and all section, figure and table numbers below refer to it.

New experiments

  • Sources of the gain (q8qP, Appendix A.2). Masking zero-advantage responses slows early learning but reaches a similar final Eval reward, and masked OPGD still clearly outperforms GRPO. An AdamW virtual step collapses, whereas the SGD step stays stable.
  • Seeds and different base models (q8qP, Section 5.2, Appendix A.1). Across three seeds, OPGD leads GRPO at every evaluated update. OPGD also outperforms GRPO on Qwen3-8B and on OLMo 3 7B Instruct.
  • Finite-step alignment (h2Mf, Appendix A.3). At the paper's step size, the actual distillation direction has cosine at least 0.995 with the first-order prediction for every group size.
  • RISE comparison (g2WF, Appendix A.5). OPGD matches RISE's early learning with half the actor updates and stays stable after RISE collapses.

Evidence from existing results

  • Compute cost (h2Mf, Table 4). With 100 updates in 31.3 hours, OPGD outperforms GRPO with 200 updates (56.6 hours) and DAPO (40.0 hours).
  • Reproducibility (h2Mf, Appendices C and D). The complete results, training settings and checkpoint rule are in the appendix.

Planned revisions

  • The revised paper states Eqs. (9–11) as a leading-order explanation of the one-step update (Section 4).
  • Table 4 now reports GPU-hours alongside update counts.
  • The revised paper includes an appendix guide and collects the reproducibility details in one place.
  • Add a schematic and pseudocode contrasting OPGD and RISE.

We hope these results address the reviewers' concerns, and we are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.