- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes OPGD: at each iteration, it takes a temporary reward-gradient step from the current model, uses the resulting token-level log-probability changes as advantages, then restores the original model for distillation, requiring no external teacher. Its contributions are the one-step teacher construction and a first-order explanation of feedback transfer across tokens and responses, together with geometric reweighting of the reward gradient through distillation. On Qwen3-4B, the best checkpoints within 100 updates improve mean Pass@1 across four mathematical benchmarks from GRPO’s 37.55% to 46.77%, reaching 48.91% when retaining the strongest 10% of token targets. Separate coding runs also show gains, but training costs are higher, and no direct comparisons with RISE or OPD² are reported.
Critical strengths
-
OPGD provides a conceptually simple way to incorporate outcome-level rewards into OPD through a one-step teacher. The method is clearly presented, with first-order analysis explaining contextual feedback and nonnegative alignment with the reward gradient.
-
OPGD achieves strong results on Qwen3-4B across mathematical reasoning and code generation, improving mean mathematical Pass@1 over GRPO by 9.2 percentage points within the same 100-update budget.
Critical weaknesses
The experimental comparison with closely related methods combining RLVR and OPD, such as RISE, is insufficient, leaving OPGD’s relative effectiveness unclear.
Questions for authors
OPGD distills into the original student after reverting the temporary reward update, whereas RISE distills into the student after its RLVR update. What is the underlying significance of this distinction, and how does it affect the resulting optimization dynamics?
2 comments
Thanks for your reply. I'll keep my score.
Official Comment by Authors
We thank the reviewer for the positive assessment and constructive feedback.
We implemented a RISE-style baseline with OPGD's model, data, rollout protocol, batch size and learning rate. The student is retained after the RLVR update, OPD uses the post-RLVR student's top-100 vocabulary support plus a tail-probability bucket, the extrapolation coefficient β is linearly decayed from 1.2 to 1.0 and the OPD objective is JSD. We omit RISE's beta decay and anchor/EMA update, so neither method uses additional stabilization.
Each RISE outer step performs one RLVR update and one OPD update, whereas OPGD performs one update. The two methods learn at similar rates over the first 30 outer steps, after which RISE collapses (Figure 12). At outer step 50 (eval reward is the mean accuracy on AIME'24 and AIME'25):
We note that our experimental setting differs in several respects from the original RISE paper, such as training dataset, batch size and max response length. More extensive, method-specific hyperparameters tuning could potentially yield better performance. However, due to the limited time and computational resources available, we were only able to conduct this initial comparison and did not perform an exhaustive tuning process.
OPD² is complementary, since it applies OPD after RL training is complete, whereas OPGD interleaves teacher construction and distillation with RLVR.
The distinction is the parameter state at which the distillation target is computed and applied. In OPGD, the reward step is temporary: we construct the virtual teacher, compute the teacher–actor log ratio, restore the actor, and distill into the original actor. Because teacher and actor share all earlier training, the log ratio is exactly the preference change from the latest reward step (Eq. 6). The actor's only update is the distillation step, which keeps nonnegative first-order alignment with the outcome gradient (Eq. 11).
In RISE, the post-RLVR student is retained and OPD is applied at that state. Each iteration therefore moves the actor twice, and the teacher–student discrepancy reflects both the dense preference signal and the state change from the RLVR update. Figure 12 shows similar early learning but stronger later stability for OPGD. We will add a schematic and pseudocode that distinguish the virtual teacher, the restored actor in OPGD, the post-RLVR student in RISE, and the final actor update.
We hypothesize that directly combining RLVR and OPD may introduce gradient interference due to their distinct optimization signals, particularly in sequential-update settings like RISE, where both gradients independently update the actor. Such interactions may destabilize training, leading to excessively long generations and eventual collapse. Moreover, both methods reuse rollouts, making the training process not strictly on-policy. In RISE, computing likelihoods using the post-RLVR policy on rollouts generated by an earlier policy introduces a policy mismatch that may bias the resulting gradient estimates.