- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SR-OPSD, which constructs a reference-anchored distillation target through geometric interpolation between a dynamic self-teacher and a frozen initial policy, and employs the Forward Rényi Divergence to control the updates of the student policy. The authors derive the conditional variational characterization and token-logit gradients under fixed contexts, and validate the method on scientific reasoning, tool use, mathematical reasoning, and code generation tasks. Experiments demonstrate that the method outperforms GRPO and existing OPSD baselines in most settings, and exhibits superior training stability on certain tasks.
Critical strengths
- The method achieves strong performance across multiple models and tasks, with notable gains particularly in scientific reasoning and code generation, demonstrating its practical utility.
- Ablation experiments show that the effect of reference anchoring is closely tied to the choice of divergence, providing valuable empirical insights for designing self-distillation targets.
Critical weaknesses
- Joint ablation experiments are restricted to a single task, making it difficult to determine whether the method's advantages are stable and general.
- The paper incorporates Rényi projection upon existing geometric interpolation and reference regularization techniques (such as those studied in -OPSD), yet lacks direct empirical comparisons with these closely related methods to clearly isolate the contribution of the Rényi divergence.
Questions for authors
- What are the main differences and additional benefits of SR-OPSD compared to -OPSD[1], and can you provide a direct comparison?
- The Rényi order in Table 6 appears inconsistent with the optimal configuration in Table 4; please clarify the hyperparameters actually used in practice.
- Can the gains from the joint ablation be reproduced on mathematical reasoning and code generation tasks?
[1] -OPSD: Deriving with Policy Optimization, Training with Self-Distillation
5 comments
Dear Reviewer bEw4:
As the discussion period is about to end in 3 days, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.
Best,
Authors of Submission #19957
Thanks for the response, the correction to Table 6, and the added discussion on -OPSD. However, citing reported numbers from -OPSD under different setups and only testing under corrupted contexts on LiveCodeBench do not fully address my concerns. Conducting controlled experiments would greatly benefit the paper and help me better discuss its merits with the other reviewers and the AC. Therefore, I will maintain my current rating for now.
Response to Reviewer bEw4
We sincerely thank the reviewer for the thoughtful review, and for the time invested in evaluating our work. We are grateful for the positive assessment and reviewer’s recognition of our contributions. We also appreciate the reviewer’s insightful comments, which help us better position the scope of the paper. We address these concerns point by point below. We hope these responses adequately address the reviewer’s concerns and that the reviewer will be willing to increase the score accordingly.
Q1/W2. Differences from β-OPSD and a direct comparison.
We thank the reviewer for highlighting this related work. Both methods connect geometric interpolation to regularized policy optimization. β-OPSD combines reverse-KL training with target scheduling and return-to-go credit assignment. SR-OPSD extends this perspective through a conditional Rényi-regularized formulation and an exact token-logit gradient analysis: α determines reference anchoring, while ρ controls target-to-student probability-ratio reweighting. Our analysis makes their joint effect on the student update explicit.
The reported training setups also differ. β-OPSD uses a frozen teacher and a stop-gradient copy of the current student as its reference. In our implementation, the self-teacher is maintained as an EMA of the student parameters, while the reference keeps the initial student parameters fixed; both are evaluated on the same privileged context. The importance of joint design is demonstrated by the Physics ablation in Table 4: anchoring changes forward-KL accuracy from 79.1 to 76.3, but changes forward Rényi accuracy at ρ=0.7 from 75.9 to 82.7. Thus, the empirical benefit depends on how anchoring and projection are combined.
Given the limited rebuttal window, we provide a comparison of the reported Qwen3-8B results on the three shared benchmarks, using Table 1 of β-OPSD [1]. Avg@12 and Avg@16 both estimate mean per-sample accuracy.
Table b-R1: Reported Qwen3-8B results on shared mathematical reasoning benchmarks.
We have now cited β-OPSD[1] in the revised manuscript.
Q2. Hyperparameters in Table 6 vs Table 4.
Thank you for identifying this inconsistency. The Physics result in Table 1 uses (α,ρ)=(0.9,0.7), consistent with Table 4; the discrepancy comes from an incorrect entry in Table 6. We will correct the corresponding entry in Table 6.
Q3/W1. Joint ablation on math and code.
The joint (α,ρ) ablation on Physics is complemented by cross-domain results using a shared anchoring strength: all mathematical-reasoning and LCB experiments use α=0.9.
We additionally evaluate reference anchoring on Qwen3-1.7B LiveCodeBench at a fixed Rényi order ρ=0.95. Under corrupted privileged context, changing α from 1.0 to 0.9 raises accuracy on held-out tests of training problems from 4.7% to 13.4%, while unseen-problem accuracy increases from 17.3% to 18.5%. Student-to-reference KL decreases from 0.092 to 0.041.
This coding ablation extends the evidence for reference anchoring beyond Physics at a fixed projection, complementing the joint study in Table 4.
Reference
[1] Xu, J., et al. β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation.