acceptodds
Under review as a conference paper at ICLR 2027

SR-OPSD: Self-Referenced On-Policy Self-Distillation

12 views

Abstract

On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target–student probability mismatches translate into updates. We propose Self-Referenced On-Policy Self-Distillation (SR-OPSD), which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward R\'enyi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the R\'enyi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bEw4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SR-OPSD, which constructs a reference-anchored distillation target through geometric interpolation between a dynamic self-teacher and a frozen initial policy, and employs the Forward Rényi Divergence to control the updates of the student policy. The authors derive the conditional variational characterization and token-logit gradients under fixed contexts, and validate the method on scientific reasoning, tool use, mathematical reasoning, and code generation tasks. Experiments demonstrate that the method outperforms GRPO and existing OPSD baselines in most settings, and exhibits superior training stability on certain tasks.

Critical strengths
  1. The method achieves strong performance across multiple models and tasks, with notable gains particularly in scientific reasoning and code generation, demonstrating its practical utility.
  2. Ablation experiments show that the effect of reference anchoring is closely tied to the choice of divergence, providing valuable empirical insights for designing self-distillation targets.
Critical weaknesses
  1. Joint ablation experiments are restricted to a single task, making it difficult to determine whether the method's advantages are stable and general.
  2. The paper incorporates Rényi projection upon existing geometric interpolation and reference regularization techniques (such as those studied in β\beta-OPSD), yet lacks direct empirical comparisons with these closely related methods to clearly isolate the contribution of the Rényi divergence.
Questions for authors
  1. What are the main differences and additional benefits of SR-OPSD compared to β\beta-OPSD[1], and can you provide a direct comparison?
  2. The Rényi order in Table 6 appears inconsistent with the optimal configuration in Table 4; please clarify the hyperparameters actually used in practice.
  3. Can the gains from the joint ablation be reproduced on mathematical reasoning and code generation tasks?

[1] β\beta-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

5 comments
AuthorsOpenReview ↗

Response to Reviewer bEw4

We sincerely thank the reviewer for the thoughtful review, and for the time invested in evaluating our work. We are grateful for the positive assessment and reviewer’s recognition of our contributions. We also appreciate the reviewer’s insightful comments, which help us better position the scope of the paper. We address these concerns point by point below. We hope these responses adequately address the reviewer’s concerns and that the reviewer will be willing to increase the score accordingly.

Q1/W2. Differences from β-OPSD and a direct comparison.

  • We thank the reviewer for highlighting this related work. Both methods connect geometric interpolation to regularized policy optimization. β-OPSD combines reverse-KL training with target scheduling and return-to-go credit assignment. SR-OPSD extends this perspective through a conditional Rényi-regularized formulation and an exact token-logit gradient analysis: α\alpha determines reference anchoring, while ρ\rho controls target-to-student probability-ratio reweighting. Our analysis makes their joint effect on the student update explicit.

  • The reported training setups also differ. β-OPSD uses a frozen teacher and a stop-gradient copy of the current student as its reference. In our implementation, the self-teacher is maintained as an EMA of the student parameters, while the reference keeps the initial student parameters fixed; both are evaluated on the same privileged context. The importance of joint design is demonstrated by the Physics ablation in Table 4: anchoring changes forward-KL accuracy from 79.1 to 76.3, but changes forward Rényi accuracy at ρ=0.7\rho=0.7 from 75.9 to 82.7. Thus, the empirical benefit depends on how anchoring and projection are combined.

  • Given the limited rebuttal window, we provide a comparison of the reported Qwen3-8B results on the three shared benchmarks, using Table 1 of β-OPSD [1]. Avg@12 and Avg@16 both estimate mean per-sample accuracy.

    Table b-R1: Reported Qwen3-8B results on shared mathematical reasoning benchmarks.

    MethodMetricAIME24AIME25HMMT25Mean
    β-OPSDAvg@1278.3370.2831.9460.18
    SR-OPSD (ours)Avg@1677.771.946.565.37

We have now cited β-OPSD[1] in the revised manuscript.

Q2. Hyperparameters in Table 6 vs Table 4.

Thank you for identifying this inconsistency. The Physics result in Table 1 uses (α,ρ)=(0.9,0.7)(\alpha,\rho)=(0.9,0.7), consistent with Table 4; the discrepancy comes from an incorrect entry in Table 6. We will correct the corresponding entry in Table 6.

Q3/W1. Joint ablation on math and code.

The joint (α,ρ)(\alpha,\rho) ablation on Physics is complemented by cross-domain results using a shared anchoring strength: all mathematical-reasoning and LCB experiments use α=0.9\alpha=0.9.

We additionally evaluate reference anchoring on Qwen3-1.7B LiveCodeBench at a fixed Rényi order ρ=0.95\rho=0.95. Under corrupted privileged context, changing α\alpha from 1.0 to 0.9 raises accuracy on held-out tests of training problems from 4.7% to 13.4%, while unseen-problem accuracy increases from 17.3% to 18.5%. Student-to-reference KL decreases from 0.092 to 0.041.

This coding ablation extends the evidence for reference anchoring beyond Physics at a fixed projection, complementing the joint study in Table 4.

Reference

[1] Xu, J., et al. β\beta-OPSD: Deriving with Policy Optimization, Training with Self-Distillation.

Dear Reviewer bEw4:

As the discussion period is about to end in 3 days, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.

Best,

Authors of Submission #19957

Thanks for the response, the correction to Table 6, and the added discussion on β\beta-OPSD. However, citing reported numbers from β\beta-OPSD under different setups and only testing α∈{0.9,1.0}\alpha \in \{0.9, 1.0\} under corrupted contexts on LiveCodeBench do not fully address my concerns. Conducting controlled experiments would greatly benefit the paper and help me better discuss its merits with the other reviewers and the AC. Therefore, I will maintain my current rating for now.

Reviewer_AYS6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SR-OPSD, which combines a self-referenced teacher target with forward Rényi divergence for on-policy self-distillation. The target is constructed as a normalized geometric interpolation between an evolving self-teacher and the frozen initial student policy, while the Rényi order controls how target-to-student probability ratios influence the update. The paper derives a variational characterization and an explicit token-logit gradient, and evaluates the method on scientific reasoning, tool use, mathematical reasoning, and code generation. The experiments suggest that reference anchoring and divergence choice interact substantially, and that their combination can outperform existing distillation objectives.

Critical strengths
  • The paper is technically well developed, with clear mathematical formulations and gradient derivations.
  • The joint consideration of target construction and divergence geometry is interesting and well motivated.
  • The ablation in Table 4 provides useful evidence. Reference anchoring improves some divergence objectives while degrading others, and the combination with forward Rényi divergence achieves the strongest performance. The evaluation also covers multiple tasks and model families.
Critical weaknesses
  • The theoretical analysis mainly characterizes the proposed objective under fixed contexts and target distributions, but does not establish why it should improve reasoning performance or stabilize the evolving student–teacher system. It is unclear whether the gains come from the proposed geometric target or from more conventional regularization effects, such as restricting policy drift or reducing update magnitude.
  • The empirical evidence for robust generalization is not fully convincing. The improvements on mathematical reasoning are relatively modest, and the main joint hyperparameter ablation is concentrated on SciKnowEval Physics only.
  • The sensitivity to the interpolation coefficient, Rényi order, and practical top-k approximation deserves more systematic evaluation.
Questions for authors
  • How does SR-OPSD compare with directly adding a KL penalty to the initial student policy? Can the authors match the actual student-reference KL drift across methods to isolate the benefit of the geometric target?
  • How transferable are the selected interpolation coefficient and Rényi order? Could the authors select them on one task and evaluate the same configuration across the other domains without retuning?
  • For LiveCodeBench v6, are evaluation problems disjoint from training problems, or only the test cases? I would be interested in results on a strictly held-out set of programming problems, particularly given the relatively large coding improvements.
  • The theoretical analysis assumes full-vocabulary, normalized distributions, whereas the implementation uses top-k approximation and token-loss clipping. How closely do the resulting practical gradients match those derived in Proposition 3.2? Is the method sensitive to the truncation threshold?
  • Does reference anchoring remain beneficial when the self-teacher's privileged information is missing, or incorrect? This could help distinguish improvements due to better use of privileged information from more general stabilization effects.
6 comments
AuthorsOpenReview ↗

Response to Reviewer AYS6 (Part 1/3)

We thank the reviewer for the constructive review, and for the time invested in evaluating our work. We are grateful for the reviewer’s constructive comments which help us sharpen the scope and presentation of the paper. We address each concern in detail below. We hope these responses adequately address the reviewer’s concerns and that the reviewer will be willing to increase the score accordingly.

W1/Q1. Geometric target anchoring versus direct KL regularization.

  • The suggested KL penalty has an exact connection to our formulation. For reverse-KL distillation with fixed conditional teacher and reference distributions, Proposition B.1 implies argminπ [KL(π∥πsf)+λ KL(π∥πref)]=πα⋆\mathrm{argmin}_{\pi}\thinspace[\mathrm{KL}(\pi\Vert\pi_{\mathrm{sf}})+\lambda\thinspace\mathrm{KL}(\pi\Vert\pi_{\mathrm{ref}})]=\pi_\alpha^\star, πα⋆∝πsfαπref1−α,α=1/(1+λ)\pi_\alpha^\star\propto\pi_{\mathrm{sf}}^\alpha\pi_{\mathrm{ref}}^{1-\alpha},\quad \alpha=1/(1+\lambda) where πsf\pi_{\mathrm{sf}} denotes the self-teacher. The geometric target is therefore the optimal policy for this KL-regularized distillation problem. Its form follows directly from the variational objective. Moreover, the entire objective equals (1+λ)KL(π∥πα⋆)+C(1+\lambda)\mathrm{KL}(\pi\Vert\pi_\alpha^\star)+C, with CC independent of π\pi, so the connection holds at the objective and gradient levels.

  • SR-OPSD further generalizes this to a Rényi-regularized policy optimization formulation. To the best of our knowledge, this joint formulation is novel in on-policy self-distillation. Proposition 3.1 shows that its objective can be written as Jα,ρ(π)=−Dρ(πα⋆∥π)+Cα,ρJ_{\alpha,\rho}(\pi)=-D_\rho(\pi_\alpha^\star\Vert\pi)+C_{\alpha,\rho}, where Cα,ρC_{\alpha,\rho} is independent of π\pi. Thus, the geometric policy πα⋆\pi_\alpha^\star emerges as the variational optimum, shared across Rényi orders for fixed teacher, reference, and α\alpha. Proposition 3.2 further characterizes the update through

    ∇zDρ(πα⋆∥π)=π−π~ρ,π~ρ(a)∝π(a)[πα⋆(a)π(a)]ρ.\nabla_z D_\rho(\pi_\alpha^\star\Vert\pi)=\pi-\widetilde{\pi}_\rho, \quad \widetilde{\pi}_\rho(a)\propto \pi(a) \left[\frac{\pi_\alpha^\star(a)}{\pi(a)}\right]^\rho.

    The order ρ\rho therefore acts as an inverse temperature on the target-to-student probability ratio, controlling the relative reweighting of tokens in the update. This gives α\alpha and ρ\rho distinct roles within the same regularized policy-optimization framework: α\alpha determines the geometric optimum, while ρ\rho controls the ratio-dependent update geometry. For a fixed student and target, varying ρ\rho generally changes the relative corrections across tokens and hence the logit-gradient direction, rather than merely rescaling the update. The ablation in Table 4 in the paper suggests that anchoring interacts with the projection objective.

  • As acknowledged in Sec. 6, our fixed-context analysis does not establish stability of the coupled student–teacher dynamics. We complement the theoretical characterization with measurements of both directions of student–reference KL (Table A-R1).

    Table A-R1. Held-out LiveCodeBench performance and measured policy drift. Values are mean ± sample standard deviation over two random seeds. Scores are Avg@16 on held-out problems. KL is averaged per response token on each student's own generated prefixes across all validation groups.

    ModelMethodHeld-out score (%)KL(π∥πref)\mathrm{KL}(\pi\Vert\pi_{\mathrm{ref}})KL(πref∥π)\mathrm{KL}(\pi_{\mathrm{ref}}\Vert\pi)
    Qwen3-1.7BSDPO15.1 ± 0.90.057 ± 0.0020.145 ± 0.001
    Qwen3-1.7BGRPO18.2 ± 0.60.159 ± 0.1110.381 ± 0.226
    Qwen3-1.7BSR-OPSD19.5 ± 0.70.052 ± 0.0100.038 ± 0.008
    Qwen3-4BSDPO20.2 ± 2.20.101 ± 0.0140.292 ± 0.054
    Qwen3-4BGRPO19.6 ± 5.60.438 ± 0.1300.606 ± 0.285
    Qwen3-4BSR-OPSD25.9 ± 1.30.148 ± 0.0190.101 ± 0.013

    Higher accuracy is not accompanied by uniformly smaller measured drift. SR-OPSD achieves higher mean held-out scores than both baselines at both model sizes. At 4B, it outperforms SDPO (25.9% versus 20.2%) despite higher student-to-reference KL (0.148 versus 0.101), while reference-to-student KL is lower at both sizes. Together with the exact reverse-KL equivalence above, these results make the regularization interpretation explicit and challenge an explanation based solely on “moving less” under the measured student-to-reference KL.

AuthorsOpenReview ↗

Response to Reviewer AYS6 (Part 2/3)

W2/W3/Q2. Transfer of (α,ρ)(\alpha, \rho) without retuning, sensitivity to (α,ρ)(\alpha, \rho) and maths performance.

  • Transfer of a fixed configuration. We directly test transfer by reusing the Physics configuration (α,ρ)=(0.9,0.7)(\alpha,\rho)=(0.9,0.7) on Chemistry and Materials without retuning either parameter (in Table A-R2). Within a 10-hour wall-clock budget, it achieves 81.5 and 79.1 Avg@16, respectively, exceeding the reported SDPO and GRPO results in both domains.

    Table A-R2. Fixed-configuration transfer on Qwen3-8B. Best Avg@16 (%) within a 10-hour wall-clock budget.

    MethodChemistryMaterials
    GRPO68.375.9
    GRPO (on-policy)62.973.1
    SDPO80.578.3
    SR-OPSD — originally reported in the paper82.879.6
    SR-OPSD — transfer of (α,ρ)=(0.9,0.7)(\alpha,\rho)=(0.9,0.7) without retuning81.579.1
  • Sensitivity to hyperparameters. Table 4 shows that all 12 tested (α,ρ)(\alpha,\rho) configurations achieve 76.3–82.7 Avg@16, exceeding SDPO (73.4) and GRPO (72.0) under the same budget. Together with the transfer results, this shows competitive performance across multiple configurations and successful reuse of one configuration across related domains.

  • Consistency on mathematics. Among the post-training methods in Table 2, SR-OPSD achieves the highest or joint-highest Avg@16 on all five mathematical benchmarks at both model sizes. It also achieves the highest or joint-highest Pass@16 in 9 of the 10 model–benchmark settings, demonstrating consistent rankings across benchmarks and scales.

Q3. LCB v6 disjointness.

Our original Table 3 uses the setting of the SDPO paper [1]: training and evaluation share problems, and evaluation uses all test cases. Following the reviewer’s suggestion, we added a strictly problem-disjoint evaluation, retraining all methods on 65 problems and testing on 66 unseen problems with zero overlap. Results are presented in Table A-R3.

Table A-R3. Problem-disjoint LiveCodeBench v6 evaluation. Held-out Avg@16 (%), reported as mean ± sample standard deviation over two random seeds. Training feedback is restricted to test cases from the training problems.

MethodQwen3-1.7BQwen3-4B
GRPO18.2 ± 0.619.6 ± 5.6
SDPO15.1 ± 0.920.2 ± 2.2
SR-OPSD19.5 ± 0.725.9 ± 1.3

SR-OPSD achieves higher mean held-out accuracy than both baselines at 1.7B and 4B. At 4B, it reaches 25.9%, compared with 20.2% for SDPO and 19.6% for GRPO, demonstrating that its relative advantage extends to unseen programming problems at these scales.

W3/Q4. Top-kk and clipping.

For SciKnowEval and LCB, we retain the remaining probability mass in a tail bucket, keeping the student, self-teacher, and reference token distributions normalized on a common (k+1)(k+1)-category space. Proposition 3.2 applies to the geometric target constructed in this reduced space. We directly measure how closely the resulting logit gradients approximate their full-vocabulary counterparts.

We conducted a controlled fixed-context diagnostic on batches of 1,024 response positions, comparing full-vocabulary and top-kk-plus-tail gradients with respect to the student logits. The relative ℓ2\ell_2 error is ∥gk−gfull∥2/∥gfull∥2\lVert g_k-g_{\mathrm{full}}\rVert_2/\lVert g_{\mathrm{full}}\rVert_2.

Table A-R4. Agreement between full-vocabulary and top-kk-plus-tail logit gradients.

ModelBatch cosine at k=20k=20Batch cosine at k=100k=100Relative ℓ2\ell_2 error at k=100k=100
Qwen3-0.6B0.9990.9990.005
Qwen3-1.7B0.9940.9990.054

These diagnostics show close agreement between top-kk-plus-tail and full-vocabulary logit gradients at the default k=100k=100 for both model sizes.

Token-loss clipping is used only in mathematics, where SR-OPSD, SDPO, and OPSD use top-kk without a tail bucket. Clipping modifies gradients where active; SciKnowEval and LCB do not use token-loss clipping.

We will include these results and discussions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer AYS6 (Part 3/3)

Q5. Corrupted privileged context

We evaluate a corrupted privileged-context setting, replacing the self-teacher’s demonstrations and feedback with content from other problems. In this ablation (Table A-R5), reference anchoring improves accuracy on held-out tests of the training problems from 4.7% to 13.4% (+8.7 points), while reducing student–reference KL from 0.092 to 0.041. Accuracy on unseen problems also increases from 17.3% to 18.5%.

Table A-R5. Results under corrupted privileged context. Qwen3-1.7B, LiveCodeBench, forward Rényi projection with ρ=0.95\rho=0.95.

MethodTraining problems: held-out tests (%)Unseen problems (%)Student–reference KL
SR-OPSD (α=0.9\alpha=0.9)13.418.50.041
SR-OPSD w/o reference anchoring (α=1\alpha=1)4.717.30.092

Scores are Avg@16. Held-out tests of training problems are excluded from the training reward. KL is KL(π∥πref)\mathrm{KL}(\pi\Vert\pi_{\mathrm{ref}}), with both policies evaluated on the original prompt and student-generated prefixes, averaged per response token across all validation groups.

Reference

[1] Hübotter, J., et al. (2026). Reinforcement Learning via Self-Distillation. ICML 2026.

Reviewer_dDz1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SR-OPSD, which combines an evolving self-teacher with a frozen initial policy through geometric mixing and trains the student using forward Rényi divergence. It provides a theoretical characterization of the objective and its gradients, alongside experiments across multiple tasks that demonstrate the method’s effectiveness and highlight the interaction between reference anchoring and the projection objective.

Critical strengths
  1. The method is simple and offers a useful perspective on the interaction between reference anchoring and the projection objective in self-distillation.
  2. The gradient analysis clearly illustrates how the different model distributions influence updates. The experiments cover multiple tasks and model scales.
Critical weaknesses
  1. The analysis primarily explains the objective formulation and local update behavior. Further discussion of the broader mechanistic insights offered by this combination would help clarify its relevance to other settings.
  2. The rationale for geometric mixing in this setting, and the distinct role of the frozen initial reference alongside an EMA self-teacher, could be explained more fully. This would also help readers assess the benefits relative to the additional training cost.
Questions for authors
  1. What motivates geometrically mixing the self-teacher and initial reference distributions rather than taking a weighted arithmetic average? A brief explanation of the expected benefits for self-distillation would be helpful.
  2. With an EMA self-teacher already in place, what issues does the frozen initial reference address that adjusting the EMA alone would not readily resolve? A brief discussion of their complementary roles would be helpful.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dDz1

We thank the reviewer for the constructive review, and for the time invested in evaluating our work. We are grateful for the reviewer’s constructive comments which help us sharpen the scope and presentation of the paper. We address each concern in detail below. We hope these responses adequately address the reviewer’s concerns and that the reviewer will be willing to increase the score accordingly.

Why geometric rather than arithmetic mixing?

  • Geometric mixing has an exact variational interpretation: for fixed conditional self-teacher and reference distributions, it is the optimizer of reverse-KL-regularized distillation (Proposition B.1). It forms a product-of-experts target: whereas arithmetic mixing averages probabilities, geometric mixing averages log-odds, allowing the reference to attenuate self-teacher preferences that it disfavors. This provides a principled way to incorporate reference information into the supervisory target. SR-OPSD further generalizes this to a Rényi-regularized policy optimization formulation. To the best of our knowledge, this joint formulation is novel in on-policy self-distillation. And our extensive experiments across a broad range of tasks, model families, and model sizes further demonstrate the effectiveness of the proposed method.

  • Proposition 3.2 expresses the effective update target as a normalized geometric mixture of the self-teacher, reference, and current student, with exponents (αρ,(1−α)ρ,1−ρ)(\alpha\rho,(1-\alpha)\rho,1-\rho). Its log-probabilities combine theirs linearly, up to a normalization term, making the joint roles of α\alpha and ρ\rho explicit.

What does the frozen reference fix that the EMA teacher cannot?

  • The EMA rate controls how quickly the self-teacher follows the student, while the initial reference remains fixed in parameter space. For α<1\alpha<1, the geometric target retains an explicit initial-policy contribution at every update. This allows the self-teacher to keep adapting while preserving a persistent reference anchor.

  • Both the reference and self-teacher policies condition on the context information. The reference supplies the initial model's interpretation of that context, while the EMA self-teacher supplies an evolving interpretation. Geometric mixing allows the reference policy to moderate self-teacher preferences that conflict with its own.

  • The ablation on Physics directly supports this complementary role: with the same EMA rate and Rényi order ρ=0.7\rho=0.7, reference anchoring raises accuracy from 75.9 to 82.7 (Table 4). Complementary empirical evidence includes response-length growth under some unanchored objectives (Fig. 7), repetition collapse under reverse KL with 88% duplicate responses (Fig. 1c), and more stable training trajectories for SR-OPSD than SDPO (Fig. 3).

We have incorporated these discussions into the revised manuscript.

Dear Reviewer dDz1:

As the discussion period is about to end in 3 days, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.

Best,

Authors of Submission #19957

Dear Reviewer dDz1:

As the discussion period is approaching its end, we would like to kindly follow up and ask whether our rebuttal has addressed your concerns. We would greatly appreciate any feedback you may have and would be happy to address any remaining questions or concerns before the discussion ends.

Best,

Authors of Submission #19957

Dear Reviewer dDz1:

We have submitted our rebuttal addressing your concerns.

We would appreciate it if the reviewer could go through our responses at your earliest convenience since the deadline is approaching. Should our replies resolve your concerns, we sincerely hope you could consider adjusting your score accordingly. Please let us know if you have any further questions.

Best,

Authors of Submission #19957

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author-Reviewer Discussion

Dear Area Chair,

Thank you for reviewing our work and coordinating the discussion. We summarize the main strengths, clarifications, and additional experimental evidence.

(i) Strengths highlighted by the reviewers.

  • Motivation: ''joint consideration of target construction and divergence geometry is interesting and well motivated'' (Reviewer AYS6).
  • Method and analysis: ''a useful perspective on the interaction between reference anchoring and the projection objective in self-distillation'' (Reviewer dDz1); ''technically well developed, with clear mathematical formulations and gradient derivations'' (Reviewer AYS6).
  • Empirical results: Strong scientific-reasoning and code-generation performance (Reviewer bEw4), with broad evaluation across tasks and models (Reviewers AYS6 and dDz1).
  • Ablations: Demonstrate the evidence and valuable insights of the joint effect of reference anchoring and the divergence objective (Reviewers AYS6 and bEw4).

(ii) Clarifications.

  • The proposed method, SR-OPSD, further generalizes towards a Rényi-regularized policy optimization formulation. To the best of our knowledge, this joint formulation is novel in on-policy self-distillation. We also clarified the Rényi-regularized formulation and its KL connection.

(iii) Additional experimental evidence.

  • Problem-disjoint LiveCodeBench Experiments (Reviewer AYS6: evaluation on unseen problems; see Tables A-R3/b-R2.) These results extend the observed gains to unseen programming problems at both model sizes.

    Problem-disjoint LiveCodeBench. We use 65 training and 66 disjoint evaluation problems. Scores are held-out Avg@16 (%), mean ± sample standard deviation over two seeds.

    MethodQwen3-1.7BQwen3-4B
    GRPO18.2 ± 0.619.6 ± 5.6
    SDPO15.1 ± 0.920.2 ± 2.2
    SR-OPSD19.5 ± 0.725.9 ± 1.3
  • Clean-context ablations and method comparisons (Reviewer bEw4: controlled evidence beyond corrupted contexts and comparison with β\beta-OPSD; see Table b-R3). SDPO with reference anchoring reaches 17.9 ± 0.1% on Qwen3-1.7B; SR-OPSD scores 1.6 percentage points higher at the same anchoring coefficient. We also added a Qwen3-1.7B β\beta-OPSD comparison using the same problem-disjoint split, two seeds, and evaluation protocol. The ablations support evidence beyond Physics.

  • Transfer without retuning (Reviewer AYS6: transferability of α\alpha and ρ\rho; see Table A-R2). Reusing (α,ρ)=(0.9,0.7)(\alpha,\rho)=(0.9,0.7) from Physics on Chemistry and Materials yields 81.5%/79.1% on Qwen3-8B, measured as Avg@16 within 10 hours. These results support reuse of the selected configuration across related scientific domains without retuning.

  • Top-kk diagnostics (Reviewer AYS6: agreement between practical and theoretical gradients; see Table A-R4). On 1,024 fixed response positions, reported gradient cosine similarities with full-vocabulary gradients are approximately 0.999 at k=100k=100 for both 0.6B and 1.7B. These diagnostics support close gradient agreement in the tested settings.

  • Corrupted contexts (Reviewer AYS6: reference anchoring under incorrect privileged information; Table A-R5). Anchoring improves accuracy from 4.7% to 13.4% on held-out tests of training problems and 17.3% to 18.5% on unseen problems. Reference anchoring remains beneficial in the tested corrupted-context setting.

(iv) Discussion status during rebuttal.

  • Reviewer bEw4 maintained 3 (weak acceptance) while requesting further controlled experiments. We subsequently provided additional clean-context results and the β\beta-OPSD comparison. We appreciate the discussions between the reviewer and us.

  • We have also responded to Reviewer AYS6 and Reviewer dDz1 but have not received discussion replies yet. We would appreciate it if the reviewers could go through our responses at their earliest convenience since the deadline is approaching.

Thank you again for your consideration.

Best,
The Authors of Submission #19957

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.