acceptodds
Under review as a conference paper at ICLR 2027

Revisiting On-Policy Self-Distillation from an RL Perspective

Abstract

Reinforcement learning with verifiable rewards (RLVR) gives a long reasoning trace only a sparse outcome reward. On-Policy Self-Distillation densifies this signal by conditioning the model on one of its own successful rollouts and distilling the resulting self-teacher into the unprompted policy, but its gains on thinking models are fragile. In this paper, we show that, even with an ideal teacher, SDPO performs a one-step behavior-regularized improvement on the value of the behavior policy, rather than maximizing the reward of the policy being improved. To correct this, we derive the exact optimum of behavior-regularized reward maximization and show that, for binary reward, it is a tilted mixture, i.e., an arithmetic mixture of the behavior policy and its success posterior. Based on this result, we propose Tilted-SDPO (T-SDPO), which distills this mixture using SDPO's prompted teacher in place of the posterior and reduces to SDPO when the mixture weight is one. Across math and science reasoning with three thinking models, T-SDPO outperforms SDPO and GRPO at an equal number of optimizer updates, improving math avg@12 over SDPO by up to points. It also keeps longer responses and higher pass@12 than SDPO.

Primary area:
reinforcement learning
Keywords:
self-distillation, reinforcement learning, on-policy distillation, self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QtCqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits Self Distillation Policy Optimization (SDPO) for RLVR post-training, which is fragile on thinking models. The papers shows that even with an ideal teacher, SDPO performs only a one-step behavior-regularized improvement on the value of the behavior policy, not maximizing the reward of the policy. To address this limitation, the proposed T-SDPO derives the exact optimum of behavior-regularized reward maximization, which is an arithmetic mixture of the behavior policy and the success posterior. Experiments on three thinking models across math and science demonstrates that T-SDPO beating SDPO and GRPO, with longer responses and preserved accuracy.

Critical strengths
  1. This paper offers an in-depth analysis of the fundamental issue that SDPO does not truly maximize the reward, and further derives a more reasonable optimization objective. The derivation is intuitive and makes sense.

  2. The experiments in this paper are relatively comprehensive, covering different models and data from different domains, and the necessary ablation studies are also conducted.

Critical weaknesses
  1. As shown in Table 5, different hyperparameters λ and η are selected for different datasets and models, which suggests that T-SDPO has limited generalizability. Moreover, the good performance in Tables 1 and 2 is obtained via overfitting, so the reported performance gains are overestimated.

  2. This work lacks comparisons with methods other than SDPO. Although the paper cites β-OPSD, HDPO, and PBSD, they are not included as experimental baselines.

Questions for authors
  1. Why are the numbers in Figure 4 and Table 1 inconsistent? For example, for the 4B model, the best result in Figure 4 is 0.592, while the corresponding metric in Table 1 is 0.605. If the discrepancy comes from run-to-run variance, then Table 1 should report the metric interval over multiple runs.
6 comments

We thank the reviewer for the positive assessment of our analysis and experiments. We address the concerns below.

W1. Different λ\lambda and η\eta across datasets and models; the gains may be overestimated.

We address this with a new experiment in G2. We fix η\eta for both SDPO and T-SDPO to a single value per domain and select λ\lambda on a held-out validation split disjoint from the test benchmarks. T-SDPO still outperforms SDPO in all six model–domain settings, with an average gain of +3.86 points against +4.23 under per-cell tuning, i.e., it retains 91% of the tuned gain.

W2. Comparison with methods other than SDPO.

We add β-OPSD as a baseline in G3, where T-SDPO outperforms it at all three mixture weights, and we discuss HDPO and PBSD there as well: HDPO targets prompts on which every rollout fails and is complementary to T-SDPO, and PBSD regularizes toward the teacher rather than the behavior policy. Note that Table 3 in the submission already compares with SDPO + KL penalty, which is the core mechanism of β-OPSD, and T-SDPO outperforms its best setting by 8.06 and 5.50 points on Qwen3-4B and Qwen3-8B.

Q1. Why are the numbers in Figure 4 and Table 1 inconsistent?

Thanks for catching this. Note first that the math panels of Figures 4 and 5 report AIME 2024 only, so the comparable entry of Table 1 is its AIME 2024 column, not the math average. The remaining 1.3-point gap comes from different runs.

To make the λ\lambda sweep directly comparable with Table 1, we additionally rerun iwith 3 seeds and report the math average over AIME 2024 and AMC 2023, exactly as in Table 1:

λ\lambda0.250.500.750.901.00 (SDPO)
Qwen3-4B62.14 ± 1.7274.27 ± 2.3375.38 ± 3.3072.14 ± 1.2267.06 ± 1.01
Qwen3-8B72.35 ± 0.5674.85 ± 0.8773.58 ± 0.0972.59 ± 0.8269.47 ± 2.31

The conclusion of Figure 4 holds under this protocol: the optimum is strictly interior on both models, and every λ∈[0.5,0.9]\lambda\in[0.5,0.9] outperforms SDPO, which is the shape Theorem 2 predicts.

We hope these new results address the reviewer's concerns. If so, we would greatly appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.

Thanks for the authors’ reply. Regarding W2, I have further questions. Why HDPO does not need to be compared against T-SDPO since it is complementary to T-SDPO? PBSD has a similar issue.

We thank the reviewer for the follow-up.

ICLR policy. Both HDPO and PBSD are arXiv-only preprints. The ICLR 2027 reviewer guidelines state that authors are not required to compare to papers that appear solely on arXiv, and that work published within two months of the submission deadline is considered contemporaneous. The latest version of PBSD appeared on August 21, 2026, only about one month before the ICLR deadline. Also, HDPO has no public implementation. We therefore focus the experimental comparison on β-OPSD, the closest of the three to our method, and discuss HDPO and PBSD below; both are cited and contrasted in the revised related work.

Method level. Beyond the policy, a head-to-head comparison with either method would not test our claim, because both differ from T-SDPO in what they are designed to do.

HDPO applies privileged self-distillation, on top of GRPO, only to prompts on which every rollout fails, whereas T-SDPO distills only on prompts with at least one success. The two methods thus act on disjoint prompts. On the prompts where T-SDPO acts, HDPO reduces to plain GRPO, which T-SDPO outperforms on math for all three models (G1). Moreover, HDPO's distillation target, by its own theorem, is the hard-threshold limit of the KL-regularized optimum, i.e., the β→0\beta\to0 limit of our Theorem 2, which is exactly SDPO's target. Our T-SDPO vs. SDPO comparison therefore already isolates the difference between HDPO's target and ours, with everything else held fixed. Since the two methods are complementary, combining them is a natural extension, which we will discuss in the final version.

PBSD replaces KL matching with a preference loss between teacher and student samples, and its reward-regularized optimum reweights the teacher. In contrast, T-SDPO regularizes toward the behavior policy, so its target stays within the support of μ\mu at every state, and it adds only one no-gradient forward pass rather than additional costly teacher sampling. PBSD is also only evaluated on instruct (non-thinking) Qwen3 models, and on the same math task its improvement over its strongest self-distillation baseline (OPSD) is marginal, +1.0, +0.5 and +0.4 points on Qwen3-8B, 4B and 1.7B (their Table 2). Our target setting is thinking models, where self-distillation is most fragile, and where T-SDPO improves over SDPO by +8.3, +5.4 and +3.8 points on math.

Reviewer_sKEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper view SDPO from RL perspective. It derives a behavior-regularized optimum for binary rewards, showing that the optimal target is an arithmetic mixture of the behavior policy and the success posterior. Auther then propose a new method TSDPO which improve performance compare with SDPO and other baselines.

Critical strengths
  1. The connection between SDPO with RL is interesting.
  2. The method is simple and require little code change on SDPO
Critical weaknesses
  1. The core of RL is Bellman optimality. Auther did connect SDPO with RL and Equ 10 and Equ 11 are the format of policy improvement. However without policy evaluation, how can SDPO maintain Bellman optimality?

  2. For table 1 and 2, auther only reports the result of math and science without statistic analysis such as std. For Table 2 result espicially, the improvement are small enough that they may fall within seed or other variance.

  3. Based on Figure 4, same model but different parameter has different behaviour on different λ\lambda and the performance varies a lot. Fine tuning the hyperparameter required more work

Questions for authors
  1. Can auther show whether its token-level mixture equal with the derived trajectory-level? The toy example does not validate the trajectory level optimum derived in Theorem 2. The theorem consider a mixture over full trajectories with the exact success posterior

  2. Can the authors show whether performance improvement arise from the tilted target, longer reasoning traces, or stronger behavior regularization?

  3. Is there a better way to determin λ\lambda?

  4. Minor typos: line 61 ", even with an ideal teacher, . "

4 comments

We thank the reviewer for finding the connection between SDPO and RL interesting and the method simple. We address the concerns below.

W1. Without policy evaluation, how can SDPO maintain Bellman optimality?

This is a good question. In our setting, policy evaluation is performed implicitly by the teacher rather than by a learned critic. More specifically, the token MDP has deterministic transitions and a single binary reward at the end, so Qμ(s,a)Q^\mu(s,a) is the probability that μ\mu's continuation succeeds after (s,a)(s,a). By Bayes' rule (Eq. 6), the success posterior satisfies q+μ(a∣s)=μ(a∣s)Qμ(s,a)/Vμ(s)q_+^\mu(a\mid s)=\mu(a\mid s)Q^\mu(s,a)/V^\mu(s), so the ratio log⁡(q+μ/μ)=log⁡Qμ−log⁡Vμ\log(q_+^\mu/\mu)=\log Q^\mu-\log V^\mu already encodes the evaluation of μ\mu. In this sense, the prompted teacher plays the role of a Monte-Carlo critic, and Eq. 10 is the corresponding improvement step.

When the teacher is updated (η>0\eta>0), evaluation and improvement alternate, and SDPO becomes a form of (generalized) policy iteration with a trust region. Note that, even then, each step improves log⁡Qμ\log Q^\mu rather than the reward, which is the bias we identify. T-SDPO instead distills the exact optimum of KL-regularized reward maximization (Theorem 2), so the regularized fixed point it targets is the standard one of KL-regularized RL (Geist et al., 2019).

W2. No statistical analysis such as std; Table 2 improvements are small.

We now report mean ± std over 3 seeds for all settings (G1). On science, T-SDPO outperforms SDPO on all three models, by +3.51, +2.05 and +2.36 points on Qwen3-4B, Qwen3-8B and Olmo-3-7B, and on Qwen3-4B and Olmo-3-7B the gap exceeds two standard deviations of either method.

W3 & Q3. Sensitivity to λ\lambda; is there a better way to determine λ\lambda?

Please see G2 for results. Regarding a better way to determine λ\lambda, Theorem 2 gives its exact form λβ(x)=cVμ(x)/(1+cVμ(x))\lambda_\beta(x)=cV^\mu(x)/(1+cV^\mu(x)), which depends on the success rate of the behavior policy at prompt xx. This quantity is available for free in our pipeline: the group of nn rollouts already provides an unbiased estimate V^μ(x)\hat V^\mu(x). We leave this variant as the future work.

Q1. Is the token-level mixture equal to the derived trajectory-level optimum?

We have clarified this in Section 4.3. The token-level conditional of the trajectory-level mixture (1−λβ)μ+λβq+μ(1-\lambda_\beta)\mu+\lambda_\beta q_+^\mu is itself a mixture of μ(⋅∣st)\mu(\cdot\mid s_t) and q+μ(⋅∣st)q_+^\mu(\cdot\mid s_t), but with a prefix-dependent weight ρt\rho_t, i.e., the posterior responsibility of the success component given the prefix sts_t. The constant λ\lambda in Eq. 13 approximates ρt\rho_t by a scalar.

This follows the same practice as SDPO itself (Hübotter et al., 2026). SDPO's target is also trajectory-level, SDPO instead uses the prompted teacher's next-token distribution as a tractable token-level surrogate. T-SDPO keeps this machinery and only mixes the teacher with μ\mu; the constant λ\lambda likewise avoids estimating success at every prefix, which would need extra rollouts. Figure 4 shows that a constant λ\lambda already outperforms SDPO for every λ∈[0.5,0.9]\lambda\in[0.5,0.9].

Q2. Does the improvement come from the tilted target, longer reasoning, or stronger regularization?

These are not competing explanations but a chain of consequences of one change. T-SDPO is not SDPO plus a length or regularization heuristic: its target is the derived optimum of behavior-regularized reward maximization (Theorem 2). The tilted target gives a better reward–regularization trade-off; keeping the mass of μ\mu on the reasoning the teacher would cut yields longer responses (Figure 3), and in turn higher accuracy. The controlled comparisons agree. (1) Stronger regularization alone does not explain the gain: SDPO + KL penalty toward μ\mu (Table 3), at its best strength, trails T-SDPO by 8.06 and 5.50 points on Qwen3-4B and Qwen3-8B. (2) The form of the target matters: β-OPSD, a geometric rather than arithmetic mixture, trails T-SDPO at every weight (G3).

Q4. Typo in line 61.

Thank you, fixed.

We hope these new results address the reviewer's concerns. If so, we would greatly appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.

Thank you for providing the response and resolve most of my concerns. The response suggests that the teacher replaces policy evaluation by directly estimating the success posterior, effectively serving as a Monte Carlo critic. However, unlike a conventional critic, the teacher is not trained to satisfy the Bellman equation. An estimate of success probability alone does not necessarily imply Bellman consistency. What theoretical justification guarantees that this teacher-based estimation can replace policy evaluation and preserve the connection to policy iteration or Bellman optimality?

We thank the reviewer for the follow-up. We make this precise below.

The critic implied by the teacher. Our MDP is the token-level MDP with deterministic transitions st+1=(st,at)s_{t+1}=(s_t,a_t), no intermediate reward, and a binary reward R(y)R(y) at the end. For any prompted teacher qq, define the critic it implies relative to μ\mu by Q^(st,at)/V^(st):=q(at∣st)/μ(at∣st)\hat Q(s_t,a_t)/\hat V(s_t) := q(a_t\mid s_t)/\mu(a_t\mid s_t), with V^(x)=Vμ(x)\hat V(x)=V^\mu(x) at the prompt and V^(st+1):=Q^(st,at)\hat V(s_{t+1}) := \hat Q(s_t,a_t). This is exactly the quantity SDPO's update uses, since its implicit reward is log⁡(q/μ)=log⁡Q^−log⁡V^\log(q/\mu)=\log\hat Q-\log\hat V (Eq. 9–10).

Which Bellman conditions hold. The finite-horizon Bellman equations for QμQ^\mu in this MDP are (a) ∑aμ(a∣s)Q(s,a)=V(s)\sum_a\mu(a\mid s)Q(s,a)=V(s), (b) Q(st,at)=V(st+1)Q(s_t,a_t)=V(s_{t+1}), and (c) the boundary condition V(sT)=R(y)V(s_T)=R(y). Condition (a) holds for Q^\hat Q for any normalized teacher, since ∑aμ(a∣s)Q^(s,a)=V^(s)∑aq(a∣s)=V^(s)\sum_a\mu(a\mid s)\hat Q(s,a)=\hat V(s)\sum_a q(a\mid s)=\hat V(s). Condition (b) holds by construction, because transitions are deterministic. The only non-trivial condition is (c). Multiplying the ratios along a full response, V^(sT)=Vμ(x) q(y)/μ(y)\hat V(s_T)=V^\mu(x)\,q(y)/\mu(y), so (c) holds if and only if q(y)=μ(y)R(y)/Vμ(x)q(y)=\mu(y)R(y)/V^\mu(x), which is the form of Assumption 1 (Eq. 8). Since the finite-horizon Bellman recursion has a unique solution, the teacher-implied critic equals QμQ^\mu exactly if and only if the teacher matches the success posterior at the response level. Note that no Bellman training is needed for this: in an episodic task with terminal reward, Qμ(s,a)=Pμ(R=1∣s,a)Q^\mu(s,a)=P_\mu(R=1\mid s,a) is itself the unique fixed point of the Bellman operator, and any exact estimate of this success probability is Bellman-consistent. Bellman training (TD) is a way to obtain such an estimate by bootstrapping, not a separate requirement on it.

Connection to policy iteration. Given an exact critic, the SDPO update is the policy-improvement step of Eq. 10 and the T-SDPO target is the exact regularized improvement of Theorem 2; with a frozen teacher (η=0\eta=0) this is a single improvement step, and with an EMA teacher it is regularized (mirror-descent) policy iteration. When Assumption 1 holds only approximately, the teacher acts as an approximate critic, and the standard approximate policy iteration analysis applies: the performance loss is bounded linearly in the evaluation error over the horizon (e.g., Bertsekas and Tsitsiklis, 1996; Geist et al., 2019 for the KL-regularized case). We do not claim Bellman optimality for the practical algorithm beyond this.

Since the other reviewers have not yet replied. We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!

Reviewer_JVfiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors analyze the Self-Distillation algorithm under a binary reward and identify a gap in which the algorithm optimizes Q^\mu rather than Q^\pi. They proposed a fix that changes the target distribution to a mixture of the conditioned and non-conditioned teacher and demonstrated the performance gains.

Critical strengths
  • The paper derivations are solid and novel. They provide a new perspective on the self-distillaiton algorithm.

  • The empirical gains compared to SDPO are significant, especially given the small cost of a single forward pass.

  • Overall good contribution, but in my eyes it needs a bit more work before it is publish-ready.

Critical weaknesses

Although I agree with the analysis in 4.1 and find it elegant, I disagree with the conclusion about one-step RL. This is true only if you fix the behavior policy and do not change it (η=0). Since in most implementations of SD the behavior policy is moving, the correct conclusion is that SD is equivalent to trust-region RL optimization (or Mirror Descent RL in the case of η=1). Therefore, I believe that statements like “even with an ideal teacher, SDPO performs a one-step behavior-regularized improvement” (from abstract\intro) are misleading.

Questions for authors
  • I wonder if Assumption 1 can be checked empirically using a small experiment - performing a rejection sampling from a model, and comparing the answer distribution to that of the same model conditioned as a teacher. In other words, checking if equation 8 is indeed true, at least in the response-level, even if not in the token-level.

  • There is an interesting interaction between the EMA and T-SDPO. According to the author’s own derivation, when η=1 (no EMA), T-SDPO should be equivalent to SDPO since \mu=\pi in equation 10. However, the target distribution (equation 14) will still be different. Where did this deviation come from? Is one of these empirically better than the other?

  • There are two known problems of SDPO with thinking models - thinking collapse due to overconfidence, and hallucinations due to references to the teacher’s context [1,2]. The analysis in Figure 3 shows that T-SDPO slightly helps with reasoning length collapse, although the lengths remain below those of GRPO. I think checking for hallucination can be interesting to see.

  • More generally, do you think T-SDPO solves the problems with SDPO in the thinking models that the previous paper pointed to? How’s the moving from optimizing Q^\mu to Q^\pi relate to these?

[1] Kim, Jeonghye, et al. "Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?." arXiv preprint arXiv:2603.24472 (2026).

[2] Kaur, Simran, et al. "Rethinking On-Policy Self-Distillation for Thinking Models." arXiv preprint arXiv:2607.05184 (2026).

2 comments

We thank the reviewer for the thoughtful and detailed review, and for finding our derivations "solid and novel" and the empirical gains "significant". The suggestions have substantially improved the paper.

W1. The "one-step RL" conclusion holds only for a fixed behavior policy.

We agree, and we thank the reviewer for this precise characterization. With η=0\eta=0, μ\mu is the base policy and SDPO is exactly a one-step improvement in the sense of one-step offline RL; with η>0\eta>0, μ\mu moves, and SDPO is a trust-region policy iteration, which reduces to mirror-descent RL at η=1\eta=1. We have revised the paper accordingly.

Note that our main conclusion is unchanged under this view: whether μ\mu is fixed or moving, each step maximizes Eπ[log⁡Qμ]−KL(π∥μ)E_\pi[\log Q^\mu]-KL(\pi\Vert\mu), whose optimum is q+μq_+^\mu, i.e., the zero-temperature limit of KL-regularized reward maximization. T-SDPO changes the per-step target to the finite-temperature optimum, which is independent of how μ\mu is updated.

Q1. Can Assumption 1 be checked empirically?

Thank you for the excellent suggestion. We ran exactly this check. On Qwen3-4B-Thinking with a frozen teacher (η=0\eta=0, so μ=π0\mu=\pi_0), we sample 16 responses from μ\mu for each of 256 DAPO training prompts and keep the correct ones, which are exact samples of q+μq_+^\mu. On the 192 prompts with at least one success, we build SDPO's prompted teacher from the first success y⋆y^\star and sample 8 responses from it.

On the 192 prompts with a successμ\muPrompted teacher qy⋆q_{y^\star}q+μq_+^\mu (rejection sampling)
Accuracy78.0%99.8%100%
TV of the answer distribution to q+μq_+^\mu0.2200.0020

At the answer level, Assumption 1 holds almost exactly. The prompted teacher puts 99.8% of its mass on correct answers, and its answer distribution is within TV 0.002 of q+μq_+^\mu, against 0.220 for μ\mu. This holds even on hard prompts: where μ\mu succeeds only 20% of the time (Vμ<0.5V^\mu<0.5), the teacher is still 99.4% correct. Note that the teacher is not simply copying y⋆y^\star: only 26% of its 8-grams appear in y⋆y^\star.

Q2. Interaction between EMA and T-SDPO at η=1\eta=1.

This is an insightful observation. At η=1\eta=1, μ=π\mu=\pi and SDPO's per-step optimum is q+πq_+^\pi, while T-SDPO's target is (1−λ)π+λqy⋆(1-\lambda)\pi+\lambda q_{y^\star}. The deviation comes from the temperature: SDPO solves the improvement step at β→0\beta\to0, which jumps to the posterior in a single step, whereas T-SDPO solves it at a finite β\beta, which moves only a fraction λ\lambda of the way. The two are no longer equivalent.

Q4. Does T-SDPO solve the problems of SDPO on thinking models?

We believe T-SDPO addresses the root cause shared by both failure modes, at least partially. Both thinking collapse and context hallucination arise because SDPO distills the teacher alone: its target is q+μq_+^\mu, which concentrates mass on short, confident continuations and on tokens supported only under the teacher's context. Theorem 2 shows that this target is the zero-temperature limit of the correct objective. At a finite temperature, the optimum retains mass on μ\mu and has the union of the two supports, so the student is never asked to drop the uncertainty-expressing tokens of μ\mu. Empirically, T-SDPO keeps substantially longer responses than SDPO and higher pass@12. Note that the remaining length gap to GRPO is expected: T-SDPO is still a distillation method whose teacher is shorter than μ\mu, and a smaller λ\lambda trades accuracy for length.

Regarding the move from QμQ^\mu to the reward: SDPO optimizes log⁡Qμ\log Q^\mu, whose optimum ignores how much probability mass the failed responses had, so it removes them entirely. Optimizing the reward under a KL budget only shifts mass in proportion to the gain, which is what keeps the reasoning of μ\mu intact.

We thank the reviewer again for the insightful questions, which have improved both the framing and the analysis of the paper. We hope our response addresses the concerns, and we would greatly appreciate it if the reviewer could consider raising the score.

We sincerely hope the reviewer could consider raising the score if no major concerns are left, that means a lot to us. We hope you all the best!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. We are glad that reviewers find the analysis of SDPO from an RL perspective "solid and novel" (JVfi), "intuitive" (QtCq) and "interesting" (sKEB), the method "simple and requires little code change" (sKEB), and the gains over SDPO "significant, especially given the small cost of a single forward pass" (JVfi). The shared concerns are about statistical significance, hyperparameter selection, and additional baselines. We address them below with new experiments, and we have revised the paper accordingly (changes marked in blue).

G1. Multiple seeds and statistical significance (QtCq, sKEB)

We rerun all main-table settings with 3 seeds and report mean ± std. T-SDPO outperforms SDPO in all six model–domain settings. The gain is largest on math: +8.32 on Qwen3-4B and +5.38 on Qwen3-8B. This suggests that the improvement of T-SDPO is not an artifact of seed variance.

Model / TaskGRPOSDPOT-SDPOΔ vs SDPO
Qwen3-4B, Math avg66.00 ± 0.7467.06 ± 1.0175.38 ± 3.30+8.32
Qwen3-8B, Math avg71.55 ± 0.1769.47 ± 2.3174.85 ± 0.87+5.38
Olmo-3-7B, Math avg67.55 ± 0.6870.14 ± 0.7975.52 ± 1.33+3.76
Qwen3-4B, Science avg41.02 ± 0.3658.30 ± 0.9361.81 ± 0.53+3.51
Qwen3-8B, Science avg41.47 ± 1.2963.28 ± 1.1265.33 ± 1.01+2.05
Olmo-3-7B, Science avg28.60 ± 1.0460.50 ± 0.3462.86 ± 0.58+2.36

G2. Robustness to hyperparameters: one shared configuration (QtCq, sKEB)

To show that the gain does not come from per-cell tuning, we fix the teacher rate for both SDPO and T-SDPO to a single value per domain, η=0\eta=0 for math and η=0.01\eta=0.01 for science (the best η\eta for SDPO), and select λ\lambda on a held-out validation split that is disjoint from the test benchmarks. T-SDPO still outperforms SDPO in all six settings, with an average gain of +3.86 points against +4.23 under per-cell tuning (G1), i.e., it retains 91% of the tuned gain. Note that Figure 4 shows that every λ∈[0.5,0.9]\lambda\in[0.5, 0.9] outperforms SDPO on both Qwen models, so the method is insensitive to λ\lambda within a wide range.

Model / TaskSDPOT-SDPO (val-selected)
Qwen3-4B, Math67.06 ± 1.0174.82 ± 2.10
Qwen3-8B, Math69.47 ± 2.3174.58 ± 0.09
Olmo-3-7B, Math70.14 ± 0.7974.49 ± 1.04
Qwen3-4B, Science57.71 ± 1.2157.94 ± 0.86
Qwen3-8B, Science62.20 ± 0.9765.20 ± 1.05
Olmo-3-7B, Science60.50 ± 0.3463.20 ± 0.61
AuthorsOpenReview ↗

General Response (cont.)

G3. Additional baselines (QtCq)

We add β-OPSD, the closest of the three methods to T-SDPO, under the same model, prompts, budget and evaluation protocol, and report it at every mixture weight we sweep rather than at a selected one. T-SDPO outperforms β-OPSD at all three weights, by 5.77 points over its best setting. Note that this comparison isolates exactly the design choice our theory makes: β-OPSD distills the normalized geometric mixture μ1−λqλ\mu^{1-\lambda}q^{\lambda} of the reference and the teacher, whereas T-SDPO distills the arithmetic mixture that Theorem 2 identifies as the exact optimum of behavior-regularized reward maximization. The geometric mixture is supported on the intersection of the two distributions and removes every token the teacher rules out, while the arithmetic one keeps the support of μ\mu (Section 4.3).

MethodAIME 2024AMC 2023Average
SDPO44.7284.7964.76
β-OPSD (λ=0.5\lambda=0.5)51.70 ± 2.8090.93 ± 1.0371.32 ± 1.59
β-OPSD (λ=0.75\lambda=0.75)53.98 ± 0.6490.03 ± 1.2672.01 ± 0.73
β-OPSD (λ=0.9\lambda=0.9)51.33 ± 1.6985.83 ± 0.3668.58 ± 1.02
T-SDPO (ours)60.5695.0077.78

HDPO and PBSD address different problems from ours, and we discuss them rather than force them into our protocol. HDPO applies privileged self-distillation only to "cliff" prompts on which every rollout fails, on top of GRPO, with a teacher conditioned on the ground-truth solution. It is therefore complementary to T-SDPO, which distills only on prompts with at least one success, so the two act on disjoint prompts and can be combined. More importantly, HDPO's theoretical result, that success-filtered privileged generation recovers the KL-regularized optimum in the hard-threshold limit, corresponds to the β→0\beta\to0 limit of our Theorem 2, where λβ=1\lambda_\beta=1 and the target is exactly SDPO's. T-SDPO is the finite-temperature counterpart, which is what keeps the mass of μ\mu. PBSD replaces KL matching with a preference loss between teacher and student samples, and its reward-regularized optimum reweights the teacher. In contrast, T-SDPO regularizes toward the behavior policy, so its target stays within the support of μ\mu at every state, and it adds only one no-gradient forward pass, whereas PBSD additionally samples from the teacher. We have added both methods, with this contrast, to the related work.

G4. Revised claim on "one-step" improvement (JVfi)

We agree that the "one-step" characterization is exact only when the behavior policy is fixed. With η=0\eta=0, μ\mu is the base model and SDPO is a one-step improvement over it; with η>0\eta>0, μ\mu tracks the student and SDPO becomes a trust-region policy iteration. We have revised the abstract, introduction and Section 4.1 accordingly (see the response to JVfi).

The difference we highlight, however, does not depend on how μ\mu is updated. Moving μ\mu changes which policy is evaluated, not what is optimized: at every step SDPO maximizes Eπ[log⁡Qμ]−KL(π∥μ)\mathbb{E}_\pi[\log Q^\mu]-\mathrm{KL}(\pi\Vert\mu), the log-success probability under the behavior policy's continuation, rather than the reward of the policy being improved. Moreover, in practice μ\mu cannot track π\pi closely, a small η\eta is needed to train stably: as η\eta grows it collapses quickly. In the regime where self-ditillation works, μ\mu therefore lags far behind π\pi, and the objective SDPO optimizes stays biased away from the reward. T-SDPO removes this bias in the per-step target itself, whichever way μ\mu is updated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.