- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits Self Distillation Policy Optimization (SDPO) for RLVR post-training, which is fragile on thinking models. The papers shows that even with an ideal teacher, SDPO performs only a one-step behavior-regularized improvement on the value of the behavior policy, not maximizing the reward of the policy. To address this limitation, the proposed T-SDPO derives the exact optimum of behavior-regularized reward maximization, which is an arithmetic mixture of the behavior policy and the success posterior. Experiments on three thinking models across math and science demonstrates that T-SDPO beating SDPO and GRPO, with longer responses and preserved accuracy.
Critical strengths
-
This paper offers an in-depth analysis of the fundamental issue that SDPO does not truly maximize the reward, and further derives a more reasonable optimization objective. The derivation is intuitive and makes sense.
-
The experiments in this paper are relatively comprehensive, covering different models and data from different domains, and the necessary ablation studies are also conducted.
Critical weaknesses
-
As shown in Table 5, different hyperparameters λ and η are selected for different datasets and models, which suggests that T-SDPO has limited generalizability. Moreover, the good performance in Tables 1 and 2 is obtained via overfitting, so the reported performance gains are overestimated.
-
This work lacks comparisons with methods other than SDPO. Although the paper cites β-OPSD, HDPO, and PBSD, they are not included as experimental baselines.
Questions for authors
- Why are the numbers in Figure 4 and Table 1 inconsistent? For example, for the 4B model, the best result in Figure 4 is 0.592, while the corresponding metric in Table 1 is 0.605. If the discrepancy comes from run-to-run variance, then Table 1 should report the metric interval over multiple runs.
6 comments
Thanks for the authors’ reply. Regarding W2, I have further questions. Why HDPO does not need to be compared against T-SDPO since it is complementary to T-SDPO? PBSD has a similar issue.
We thank the reviewer for the follow-up.
ICLR policy. Both HDPO and PBSD are arXiv-only preprints. The ICLR 2027 reviewer guidelines state that authors are not required to compare to papers that appear solely on arXiv, and that work published within two months of the submission deadline is considered contemporaneous. The latest version of PBSD appeared on August 21, 2026, only about one month before the ICLR deadline. Also, HDPO has no public implementation. We therefore focus the experimental comparison on β-OPSD, the closest of the three to our method, and discuss HDPO and PBSD below; both are cited and contrasted in the revised related work.
Method level. Beyond the policy, a head-to-head comparison with either method would not test our claim, because both differ from T-SDPO in what they are designed to do.
HDPO applies privileged self-distillation, on top of GRPO, only to prompts on which every rollout fails, whereas T-SDPO distills only on prompts with at least one success. The two methods thus act on disjoint prompts. On the prompts where T-SDPO acts, HDPO reduces to plain GRPO, which T-SDPO outperforms on math for all three models (G1). Moreover, HDPO's distillation target, by its own theorem, is the hard-threshold limit of the KL-regularized optimum, i.e., the limit of our Theorem 2, which is exactly SDPO's target. Our T-SDPO vs. SDPO comparison therefore already isolates the difference between HDPO's target and ours, with everything else held fixed. Since the two methods are complementary, combining them is a natural extension, which we will discuss in the final version.
PBSD replaces KL matching with a preference loss between teacher and student samples, and its reward-regularized optimum reweights the teacher. In contrast, T-SDPO regularizes toward the behavior policy, so its target stays within the support of at every state, and it adds only one no-gradient forward pass rather than additional costly teacher sampling. PBSD is also only evaluated on instruct (non-thinking) Qwen3 models, and on the same math task its improvement over its strongest self-distillation baseline (OPSD) is marginal, +1.0, +0.5 and +0.4 points on Qwen3-8B, 4B and 1.7B (their Table 2). Our target setting is thinking models, where self-distillation is most fragile, and where T-SDPO improves over SDPO by +8.3, +5.4 and +3.8 points on math.
We thank the reviewer for the positive assessment of our analysis and experiments. We address the concerns below.
W1. Different λ and η across datasets and models; the gains may be overestimated.
We address this with a new experiment in G2. We fix η for both SDPO and T-SDPO to a single value per domain and select λ on a held-out validation split disjoint from the test benchmarks. T-SDPO still outperforms SDPO in all six model–domain settings, with an average gain of +3.86 points against +4.23 under per-cell tuning, i.e., it retains 91% of the tuned gain.
W2. Comparison with methods other than SDPO.
We add β-OPSD as a baseline in G3, where T-SDPO outperforms it at all three mixture weights, and we discuss HDPO and PBSD there as well: HDPO targets prompts on which every rollout fails and is complementary to T-SDPO, and PBSD regularizes toward the teacher rather than the behavior policy. Note that Table 3 in the submission already compares with SDPO + KL penalty, which is the core mechanism of β-OPSD, and T-SDPO outperforms its best setting by 8.06 and 5.50 points on Qwen3-4B and Qwen3-8B.
Q1. Why are the numbers in Figure 4 and Table 1 inconsistent?
Thanks for catching this. Note first that the math panels of Figures 4 and 5 report AIME 2024 only, so the comparable entry of Table 1 is its AIME 2024 column, not the math average. The remaining 1.3-point gap comes from different runs.
To make the λ sweep directly comparable with Table 1, we additionally rerun iwith 3 seeds and report the math average over AIME 2024 and AMC 2023, exactly as in Table 1:
The conclusion of Figure 4 holds under this protocol: the optimum is strictly interior on both models, and every λ∈[0.5,0.9] outperforms SDPO, which is the shape Theorem 2 predicts.
We hope these new results address the reviewer's concerns. If so, we would greatly appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.