acceptodds
Under review as a conference paper at ICLR 2027

RUC-SDPO: Reward-Uplift-Calibrated Self-Distillation Policy Optimization

Abstract

Reinforcement learning with verifiable rewards (RLVR) enables scalable post-training, with Group Relative Policy Optimization (GRPO) widely used to translate verifier outcomes into policy updates. However, GRPO assigns the same group-relative advantage to every token in a trajectory, resulting in coarse credit assignment; its reward-driven update vanishes when all trajectories in a group receive identical rewards. Privileged on-policy self-distillation can provide dense token-level guidance from training-only context, but applying it indiscriminately may impose redundant, noisy, or reward-irrelevant preferences. We introduce Reward-Uplift-Calibrated Self-Distillation Policy Optimization (RUC-SDPO), a reward-anchored extension of GRPO that calibrates privileged self-distillation by the estimated task-level utility of the privileged context. Specifically, RUC-SDPO retains GRPO as the primary reward anchor on every sample, and uses reward-uplift calibration to estimate the verifier-reward uplift of privileged conditioning over deployment-matched conditioning. It maps only positive uplift to a bounded detached weight, and scales token-level distillation accordingly. The method requires no separate teacher model, learned reward model, or step-level annotations. Across mathematical reasoning and tool-call generation with Qwen3-4B, Qwen3-8B, and Qwen3-14B, RUC-SDPO outperforms a strengthened GRPO baseline, attaining the best average accuracy in all nine mathematical reasoning model–benchmark settings and the best score in eight of the nine tool-call settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Fine-Tuning, Post-Training, Model Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GS69OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RUC-SDPO, which combines GRPO with privileged on-policy self-distillation calibrated by estimated verifier-reward uplift. The method estimates the benefit of privileged conditioning using additional rollouts and uses positive uplift to continuously weight an auxiliary token-level JSD objective. Experiments across Qwen3 models on mathematical and tool-call tasks demonstrate consistent improvements over GRPO baseline.

Critical strengths
  1. The proposed weighting mechanism provides a task-level criterion for regulating privileged self-distillation, rather than relying solely on correctness or teacher confidence.
  2. The proposed RUC-SDPO shows improvement over GRPO across mathematical and tool-call benchmarks.
Critical weaknesses
  1. Eqs. (3–5) compare original-policy rewards against privileged EMA-policy rewards in the actual implementation, potentially introducing bias in reward-uplift estimation. Independent finite-sample reward estimates can also produce positive uplift even when privileged conditioning provides no genuine improvement.
  2. SRPO and HDPO already combine group-relative RL with selective privileged self-distillation. Although RUC-SDPO introduces a distinct reward-uplift weighting mechanism, neither method is discussed or evaluated.
  3. Although Eqs.6–7 motivate context-specific reward-uplift weighting, Appendix D aggregates rewards at the prompt level and assigns the same weight to all privileged contexts. Consequently, harmful contexts may still receive positive distillation weights when other contexts provide sufficient uplift.
Questions for authors
  1. Is there evidence that positive uplift corresponds to more informative teacher distributions along these student trajectories?
  2. How does performance change when using only the correctness label, only the sibling response, or both? This would clarify which information is responsible for the improvements.
  3. How sensitive is the method to the number of additional privileged rollouts (K)? Does increasing (K) improve the reliability of uplift estimation and downstream performance?
7 comments
AuthorsOpenReview ↗

Response to Reviewer GS69 (Part 1)

Thank you for your constructive feedback and for recognizing our task-level weighting criterion for regulating privileged self-distillation and the consistent improvements over GRPO across mathematical and tool-call benchmarks. Below, we address each weakness and question in detail, providing a new controlled comparison against SRPO, an ablation on privileged input components, a sensitivity study on KK, and clarifications on EMA calibration and prompt-level aggregation.

W1: EMA Calibration and Finite-Sample Noise

Our EMA teacher follows the teacher-regularization design in SDPO (Hübotter et al., 2026), which introduces EMA for training stability and supports this choice through its teacher-update ablation. Related evidence from SDFT (Shenfeld et al., 2026) shows that using the current student directly as the teacher amplifies noisy updates, whereas an EMA teacher remains stable across steps while still keeping pace with the student's improvement.

In RUC-SDPO, we use an EMA update rate of 0.05 (Table 8). Calibration rollouts use the same EMA policy to estimate the uplift of the teacher that actually provides the distillation targets. The estimate is therefore defined for this EMA teacher relative to the current student. We have updated Section 3.2 (after Eq. (5)) of the revision to explicitly state that, following SDPO and SDFT, the teacher view uses the EMA policy for stability and the calibration rollouts use the same EMA policy to estimate the uplift of that teacher.

For finite-sample calibration, we pool rewards across valid privileged targets of each prompt and convert the estimated gap into a bounded, detached weight. Positive estimates can still arise from sampling noise, but one that is small relative to the remaining reward margin receives a proportionally small weight, rather than the full weight that binary gating would assign (Table 3). We have clarified in Section 3.2 that the weight represents estimated prompt-level utility.

W2: Comparison with SRPO and HDPO

In Section 2 (Related Work) of the submission, SRPO is already cited and discussed as Li et al. (2026) on sample routing, and HDPO as Ding (2026) on distillation for all-failed prompts. In the revision, we have updated Section 2 to explicitly name both methods and further clarify how they differ from RUC-SDPO.

Methodologically, RUC-SDPO differs from SRPO and HDPO in both how the distillation weight is determined and what privileged information is used:

  • SRPO routes each sample exclusively to either GRPO or distillation and weights distilled tokens by teacher entropy, which reflects the teacher's output confidence alone.
  • HDPO applies a binary gate that distills only when all student rollouts fail, and supplies the teacher with the reference solution.

Neither method measures whether conditioning on the privileged context actually improves task reward over the student view. In contrast, RUC-SDPO retains GRPO on every original student rollout, uses a sibling response and its verifier label rather than the reference solution as privileged input, and derives the auxiliary distillation weight directly from the estimated verifier-reward uplift of the privileged teacher view over the student view.

To evaluate the teacher-entropy weighting rule under a controlled setup, we compare RUC-SDPO against an SRPO control on Qwen3-8B for mathematical reasoning. We replace RUC-SDPO's reward-uplift weight with SRPO's teacher-entropy-based weighting. Under the same 12.2-hour wall-clock budget, RUC-SDPO completes 120 training steps and the SRPO control completes 130 training steps. Table E below reports MEAN@8, where ± values denote standard deviations across three evaluation seeds.

Table E. Controlled comparison with teacher-entropy weighting (SRPO control) on Qwen3-8B for mathematical reasoning under a matched wall-clock budget (MEAN@8 ± std over 3 evaluation seeds).

MethodTraining stepsMATH-500AIME 2026HMMT Feb. 2026
RUC-SDPO12086.73±0.5047.78±1.9229.42±0.58
SRPO control13085.20±0.7044.44±1.4626.26±1.53

Within this budget, RUC-SDPO at step 120 outperforms the SRPO control at step 130 on all three benchmarks.

We do not add an end-to-end HDPO run because HDPO gives the teacher the reference solution, whereas RUC-SDPO gives the teacher a sibling attempt and its verifier label; any difference between HDPO and RUC-SDPO could therefore come either from the weighting rule or from the teacher's access to the reference solution. For the weighting rule itself, HDPO applies full distillation weight via a binary gate, which is similar to the GRPO+Binary Gate method in Table 3. We already show that continuous calibration outperforms binary gating on Qwen3-8B across the three mathematical reasoning benchmarks in MEAN@8 (MATH-500: 87.0 vs. 86.4; AIME 2026: 48.3 vs. 46.8; HMMT February 2026: 30.7 vs. 29.4).

AuthorsOpenReview ↗

Response to Reviewer GS69 (Part 2)

W3: Prompt-Level Aggregation vs. Selectivity

As noted in Section 3.2 and Appendix D.2, we average the privileged rollout rewards across all valid contexts of the same prompt so the shared weight estimates the prompt's average privileged utility. With binary rewards and K=1K=1, a per-context estimate J^f(x,c(i))∈0,1\widehat{J}_f(x,c^{(i)}) \in \\{0, 1\\} has high sampling variance and collapses the weight in Eqs. (6)–(7) to a binary 0/10/1 gate. Therefore, we adopt prompt-level aggregation to reduce this variance and produce continuous weights.

We agree that this design trades context-specific selectivity for a lower-variance estimate: an individually harmful context can receive a positive weight if other contexts for the same prompt yield strong uplift. Even in that case, the harmful context's lower privileged reward pulls down the shared prompt weight, and the GRPO anchor remains active on every trajectory. In the revision, we have explicitly labeled this shared weight after Eqs. (6)–(7) in Section 3.2 as prompt-level to match Appendix D.2.

Q1: Evidence for Positive-Uplift Guidance

Tables 2 and 3 provide empirical evidence that positive uplift identifies useful teacher guidance. Across both ablations, we keep the privileged-context construction rule and the JSD objective fixed while varying only whether and how positive uplift weights the distillation loss:

  • Without the GRPO anchor (Table 2), weighting the JSD along student trajectories by positive uplift (RUC-SD) raises unweighted average MEAN@8 from 45.3 (SDPO) to 52.3.
  • With the GRPO anchor fixed (Table 3), unweighted average MEAN@8 is 53.3 for GRPO, drops to 48.8 when distilling from the privileged teacher without uplift calibration (GRPO+SDPO), and improves to 54.2 with binary uplift gating (GRPO+Binary Gate) and 55.3 with continuous uplift weighting (RUC-SDPO).

Together, these comparisons indicate that the same privileged teacher construction harms learning when applied uniformly, and improves learning once gated or weighted by positive uplift.

Q2: Ablation on Privileged Input Components

To isolate which component of the privileged input drives the improvement, we conduct a controlled ablation on Qwen3-8B for mathematical reasoning (Table F). By default, the teacher receives a non-self sibling response together with its verifier correctness label. In the Sibling-only variant, we keep the same sibling response and replace the correctness heading with the neutral heading "Another attempt." The training data, number of steps, sibling-selection rule, and hyperparameters stay the same, including λSD=0.01\lambda_\mathrm{SD}=0.01, and the uplift weights are computed with this modified input, while GRPO serves as the baseline without a privileged teacher.

We do not test a label-only variant because the student rollout's own correctness label is already used by GRPO, whereas the label in the teacher prompt refers to the sibling attempt; providing that label without the sibling response would state an outcome without showing the attempt itself. Table F below reports MEAN@8 at the final checkpoint, where ± values denote standard deviations across three evaluation seeds.

Table F. Ablation on privileged input components on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds).

Privileged inputMATH-500AIME 2026HMMT Feb. 2026
GRPO (no privileged input)85.94±0.2945.69±1.2528.28±0.79
Sibling-only86.20±0.3046.39±0.6427.90±0.58
Sibling response and verifier label87.03±0.1648.33±0.8330.68±0.76

Sibling-only remains close to GRPO on all three benchmarks, whereas pairing the sibling response with its verifier label raises performance clearly above both Sibling-only and GRPO.

AuthorsOpenReview ↗

Response to Reviewer GS69 (Part 3)

Q3: Sensitivity to Calibration Rollouts KK

To evaluate sensitivity to the number of additional privileged rollouts KK, we compare K=2K=2 against our default K=1K=1 run from Table 1 on Qwen3-8B for mathematical reasoning (Table G). All other settings, including the training data, number of optimization steps, prompt-level aggregation, G=8G=8, and λSD=0.01\lambda_\mathrm{SD}=0.01, remain unchanged.

For a fixed policy and fixed contexts, averaging K=2K=2 independent calibration rollouts lowers per-context sampling variance compared with K=1K=1; however, because prompt-level pooling already averages across up to G=8G=8 valid privileged contexts per prompt, K=1K=1 already pools up to 8 calibration rollouts per prompt (versus 16 at K=2K=2). Table G below reports MEAN@8 at the final checkpoint, where ± values denote standard deviations across three evaluation seeds.

Table G. Sensitivity of RUC-SDPO to the number of additional privileged rollouts KK on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds).

KKMATH-500AIME 2026HMMT Feb. 2026
187.03±0.1648.33±0.8330.68±0.76
287.04±0.1048.61±0.2430.81±0.58

The results hardly improve when increasing KK from 1 to 2; however, the training time rises from 15.45 hours to 17.33 hours, or from 371 to 416 seconds per step, an increase of 1.88 hours (12%). The added time is due to the extra calibration rollout. Because prompt-level pooling already averages up to 8 calibration rollouts per prompt at K=1K=1, doubling KK adds training time without improving downstream accuracy beyond the evaluation standard deviations. We therefore keep K=1K=1 as the default.

Reviewer_eRsQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an additive combination of GRPO and SDPO, where the SDPO part is weighted by its estimated reward improvement from conditioning on privileged information. It achieves consistent but modest gains over GRPO, with roughly 27% additional training cost.

Critical strengths

S1: The method uses verifier feedback to selectively weight self-distillation, directly testing whether privileged context improves task performance.

S2: Evaluation covers three model sizes and two task families. The ablations separate the effects of gating and continuous weighting.

Critical weaknesses

The reported error bars reflect evaluation seeds rather than independent training runs, leaving the robustness of the modest gains over GRPO unclear. The gains over GRPO are consistent but modest, and the reported error bars don't reflect the variability across independent training runs.

RUC-SDPO adds approximately 27% additional training time, but the paper does not establish if the gains persist gainst GRPO under matched compute budget.

The contribution appears incremental relative to prior work combining GRPO with gated self-distillation, particularly EDGE’s reward-improvement gating.

Questions for authors

Do the gains over GRPO persist across independent training seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer eRsQ

Thank you for your thoughtful review and for recognizing our evaluation across model scales and task families as well as our ablations on gating and continuous weighting. Below, we address each weakness and question in detail with a new independent training run, a matched-budget comparison, and a discussion of EDGE.

W1: Robustness Across Training Seeds

We train GRPO and RUC-SDPO once more on Qwen3-8B for mathematical reasoning, using the same data, schedule, and method-specific hyperparameters. Each final checkpoint is evaluated with three seeds. Table C reports MEAN@8. Training seed 42 is the original run in Table 1. Training seed 43 is the additional run.

Table C. Independent training runs on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds per run).

MethodTraining seedTraining stepsMATH-500AIME 2026HMMT Feb. 2026Avg.
GRPO4215085.94±0.2945.69±1.2528.28±0.7953.30
GRPO4315085.98±0.1345.97±0.6428.16±0.5853.37
RUC-SDPO4215087.03±0.1648.33±0.8330.68±0.7655.35
RUC-SDPO4315087.04±0.1548.19±0.6431.06±0.6655.43

RUC-SDPO outperforms GRPO on every benchmark in both independent training runs. The unweighted average MEAN@8 scores are 55.35 and 55.43 for RUC-SDPO versus 53.30 and 53.37 for GRPO, corresponding to consistent gains of +2.05 and +2.06 percentage points across training seeds.

W2: Matched Compute-Budget Comparison

On Qwen3-8B for mathematical reasoning, we compare RUC-SDPO with GRPO under one wall-clock budget of 12.2 hours that GRPO takes to complete 150 training steps. Within this budget, RUC-SDPO completes 120 training steps, and we evaluate this checkpoint. Table D below reports MEAN@8, where ± values denote standard deviations across three evaluation seeds.

Table D. Controlled comparison on Qwen3-8B for mathematical reasoning under a matched wall-clock budget (MEAN@8 ± std over 3 evaluation seeds).

MethodTraining stepsMATH-500AIME 2026HMMT Feb. 2026
GRPO15085.94±0.2945.69±1.2528.28±0.79
RUC-SDPO12086.73±0.5047.78±1.9229.42±0.58

RUC-SDPO at step 120 outperforms GRPO at step 150 on all three benchmarks, confirming that the gains over GRPO persist under a matched compute budget.

W3: Positioning Relative to EDGE

EDGE is a closely related contemporaneous work whose arXiv version appeared approximately one month before the ICLR submission deadline. We missed this paper at the time of submission. We have included it in the related-work discussion and made the relationship between the two methods explicit.

Both methods use reward improvement to regulate privileged distillation. RUC-SDPO uses normalized, continuous reward-uplift weights to calibrate the auxiliary distillation strength while retaining GRPO on the original unprivileged rollout group. EDGE uses a binary gain gate and also incorporates admitted privileged rollouts into the RL update. Our contribution focuses on this continuous calibration design within the reward-anchored framework.

Table 3 provides an internal comparison under the same GRPO anchor and JSD objective. On Qwen3-8B, continuous calibration achieves 55.3 in unweighted average MEAN@8 across MATH-500, AIME 2026, and HMMT February 2026, compared with 54.2 for binary gating (the same type of 0/1 gain gate used in EDGE) and 48.8 for uncalibrated distillation. This ablation supports the calibration design in the reported setting and complements the methodological comparison with EDGE.

Q: Persistence Across Training Seeds

Please refer to the reply to W1 (Table C). Under different training seeds, GRPO obtains unweighted average MEAN@8 scores of 53.30 and 53.37, and RUC-SDPO obtains 55.35 and 55.43. RUC-SDPO consistently outperforms GRPO by 2.05 and 2.06 percentage points under both training seeds.

Thank you for the response and for providing additional experiments. My concerns about robustness and matched compute are now largely addressed. My reservation about the significance of the contribution remains. Even though the reported gains are consistent, I find them modest and the methodological advance incremental. Thus, I will keep my rating at 2.

I want to note that this is a borderline judgement, and I would not oppose acceptance if another reviewer see the contributions as sufficiently valuable.

Thank you for carefully reviewing our rebuttal and confirming that the concerns regarding training-seed robustness and matched-compute effectiveness are largely addressed. We appreciate your candid assessment.

Reviewer_8MkkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RUC-SDPO, which balances the verifier's reward signals and the self-distilled teacher's signals based on the teacher's credibility. RUC-SDPO addresses the concern that the self-distilled teacher may have incorrect reward signals in its log probabilities by adopting the verifier's original reward signal when the teacher is less credible. Across the model sizes in the Qwen3 family, RUC-SDPO demonstrates superior performance over the plain SDPO and GRPO across multiple math reasoning tasks.

Critical strengths

Strength

  1. The motivation for the proposed method is straightforward. RUC-SDPO leverages the plain verifier signal as a weighted fallback when the conditioned teacher is not credible as guidance.

  2. RUC-SDPO outperforms GRPO and SDPO in the majority of cases, with statistically significant gaps across model sizes from 4B to 14B.

  3. Clear ablation studies shown in Tables 2 and 3 explain the necessity of the methodological design of RUC-SDPO.

Critical weaknesses
  1. Missing baselines around difficulty-aware self-distillation methods: The main experiments compare RUC-SDPO against GRPO and SDPO, although several recent studies address difficulty awareness and selective distillation at the GRPO and SDPO levels. While reward-uplift calibration has a clear motivation, the experiments should establish its advantages over simpler approaches based on signals such as pass rate, especially given the non-trivial increase in training time. Closely related methods such as SC-SDPO [1], SRPO [2], and HDPO [3] are relevant comparison baselines, provided that differences in privileged information are controlled or explicitly discussed.

  2. Additional ablations on λSD\lambda_{SD}: The authors fix λSD\lambda_{SD} to 0.01 throughout the paper, including the ablation studies in Tables 2 and 3. Since uu is often below 1, the contribution of LRUC−SDL_{RUC-SD} may be small; using the average u=0.272u=0.272 in Table 4 gives an average loss coefficient of approximately 0.00272, although this alone does not determine its gradient contribution. Since the paper’s novelty largely lies in reward-uplift calibration of privileged self-distillation, sensitivity to λSD\lambda_{SD}, including larger values, should be examined.

References

[1] Liu et al., 2026, "Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning." (ArXiv)

[2] Li et al., 2026, "Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing." (COLM 2026)

[3] Ding, Ken, 2026, "HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation." (ArXiv)

Questions for authors
  1. How is RUC-SDPO better compared to difficulty-aware self-distillation methods like SC-SDPO, HDPO, and SRPO?

  2. Given the additional 27% in per-step training time, how does the accuracy gain at the same training time look with RUC-SDPO compared to the baselines? Is it more effective with the limited training time?

  3. In Table 9, thinking mode was disabled during the evaluation. Is the thinking mode enabled during training?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8Mkk (Part 1)

Thank you for your constructive feedback and for recognizing our motivation, the consistent improvements across model scales, and the clarity of our ablation studies. Below, we address each weakness and question in detail, providing new controlled baseline comparisons, λSD\lambda_\mathrm{SD} sensitivity results, and clarification on the training setup.

W1: Controlled Baseline Comparisons

To compare against SC-SDPO and SRPO under identical privileged information, we conduct a controlled experiment on Qwen3-8B for mathematical reasoning (trained on OpenR1-Math) that isolates the weighting signal each method uses.

We use one wall-clock budget of 12.2 hours that GRPO takes to complete 150 training steps, and evaluate the checkpoint saved inside that budget. The SC-SDPO control replaces RUC-SDPO's reward-uplift weight with SC-SDPO's pass-rate-based weighting. The SRPO control replaces it with SRPO's teacher-entropy-based weighting. Within this budget, RUC-SDPO completes 120 training steps, while both the SC-SDPO control and the SRPO control complete 130 training steps. Table A below reports MEAN@8 on three mathematical reasoning benchmarks. The ± values are standard deviations across three evaluation seeds.

Table A. Controlled comparison on Qwen3-8B for mathematical reasoning under a matched wall-clock budget (MEAN@8 ± std over 3 evaluation seeds).

MethodTraining stepsMATH-500AIME 2026HMMT Feb. 2026
GRPO15085.94±0.2945.69±1.2528.28±0.79
RUC-SDPO12086.73±0.5047.78±1.9229.42±0.58
SC-SDPO control13086.40±0.4045.56±1.5829.29±1.53
SRPO control13085.20±0.7044.44±1.4626.26±1.53

Within this budget, RUC-SDPO outperforms GRPO on all three benchmarks. It also outperforms the SRPO control on all three benchmarks, and outperforms the SC-SDPO control on MATH-500 and AIME 2026 while matching it on HMMT February 2026.

We do not add an HDPO run for the following reason. HDPO gives the teacher the reference solution and distills only when all student rollouts fail, while RUC-SDPO gives the teacher a sibling attempt and its verifier label, not the reference solution. The difference between HDPO and RUC-SDPO could therefore come either from the weighting rule or from the teacher's access to the reference solution.

For the weighting rule itself, HDPO applies full distillation weight via a binary gate. It is similar to the GRPO+Binary Gate method in Table 3. We already show that continuous calibration outperforms binary gating on Qwen3-8B across the three mathematical reasoning benchmarks in MEAN@8 (MATH-500: 87.0 vs. 86.4; AIME 2026: 48.3 vs. 46.8; HMMT February 2026: 30.7 vs. 29.4).

W2: Sensitivity to λSD\lambda_\mathrm{SD}

Even with a small average coefficient, the per-token JSD is nonnegative at every token, whereas the GRPO advantage vanishes on all-correct and all-incorrect groups, so the auxiliary term still provides a substantive gradient signal. To directly test sensitivity to larger values, we keep λSD=0.01\lambda_\mathrm{SD}=0.01 in the main experiments and additionally evaluate λSD=0.03\lambda_\mathrm{SD}=0.03 on Qwen3-8B for mathematical reasoning with all remaining settings unchanged, reporting MEAN@8 at the final checkpoint.

Table B. Sensitivity of RUC-SDPO to λSD\lambda_{\mathrm{SD}} on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds).

λSD\lambda_{\mathrm{SD}}MATH-500AIME 2026HMMT Feb. 2026
0.0187.03±0.1648.33±0.8330.68±0.76
0.0387.17±0.2449.31±0.8731.82±0.38

Raising λSD\lambda_{\mathrm{SD}} from 0.01 to 0.03 increases MEAN@8 by +0.14, +0.98, and +1.14 points on MATH-500, AIME 2026, and HMMT Feb. 2026, respectively, confirming that the auxiliary term contributes positively at these scales.

AuthorsOpenReview ↗

Response to Reviewer 8Mkk (Part 2)

Q1: Advantages Over Related Methods

The key advantage of RUC-SDPO over SC-SDPO, HDPO, and SRPO is that its distillation weight is computed directly from the estimated verifier-reward uplift of the privileged teacher view over the student view, rather than from a single-view heuristic:

  • SC-SDPO weights distillation using the student pass rate, which reflects prompt difficulty under the student view alone.
  • HDPO applies a binary gate that distills only when all student rollouts fail, and supplies the teacher with the reference solution.
  • SRPO routes each sample exclusively to either GRPO or distillation and weights distilled tokens by teacher entropy, which reflects the teacher's output confidence alone.

None of these methods measures whether conditioning on the privileged context actually improves task reward over the student view. In contrast, RUC-SDPO retains GRPO on every sample and derives the auxiliary distillation weight directly from the estimated reward uplift between the teacher and student views.

Empirically, this advantage is supported by our controlled comparisons against the SC-SDPO and SRPO controls in our response to W1 (Table A) and against binary gating in Table 3.

Q2: Matched-Time Training Effectiveness

Table A in our response to W1 reports the matched 12.2-hour wall-clock comparison on Qwen3-8B for mathematical reasoning. Within this identical time budget, RUC-SDPO completes 120 training steps, compared with 150 for GRPO and 130 for the SC-SDPO and SRPO controls. Even with fewer steps, RUC-SDPO still outperforms GRPO and the SRPO control on all three benchmarks, and outperforms the SC-SDPO control on MATH-500 and AIME 2026 while matching it on HMMT February 2026.

Q3: Thinking Mode During Training

Thinking mode was disabled during training for RUC-SDPO, GRPO, and SDPO. All three methods use a chat template with enable_thinking set to false, so rollouts are generated without a thinking trace. We have explicitly added this training setting to Table 7 (Appendix C) of the revision.

Dear Reviewer 8Mkk,

As the discussion period is nearing its close, we wanted to gently follow up on our responses above, where we provided the matched-time controlled comparisons against SC-SDPO and SRPO (Table A), the λSD=0.03\lambda_{\mathrm{SD}}=0.03 sensitivity ablation (Table B), and the clarification on thinking mode during training in Table 7 of the revision. Could you kindly let us know whether our responses have addressed your concerns, or if there are any remaining questions we can clarify? We would be glad to respond promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.