- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RUC-SDPO, which combines GRPO with privileged on-policy self-distillation calibrated by estimated verifier-reward uplift. The method estimates the benefit of privileged conditioning using additional rollouts and uses positive uplift to continuously weight an auxiliary token-level JSD objective. Experiments across Qwen3 models on mathematical and tool-call tasks demonstrate consistent improvements over GRPO baseline.
Critical strengths
- The proposed weighting mechanism provides a task-level criterion for regulating privileged self-distillation, rather than relying solely on correctness or teacher confidence.
- The proposed RUC-SDPO shows improvement over GRPO across mathematical and tool-call benchmarks.
Critical weaknesses
- Eqs. (3–5) compare original-policy rewards against privileged EMA-policy rewards in the actual implementation, potentially introducing bias in reward-uplift estimation. Independent finite-sample reward estimates can also produce positive uplift even when privileged conditioning provides no genuine improvement.
- SRPO and HDPO already combine group-relative RL with selective privileged self-distillation. Although RUC-SDPO introduces a distinct reward-uplift weighting mechanism, neither method is discussed or evaluated.
- Although Eqs.6–7 motivate context-specific reward-uplift weighting, Appendix D aggregates rewards at the prompt level and assigns the same weight to all privileged contexts. Consequently, harmful contexts may still receive positive distillation weights when other contexts provide sufficient uplift.
Questions for authors
- Is there evidence that positive uplift corresponds to more informative teacher distributions along these student trajectories?
- How does performance change when using only the correctness label, only the sibling response, or both? This would clarify which information is responsible for the improvements.
- How sensitive is the method to the number of additional privileged rollouts (K)? Does increasing (K) improve the reliability of uplift estimation and downstream performance?
7 comments
Response to Reviewer GS69 (Part 2)
W3: Prompt-Level Aggregation vs. Selectivity
As noted in Section 3.2 and Appendix D.2, we average the privileged rollout rewards across all valid contexts of the same prompt so the shared weight estimates the prompt's average privileged utility. With binary rewards and , a per-context estimate has high sampling variance and collapses the weight in Eqs. (6)–(7) to a binary gate. Therefore, we adopt prompt-level aggregation to reduce this variance and produce continuous weights.
We agree that this design trades context-specific selectivity for a lower-variance estimate: an individually harmful context can receive a positive weight if other contexts for the same prompt yield strong uplift. Even in that case, the harmful context's lower privileged reward pulls down the shared prompt weight, and the GRPO anchor remains active on every trajectory. In the revision, we have explicitly labeled this shared weight after Eqs. (6)–(7) in Section 3.2 as prompt-level to match Appendix D.2.
Q1: Evidence for Positive-Uplift Guidance
Tables 2 and 3 provide empirical evidence that positive uplift identifies useful teacher guidance. Across both ablations, we keep the privileged-context construction rule and the JSD objective fixed while varying only whether and how positive uplift weights the distillation loss:
- Without the GRPO anchor (Table 2), weighting the JSD along student trajectories by positive uplift (RUC-SD) raises unweighted average MEAN@8 from 45.3 (SDPO) to 52.3.
- With the GRPO anchor fixed (Table 3), unweighted average MEAN@8 is 53.3 for GRPO, drops to 48.8 when distilling from the privileged teacher without uplift calibration (GRPO+SDPO), and improves to 54.2 with binary uplift gating (GRPO+Binary Gate) and 55.3 with continuous uplift weighting (RUC-SDPO).
Together, these comparisons indicate that the same privileged teacher construction harms learning when applied uniformly, and improves learning once gated or weighted by positive uplift.
Q2: Ablation on Privileged Input Components
To isolate which component of the privileged input drives the improvement, we conduct a controlled ablation on Qwen3-8B for mathematical reasoning (Table F). By default, the teacher receives a non-self sibling response together with its verifier correctness label. In the Sibling-only variant, we keep the same sibling response and replace the correctness heading with the neutral heading "Another attempt." The training data, number of steps, sibling-selection rule, and hyperparameters stay the same, including , and the uplift weights are computed with this modified input, while GRPO serves as the baseline without a privileged teacher.
We do not test a label-only variant because the student rollout's own correctness label is already used by GRPO, whereas the label in the teacher prompt refers to the sibling attempt; providing that label without the sibling response would state an outcome without showing the attempt itself. Table F below reports MEAN@8 at the final checkpoint, where ± values denote standard deviations across three evaluation seeds.
Table F. Ablation on privileged input components on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds).
| Privileged input | MATH-500 | AIME 2026 | HMMT Feb. 2026 |
|---|---|---|---|
| GRPO (no privileged input) | 85.94±0.29 | 45.69±1.25 | 28.28±0.79 |
| Sibling-only | 86.20±0.30 | 46.39±0.64 | 27.90±0.58 |
| Sibling response and verifier label | 87.03±0.16 | 48.33±0.83 | 30.68±0.76 |
Sibling-only remains close to GRPO on all three benchmarks, whereas pairing the sibling response with its verifier label raises performance clearly above both Sibling-only and GRPO.
Response to Reviewer GS69 (Part 3)
Q3: Sensitivity to Calibration Rollouts
To evaluate sensitivity to the number of additional privileged rollouts , we compare against our default run from Table 1 on Qwen3-8B for mathematical reasoning (Table G). All other settings, including the training data, number of optimization steps, prompt-level aggregation, , and , remain unchanged.
For a fixed policy and fixed contexts, averaging independent calibration rollouts lowers per-context sampling variance compared with ; however, because prompt-level pooling already averages across up to valid privileged contexts per prompt, already pools up to 8 calibration rollouts per prompt (versus 16 at ). Table G below reports MEAN@8 at the final checkpoint, where ± values denote standard deviations across three evaluation seeds.
Table G. Sensitivity of RUC-SDPO to the number of additional privileged rollouts on Qwen3-8B for mathematical reasoning (MEAN@8 ± std over 3 evaluation seeds).
| MATH-500 | AIME 2026 | HMMT Feb. 2026 | |
|---|---|---|---|
| 1 | 87.03±0.16 | 48.33±0.83 | 30.68±0.76 |
| 2 | 87.04±0.10 | 48.61±0.24 | 30.81±0.58 |
The results hardly improve when increasing from 1 to 2; however, the training time rises from 15.45 hours to 17.33 hours, or from 371 to 416 seconds per step, an increase of 1.88 hours (12%). The added time is due to the extra calibration rollout. Because prompt-level pooling already averages up to 8 calibration rollouts per prompt at , doubling adds training time without improving downstream accuracy beyond the evaluation standard deviations. We therefore keep as the default.
Response to Reviewer GS69 (Part 1)
Thank you for your constructive feedback and for recognizing our task-level weighting criterion for regulating privileged self-distillation and the consistent improvements over GRPO across mathematical and tool-call benchmarks. Below, we address each weakness and question in detail, providing a new controlled comparison against SRPO, an ablation on privileged input components, a sensitivity study on K, and clarifications on EMA calibration and prompt-level aggregation.
W1: EMA Calibration and Finite-Sample Noise
Our EMA teacher follows the teacher-regularization design in SDPO (Hübotter et al., 2026), which introduces EMA for training stability and supports this choice through its teacher-update ablation. Related evidence from SDFT (Shenfeld et al., 2026) shows that using the current student directly as the teacher amplifies noisy updates, whereas an EMA teacher remains stable across steps while still keeping pace with the student's improvement.
In RUC-SDPO, we use an EMA update rate of 0.05 (Table 8). Calibration rollouts use the same EMA policy to estimate the uplift of the teacher that actually provides the distillation targets. The estimate is therefore defined for this EMA teacher relative to the current student. We have updated Section 3.2 (after Eq. (5)) of the revision to explicitly state that, following SDPO and SDFT, the teacher view uses the EMA policy for stability and the calibration rollouts use the same EMA policy to estimate the uplift of that teacher.
For finite-sample calibration, we pool rewards across valid privileged targets of each prompt and convert the estimated gap into a bounded, detached weight. Positive estimates can still arise from sampling noise, but one that is small relative to the remaining reward margin receives a proportionally small weight, rather than the full weight that binary gating would assign (Table 3). We have clarified in Section 3.2 that the weight represents estimated prompt-level utility.
W2: Comparison with SRPO and HDPO
In Section 2 (Related Work) of the submission, SRPO is already cited and discussed as Li et al. (2026) on sample routing, and HDPO as Ding (2026) on distillation for all-failed prompts. In the revision, we have updated Section 2 to explicitly name both methods and further clarify how they differ from RUC-SDPO.
Methodologically, RUC-SDPO differs from SRPO and HDPO in both how the distillation weight is determined and what privileged information is used:
Neither method measures whether conditioning on the privileged context actually improves task reward over the student view. In contrast, RUC-SDPO retains GRPO on every original student rollout, uses a sibling response and its verifier label rather than the reference solution as privileged input, and derives the auxiliary distillation weight directly from the estimated verifier-reward uplift of the privileged teacher view over the student view.
To evaluate the teacher-entropy weighting rule under a controlled setup, we compare RUC-SDPO against an SRPO control on Qwen3-8B for mathematical reasoning. We replace RUC-SDPO's reward-uplift weight with SRPO's teacher-entropy-based weighting. Under the same 12.2-hour wall-clock budget, RUC-SDPO completes 120 training steps and the SRPO control completes 130 training steps. Table E below reports MEAN@8, where ± values denote standard deviations across three evaluation seeds.
Table E. Controlled comparison with teacher-entropy weighting (SRPO control) on Qwen3-8B for mathematical reasoning under a matched wall-clock budget (MEAN@8 ± std over 3 evaluation seeds).
Within this budget, RUC-SDPO at step 120 outperforms the SRPO control at step 130 on all three benchmarks.
We do not add an end-to-end HDPO run because HDPO gives the teacher the reference solution, whereas RUC-SDPO gives the teacher a sibling attempt and its verifier label; any difference between HDPO and RUC-SDPO could therefore come either from the weighting rule or from the teacher's access to the reference solution. For the weighting rule itself, HDPO applies full distillation weight via a binary gate, which is similar to the
GRPO+Binary Gatemethod in Table 3. We already show that continuous calibration outperforms binary gating on Qwen3-8B across the three mathematical reasoning benchmarks in MEAN@8 (MATH-500: 87.0 vs. 86.4; AIME 2026: 48.3 vs. 46.8; HMMT February 2026: 30.7 vs. 29.4).