- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes rubric-guided self-distillation (RSD) to improve model training using rubrics. The method alternates reinforcement learning and SFT. To prepare for SFT, it selects weak queries and adds rubric guidance to the prompts, allowing the same model to generate demonstrations. High-quality demonstrations are selected based on rubric scores, and the rubric hints are removed from the prompts before SFT. This cycle is repeated to improve the model’s performance. Experiments on Qwen3-8B, 14B, and 32B demonstrate performance gains from this training approach.
Critical strengths
- Although generating a set of task-specific rubrics may require the assistance of an additional, stronger model, these rubrics are reusable, and low-scoring trajectories can be filtered out based on them.
- Rubric hints are used when generating demonstrations, while the original, unhinted prompts are restored during SFT, and complete tool trajectories are learned. This training pipeline is well-designed and aligns with the ultimate goal of deployment without hints.
- The experiments cover multiple model sizes and benchmarks.
Critical weaknesses
- The proposed method combines ideas such as rubric guidance, self-generated demonstrations by the model, filtering high-quality trajectories, removing hints for SFT, and alternating SFT with RL training. Since related work already exists for these basic ideas, the novelty is somewhat incremental. Moreover, the experiments show that SFT-only already captures a large portion of the gains; the clearest contribution is that rubric-guided demonstration learning is effective, while the additional contributions of some other design choices remain unclear.
- The proposed method relies heavily on the content of the rubrics. In this paper, the rubrics simultaneously control three things: the RL reward, guidance when generating demonstrations, and the filtering of demonstrations. Consequently, the final trained model is highly dependent on how the rubrics are designed.
- The comparison with baseline methods does not adequately control for the intensity of supervised training. RSD trains for up to eight epochs on the demonstrations, whereas the APTER and ReLIFT adaptations use only one epoch, and the SFT learning rates also differ. The paper does not sufficiently justify these differences in settings, making it difficult to determine whether the performance gains come from the RSD training framework or simply from more extensive supervised training.
Questions for authors
- Why does RSD train for up to eight epochs on the demonstrations, while the two baseline adaptations (APTER and ReLIFT) use only one epoch, and the SFT learning rates also differ? The paper only explains the motivation for dedicated SFT phases, but I believe this is insufficient to attribute the performance advantage to the RSD training framework rather than to more extensive supervised training.
- Given that SFT-only already achieves a large portion of the gains, compared with merely learning from demonstrations, how much additional gain do the designs in RSD—such as failure-guided query selection and alternating SFT with RL training—each contribute?
- The random query selection variant is evaluated at round 19, while RSD-8B is evaluated at round 56. Why are different rounds used for comparison?
4 comments
The new controls align the amount and source of supervision while preserving each method's consolidation rule. Both APTER-SD and ReLIFT-SD complete 6,105,768 supervised assistant-token exposures, eight SFT epochs, and 228 RL updates. ReLIFT-SD uses 13 SFT phases and 176 SFT updates; APTER-SD uses 144 SFT updates. The reported RSD reference is 5.82 RQA points and 4.87 DRB-II points above APTER-SD at this matched exposure.
The shared SFT configuration specifies effective batch size 16, Adam , peak learning rate , cosine scheduling, and a configured minimum learning rate of . Schedule lengths and realized learning-rate trajectories remain method-specific.
| Method | RQA | DRB | DRB-II | LSB | GPQA |
|---|---|---|---|---|---|
| APTER-SD | 53.53 | 26.01 | 12.58 | 27.27 | 31.25 |
| ReLIFT-SD | 22.42 | 7.15 | 1.59 | 22.52 | 14.25 |
| RSD-8B (ours) | 59.35 | 29.56 | 17.45 | 28.79 | 34.25 |
The supervision audit shows why token exposure and corrective content are distinct. ReLIFT-SD learns 405 demonstrations covering 80 of the 920 training queries. Under the saved judge scores, 280 score no higher than the best preceding student answer, including 160 that score lower. These demonstrations account for 73.2% of the supervised-token exposure. The selection rule asks whether any student attempt satisfies every criterion, while admission requires a new mean rubric score of at least 0.8. An accepted demonstration can therefore score below an answer already available from the student.
This audit concerns the rubric-judged self-distillation adaptation of ReLIFT, whose original formulation verifies external demonstrations. It identifies the quality of the admitted targets as a concrete issue in this setting.
RSD supplies the complete rubric and additionally emphasizes criteria the current student group satisfies less often. Scores from unhinted rollouts determine this extra emphasis; all task criteria remain in the guidance. The current policy retries selected weak queries with that context, and high-reward trajectories accumulate for periodic, multi-epoch SFT. The added guidance is removed from the training input, and assistant actions across the tool interaction are learned under the original query before unhinted RL resumes. The methodological distinction concerns how the full rubric and targeted emphasis shape self-generated supervision; the matched-exposure controls compare the complete training procedures, including their collection and consolidation rules.
If these clarifications and additional results resolve your concerns, we would be grateful if you would consider raising your score.
Thank you for the responses and experiments, which address my concern about differences in the intensity of supervised training and additional gains from alternating SFT with RL training. I would like to raise my rating.
Thank you for your careful review and positive assessment. We are glad that our responses and additional experiments addressed your concerns.
Thank you for asking us to separate the effects of supervision strength, selection, and scheduling. The completed baseline controls and our clarified interpretation of the ablations now provide a more precise account of what the results establish.
Self-distillation under a matched supervision budget
We make the APTER/ReLIFT comparison consistent with RSD by using self-distillation throughout. Demonstrations are generated by each method's current student policy; GLM-5.3-Flash is used only for independent judging. Both adaptations use eight SFT epochs and a matched 6,105,768 supervised assistant-token exposure budget.
The SFT optimizer settings are also aligned: effective batch size 16, Adam β2=0.98, peak learning rate 10−5, and cosine decay with a 10−6 floor. The timing of consolidation follows each method: RSD consolidates after zero-indexed RL rounds 19 and 39, APTER triggers repair from accumulated criterion failures, and ReLIFT consolidates its 32-demonstration buffer. Matching supervised-token exposure preserves these method-specific triggers rather than forcing identical phase boundaries.
APTER-SD has completed 6,105,768 actual, checkpoint-committed supervised assistant-token exposures:
Notes: SD denotes current-student self-distillation. APTER-SD uses the cleaned 920-query training set, 57 RL rounds, and expanded candidate collection to reach the matched supervised-token exposure. RSD is the reported paper reference; superscripts give RSD minus APTER-SD. All benchmark inputs remain in the denominator. LSB includes one external-search service failure counted as zero; the other four benchmark checks found no infrastructure failures.
The reported RSD reference retains 5.82 RQA points and 4.87 DRB-II points of advantage under this matched exposure. The result supports the importance of demonstration construction alongside supervision quantity: APTER retains its category-level repair trigger, while RSD uses query-specific rollout feedback to guide research and consolidate accepted trajectories. We will add the ReLIFT self-distillation results within 24 hours.
Selection strategy
Random follows the paper's core RSD training hyperparameters and uses uniform sampling from the pending weak-query queue.
Source and use of the training rubrics
We reuse the training queries and GPT-5-generated rubrics from Deep Research as Rubric for Reinforcement Learning. These rubrics are supplied with the dataset and remain fixed throughout our training. The baseline adaptations reuse the existing rubric text for the retained queries.
Rubric content defines the training objective. Its three uses in RSD target the same task requirements: scoring attempts, guiding a new attempt toward missed requirements, and filtering trajectories for SFT. Holding the existing rubric text fixed allows us to compare how the methods use that information. Rubric-based GRPO uses it for rewards; RSD also uses criterion-level feedback to guide the current student in constructing trajectories that are learned under the original query, with the hints removed. On the external benchmarks, the judge uses each benchmark's own evaluation criteria, and the policy receives no rubric hints.