- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper propose ORSD(Online Rubric Self-Distillation), a self-distillation method using rubrics for token level supervision. The method uses the policy, which is being optimized, to generate bundles of rubrics, and a verifier to evaluate the trajectories on each rubric. Next, the rubric bundle that has highest spearman correlation with golden reward is selected, and used as additional infomation in on policy self distillation.
Critical strengths
The paper proposed a interesting method that uses rubrics to generate token level supervision for GRPO.
Critical weaknesses
Major Weakness
W1: Unclear motivation of policy-generated rubric
Why should the policy, which is use to generate trajectories, be used to generate rubrics as well? Normally, a good trajectory-generator is not necessarily a good rubircs generator. For example, a coding LLM is good at generating code, but may not be good at generating natural language rubrics. How was the quality of rubrics generate by policy? Will the policy generate better rubric with the process of training?
The experiment with a fixed rubric generator(for example, a capable LLM) / human rubric [2] should be added as well.
W2: Unclear about motivation for using rubrics.
The rubrics is originally used to replace golden answer needed in GRPO/RLVR [1], in senario which golden answer is hard to obtain. But in this paper, rubrics are used as context to generate better answers. What if the rubrics are used as actual reward? Can this method be used when golden answer is missing?
W3: Lack overhead analysis.
During training, ORSD require generating rubrics, verifying rubrics, calculating correlation and so on. What is the overhead cost of these steps(over different rubrics bundle number m)? Does is significantly slow down training?
W4: Lack ablation study.
- Bundle number m: only m = 1(w/o Rubric–Reward Alignment) and 4 are presented. What about other m? What is the cost of different m(in W3)?
- How to decide the number of rubrics inside one bundle? Does this affect the result? What happens if only some rubrics in a boundle are accurate, while others are wrong?
Minor Weakness:
- Presentation is really hard to follow. Background is not properly introduced, like Rubrics, GRPO and self-distillation/OPSD, which makes it hard for people to understand.
- Section 3.1 "problem setup" do not set up the problem in the paper.
- Figure 1 is crowded and not infomative. Robots here are just visual distraction.
- Figure 2 also not infomative, the result can be shown in a small table or a paragraph. Consider adding more data.
- Experiment only done in Qwen model family. What about other model families?
[1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
[2] HealthBench: Evaluating Large Language Models Towards Improved Human Health
Questions for authors
See weakness
6 comments
Response to Reviewer bmB3 (2/3)
[W2] Why use rubrics as context rather than rewards, and can ORSD work without gold answers?
(1) In ORSD, rubric context is used to obtain token-level log-probability differences on trajectories that have already been sampled. We score the same tokens with and without the selected rubric, then use the differences to refine their GRPO advantages. No new answer is generated during this replay. Using rubric satisfaction directly as a reward would instead change the training signal to criterion compliance, whereas the role we study is to derive token-specific guidance while retaining outcome-based supervision.
(2) As for whether ORSD can work without gold answers or equivalent outcome supervision, the current method requires reliable correctness feedback. This can come from a gold answer or, in ALFWorld and WebShop, the environment. Rubric scoring measures whether a trajectory satisfies a criterion. An unmet criterion can still be informative when its violation consistently accompanies task failure. Without external outcome feedback, however, rewarding rubric satisfaction alone could encourage the policy to satisfy its own criteria without achieving the actual task objective.
(3) Our rubric–reward alignment mechanism therefore selects bundles whose satisfaction scores agree with observed outcomes. The gold answer is not supplied as privileged context, and outcome rewards are excluded from the rubric-generation prompt. Instead, the selected rubric conveys process guidance derived from the trajectories, while outcome feedback anchors its selection to task success. We clarify this supervision requirement in Appendix C.4.
[W3] What is the overhead of rubric generation, verification, and selection, and does it significantly slow down training?
(1) We measured the additional components and complete training runtime for Qwen3-4B on WebShop. Table R3 reports both candidate counts. Generation, verification, and privileged replay account for almost all of the additional component time, while correlation computation and selection take negligible time in these runs. Appendix C.5 provides the cost and workload records.
Table R3. Measured component costs and runtime over 300 training steps on four GPUs. Both runs collect 19,200 environment trajectories. Training runtime excludes validation and checkpoint saving. At M=4, additional components account for 17.54% of measured training time.
| Component | M=4 | M=6 |
|---|---|---|
| Policy-generated rubrics | 10.24 min | 15.53 min |
| Verifier waiting time | 23.17 min | 20.04 min |
| Spearman computation and selection | 0.09 s | 0.16 s |
| Rubric parsing and preparation | 0.56 min | 0.88 min |
| Privileged log-probability replay | 36.95 min | 49.59 min |
| Total additional component time | 70.92 min | 86.04 min |
| Average per training step | 14.18 s | 17.21 s |
| Complete training runtime | 6.74 h | 16.92 h |
| Environment rollout time | 2.74 h | 11.49 h |
(2) The complete runtime difference cannot be attributed solely to candidate count. As Table R3 shows, most of the increase occurs during environment rollout, alongside longer responses and more groups triggering rubric generation. Appendix C.5 reports these workload differences separately from the component costs. Both runs use the same environment-interaction budget. The additional rubric operations occur only during training, while inference uses the trained policy directly.
[W4.1] How do other candidate counts affect performance and cost?
We added a larger candidate pool and a random-selection control to distinguish the value of more candidates from that of reward-aligned selection. Table R4 shows that alignment-based selection clearly improves performance at a fixed candidate count. Expanding the aligned candidate pool produces only a small success gain and does not improve task score. These comparisons appear in Table 2 and Figure 2(a) of the revised paper. Table R4 also includes rubric pruning, discussed under W4.2.
The alignment curves in Figure 5, Appendix C.6, are consistent with this result. Table R5 summarizes the small change in selected-bundle alignment alongside the increase in measured component time. The component breakdown is already provided in Table R3. Together, these observations support the default candidate count within the tested range.
Table R4. Candidate-count, selection, and rubric-pruning ablations with Qwen3-4B on WebShop.
| Setting | Score | Success (%) |
|---|---|---|
| Random selection, M=4 | 84.94 | 69.90 |
| Alignment selection, M=4, full bundle | 89.02 | 77.05 |
| Alignment selection, M=6 | 88.67 | 77.25 |
| Alignment selection, M=4, two highest-alignment items | 88.04 | 76.25 |
Table R5. Alignment and marginal cost when increasing the candidate count.
| Comparison | Value |
|---|---|
| Selected-bundle correlation, M=4 | 0.7884 |
| Selected-bundle correlation, M=6 | 0.7964 |
| Increase in additional component time, M=4 to M=6 | 21.32% |
Response to Reviewer bmB3 (3/3)
[W4.2] How is the number of criteria per bundle chosen, and what happens when some criteria are less accurate?
(1) Our generation prompt requests 5–8 criteria per bundle. To test sensitivity to lower-alignment items, we prune the selected bundle to its two highest-alignment criteria at M=4. Each criterion is ranked by the Spearman correlation between its verifier scores across trajectories and the corresponding outcome rewards. The results appear in Figure 2(a) and Section 4.3 of the revised paper, with the construction described in Appendix C.4.
(2) Table R4 shows that removing lower-alignment criteria does not improve performance over the complete selected bundle. Together with the random-selection comparison, this supports the effectiveness of reward-aligned bundle selection when individual criteria differ in alignment. The selection mechanism evaluates the bundle as a whole, and these results support retaining its combined process guidance rather than requiring every criterion to achieve high individual alignment.
[Minor 1] The background is hard to follow, and Section 3.1 does not establish the problem setup.
(1) We have revised the Introduction to introduce rubrics and their use in prior work, including Rubrics as Rewards [1] and HealthBench [2], and to explain how rubric-based supervision relates to the credit-assignment problem.
(2) Section 3.1 is now titled “Preliminaries” and explains GRPO and OPSD before introducing ORSD. The transition to Section 3.2 makes their connection explicit: GRPO provides a shared trajectory-level advantage, while privileged context offers a way to introduce token-level supervision. ORSD constructs that context from the policy’s rollout experience.
[Minor 2] Figure 1 is crowded, and Figure 2 provides too little information.
(1) We have revised Figure 1 by reducing the robot icons, enlarging the text, and improving the spacing and arrow routing to make the training procedure easier to follow.
(2) We have consolidated the token-credit, candidate-count, rubric-pruning, generator/verifier, and alpha-sensitivity comparisons into the three-panel Figure 2. This presents the original sensitivity analysis alongside the additional experiments in a single figure, with numerical results labeled directly.
[Minor 3] What about model families beyond Qwen?
We have extended the evaluation to Llama3.2-3B-Instruct on both benchmarks, comparing ORSD with GRPO, GRPO+OPSD, and standalone OPSD. Table R6 shows that ORSD outperforms all three baselines on both environments. These results provide evidence beyond the Qwen family and are included in Table 1 of the revised paper, with evaluation settings in Appendix C.4.
Table R6. Llama3.2-3B-Instruct success (%) on WebShop and ALFWorld at step 300 under mean@4.
| Method | WebShop | ALFWorld |
|---|---|---|
| GRPO | 76.05 | 87.68 |
| GRPO+OPSD | 67.10 | 91.07 |
| OPSD | 2.57 | 5.71 |
| ORSD | 79.08 | 93.21 |
References
[1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
[2] HealthBench: Evaluating Large Language Models Towards Improved Human Health
[3] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. 2026. “Generative Reward Model” paragraph under Section 5.1.1 (“Specialist Training”). Paper.
Response to Reviewer bmB3 (1/3)
Thank you for the constructive feedback. We have added comparisons with an external rubric generator and alternative verifier, candidate-count and rubric-pruning ablations, training-cost measurements, and experiments with Llama3.2-3B-Instruct. These results support the effectiveness of policy-generated guidance and clarify the trade-off between performance and additional computation. We have also revised the background and method presentation to explain the role of rubrics more clearly.
[W1.1] Why use the trajectory policy to generate rubrics, and how does it compare with a fixed external generator?
(1) The motivation for using the current policy is to derive guidance from the behavior it is currently learning to improve. A related precedent is DeepSeek-V4 [3], which uses the actor itself as a generative reward model and jointly optimizes its generation and judgment capabilities. The DeepSeek-V4 report explains that this shared-model design brings the model’s reasoning capabilities into its evaluation process, supporting the possibility of using the same model for generation and evaluation. In ORSD, the policy compares trajectories for the same task and proposes criteria that capture their behavioral differences. The verifier and reward-alignment selection then assess which candidate bundle best distinguishes the observed outcomes. This motivates using the policy to propose guidance while testing its usefulness through the comparisons below.
(2) Table R1 separates the effects of rubric generation and verification. With the verifier held fixed, current-policy generation outperforms DeepSeek generation under both verifiers. This supports using the policy’s own rollout experience to construct useful rubric context.
With the generator held fixed, DeepSeek verification improves downstream success for both generators. Thus, policy-generated guidance remains effective across verifiers, while verifier choice provides an additional source of improvement. The best combination uses current-policy generation with DeepSeek verification. These comparisons appear in Figure 2(b), with experimental settings in Appendix C.4.
Table R1. WebShop success (%) with Qwen3-4B under different rubric generators and verifiers.
[W1.2] How does the quality of policy-generated rubrics change during training?
We compared Qwen3-4B checkpoints on three fixed WebShop tasks with identical stored trajectories, M=4, and DeepSeek verification. Table R2 shows rubric excerpts for the same footwear task, illustrating changes in attribute checks, concrete actions, and error avoidance. Both selection and held-out correlations peak at step 100 and vary across checkpoints, reinforcing the need to assess candidate alignment at each update rather than assuming that later rubrics are better. Appendix C.7 provides the evaluation protocol and results for all seven checkpoints.
Table R2. Verbatim excerpts from selected rubrics for the same WebShop task. Two criteria are shown per checkpoint, with original item numbers retained. The task requires men's slip-resistant loafers in color black2626, size 14, priced below $70.