- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why scores can rise while response quality declines during reinforcement learning. Awarding points for each criterion separately may lead the model to focus on easy criteria while overlooking whether its response is appropriate. To address this, ProRubric divides the scoring criteria into groups. A response receives credit for a group only when it meets all criteria in that group and does not trigger the group's failure clause. The paper compares how different scoring methods affect response quality.
Critical strengths
- The paper includes controlled experiments that keep the scoring criteria unchanged and vary only the scoring method. This helps explain how the scoring method affects response quality.
- The experiments test Qwen3-4B and Qwen3-8B and report results from multiple training runs. The paper also uses different judge models for evaluation.
- In controlled experiments on science tasks, ProRubric performs better than simple scoring methods such as Implicit.
Critical weaknesses
- The theoretical analysis does not yet explain which groupings are better for improving response quality.
Under simplifying assumptions, the authors show that requiring all conditions in a group to be met before awarding credit can reduce cases where a model earns a high score by satisfying only easy criteria. However, further analysis is needed to explain how to tell whether one grouping is better than another.
- ProRubric performs much worse than supervised fine-tuning (SFT) on the appropriateness of medical responses.
On HealthBench-consensus, Qwen3-4B scores 36.8 with ProRubric and 66.3 with SFT. The corresponding scores for Qwen3-8B are 45.8 and 75.6. In addition, ProRubric's HealthBench scores in Table 2 are slightly lower than those of SFT. The paper also does not measure the cost of using the judge model during training.
- ProRubric shows no clear advantage over the simple scoring method Implicit on the appropriateness of medical responses.
In Table 3, the 4B model scores 36.7 with Implicit and 36.8 with ProRubric, which are almost the same. For the 8B model, Implicit has a higher reported average score of 47.6, compared with 45.8 for ProRubric.
- The main experiments do not provide a complete comparison between ProRubric and the simple scoring method Implicit.
Table 2 does not provide complete results for Implicit on the seven benchmarks.
Questions for authors
-
Can the theoretical analysis explain how to tell whether one grouping is better than another for improving response quality?
-
Why does ProRubric score much lower than SFT on the appropriateness of medical responses? What practical advantages does ProRubric offer for these tasks?
-
How do you explain ProRubric's similar or lower scores compared with Implicit on the appropriateness of medical responses?
-
How does Implicit perform on the seven benchmarks in Table 2?
4 comments
Response to Reviewer 2vjB (Part 2/2: Q3/W3 & Q4/W4)
[Q3 & W3] How to explain ProRubric vs. Implicit on medical consensus?
Implicit is also non-additive, so a tie on medical consensus fits our mechanism, while ProRubric leads on probes, HealthBench-full, and MedQA. Table R9 reports the complete 8B medical aggregation controls across both appropriateness (c) and coverage (g).
Table R9: 8B medical aggregation controls (3 seeds ± sd; cf. revised Table 17)
| 8B Medical Arm (DeepSeek-V4-Pro) | Appropriateness (c) ± sd | Coverage (g) ± sd |
|---|---|---|
| Base (untrained) | 62.5 | 31.6 |
| Rubric-RL (additive, ) | 29.5 ± 1.1 | 36.5 ± 0.6 |
| raw-AND (verbatim grouping, ) | 40.9 ± 3.0 | 39.5 ± 1.0 |
| Implicit (holistic score) | 47.6 ± 1.0 | 39.4 ± 0.5 |
| ProRubric (protocol conjunction, ) | 45.8 ± 1.8 | 40.1 ± 1.1 |
At 8B, all three non-additive methods substantially outperform additive Rubric-RL on both axes (+11.4 to +18.1 on ; +2.9 to +3.6 on ). While Implicit is slightly higher on 8B consensus appropriateness (and tied at 4B in Table R5), ProRubric achieves the highest 8B coverage and outperforms Implicit in three ways:
- Dedicated 4-domain appropriateness probe (Table 14): ProRubric beats Implicit across medicine (+2.0*), science (+5.7*), and dialogue (+1.6, per-seed +1.1/+1.3/+2.4; Table R5).
- Reward discriminability and parse reliability: On medical prompts, Implicit gives Base responses a generous baseline score (62.5/100) and achieves lower substance-vs.-padding discriminability than ProRubric (36.5% vs. 51.4%), while suffering more training parse-failure dead groups (1.00–1.46% vs. 0.000–0.016%) and returning an opaque scalar instead of per-dimension verdicts.
- Clinical MCQ accuracy and 7-benchmark average: ProRubric leads Implicit on MedQA at both scales (65.1 vs. 64.1 at 4B, winning on all 3 seeds, and 69.9 vs. 69.1 at 8B) and achieves the highest 7-benchmark average (see Q4 below).
Revision: Added the complete 8B aggregation controls ( and ) in Section 5.3 and Appendix D.2.
[Q4 & W4] How does Implicit perform across all seven benchmarks in Table 2?
Completing all 8B Implicit runs shows that ProRubric achieves the highest 7-benchmark average across all methods at both 4B (46.9) and 8B (52.2). Table R10 compares ProRubric against Base and all baselines across the seven benchmarks at both scales (highest score per scale and column in bold).
Table R10: Seven-benchmark comparison across all methods (3-seed means; cf. revised Tables 2 and 8)
| Method | WB | Cr-v3 | AH | HB-full | MedQA | GPQA | ResQA | Avg. |
|---|---|---|---|---|---|---|---|---|
| 4B Base | 49.2 | 27.9 | 12.9 | 26.1 | 65.2 | 41.1 | 51.8 | 39.2 |
| 4B + SFT | 63.9 | 35.5 | 21.6 | 35.1 | 59.5 | 36.0 | 63.2 | 45.0 |
| 4B + OPSD | 49.7 | 27.7 | 9.2 | 25.1 | 60.8 | 36.4 | 47.5 | 36.6 |
| 4B + Rubric-RL | 64.6 | 35.0 | 17.3 | 31.9 | 62.6 | 44.7 | 62.9 | 45.6 |
| 4B + Implicit | 66.7 | 37.9 | 19.6 | 32.5 | 64.1 | 42.8 | 61.2 | 46.4 |
| 4B + RuscaRL | 63.7 | 34.0 | 18.2 | 31.9 | 62.0 | 45.6 | 61.3 | 45.2 |
| 4B + ProRubric | 67.1 | 37.7 | 18.6 | 34.7 | 65.1 | 43.9 | 61.1 | 46.9 |
| 8B Base | 53.9 | 36.0 | 19.1 | 31.4 | 69.0 | 42.9 | 55.9 | 44.0 |
| 8B + SFT | 68.2 | 43.8 | 30.6 | 40.4 | 67.8 | 31.0 | 67.4 | 49.9 |
| 8B + OPSD | 48.1 | 23.2 | 9.7 | 27.0 | 67.7 | 40.2 | 47.2 | 37.6 |
| 8B + Rubric-RL | 68.1 | 38.9 | 24.8 | 36.4 | 68.9 | 49.8 | 65.8 | 50.4 |
| 8B + Implicit | 68.4 | 42.1 | 25.7 | 37.2 | 69.1 | 50.5 | 65.1 | 51.2 |
| 8B + RuscaRL | 68.0 | 38.6 | 24.0 | 36.5 | 69.7 | 45.5 | 66.2 | 49.8 |
| 8B + ProRubric | 69.1 | 43.1 | 28.6 | 39.9 | 69.9 | 50.0 | 64.9 | 52.2 |
As Table R10 shows, ProRubric achieves the highest overall average across all methods at both 4B and 8B. Compared with the runner-up Implicit, ProRubric leads on HealthBench-full (+2.2 at 4B, +2.7 at 8B), WritingBench, and MedQA at both scales, as well as 8B Creative-v3 (+1.0) and 8B Arena-Hard (+2.9), while the remaining minor differences ( point) fall within cross-seed noise.
Revision: Added the complete 4B and 8B Implicit rows directly below Rubric-RL in Section 5.2 and per-seed breakdowns in Appendix C.1.
The additional SFT-initialized experiments have addressed my concerns regarding SFT. I have raised my rating from 2 to 3 (weak acceptance).
Dear Reviewer 2vjB,
We sincerely appreciate your feedback and your decision to raise the score! We are delighted that the new SFT-initialized experiments resolved your concerns. We will integrate these experiments and insights into the final version of our paper. Thank you again for your valuable time and constructive suggestions!
Best regards,
The Authors
Response to Reviewer 2vjB (Part 1/2: Q1/W1 & Q2/W2)
We thank Reviewer 2vjB for recognizing the practical relevance of the reward-hacking failure mode, our controlled science experiments where ProRubric clearly outperforms Implicit, and our systematic analysis. Below we provide point-by-point responses to your questions (all results are 3-seed means unless noted).
Eq. 3 yields a two-step pre-training test on Base rollouts that separates ungrouped, coarse, verbatim, and protocol groupings before RL. Under Eq. 3, the marginal reward gradient of item i in group Ik is
∂E[R~]/∂pi=∑mw~mw~k∏j∈Ik,j=ipj,
where pj is the Base pass rate. Before RL, we check two training-free diagnostics on Base rollouts (Table R7):
Table R7: Pre-RL diagnostics vs. post-RL performance (cf. revised Tables 1b, 17, and 21)
Diagnostic (i) separates K≈30 and K=1 from K≈4.6, while Diagnostic (ii) separates ProRubric from raw-AND at K≈4.6, matching post-RL gains across 4B, 8B (Table R7), and science (Table R5).
Revision: Added the two-step pre-training grouping test in Section 4.2 and Appendix D.6.
SFT and ProRubric play complementary roles and stack positively, while ProRubric uses fewer total judge tokens than Rubric-RL.
Table R8: Initializing RL from SFT (4B medicine, 3 seeds; cf. revised Table 19)
Revision: Added SFT + ProRubric in Appendix D.2 and same-unit judge-token accounting in Appendix A.4 (cf. revised Table 4).