acceptodds
Under review as a conference paper at ICLR 2027

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

14 views

Abstract

Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one another: a policy that misses the one decision that matters can buy the points back with advice nobody asked for. On clinical consultation, such a policy scores higher and answers worse. Rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model. The medical criteria are not to blame. Grouped so that they must hold together, the same criteria, unchanged to the word, recover a third of the loss; shorter answers recover almost none. We therefore propose **Pro**tocol-level **Rubric**s (**ProRubric**), which keeps what the criteria ask for and changes how they are aggregated. It groups a checklist into a few protocol-level dimensions. A dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is done once, offline, and leaves the optimizer unchanged. ProRubric raises appropriateness by 10.8 points without losing coverage and has the best seven-benchmark average at both scales. Reward validity is set not only by what a rubric verifies, but by how it aggregates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rubric-based reinforcement learning, reward design, reward hacking, LLM-as-a-judge, post-training, alignment, RLHF

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2vjBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why scores can rise while response quality declines during reinforcement learning. Awarding points for each criterion separately may lead the model to focus on easy criteria while overlooking whether its response is appropriate. To address this, ProRubric divides the scoring criteria into groups. A response receives credit for a group only when it meets all criteria in that group and does not trigger the group's failure clause. The paper compares how different scoring methods affect response quality.

Critical strengths
  1. The paper includes controlled experiments that keep the scoring criteria unchanged and vary only the scoring method. This helps explain how the scoring method affects response quality.
  2. The experiments test Qwen3-4B and Qwen3-8B and report results from multiple training runs. The paper also uses different judge models for evaluation.
  3. In controlled experiments on science tasks, ProRubric performs better than simple scoring methods such as Implicit.
Critical weaknesses
  1. The theoretical analysis does not yet explain which groupings are better for improving response quality.

Under simplifying assumptions, the authors show that requiring all conditions in a group to be met before awarding credit can reduce cases where a model earns a high score by satisfying only easy criteria. However, further analysis is needed to explain how to tell whether one grouping is better than another.

  1. ProRubric performs much worse than supervised fine-tuning (SFT) on the appropriateness of medical responses.

On HealthBench-consensus, Qwen3-4B scores 36.8 with ProRubric and 66.3 with SFT. The corresponding scores for Qwen3-8B are 45.8 and 75.6. In addition, ProRubric's HealthBench scores in Table 2 are slightly lower than those of SFT. The paper also does not measure the cost of using the judge model during training.

  1. ProRubric shows no clear advantage over the simple scoring method Implicit on the appropriateness of medical responses.

In Table 3, the 4B model scores 36.7 with Implicit and 36.8 with ProRubric, which are almost the same. For the 8B model, Implicit has a higher reported average score of 47.6, compared with 45.8 for ProRubric.

  1. The main experiments do not provide a complete comparison between ProRubric and the simple scoring method Implicit.

Table 2 does not provide complete results for Implicit on the seven benchmarks.

Questions for authors
  1. Can the theoretical analysis explain how to tell whether one grouping is better than another for improving response quality?

  2. Why does ProRubric score much lower than SFT on the appropriateness of medical responses? What practical advantages does ProRubric offer for these tasks?

  3. How do you explain ProRubric's similar or lower scores compared with Implicit on the appropriateness of medical responses?

  4. How does Implicit perform on the seven benchmarks in Table 2?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2vjB (Part 1/2: Q1/W1 & Q2/W2)

We thank Reviewer 2vjB for recognizing the practical relevance of the reward-hacking failure mode, our controlled science experiments where ProRubric clearly outperforms Implicit, and our systematic analysis. Below we provide point-by-point responses to your questions (all results are 3-seed means unless noted).

[Q1 & W1] How can one tell before RL whether one grouping is better than another?

Eq. 3 yields a two-step pre-training test on Base rollouts that separates ungrouped, coarse, verbatim, and protocol groupings before RL. Under Eq. 3, the marginal reward gradient of item ii in group Ik\mathcal I_k is

∂E[R~]/∂pi=w~k∑mw~m∏j∈Ik,j≠ipj,\partial \mathbb{E}[\tilde R]/\partial p_i = \frac{\tilde w_k}{\sum_m \tilde w_m} \prod_{j \in \mathcal I_k, j \ne i} p_j,

where pjp_j is the Base pass rate. Before RL, we check two training-free diagnostics on Base rollouts (Table R7):

  1. Diagnostic (i) — Eliminating item-naming payout (ΔRname\Delta R_{\text{name}}) while bounding GRPO ties (selecting KK): Under additive scoring (K≈30K \approx 30), easy mention items (pi≈1p_i \approx 1) always pay full weight, rewarding empty item-naming (+7.3*). Grouping criteria into K≈4.6K \approx 4.6 blocks so high-pip_i items share a group with harder low-pjp_j items scales their payout by ∏j∈Ik∖{i}pj\prod_{j \in \mathcal{I}_k \setminus \{i\}} p_j, collapsing item-naming payout to noise in raw-AND and ProRubric while keeping GRPO ties moderate at 25–37% (vs. 61–63% dead-group ties and 33.5 coverage at K=1K = 1).
  2. Diagnostic (ii) — Penalizing needless tests (ΔRtest\Delta R_{\text{test}}) and discriminability at K≈4.6K \approx 4.6: Between raw-AND and ProRubric at K≈4.6K \approx 4.6, synthesizing each group with a failure clause actively penalizes needless tests (−5.8* vs. −1.4 n.s.) and doubles reward discriminability (51.4% vs. 22.9%).

Table R7: Pre-RL diagnostics vs. post-RL performance (cf. revised Tables 1b, 17, and 21)

Grouping DesignTie RateName / Test4B Med. c / g8B Med. c / g
Rubric-RL (K≈30K \approx 30)0–2%+7.3* / +0.326.0 / 32.129.5 / 36.5
K=1K = 1 (single group)61–63%(all-or-nothing)37.0 / 33.5(4B ablation)
raw-AND (K≈4.6K \approx 4.6)25–37%−0.3 / −1.434.9 / 33.840.9 / 39.5
ProRubric (K≈4.6K \approx 4.6)25–37%+2.2 / −5.8*36.8 / 35.045.8 / 40.1

Diagnostic (i) separates K≈30K \approx 30 and K=1K = 1 from K≈4.6K \approx 4.6, while Diagnostic (ii) separates ProRubric from raw-AND at K≈4.6K \approx 4.6, matching post-RL gains across 4B, 8B (Table R7), and science (Table R5).

Revision: Added the two-step pre-training grouping test in Section 4.2 and Appendix D.6.

[Q2 & W2] Why does SFT score higher on consensus, and what is the judge cost?

SFT and ProRubric play complementary roles and stack positively, while ProRubric uses fewer total judge tokens than Rubric-RL.

  1. SFT on consensus and SFT + ProRubric stacking: Released SFT targets (26,194 rows) are distilled from frontier external teachers (62% GPT-5.1, 38% Gemini-3-Pro, best-of-6), importing a concise teacher style onto consensus prompts (66.3 vs. 36.8 at 4B; 75.6 vs. 45.8 at 8B), whereas on-policy ProRubric stays within 0.5 points of SFT on HealthBench-full (34.7 vs. 35.1 at 4B; 39.9 vs. 40.4 at 8B). Moreover, off-policy SFT drops below Base on 4B/8B MedQA (59.5 / 67.8 vs. 65.2 / 69.0) and GPQA-Diamond (36.0 / 31.0 vs. 41.1 / 42.9), where ProRubric matches or beats Base (65.1 / 69.9 and 43.9 / 50.0; Table R10). Initializing RL from SFT (SFT + ProRubric; Table R8) combines both—lifting coverage by +8.1 and appropriateness by +1.6—while SFT + Rubric-RL degrades appropriateness (−1.9).

Table R8: Initializing RL from SFT (4B medicine, 3 seeds; cf. revised Table 19)

4B Medical Arm (DeepSeek-V4-Pro)HealthBench-full (g)Appropriateness (c)Change vs. SFT (g / c)
SFT (teacher distillation)35.166.3(SFT baseline)
SFT + Rubric-RL (additive)40.7 ± 1.364.4+5.6 / −1.9
SFT + ProRubric (protocol)43.3 ± 0.167.9+8.1 / +1.6 (+3.5 vs. additive)
  1. Lower judge-token budget: Every RL method makes 153,600 online judge calls per run (64 prompts × 8 rollouts × 300 steps). Compressing ∼30 items into 4.6 dimensions shrinks the rubric prompt by 38.0% in medicine and 35.7% in dialogue, cutting input tokens per medical call by 12.9% (3,315 to 2,888 tokens, saving ∼65.6M input tokens per run, or ∼197M across 3 seeds) alongside shorter rollouts. One-time offline synthesis takes 12,515 calls (∼28M total tokens, 8.1% of one run's calls)—under half the 65.6M input tokens saved in a single run.

Revision: Added SFT + ProRubric in Appendix D.2 and same-unit judge-token accounting in Appendix A.4 (cf. revised Table 4).

AuthorsOpenReview ↗

Response to Reviewer 2vjB (Part 2/2: Q3/W3 & Q4/W4)

[Q3 & W3] How to explain ProRubric vs. Implicit on medical consensus?

Implicit is also non-additive, so a tie on medical consensus fits our mechanism, while ProRubric leads on probes, HealthBench-full, and MedQA. Table R9 reports the complete 8B medical aggregation controls across both appropriateness (c) and coverage (g).

Table R9: 8B medical aggregation controls (3 seeds ± sd; cf. revised Table 17)

8B Medical Arm (DeepSeek-V4-Pro)Appropriateness (c) ± sdCoverage (g) ± sd
Base (untrained)62.531.6
Rubric-RL (additive, K≈30K \approx 30)29.5 ± 1.136.5 ± 0.6
raw-AND (verbatim grouping, K≈4.6K \approx 4.6)40.9 ± 3.039.5 ± 1.0
Implicit (holistic score)47.6 ± 1.039.4 ± 0.5
ProRubric (protocol conjunction, K≈4.6K \approx 4.6)45.8 ± 1.840.1 ± 1.1

At 8B, all three non-additive methods substantially outperform additive Rubric-RL on both axes (+11.4 to +18.1 on cc; +2.9 to +3.6 on gg). While Implicit is slightly higher on 8B consensus appropriateness (and tied at 4B in Table R5), ProRubric achieves the highest 8B coverage and outperforms Implicit in three ways:

  1. Dedicated 4-domain appropriateness probe (Table 14): ProRubric beats Implicit across medicine (+2.0*), science (+5.7*), and dialogue (+1.6, per-seed +1.1/+1.3/+2.4; Table R5).
  2. Reward discriminability and parse reliability: On medical prompts, Implicit gives Base responses a generous baseline score (62.5/100) and achieves lower substance-vs.-padding discriminability than ProRubric (36.5% vs. 51.4%), while suffering more training parse-failure dead groups (1.00–1.46% vs. 0.000–0.016%) and returning an opaque scalar instead of per-dimension verdicts.
  3. Clinical MCQ accuracy and 7-benchmark average: ProRubric leads Implicit on MedQA at both scales (65.1 vs. 64.1 at 4B, winning on all 3 seeds, and 69.9 vs. 69.1 at 8B) and achieves the highest 7-benchmark average (see Q4 below).

Revision: Added the complete 8B aggregation controls (cc and gg) in Section 5.3 and Appendix D.2.

[Q4 & W4] How does Implicit perform across all seven benchmarks in Table 2?

Completing all 8B Implicit runs shows that ProRubric achieves the highest 7-benchmark average across all methods at both 4B (46.9) and 8B (52.2). Table R10 compares ProRubric against Base and all baselines across the seven benchmarks at both scales (highest score per scale and column in bold).

Table R10: Seven-benchmark comparison across all methods (3-seed means; cf. revised Tables 2 and 8)

MethodWBCr-v3AHHB-fullMedQAGPQAResQAAvg.
4B Base49.227.912.926.165.241.151.839.2
4B + SFT63.935.521.635.159.536.063.245.0
4B + OPSD49.727.79.225.160.836.447.536.6
4B + Rubric-RL64.635.017.331.962.644.762.945.6
4B + Implicit66.737.919.632.564.142.861.246.4
4B + RuscaRL63.734.018.231.962.045.661.345.2
4B + ProRubric67.137.718.634.765.143.961.146.9
8B Base53.936.019.131.469.042.955.944.0
8B + SFT68.243.830.640.467.831.067.449.9
8B + OPSD48.123.29.727.067.740.247.237.6
8B + Rubric-RL68.138.924.836.468.949.865.850.4
8B + Implicit68.442.125.737.269.150.565.151.2
8B + RuscaRL68.038.624.036.569.745.566.249.8
8B + ProRubric69.143.128.639.969.950.064.952.2

As Table R10 shows, ProRubric achieves the highest overall average across all methods at both 4B and 8B. Compared with the runner-up Implicit, ProRubric leads on HealthBench-full (+2.2 at 4B, +2.7 at 8B), WritingBench, and MedQA at both scales, as well as 8B Creative-v3 (+1.0) and 8B Arena-Hard (+2.9), while the remaining minor differences (≤1.0\le 1.0 point) fall within cross-seed noise.

Revision: Added the complete 4B and 8B Implicit rows directly below Rubric-RL in Section 5.2 and per-seed breakdowns in Appendix C.1.

The additional SFT-initialized experiments have addressed my concerns regarding SFT. I have raised my rating from 2 to 3 (weak acceptance).

Dear Reviewer 2vjB,

We sincerely appreciate your feedback and your decision to raise the score! We are delighted that the new SFT-initialized experiments resolved your concerns. We will integrate these experiments and insights into the final version of our paper. Thank you again for your valuable time and constructive suggestions!

Best regards,

The Authors

Reviewer_jokbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a divergence between rubric coverage and answer appropriateness during reinforcement learning. It argues that independently rewarded checklist items can compensate for a missed important decision. ProRubric groups criteria into protocol-level dimensions and adds failure conditions. The experiments compare alternative aggregations and investigate several explanations for the observed degradation. Appropriateness is assessed by LLM judges using held-out criteria, rather than by direct clinical evaluation.

Critical strengths
  1. The paper does more than document a reward-hacking example. The matched-wording controls intervene on how the criteria are scored, and the length, weighting, and regularization comparisons test plausible alternatives. Taken together, these experiments provide a reasonably convincing case that the scored unit matters. I found the medical evidence stronger than the broader cross-domain interpretation, but it is a worthwhile result.

  2. The evaluation does not present the proposed method as a complete solution. It reports the remaining gap to the base model on appropriateness, competitive simpler aggregation rules, and losses on tasks that reward exhaustive coverage. Multiple judges and physician-label calibration add useful checks. These limitations make the paper more informative about when the proposed change is likely to help.

Critical weaknesses
  1. The specific benefit of protocol synthesis is harder to establish than the benefit of changing aggregation. Implicit aggregation is already competitive in the medical experiments. Where synthesis helps, it also changes the wording and introduces explicit failure conditions, so the intervention is no longer purely an aggregation change. The assignment checks in Appendix A.3 do not verify that every requirement survives semantically. The incentive argument also simplifies aspects of the executed reward. I would narrow the mechanism claim accordingly: the controls support a role for aggregation, but do not establish that this particular synthesized, conjunctive recipe is necessary.

  2. Some quality conclusions remain dependent on the evaluation procedure. The principal evaluator also generates the proposed rubrics. The alternate-generator and cross-family checks are helpful, but substantial disagreement remains in some domains. The physician meta-evaluation assesses the judge on an existing corpus, rather than adjudicating the new policies' answers. In addition, common-item evaluation removes prompts with failed judgments. That maintains pairing but does not rule out selection related to length or difficulty. These concerns qualify the breadth of the quality claim; they are less damaging to the more consistent medical comparison.

Questions for authors
  1. Was semantic preservation checked beyond assigning each original criterion to a group? Please report any audit and distinguish inherited requirements from newly introduced failure conditions.
  2. Can the comparison with implicit aggregation, raw grouping, and one-to-one rewriting be summarized using matched per-seed results? Where does synthesis provide a consistent additional benefit?
  3. What are the per-method judgment-failure rates, and do the main conclusions persist on less restrictive pairwise common sets? Please report existing sensitivity checks if available.
  4. How should the simplified incentive argument be related to clipping, penalty criteria, and overlong reward shaping in the actual training objective?
  5. Which checkpoints were used for the main comparisons, and what validation signals selected them? Was the held-out appropriateness evaluation involved in selection?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jokb (Part 1/2: Q1/W1 & Q2)

We thank Reviewer jokb for recognizing the clarity of our problem formulation, our matched-wording and confounder controls, and our multi-judge evaluation across model scales. Below we provide point-by-point responses to your questions (all results are 3-seed means unless noted).

[Q1 & W1a] Was semantic preservation checked beyond group assignment?

Yes. An independent GPT-5.6 audit finds 97.8% of medical criteria fully or partially preserved (54.1% fully), and 93.0% of failure conditions inherited. We audited 100 medical prompts (1,663 failure conditions across 464 dimensions) and 100 science prompts (1,118 failure conditions across 380 dimensions; Table R4).

Table R4: Semantic preservation audit (GPT-5.6, 200 prompts; cf. revised Table 22a)

Audit MetricMedicineScience
Atomic criteria fully / partially preserved54.1% / 43.7%88.5% / 11.3%
Atomic criteria omitted2.2%0.1%
Failure conditions inherited93.0% (1,544 / 1,663)88.0% (984 / 1,118)
New — integrative / scope & anti-padding55% / 22% of new47% / 10% of new
New — domain content (% of all)1.7%5.2%

In medicine, partial preservation reflects abstracting ~30 granular items (illustrative examples, 11.3% duplicates, 7.2% format items) into 4.6 phase-level blocks, with only 2.2% omitted (mostly duplicate/format items). Crucially, safety and scope constraints already exist in the original checklists as compensable items; ProRubric converts them into non-compensable vetoes while adding synthesis and scope guardrails in the remaining 7–12% of clauses. Acting as conjunctive vetoes, the failure clauses as a whole add +2.3 over No-Veto (lost under additive Flat-Veto; Table R2), and retraining science after deleting all newly introduced domain conditions (No-New-Content) retains 76% of ProRubric's gain (+5.71 ± 3.53 over Rubric-RL; per-seed +2.5 / +5.6 / +9.0, positive on all three seeds; Table R1).

Revision: Added the GPT-5.6 semantic audit and science No-New-Content retraining (cf. revised Table 22b) in Appendices D.6–D.7.

[W1b] How do conjunctive grouping and protocol synthesis each contribute to ProRubric?

Conjunctive grouping provides the primary foundation, while protocol synthesis and failure clauses deliver essential gains as scale and task complexity grow. First, under additive scoring, one-to-one rewriting in ProRubric's style (Rewritten) yields only +1.9 on the science probe (n.s. on all 3 seeds) vs. +7.5 for ProRubric, so wording alone is ineffective. Second, while raw-AND and No-Veto capture ~80% of the 4B medical gain, full ProRubric consistently beats them on harder settings: it beats verbatim raw-AND on 8B medicine (Table R9) and science (+7.5 vs. +2.6), achieves higher 4B coverage than global gating without needing a 5-class classifier (Table R3), doubles reward discriminability (51.4% vs. 22.9%), and cuts rubric prompt length by 38.0% (−12.9% judge input tokens; Appendix A.4).

Revision: Clarified the respective roles of conjunctive grouping and protocol synthesis in the Abstract and Sections 1, 4.1, and 5.3.

[Q2 & W1c] How does ProRubric compare per seed with Implicit, raw-AND, and rewriting?

Across matched seeds, ProRubric beats rewriting and raw-AND, and leads Implicit on the 4-domain probe, HealthBench-full, and MedQA. Table R5 reports per-seed scores across s42 / s43 / s44 and 3-seed means for the four cross-domain methods.

Table R5: Per-seed comparison across s42/s43/s44 (cf. revised Tables 8, 14, 20, and 21)

ArmMed. (c)Med. ProbeSci. ProbeMedQA (4B)
Rubric-RL24.3/28.5/25.1 (26.0)18.9/23.5/21.0 (21.1)41.0/40.6/41.5 (41.0)63.2/61.2/63.5 (62.6)
raw-AND38.9/32.0/33.9 (34.9)29.943.663.3
Implicit36.8/36.7/36.8 (36.7)31.4/32.4/33.4 (32.4)44.2/42.8/41.5 (42.8)65.0/64.1/63.2 (64.1)
ProRubric39.8/36.2/34.5 (36.8)39.2/33.4/30.5 (34.4)51.0/45.6/49.0 (48.5)66.1/65.0/64.2 (65.1)

In domain-specific ablations, medical No-Veto scores 35.8/34.2/33.6 on cc (mean 34.5; Med. Probe 33.4), while science Rewritten scores 43.1/42.6/43.2 (mean 43.0, +2.1/+2.0/+1.7 over Rubric-RL, n.s. on all 3 seeds, vs. +10.0/+5.0/+7.5 for ProRubric). Against Implicit, ProRubric is tied on 4B medical appropriateness (Table R5; Implicit is slightly higher on 8B cc in Table R9), while leading on the science (+6.8*/+2.8/+7.5*, mean +5.7), medical (+2.0*), and dialogue (+1.1/+1.3/+2.4, mean +1.6) probes, on 4B/8B HealthBench-full (Table R10), across all 3 seeds on 4B MedQA (65.1 vs. 64.1), and on the 8B MedQA mean (69.9 vs. 69.1).

Revision: Added per-seed comparisons across benchmarks and probes in Appendices C.1 and D.4.

AuthorsOpenReview ↗

Response to Reviewer jokb (Part 2/2: Q3/W2, Q4 & Q5)

[Q3 & W2] What are the judgment failure rates, common-set shifts, and cross-judge checks?

Evaluation grading succeeds on >99.6% of prompts, unrestricted pools shift all Table 16 contrasts by ≤0.50\le 0.50 points, and three independent proxies confirm cross-judge consistency.

  1. Per-method failure rates and length/difficulty on excluded prompts: On HealthBench-full, per-seed grading coverage is 100.00% for Rubric-RL, 99.95–99.98% for Implicit, and 99.61–100.00% for ProRubric; in training, Implicit loses 1.00–1.46% of medical GRPO groups to parse failures vs. 0.000–0.016% for ProRubric. On HealthBench-consensus (3,671 prompts), primary arms grade 96.6–100.0% of prompts; the Table 3 common set (3,141 prompts, 85.6%) is bounded only by the subset evaluated for No-Veto s43/s44 (Table 6). Unrestricted pairwise pools (90.3–99.9%) shift all five Table 16 contrasts by ≤0.50\le 0.50 points (≤0.59\le 0.59 across all auxiliary checks) with unchanged significance. Although the 530 excluded prompts are slightly shorter (572 vs. 661 chars) and harder (Base 48.2 vs. 54.2), ProRubric still leads Rubric-RL on their 489 pairwise-graded prompts by +7.16 ([+5.32, +9.03], p<0.001p < 0.001).
  2. Generator swap and science judges: Replacing the rubric generator (DeepSeek-V4-Pro → Doubao-lite) shifts appropriateness by <2 points (38.1 vs. 39.8; Table 10), and generator-free raw-AND and Implicit show the same jump over Rubric-RL. In science (Table 1a), holistic DeepSeek-V4-Pro favors derivation completeness over brevity, whereas grading the 4 appropriateness criteria individually on the same answers (Table 14 / 15a) shows 0–1% non-redundancy pass and −11.2 for Rubric-RL—matching GPT-5.6-luna once completeness cannot offset redundancy.
  3. Physician calibration and multi-judge validation: HealthBench's meta-evaluation (29,511 response–criterion pairs with 60,896 physician labels; Appendix B.1, Table 5) calibrates DeepSeek-V4-Pro (balanced F1 0.62 vs. 0.65 inter-physician; Section 7 and Appendix B.1 note that new policy outputs are not physician-adjudicated). We validate policies across evaluators via 8,053 physician-written consensus criteria, GPT-5.6-luna re-grading (Table 11: Spearman ρ=0.84–0.91\rho = 0.84\text{--}0.91), and answer-keyed MedQA.

Revision: Stated the physician limitation in Section 7 and Appendix B.1; added full-pool sensitivity in Appendix B.2 (cf. revised Table 16) and science judge analysis in Appendix D.4.

[Q4 & W1d] How do overlong shaping, penalty criteria, and clipping affect Eq. 3?

All three settings apply identically across arms and preserve within-group reward ordering under Eq. 3.

  1. Overlong shaping: The length penalty is 0 up to 4,096 tokens, well above the mean response lengths of both Rubric-RL (~2,900 tokens) and ProRubric (~2,400 tokens), so any upper-tail shaping acts only against the longer additive baseline.
  2. Negative criteria: Negative-weight items are 0% of our 375,954 base training criteria and 0% of the 8,053 consensus criteria (appearing only in the auxiliary Weighted ablation), and Tables 1b and 13 evaluate the executed reward directly on responses.
  3. PPO clipping and GRPO normalization: PPO ratio clipping (0.20 / 0.28) bounds step size without changing any incentive's sign, and group normalization (r−μgroup)/(σgroup+ϵ)(r - \mu_{\text{group}}) / (\sigma_{\text{group}} + \epsilon) is a positive affine transform per group that preserves within-group reward ordering.

Revision: Added these three verifications in Appendix A.4 (cf. revised Table 4).

[Q5] Which checkpoints were evaluated, and was held-out appropriateness used for selection?

Every RL result is evaluated at the fixed final checkpoint (step 300) with zero validation selection. Held-out appropriateness was never used to select checkpoints or hyperparameters (RuscaRL and OPSD use their published step budgets; Appendix A.4). As Table R6 shows, ProRubric scores higher on appropriateness at earlier steps (e.g., steps 50 and 150)—had we selected on held-out appropriateness, we would have picked an earlier step rather than fixed step 300. From step 50 to 300, ProRubric leads Rubric-RL by +6.6 to +14.4 points.

Table R6: Trajectory across 50-step checkpoints (4B medicine, 3 seeds; cf. revised Table 18)

StepTrain Pass (Atom. / Prot., %)ProRubric (c)Rubric-RL (c)Diff.
047.2 / 33.454.154.10.0
5056.7 / 45.850.335.9+14.4
10066.1 / 57.541.034.4+6.6
15068.0 / 61.641.330.4+11.0
20070.3 / 64.540.127.8+12.3
25071.5 / 67.036.625.3+11.3
30073.2 / 69.536.826.0+10.8

Revision: Documented fixed-step evaluation in Appendix A.4 and added 50-step trajectories in Appendix D.2.

Reviewer_jokbOpenReview ↗

Official Comment by Reviewer jokb

Thanks to the author for the rebuttal. After reading the rebuttal carefully, I will maintain the score.

Reviewer_5xnjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper agues that one of the main sources of exploitation in Rubric-based RL in using aggregation of credited criteria as the reward function. The summation can dilute the effect of important criteria (final decision) by summing up other easily satisfied ones. Effectively, the policy can pad answers optimizing for recall instead of getting the key decision right. On medical dataset from RubricHub, rubric-rl improves helathbench rubric coverage scores while reducing appropriateness on a held-out physician-consensus criteria even lower than the untrained base policy. The proposed method ProRubric, regroups the original fine-grained checklist into protocol-phase dimensions that earn credit only and only if all their criteria are met. This raises appropriateness by 10.8 points compared rubric-rl without losing coverage on multiple domains at 4B and 8B scales.

Critical strengths
  1. This paper studies an important side-effect of rubric-based RL training where training with rubric-based judges results in policy that optimize for recall of criteria while failing at important aspects of a response quality. They demonstrate that untrained policy can score higher than a RL trained policy on important aspects like the decisions the patients need to make.
  2. They conduct a thorough analysis studying confounders like response length, KL, reweighting and sft on the same data making a strong case that aggregation/summing drives the collapse.
  3. They conduct experiments on 2 different policies and on 7 different domains demonstrating the effectiveness of the proposed method.
Critical weaknesses
  1. Training using the reward function from ProRubric method still reduces appropriateness compared to untrained model and GPT-5.6-luna prefers the untrained models overProRubric in 80% of decided medical pairs (Table 7). So I would not say that using ProRubric resolves reward hacking.
  2. The statement that "the criteria is NOT to blame" is overstated in the paper. Adding one appropriateness criterion under plain additive aggregation raises appropriateness from 26.0 to 34.2 (Appendix D.3), nearly matching raw-AND (34.9). ProRubric’s failure clauses also add new content, and removing them costs 2.3 points.
Questions for authors
  1. The paper argues aggregation, not the criteria, causes the collapse, contrasting with Mahmoud et al. (2026). Yet adding one appropriateness criterion under plain additive aggregation recovers about as much (26.0 → 34.2, Appendix D.3) as grouping the unchanged criteria (raw-AND, 34.9). ProRubric’s failure clauses also add new content, and removing them costs 2.3 points. Would the authors agree the two explanations are complementary? Could you test failure clauses as ordinary additive criteria to separate their effect from conjunction?

  2. The Weighted ablation (×3 on criteria tagged critical) does not improve appropriateness (25.3 vs. 26.0). How were critical criteria identified, and how reliable is that classification? Does a much stronger weighting, or gating the reward on critical criteria (zero reward if a critical criterion fails), change this result? At the extreme, gating would resemble ProRubric’s conjunction, so this would help show whether the gain comes from conjunction itself or simply from making critical criteria matter more.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5xnj (Part 1/2: Q1/W2)

We thank Reviewer 5xnj for recognizing our confounder analysis and multi-benchmark evaluation across model scales, and for the presentation feedback—we have streamlined the exposition, unified arm names, and simplified table comparisons throughout the revised paper. Below we respond point by point (all results are 3-seed means unless noted).

[Q1 & W2] Are criterion content and aggregation complementary, and do additive failure clauses work?

Yes, they are complementary; but conjunction drives ~80% of the gain, and failure clauses help only as vetoes. Adding the shared appropriateness item yields diminishing returns as aggregation becomes more structured, lifting Rubric-RL by +8.3 (26.0 to 34.2) and ProRubric by +6.3 (36.8 to 43.1); even with that item, additive scoring plateaus at 34.2, ~9 points below conjunctive arms (43.1–44.1). Importantly, constructing +1 Appropriate requires oracle prior knowledge of the held-out evaluation dimension (violating blind evaluation), so we use it strictly as a diagnostic ablation rather than a main baseline.

Across both medicine and science, isolating conjunctive structure from new domain text shows that conjunction drives the vast majority of the improvement: verbatim grouping (raw-AND) and stripping failure clauses (No-Veto) retain 82% and 79% of the 4B medical gain, and deleting all newly introduced domain conditions identified by a GPT-5.6 audit (5.2% of clauses) before retraining 3 seeds in science (No-New-Content) retains 76% of the gain (+5.71 ± 3.53, positive on all 3 seeds: +2.5 / +5.6 / +9.0; Table R1).

Table R1: Decomposition of gains (cf. revised Tables 3 and 22b)

Domain (Rubric-RL → ProRubric)Control ArmScoreGain Retained
Medicine, 4B (c: 26.0 → 36.8, +10.8)raw-AND (zero new text)34.9+8.9 (82%)
Medicine, 4B (c: 26.0 → 36.8, +10.8)No-Veto (no failure clauses)34.5+8.5 (79%)
Science probe (41.0 → 48.5, +7.5)No-New-Content (inherited only)46.8+5.7 (76%)

Two further 3-seed ablations on 4B medicine separate weight concentration from conjunctive vetoes (Table R2). In +1 Appropriate, the added item carries 18.0% of positive weight (6.6× the atomic mean); setting it to the ordinary mean atomic weight (3.4%, Atomic-Weight Appr.) eliminates 89% of its gain (−7.3 ± 0.3). Conversely, detaching ProRubric's failure clauses into independent additive items (Flat-Veto, 9.0 items/prompt) loses the entire veto benefit and reverts to the level of No-Veto.

Table R2: Functional-form ablations (4B medicine, 3 seeds; cf. revised Table 20)

4B Medical ArmAppr. (c) ± sdDoubao-lite (c)Paired Contrast
Rubric-RL26.0 ± 2.269.0(baseline)
+1 Appropriate (18.0% wt.)34.2 ± 0.571.7+8.3 vs. Rubric-RL
Atomic-Weight Appr. (3.4% wt.)26.9 ± 0.369.5−7.3 ± 0.3 vs. +1 Appr.
No-Veto34.5 ± 1.173.5+8.5 vs. Rubric-RL
ProRubric36.8 ± 2.775.7+2.3 ± 0.3 vs. No-Veto
Flat-Veto34.0 ± 2.969.0−2.8 ± 0.4 vs. ProRubric

Revision: Revised the Abstract and Sections 1 and 5.3; added the functional-form ablations in Appendix D.3 (cf. revised Table 20) and the semantic/content audits in Appendices D.6–D.7 (cf. revised Table 22).

AuthorsOpenReview ↗

Response to Reviewer 5xnj (Part 2/2: Q2 & W1)

[Q2] How are critical criteria identified, and does ×10 weighting or gating help?

Weighting is not the mechanism; non-compensability is: raising the same CRITICAL labels to ×10 under a sum fails (27.8, n.s.), whereas turning those identical labels into a gate jumps to 37.3 (+12.0). In Weighted (×3), Doubao-lite classifies medical criteria into CRITICAL (13.5%, ×3), DUPLICATE (11.3%, ×0), FORMAT (7.2%, ×0), OVERCLINICAL (2.6%, ×−1), and regular items (×1). Re-annotating 100 prompts (3,003 criteria) across three external annotators (two DeepSeek-V4-Pro endpoints and cross-family GPT-5.6-luna) yields 90.0% mean binary agreement on CRITICAL (92.0%, 89.1%, 89.0%), and all 60 substantive incentive cells (∣Δreward∣≥0.05|\Delta\text{reward}| \ge 0.05) preserve their exact sign and rank. Crucially, category labels affect only the Weighted and Critical-Gate controls—ProRubric, raw-AND, and No-Veto use no category labels at all—and because all three external annotators mark fewer items critical (9.0–12.8% vs. 15.2%), stricter labels keep additive Weighted even closer to unweighted Rubric-RL, strengthening our conclusion.

Table R3: Stronger weighting vs. critical gating (4B medicine, 3 seeds; cf. revised Table 21a)

4B Medical ArmAppr. (c) ± sdCoverage (g)Paired Contrast
Rubric-RL26.0 ± 2.232.1(baseline)
Weighted (×3)25.3 ± 1.532.0−0.7 vs. Rubric-RL
Weighted (×10)27.8 ± 1.631.9+1.8 vs. Rubric-RL (n.s.)
K=1K = 1 (single group)37.0 ± 1.933.5+11.0 vs. Rubric-RL (61–63% ties)
raw-AND (K≈4.6K \approx 4.6)34.9 ± 3.633.8+8.9 vs. Rubric-RL
Critical-Gate37.3 ± 1.233.6+12.0 vs. Weighted (×3)
ProRubric (K≈4.6K \approx 4.6)36.8 ± 2.735.0+10.8 vs. Rubric-RL

As Table R3 shows, blocking compensation is what stops reward hacking, but global all-or-nothing gating (K=1K = 1 and Critical-Gate) sacrifices checklist coverage (33.5–33.6). ProRubric (K≈4.6K \approx 4.6) matches their appropriateness (36.8, paired Δ=+0.4\Delta = +0.4 vs. Critical-Gate) while achieving the highest coverage (35.0) without needing a criticality classifier.

Revision: Added the 3-annotator CRITICAL audit in Appendix A.5 and the weighting vs. gating ablations in Appendix D.6.

[W1] Does ProRubric still lose to the untrained Base model?

We agree: ProRubric mitigates rather than eliminates reward hacking relative to Base, and we now foreground this boundary in Sections 1 and 5.3. As noted in our title (Mitigating Reward Hacking...) and Sections 4.2 and 5.4, ProRubric eliminates compensation within each dimension but still sums additively across K≈4.6K \approx 4.6 dimensions. Four findings place the comparison with untrained Base in context:

  1. Equal-budget blind pairwise evaluation: At unregularized step 300 (β=0\beta = 0), raising coverage from Base's 26.6 to 35.0 expands response length from 3.0k to 9.8k characters, incurring redundancy penalties on brief consultations. Under an equal 2,048-token generation ceiling that controls for response length (Figure 4b, DeepSeek-V4-Pro), ProRubric wins 83% (medicine) and 79% (science) of decided blind pairs against Rubric-RL, and 64% (medicine) and 99% (science) against Base.
  2. Unregularized step-300 pairwise results across judges: Even on unregularized step-300 responses (Table 7, 300 pairs across 3 seeds), ProRubric substantially narrows the gap to Base relative to Rubric-RL under both rubric-free judges: Base's win rate drops by half under DeepSeek-V4-Pro (65.7% to 33.1%) and drops from 93.6% to 80.2% in medicine and from 90.3% to 59.7% in science under GPT-5.6-luna.
  3. Coverage–appropriateness frontier across KL coefficients and steps: Untrained Base scores 54.1 on appropriateness by giving brief, low-coverage answers (26.6). Across active KL regularization (Figure 3b, Table 10) and earlier checkpoints (Figure 5), ProRubric traces a consistently higher coverage–appropriateness frontier than Rubric-RL (e.g., retaining 48.0 vs. 27.9 appropriateness at β=0.01\beta = 0.01 while lifting coverage to 30.5 vs. Base 26.6).
  4. Answer-keyed multiple-choice accuracy (MedQA and GPQA-Diamond): On answer-keyed benchmarks free of judge or length priors (Table 2, Appendix C.1), additive Rubric-RL drops below Base on both 4B MedQA (62.6 vs. 65.2) and 8B MedQA (68.9 vs. 69.0). At step 300, ProRubric preserves Base accuracy on 4B MedQA (65.1 vs. 65.2, +2.5 over Rubric-RL), improves over Base on 8B MedQA (69.9 vs. 69.0, +1.0 over Rubric-RL), and lifts GPQA-Diamond over Base at both scales (43.9 vs. 41.1 at 4B; 50.0 vs. 42.9 at 8B).

Revision: Highlighted the cross-dimension boundary in Sections 1 and 5.3, and expanded the pairwise analysis in Appendix B.3 (cf. revised Table 7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their constructive feedback. We first highlight the strengths recognized across the reviews:

  • Diagnosing reward hacking in Rubric-RL: Reviewers recognized our study of a timely, practical failure mode where rubric-based RL optimizes for checklist recall at the expense of response appropriateness (Reviewers 5xnj, jokb, 2vjB).
  • Controlled interventions isolating credit aggregation: Reviewers commended our confounder analyses across length, KL regularization, re-weighting, and SFT (Reviewers 5xnj, jokb, 2vjB), as well as our matched-wording controls (raw-AND) and multi-judge / physician-calibrated evaluations (Reviewers jokb, 2vjB).
  • Broad empirical validation and candid trade-off analysis: Reviewers highlighted our evaluation across two policy scales (Qwen3-4B and Qwen3-8B) and 7 benchmarks (Reviewers 5xnj, jokb, 2vjB)—noting ProRubric's clear advantage over Implicit in controlled science experiments (Reviewer 2vjB)—and our transparent reporting of trade-offs (Reviewer jokb).

Below we summarize our new 3-seed experiments and audits, which directly resolve the soundness and mechanism questions raised in the reviews (point-by-point responses follow below; all results are 3-seed means unless noted):

  • Conjunctive aggregation vs. criterion content (Reviewers 5xnj, jokb): Conjunctive structure drives ~80% of ProRubric's gain, and failure clauses help only as vetoes. An independent GPT-5.6 audit finds that 93.0% of medical and 88.0% of science failure conditions are inherited directly from the original checklists (Table R4). Zero-new-text conjunction (raw-AND, 82% retained), stripping failure clauses (No-Veto, 79%), and retraining science without new domain conditions (No-New-Content, 76%, positive on all 3 seeds) confirm that conjunctive structure accounts for most of the gain (Table R1). Conversely, equalizing the weight of +1 Appropriate loses 89% of its gain, and scoring ProRubric's failure clauses additively (Flat-Veto) drops back to the No-Veto baseline (Table R2).
  • Pre-RL grouping diagnostics and ×10 weighting vs. critical gating (Reviewers 5xnj, 2vjB): Eq. 3 yields a two-step pre-training test on Base rollouts, and critical gating succeeds where ×10 weighting fails. Before RL, checking whether grouping suppresses empty item-naming while keeping GRPO ties moderate (Diagnostic i) and whether failure clauses penalize needless tests with high discriminability (Diagnostic ii) separates K≈30K \approx 30, K=1K = 1, raw-AND, and ProRubric in agreement with post-RL rankings across 4B, 8B, and science (Table R7). Meanwhile, raising additive CRITICAL weights to ×10 fails (27.8, n.s.), whereas turning the exact same CRITICAL labels into a global gate (Critical-Gate) jumps by +12.0 points to 37.3 (Table R3).
  • Complete 8B Implicit results and SFT + ProRubric stacking (Reviewers jokb, 2vjB): ProRubric achieves the highest 7-benchmark average at both 4B (46.9) and 8B (52.2), and stacks positively with SFT. Completing all 8B Implicit runs confirms that ProRubric leads across scales while avoiding off-policy SFT's degradation on MedQA and GPQA (Table R10). Moreover, initializing RL from the SFT checkpoint (SFT + ProRubric) improves both coverage (+8.1) and appropriateness (+1.6), whereas SFT + Rubric-RL degrades appropriateness (Table R8).
  • Fixed checkpoint evaluation and trajectory stability (Reviewers 5xnj, jokb): Every RL run is evaluated at fixed step 300 with zero validation selection. Across all 50-step checkpoints from step 50 to 300, ProRubric consistently leads Rubric-RL by +6.6 to +14.4 points (Table R6).

Summary of Revisions

  • Mechanism and scope claims (Abstract, Sections 1, 4.1, 4.2, 5.3, 7): Qualified the Abstract to "The medical criteria alone are not to blame," and revised Section 1 to "In medicine and dialogue, changing only the aggregation moves the collapse while the criteria stay fixed; in science, where criteria are substantive, both grouping and synthesis matter." Clarified the roles of grouping vs. synthesis, streamlined exposition and table naming, and noted in Section 7 and Appendix B.1 that no physician directly adjudicated the new policies' answers.
  • Completed main and 8B tables (Section 5.2, Table 2; Appendix C.1, Table 8; Appendix D.2, Table 17): Added complete 4B and 8B Implicit rows in Table 2 and Table 8, and full 8B medical aggregation controls (cc and gg) in Table 17.
  • New ablations and audits (Appendices A.5, B.2, D.2, D.3, D.6, D.7): Added full-pool sensitivity (Table 16), 50-step trajectories (Table 18), SFT + ProRubric stacking (Table 19), Atomic-Weight Appropriate and Flat-Veto ablations (Table 20), Weighted ×10 and Critical-Gate controls (Table 21), GPT-5.6 semantic and No-New-Content audits (Table 22), and the 3-model CRITICAL audit (Appendix A.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.