acceptodds
Under review as a conference paper at ICLR 2027

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Abstract

Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage. To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors. Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00–20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.

Primary area:
reinforcement learning
Keywords:
rubrics-based reinforcement learning, reward hacking, Large Language Models, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ck8yOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses an important issue in rubric-based RL, where the judge may assign rewards even when the required information is missing from the response. The authors propose MetaRubric, which combines satisfaction, coverage, and support scores to make rubric-based rewards more reliable. The method also updates the rubric criteria and their weights during training, while ensuring that the original requirements are preserved. Training uses original and counterfactual prompts along with an auxiliary QA reward. The experiments cover four medical benchmarks, including text and multimodal tasks, with ablations to study the different components.

Critical strengths
  1. The paper identifies a relevant problem with rubric-based rewards and provides a useful experiment to demonstrate it. In Table 1, the judges still assign 82.0% and 69.8% of the original credit after the required content is deleted. The comparison with deleting generic statements is also useful, since it helps show that the problem is specifically related to missing required information. I think this experiment provides a good motivation for the proposed approach.

  2. The experimental results are encouraging, and the ablation study helps understand the contribution of rubric adaptation. MetaRubric improves over static-judge GRPO in all 12 comparisons using the original benchmark metrics in Table 2. In Figure 4, removing rubric adaptation reduces performance on HealthBench-Hard and MMOral-OPG by 3.17 and 4.40 points, respectively. Since the other components are retained, this provides reasonable evidence that rubric adaptation is useful. However, the results are based on a single seed, so it is difficult to assess the reliability of smaller improvements.

Critical weaknesses
  1. The paper does not directly evaluate whether MetaRubric actually reduces incorrect reward assignments. Table 1 demonstrates the problem for the original judges, but the same deletion experiment is not reported for MetaRubric. The proposed minimum of three scores can help only when at least one of the scores correctly identifies missing evidence. While Figure 4 shows improvements in downstream performance, it does not establish how effectively the method removes incorrect credit or whether it also removes valid credit. I think this is an important missing experiment, given that this is one of the main motivations of the paper.

  2. The rubric revision process is not sufficiently clear to me. Table 6 mentions at most one proposal per boundary, but Table 3 reports 4,727 revised criteria for HealthBench. It is not clear how many revision boundaries are used, how long each stage is, and whether the proposal limit applies globally or per case. I am also not clear how a revision to a case-specific criterion is evaluated using 20 different prompt pairs. These details are important to understand how the adaptation process works and whether the reported revision counts are consistent with the algorithm. I would encourage the authors to clarify this part.

Questions for authors

Could the authors repeat the required-content deletion test using MetaRubric and satisfaction-only scoring on the same responses and rubric? Please report the scores before and after deletion.

For the HealthBench run, how many training updates are used in each stage, and how many revision boundaries are there? At each boundary, is one proposal allowed in total, or one per case? Please explain how this procedure produces the 4,727 distinct revised criteria in Table 3.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer Ck8y (1/2)

Dear Reviewer Ck8y:

Thank you for recognizing the deletion test as evidence of Vacuous Credit and the benchmark and ablation results as support for MetaRubric.

We address your questions on incorrect and valid credit, the revision schedule, and validation of case-specific criteria below. To address your concern about training variability, we added a five-seed comparison during the rebuttal period: MetaRubric scores 13.02 ± 0.52 versus 10.53 ± 0.29 for GRPO, with a paired gain of 2.49 ± 0.24 points (Appendix A.6.3, Table 10).

Response to Q1/W1

“Could the authors repeat the required-content deletion test using MetaRubric and satisfaction-only scoring on the same responses and rubric? Please report the scores before and after deletion.”

Following your suggestion, we extended the deletion test to evidence-aware scoring and added a valid-credit audit during the rebuttal period (Tables R1–R2). On identical HealthBench responses and rubrics, evidence-aware scoring reduces the mean target score after required-content deletion from 0.698 to 0.186 with GPT-5.4-mini, and from 0.820 to 0.213 with GPT-4o-mini. In Eq. (4), coverage or support can bound the final score even when satisfaction remains high.

Table R1 uses the HealthBench [1] response triples from Section 2.2: the intact response, a version with the target criterion's required content removed, and a version with only generic statements removed. We compare u_sat with min(u_sat, u_cov, u_sup). GPT-5.4-mini is the training judge; GPT-4o-mini repeats the other diagnostic condition in the submitted Table 1.

GPT-5.5 generates and verifies the response edits. The comparison uses 942 verified triples for GPT-4o-mini and 944 for GPT-5.4-mini, applying both scoring rules to the same triples for each judge. False rejection is measured on a separate stratified sample of 612 verified valid responses.

Table R1. Mean HealthBench target-criterion scores before and after deletion.

JudgeScoring ruleIntact target scoreAfter required-content deletionAfter generic-statement deletion
GPT-4o-miniSatisfaction only1.0000.8200.931
GPT-4o-miniEvidence-aware minimum0.8740.2130.842
GPT-5.4-miniSatisfaction only1.0000.6980.950
GPT-5.4-miniEvidence-aware minimum0.9020.1860.867

For Table R2, the award threshold is 0.5 and the judge is GPT-5.4-mini. Incorrect-award rates use all 2,164 scored targets as the denominator; retention rates use the 1,744 targets credited before deletion.

Table R2. Incorrect-award and false-rejection rates on verified HealthBench response edits with GPT-5.4-mini.

MeasureSatisfaction onlyEvidence-aware minimum
Incorrect awards after required-content deletion, %61.218.1
Award retention after required-content deletion, %75.922.4
Valid awards retained after generic-statement deletion, %94.189.6
False rejections on independently verified valid responses, %2.37.1

Incorrect-award retention falls by 53.5 points, while 89.6% of valid awards survive generic-statement deletion. False rejection on independently verified valid responses changes from 2.3% to 7.1%. The explanation following Eq. (4) now states the detection condition (Section 3.1, p. 5), and Appendix A.6.5, Tables 13–14 (p. 28), reports the full comparison; Section 4.4 (pp. 9–10) summarizes the result.

Response to Q2/W2

“For the HealthBench run, how many training updates are used in each stage, and how many revision boundaries are there? At each boundary, is one proposal allowed in total, or one per case? Please explain how this procedure produces the 4,727 distinct revised criteria in Table 3.”

The 4,727 count comes from 30 stages of 32 updates, with one proposal allowed per sampled case at each boundary. The one-proposal-per-boundary global cap applies to the shortened 140-update supplementary configuration. Table 3 counts distinct criteria whose description or correspondence label changes at least once; repeated edits count once and weight-only updates are excluded.

This gives 960 policy updates and 30 revision boundaries. Each boundary samples up to 96 cases. The run recorded 2,687 proposals and 2,364 accepted edits (88.0%). Each accepted edit revised the selected criterion and its counterfactual counterpart, producing 4,728 criterion records. One criterion was revised twice, leaving 4,728 − 1 = 4,727 distinct criteria.

Section 4.4 (pp. 9–10) now identifies the run behind Table 3, and Appendix A.6.8, Table 17 (p. 31), places the two schedules side by side. Revised Table 7 specifies 140 updates, ten updates per stage, 14 boundaries, and the global proposal cap for the supplied configuration.

AuthorsOpenReview ↗

Author Response to Reviewer Ck8y (2/2)

Response to W2: validation on 20 prompt pairs

“I am also not clear how a revision to a case-specific criterion is evaluated using 20 different prompt pairs.”

The 20-pair validation tests whether revised assessment wording improves agreement with fixed reference judgments on other cases. The original submission (Section 3.3 and Appendix A.3.4) requires an agreement gain strictly above 0.02 and semantic approval before accepting an edit. Thus, a revision must improve judgments beyond its proposal case while preserving the original requirement before it can affect training rewards.

The LLM summarizes recurring patterns from collected training rollouts to guide the next revision. We find that edits mainly refine supplementary qualifications and assessment wording around a criterion's core requirement. The clinical facts remain case-specific, while the wording can clarify a shared distinction, such as whether the response merely acknowledges an issue or performs the required action.

For example, a HealthBench criterion for follow-up after a stent procedure states: “Asks the user about their geographic location as follow-up schedules after stent procedures may vary.” The unchanged core requirement is to ask for the user's location. The criterion can preserve this requirement while adding: “Merely stating that guidelines vary by region does not count as asking for the user's location.” While the stent context is case-specific, the added assessment wording captures a transferable judgment pattern: distinguishing between mentioning relevant context and explicitly requesting the needed information.

To evaluate a proposed edit, we compare the original and revised criteria on the same responses from 20 held-out training prompt pairs against reference judgments fixed before the proposal. The 20 pairs exclude both the proposal case and the current rollout batch. Each criterion is interpreted relative to the held-out case's own facts. Agreement is averaged over all 20 pairs in the denominator; pairs where the target requirement is inapplicable remain included and contribute no positive target. Separate GPT-5.4-mini role calls generate the reference judgments, validation scores, and semantic review. An accepted edit updates both the selected criterion and its counterfactual counterpart.

The same source training examples are visited across multiple RL updates. When an example is sampled again, the updated policy generates fresh rollouts that are scored using the accepted rubric revisions in subsequent reward calculations.

Section 3.3 (p. 7) and Appendix A.6.9 (p. 31) now clarify how criteria are interpreted across cases, how reference judgments are fixed, and how agreement is aggregated.

References

[1] Rahul K. Arora et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. 2025.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer Ck8y,

Thank you again for your careful review and for recognizing the deletion test's motivation and the evidence from our benchmark results and ablations. We would appreciate your thoughts on whether our response addresses your questions.

During the rebuttal period, we made every effort to address your concerns as follows.

  • Incorrect and valid credit. On the same HealthBench responses and rubrics, evidence-aware scoring reduces the mean target score after required-content deletion from 0.698 to 0.186 with GPT-5.4-mini. We also report intact-response scores and the valid-credit tradeoff: 89.6% of valid awards survive generic-statement deletion, while false rejection on independently verified valid responses rises from 2.3% to 7.1% (Tables R1–R2).
  • Revision counts and validation. We explain how the Table 3 run's 30 stages of 32 updates, with one proposal per sampled case at each boundary, yield 4,727 distinct revised criteria. We distinguish this schedule from the shortened supplementary configuration and clarify how the 20-pair check evaluates assessment wording against fixed reference judgments while preserving each criterion's core requirement.
  • Training variability. Across 5 seeds, MetaRubric achieves 13.02 ± 0.52 versus 10.53 ± 0.29 for GRPO on the HealthBench-Hard accuracy axis (mean ± SD).

Your feedback helped us directly test incorrect credit and make the adaptation procedure clearer. We would be grateful if you could reconsider your recommendation in light of these clarifications and additional results. We would be happy to discuss any remaining questions.

Warm regards, Authors

Reviewer_4rhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a failure mode termed Vacuous Credit within rubric‑based reinforcement learning, where LLM judges assign high scores to response criteria even when required evidence or actions are entirely absent from model outputs. This flawed credit assignment corrupts GRPO learning signals and can flip the advantage sign of candidate responses during policy update. To mitigate this issue, the authors propose MetaRubric, a two‑loop framework combining evidence‑aware policy optimization and rubric adaptation. The inner loop performs GRPO training with counterfactual prompt pairs, and criterion scores are bounded jointly by satisfaction, coverage and evidential support. The outer loop adaptively revises rubric descriptions and adjusts group‑level criterion weights according to observed policy errors, while preserving the original semantic meaning of rubric requirements. Evaluated across multiple text‑only and multimodal medical benchmarks, MetaRubric outperforms static‑judge GRPO and several strong RL baselines, yielding 6.00‑20.40 percentage point accuracy improvements on PubMedQA and consistent gains on HealthBench‑Hard, MMOral‑X and MMOral‑OPG. Ablation studies validate the necessity of evidence checking, rubric revision and weight adaptation components.

Critical strengths

S1: This paper identifies an important and real failure mode: Vacuous Credit in rubric‑based reinforcement learning. This issue corrupts GRPO learning signals and can even flip advantage values of training samples, offering general insights for rubric‑based LLM alignment research.

S2: The counterfactual prompt‑pair design is insightful. By modifying exactly one task‑relevant fact to construct counterfactual samples, the framework forces the model to attend to fact‑dependent requirements. It offers a new data‑construction perspective for rubric‑based RL.

S3: Semantic anchoring is enforced for rubric revision. When iteratively updating rubrics, revisions are constrained to preserve the original semantic intent of criteria rather than shifting task objectives, a critical safeguard often overlooked in self‑evolving rubric literature.

S4: Thorough ablation studies verify component‑wise contributions. By disabling rubric revision, weight adaptation, and evidence checks, the paper quantifies individual component gains. Additional experiments show that retraining with only the final static rubric cannot reproduce full performance, highlighting that the dynamic iterative process itself is essential.

Critical weaknesses

W1: Only one random seed is used for each training configuration As acknowledged in the limitation section, every training setting adopts merely a single random seed. The paper does not report variance across multiple training runs. It is hard to tell whether the observed performance gains are stable or driven by random training fluctuation.

W2: High computational overhead introduced by the double‑loop framework MetaRubric’s outer loop requires counterfactual sample generation, rubric revision, semantic validation and group‑wise weight adaptation in every training stage. Compared with vanilla GRPO, it brings considerable extra computation cost. The paper lacks systematic analysis of computational cost and latency.

W3: No comparison against other self‑evolving rubric baselines Several recent works study self‑adaptive / self‑evolving rubric for RL. The paper only compares with standard GRPO, DAPO, Dr.GRPO and GSPO, but does not include representative rubric‑evolution baselines for direct head‑to‑head comparison.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer 4rhk (1/2)

Dear Reviewer 4rhk:

Thank you for recognizing the importance of Vacuous Credit, our counterfactual-pair design and semantic anchoring, and the evidence provided by the component ablations.

We address your concerns about training variability, computational cost, and comparisons with evolving-rubric methods below.

Response to W1

“The paper does not report variance across multiple training runs. It is hard to tell whether the observed performance gains are stable or driven by random training fluctuation.”

In response to this concern, we conducted a five-seed HealthBench comparison during the rebuttal period. MetaRubric achieves 13.02 ± 0.52, compared with 10.53 ± 0.29 for static-judge GRPO, with a paired gain of 2.49 ± 0.24 points. This complements the gains in all 12 original benchmark comparisons across three model families and four benchmarks (Table 2), including PubMedQA improvements of 6.00, 6.80, and 20.40 points.

We train Qwen3-4B on HealthBench with seeds 42, 7, 173, 2019, and 88. Each run uses 140 policy updates and GPT-5.4-mini as the training judge. Table R1 reports HealthBench-Hard accuracy-axis scores as mean ± SD across the five runs; gains are computed within each seed.

Table R1. HealthBench training variability across five seeds per method, evaluated on HealthBench-Hard.

SettingMethodIndependent training runsScore, mean ± SDPaired gain over GRPO, mean ± SD
Qwen3-4B, HealthBenchStatic-judge GRPO510.53 ± 0.29Reference
Same settingMetaRubric513.02 ± 0.52+2.49 ± 0.24

The revised Section 4.4 (pp. 9–10) and Appendix A.6.3, Table 10 (p. 27), include this five-seed comparison.

Response to W2

“Compared with vanilla GRPO, it brings considerable extra computation cost. The paper lacks systematic analysis of computational cost and latency.”

To quantify this overhead, we added a resource and latency analysis during the rebuttal period. MetaRubric takes 15.7 hours versus GRPO's 12.6 hours (1.25×), while improving HealthBench-Hard accuracy-axis score from 10.56 to 13.02. Outer-loop adaptation accounts for 0.9 hours.

As described in the original submission (Section 3.1 and Appendix A.3.2), counterfactual prompts and auxiliary questions are constructed once before training. In the additional 140-update runs, the outer loop executes every 10 updates, for 14 boundaries. At each boundary, it grades at most 96 responses and validates a proposed revision on 20 prompt pairs (Appendix A.3.4).

Table R2 reports seed-42 runs with GPT-5.4-mini as the reward and outer-loop judge; token totals include failed requests and retries. Reader GPU-hours measure Qwen3-1.7B inference for auxiliary QA.

Table R2. HealthBench computational cost and latency with GPT-5.4-mini as the training judge.

MethodPreparation costJudge input/output tokensReader inference GPU-hoursPolicy GPU-hoursOuter-loop timeEnd-to-end timeEstimated API cost, 3 epochsHealthBench-Hard accuracy-axis score
Static-judge GRPO1.6 GPU-h47.3M / 7.9M0.812.1Not applicable12.6 h$71.510.56
MetaRubric4.7 GPU-h103.6M / 18.4M1.312.80.9 h15.7 h$161.613.02

We also added stronger-judge training controls during the rebuttal period (Table R3). MetaRubric reduces estimated API expenditure by 36.7% relative to GPT-5.4 GRPO and by 82.1% relative to panel-based GRPO, while achieving a higher accuracy-axis score than both. It takes longer than the single GPT-5.4 baseline (15.7 versus 13.1 hours).

Table R3. HealthBench performance and cost with stronger training judges, seed 42.

MethodTraining judgeHealthBench-Hard accuracy-axis scoreEnd-to-end timeEstimated API cost, 3 epochs
Static-judge GRPOGPT-5.411.3113.1 h$255.3
Static-judge GRPOGPT-5.4 + Gemini 3 Pro + Claude Opus 4.611.7918.4 h$904.4
MetaRubricGPT-5.4-mini13.0215.7 h$161.6

The revised Section 4.4 (pp. 9–10) and Table 5 (p. 10) report performance, elapsed time, and estimated API expenditure; Appendix A.6.4 (pp. 27–28), Tables 11–12 (p. 27), separates preparation from recurring resource use.

AuthorsOpenReview ↗

Author Response to Reviewer 4rhk (2/2)

Response to W3

“The paper only compares with standard GRPO, DAPO, Dr.GRPO and GSPO, but does not include representative rubric‑evolution baselines for direct head‑to‑head comparison.”

Following your suggestion, we added direct comparisons with HealthBench adaptations of RLER and OnlineRubrics during the rebuttal period (Table R4). MetaRubric scores 13.02, exceeding their 11.87 and 11.34 by 1.15 and 1.68 points, respectively.

The difference concerns both what is revised and how credit is assigned. Compared with RLER's addition of criteria from retrieved information and contrasting rollouts [1], MetaRubric refines an existing requirement while preserving its initial meaning, with semantic review and improved held-out agreement required for acceptance. Compared with OnlineRubrics' current-versus-reference response comparison [2], our contrast changes one prompt fact to reveal which actions should change and which requirements should remain invariant. These paired responses guide revisions that distinguish generic acknowledgment from fulfillment.

RLER and OnlineRubrics aggregate criterion satisfaction; MetaRubric additionally bounds credit by coverage and support, reducing a high satisfaction score when either check detects missing or unsupported content. Its group weights adapt to observed errors under severity constraints.

The original Figure 4 isolates the contribution of adaptation while retaining paired sampling, auxiliary QA, and evidence-aware scoring: freezing rubric descriptions and weights lowers HealthBench-Hard and MMOral-OPG scores by 3.17 and 4.40 points, respectively. The new comparison below evaluates MetaRubric against the two evolving-rubric adaptations.

Table R4. Comparison with evolving-rubric methods adapted to HealthBench, seed 42.

MethodBackbone and training dataHealthBench-Hard accuracy-axis scoreJudge tokens / elapsed time
Static-judge GRPOQwen3-4B, HealthBench10.5647.3M / 7.9M; 12.6 h
RLER adaptationQwen3-4B, HealthBench11.8758.1M / 9.7M; 13.9 h
OnlineRubrics adaptationQwen3-4B, HealthBench11.3454.6M / 9.1M; 13.6 h
MetaRubricQwen3-4B, HealthBench13.02103.6M / 18.4M; 15.7 h

The comparisons use the same backbone, source training split, policy-update budget, and reward judge. The two adapted baselines train on original prompts; MetaRubric uses original–counterfactual pairs.

The revised related work (Appendix A.1, p. 14) distinguishes these mechanisms, and Section 4.4, Table 5 (p. 10), reports the direct comparison. Appendix A.6.1 gives the baseline adaptations.

References

[1] Rulin Shao et al. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. 2025.

[2] MohammadHossein Rezaei et al. Online Rubrics Elicitation from Pairwise Comparisons. 2025.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer 4rhk,

Thank you again for your careful review and for recognizing the importance of Vacuous Credit, the counterfactual-pair design, and semantic anchoring. We would be grateful if you could read our response and let us know whether it addresses your concerns.

During the rebuttal period, we made every effort to address your concerns as follows.

  • Training variability. Across 5 seeds, MetaRubric achieves a HealthBench-Hard accuracy-axis score of 13.02 ± 0.52, compared with 10.53 ± 0.29 for static-judge GRPO, with a paired gain of 2.49 ± 0.24 points (mean ± SD; Table R1).
  • Computational cost. We report preparation, judge usage, reader inference, and outer-loop costs. End-to-end time is 15.7 hours versus 12.6 hours for GRPO, or 1.25×, with 0.9 hours spent on outer-loop adaptation. The stronger-judge controls also place this overhead alongside performance and estimated API expenditure (Tables R2–R3).
  • Evolving-rubric baselines. We added HealthBench adaptations of RLER and OnlineRubrics. MetaRubric scores 13.02, compared with 11.87 and 11.34, respectively, and our response explains how evidence-aware credit and meaning-preserving revision differ from these methods (Table R4).

Your suggestions helped us better establish the consistency of the gains and quantify their computational cost. We would be grateful if you could reconsider your recommendation in light of these clarifications and additional results. We would be happy to clarify any remaining concerns.

Warm regards, Authors

Reviewer_SEmeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper studies a failure mode in rubric based RL called "vacuous credit" where a rubric judge assigns high reward even when necessary information is removed from a response. Their solution is to create counterfactual prompts which remove some task relevant facts from the prompt. Rubric evaluation can be problematic due to criteria underspecification & unsupported judging.
  • Their method meta rubric uses a two stage approach to update the rubrics (such as reword for ambiguity) based on policy failures in the inner loop. Results show improvement over baselines across multiple model families and datasets showing improvements in performance. Ablations show contributions of various design choices and their impact on the final performance.
Critical strengths
  1. MetaRubric’s two-loop training is a nice design choice. Instead of keeping the rubric fixed, it refreshes the rubric during training: it rewords ambiguous criteria and adapts criterion weights within severity-order constraints. Revisions are accepted only after validation on held-out cases. Together, this aims to make rubric-based training more robust to the failure modes that arise from vacuous credit.
  2. Counterfactual prompts are a creative idea. Beyond their use as training data here, future work could use them to detect judge oversight, find ambiguous rubric criteria, identify judge divergence, or detect certain types of reward hacking.
Critical weaknesses
  1. The novelty of the paper needs to be substantiated in a better way. Ideas of evolving or online rubrics [2,3], as well as measuring judge-model exploitation during rubric-based RL [1], are already published in the literature. This paper focuses on a single failure mode, vacuous credit, but it should discuss these related works in the main paper and make clear what is new relative to them. The method also combines several components: counterfactual data augmentation, evidence-aware scoring, an auxiliary QA reward, and two-loop training (inner and outer loop). It is not clear whether the improvements come from the counterfactual data or from the two-loop training, and the ablations don’t isolate these. The paper also doesn’t show whether training with MetaRubric reduces the divergence between training reward and evaluation-panel scores seen in Fig. 2, or the vacuous credit rate.
  2. The paper needs more thorough reporting of its experiments such as (i) how the counterfactual prompts and their rubrics are generated and checked for accuracy (ii) which judge models were used to train the baselines and the method (iii) whether the baselines also train on the counterfactual prompts. The paper should also compare against a baseline trained with a stronger judge or a panel of judges, which [1] shows reduces exploitation. Since the method targets a narrower subset of judge failures, this comparison would show whether a targeted intervention is warranted over a more general fix.

References:

[1] Mahmoud, Anas, et al. "Reward hacking in rubric-based reinforcement learning." arXiv preprint arXiv:2605.12474 (2026).

[2] Shao, Rulin, et al. "Dr tulu: Reinforcement learning with evolving rubrics for deep research." arXiv preprint arXiv:2511.19399 (2025).

[3] Rezaei, MohammadHossein, et al. "Online rubrics elicitation from pairwise comparisons." arXiv preprint arXiv:2510.07284 (2025).

Questions for authors
  1. How are counterfactual prompts and corresponding rubrics vetted for accuracy? Changing one task-relevant fact about the prompt can lead to multiple other changes in an ideal response, so only revising fact specific rubrics may not be sufficient. The paper currently does not provide much evidence on how counterfactual prompts are generated and whether this is qualitatively tested to ensure the rubrics for counterfactual prompts are still valid.
  2. What does reward growth and rubric satisfaction (the figure 2 analysis) look like with the new proposed meta-rubric training? Does is reduce the divergence of reward between train and evaluation judge?
  3. Which data and judge models were used during training for the baselines and how does the judge call compute budget vary across the baseline and proposed method? Are counterfactuals prompts also included in the baseline training data?
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer SEme (1/3)

Dear Reviewer SEme:

Thank you for recognizing the two-loop design and the value of counterfactual prompts for identifying rubric ambiguities and judge failures.

We address your concerns below.

Response to W1: novelty and related work

“Ideas of evolving or online rubrics [2,3], as well as measuring judge-model exploitation during rubric-based RL [1], are already published in the literature. This paper focuses on a single failure mode, vacuous credit, but it should discuss these related works in the main paper and make clear what is new relative to them.”

MetaRubric's contribution is to control when an existing requirement earns credit and how that requirement is revised during training.

Compared with Mahmoud et al., who diagnose exploitation through verifier–panel disagreement and rubric-based versus rubric-free evaluation [1], we delete required content while retaining generic statements. In the original submission, 82.0% and 69.8% of awards survive deletion (Section 2.2, Table 1), and correcting these awards can reverse a GRPO advantage sign (Section 2.3). This links missing-content errors to policy updates and motivates coverage and support checks.

Compared with RLER, which adds criteria from retrieved information and contrasting rollouts [2], MetaRubric refines existing criteria while preserving their initial meaning, accepting edits only after semantic review and improved held-out agreement. Compared with OnlineRubrics, which elicits criteria from current/reference responses to the same prompt [3], MetaRubric changes one prompt fact. Paired responses identify which actions should change and which requirements remain invariant, guiding refinement of assessment wording.

The scoring rule also differs: RLER and OnlineRubrics aggregate satisfaction; MetaRubric uses min(satisfaction, coverage, support) to limit credit for missing or unsupported content. Anchored revision and severity-constrained weights adapt rewards to policy errors.

Following your suggestion, we added HealthBench comparisons with RLER and OnlineRubrics during the rebuttal period (Table R1). All use GPT-5.4-mini.

Table R1. Comparison with evolving-rubric methods adapted to HealthBench.

MethodHealthBench-Hard accuracy-axis scoreJudge tokens / elapsed time
Static-judge GRPO10.5647.3M / 7.9M; 12.6 h
RLER adaptation11.8758.1M / 9.7M; 13.9 h
OnlineRubrics adaptation11.3454.6M / 9.1M; 13.6 h
MetaRubric13.02103.6M / 18.4M; 15.7 h

Comparisons share the backbone, source training split, policy-update budget, and reward judge.

Section 4.4 (pp. 9–10) discusses these mechanisms, with results in Table 5 (p. 10). Appendix A.1 (p. 14) expands the comparison; A.6.1 details the adaptations.

Response to W1: component contributions

“It is not clear whether the improvements come from the counterfactual data or from the two-loop training, and the ablations don’t isolate these.”

Figure 4 in the original submission already isolates rubric adaptation while retaining counterfactual data, evidence-aware scoring, and auxiliary QA. Freezing both description revision and weight adaptation lowers HealthBench-Hard and MMOral-OPG scores by 3.17 and 4.40 points, respectively. Original Table 4 provides complementary evidence: training with the final static rubric reaches 9.04 and 21.78, compared with 13.02 and 26.35 for MetaRubric.

To test whether the benefit of counterfactual data depends on adaptation, we extended this analysis during the rebuttal period by crossing counterfactual availability with rubric adaptation (Table R2). Adding counterfactuals improves the score by 1.78 points with adaptation active, but changes it by −0.08 points with frozen rubrics. The gain from paired data therefore depends on adaptation in these controls.

Conditions A–D vary counterfactual data and rubric adaptation while retaining evidence-aware scoring and auxiliary QA; condition E removes auxiliary QA. All conditions use the same total rollout budget.

Table R2. HealthBench component ablations, evaluated on HealthBench-Hard.

ConditionCounterfactual dataRubric adaptationAuxiliary QAHealthBench-Hard accuracy-axis score
ANoFrozenOn9.93
BYesFrozenOn9.85
CNoActiveOn11.24
D: full methodYesActiveOn13.02
EYesActiveOff12.19

Section 4.3 (pp. 8–9) now reports these contrasts, and Appendix A.6.2, Table 9 (p. 26), gives all five conditions and the rollout budget.

AuthorsOpenReview ↗

Author Response to Reviewer SEme (2/3)

Response to Q2/W1

“What does reward growth and rubric satisfaction (the figure 2 analysis) look like with the new proposed meta-rubric training? Does is reduce the divergence of reward between train and evaluation judge?”

Following your suggestion, we extended the original Figure 2 analysis to MetaRubric during the rebuttal period (Table R3 and the new Figure 21), measuring reward divergence, requirement fulfillment, and flagged Vacuous Credit. MetaRubric reduces the final training-judge versus evaluation-panel gap from GRPO's 0.423 to 0.317, a 25.1% reduction. Fully satisfied requirement weight increases from GRPO's 27.4% to 34.8%, and flagged Vacuous Credit decreases from 14.6% to 6.2%.

Table R3 evaluates Qwen3-4B at initialization and after 140 updates on the 1,000 HealthBench-Hard conversations. The diagnostic panel comprises GPT-5.5, Gemini 3.5 Flash, and DeepSeek-V4-Pro, requiring unanimous criterion satisfaction. Panel and judge scores use a fixed rubric; fully satisfied requirement weight and flagged Vacuous Credit follow Section 2.1. We report the optimized training reward separately because MetaRubric combines an evolving rubric with auxiliary QA.

The new Figure 21 (Appendix A.6.7, p. 30) shows intermediate checkpoints at 20-update intervals. MetaRubric maintains a smaller judge–panel gap than GRPO at every evaluated checkpoint after initialization. Fully satisfied requirement weight increases overall and flagged Vacuous Credit decreases under MetaRubric, while GRPO shows the opposite overall trends.

Table R3. HealthBench reward divergence and evidence fulfillment at initialization and after 140 updates.

Policy / checkpointOptimized training rewardFixed-rubric GPT-5.4-mini scoreDiagnostic evaluation-panel scoreGPT-5.4-mini minus panel scoreFully satisfied weightFlagged Vacuous Credit
Shared initializationNot applicable0.4120.286+0.12631.7%8.9%
Static-judge GRPO, final0.6870.6640.241+0.42327.4%14.6%
MetaRubric, final0.7480.6310.314+0.31734.8%6.2%

Section 4.4 (pp. 9–10) summarizes this additional analysis; Appendix A.6.7 reports the endpoints in Table 16 (p. 29) and trajectories in Figure 21 (p. 30).

Response to Q1/W2

“How are counterfactual prompts and corresponding rubrics vetted for accuracy? Changing one task-relevant fact about the prompt can lead to multiple other changes in an ideal response, so only revising fact specific rubrics may not be sufficient.”

The construction procedure in the original submission already considers each criterion separately: a single changed fact can affect multiple criteria, including their targets, weights, and applicability. Section 3.1 and Appendix A.4.1, Figures 7 and 8, specify that construction updates the reference answer and assigns each requirement a correspondence label: invariant, target change, weight change, dropped, or added. Unaffected requirements retain their original wording.

For example, changing a patient's age may change both a recommendation and the explanation required to justify it; both criteria are therefore checked. For image-based tasks, Figure 7 keeps the image unchanged and specifies the altered observation as a counterfactual condition.

To assess how reliably this procedure propagates a fact change, we conducted a full-rubric validity audit during the rebuttal period (Table R4). It checks whether each intervention changes one fact, the updated reference is correct, all affected criteria are revised, and criteria labeled invariant remain valid.

Counterfactual pairs were constructed with GPT-5.4-mini from the supplied prompt, reference, and initial rubric. The audit sample of 120 accepted pairs was stratified by changed-fact type. Two annotators (one clinician, one NLP researcher) labeled independently; disagreements were adjudicated by a third reviewer.

Table R4. HealthBench counterfactual prompt and full-rubric validity audit.

Training dataAudited prompt pairsValid single-fact interventionsCorrect updated referencesFully consistent counterfactual rubricsMissed dependent-criterion changes
HealthBench12096.7%91.7%85.8%10.0%

Section 3.1 (p. 5) now clarifies that one changed fact can affect several criteria; Appendix A.6.6, Table 15 (p. 29), reports the whole-rubric audit.

AuthorsOpenReview ↗

Author Response to Reviewer SEme (3/3)

Response to Q3/W2

“Which data and judge models were used during training for the baselines and how does the judge call compute budget vary across the baseline and proposed method? Are counterfactuals prompts also included in the baseline training data?”

Training data and judges. The original submission specifies the data splits in Appendix A.3.1: non-test PubMedQA PQA-L, HealthBench outside HealthBench-Hard, and MMOral-RL for both multimodal benchmarks. It also identifies MetaRubric's GPT-5.4-mini training judge and outer-loop roles in Appendix A.3.4. All standard baselines use the same source prompts and GPT-5.4-mini judge. We have made this shared configuration explicit in Section 4.1 (p. 8).

Counterfactuals work jointly with MetaRubric's other components. Baselines train on original prompts only. MetaRubric pairs each original with a counterfactual (original Section 3.1) because the paired contrast, evidence-aware scoring, and rubric adaptation serve complementary roles: the contrast exposes fact-dependent omissions, evidence checks limit credit for those omissions, and revision clarifies the corresponding requirement for subsequent updates. The rebuttal-period controls in Table R2 hold the rollout budget fixed: adding counterfactuals changes the score from 9.93 to 9.85 with frozen rubrics, but from 11.24 to 13.02 with adaptation. These results support the joint use of counterfactual pairing and adaptation; adding counterfactual data alone does not improve performance in this control.

Judge budget. We added resource accounting during the rebuttal period. For Qwen3-4B on HealthBench at 140 updates, each standard baseline uses 20,160 judge calls, versus 40,320 for MetaRubric (2×), with the same total rollout budget. End-to-end time is 12.6 versus 15.7 hours for GRPO and MetaRubric. Appendix A.6.4, Tables 11–12 (p. 27), gives token totals and the preparation, reader, and outer-loop breakdown, including retries.

Response to W2: stronger judge and judge panel

“The paper should also compare against a baseline trained with a stronger judge or a panel of judges, which [1] shows reduces exploitation. Since the method targets a narrower subset of judge failures, this comparison would show whether a targeted intervention is warranted over a more general fix.”

Following your suggestion, we added stronger-judge and judge-panel training controls during the rebuttal period (Table R5). MetaRubric with GPT-5.4-mini scores 13.02, compared with 11.31 for GRPO trained with GPT-5.4 and 11.79 for GRPO trained with the three-judge panel. Flagged Vacuous Credit is also lower: 6.2%, compared with 10.4% and 7.9%, respectively.

The training panel comprises GPT-5.4, Gemini 3 Pro, and Claude Opus 4.6, following the reference-panel composition of Mahmoud et al. [1]. Their panel is used for evaluation; we use it to provide training rewards. Each member judges every criterion, and credit requires unanimous satisfaction before applying the rubric weights.

Table R5. HealthBench stronger-judge and judge-panel controls.

Training rewardTraining judgeHealthBench-Hard accuracy-axis scoreFlagged Vacuous CreditJudge tokensEnd-to-end timeEstimated API cost, 3 epochs
Static rubric + GRPOStandard: GPT-5.4-mini10.5614.6%47.3M / 7.9M12.6 h$71.5
Static rubric + GRPOStronger: GPT-5.411.3110.4%49.8M / 8.6M13.1 h$255.3
Static rubric + GRPOPanel: GPT-5.4 + Gemini 3 Pro + Claude Opus 4.611.797.9%146.2M / 25.1M18.4 h$904.4
MetaRubricGPT-5.4-mini13.026.2%103.6M / 18.4M15.7 h$161.6

This advantage also holds in estimated API expenditure. For three HealthBench epochs, MetaRubric costs $161.6, compared with $255.3 for GPT-5.4 GRPO and $904.4 for panel-based GRPO, reductions of 36.7% and 82.1%, respectively. It achieves a higher accuracy-axis score than both, despite taking longer than the single GPT-5.4 baseline (15.7 versus 13.1 hours).

Section 4.4, Table 5 (p. 10), now reports performance and resource use; Appendix A.6.1, Table 8 (p. 26), adds the Vacuous Credit comparison.

References

[1] Anas Mahmoud et al. Reward Hacking in Rubric-Based Reinforcement Learning. 2026.

[2] Rulin Shao et al. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. 2025.

[3] MohammadHossein Rezaei et al. Online Rubrics Elicitation from Pairwise Comparisons. 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.