- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses an important issue in rubric-based RL, where the judge may assign rewards even when the required information is missing from the response. The authors propose MetaRubric, which combines satisfaction, coverage, and support scores to make rubric-based rewards more reliable. The method also updates the rubric criteria and their weights during training, while ensuring that the original requirements are preserved. Training uses original and counterfactual prompts along with an auxiliary QA reward. The experiments cover four medical benchmarks, including text and multimodal tasks, with ablations to study the different components.
Critical strengths
-
The paper identifies a relevant problem with rubric-based rewards and provides a useful experiment to demonstrate it. In Table 1, the judges still assign 82.0% and 69.8% of the original credit after the required content is deleted. The comparison with deleting generic statements is also useful, since it helps show that the problem is specifically related to missing required information. I think this experiment provides a good motivation for the proposed approach.
-
The experimental results are encouraging, and the ablation study helps understand the contribution of rubric adaptation. MetaRubric improves over static-judge GRPO in all 12 comparisons using the original benchmark metrics in Table 2. In Figure 4, removing rubric adaptation reduces performance on HealthBench-Hard and MMOral-OPG by 3.17 and 4.40 points, respectively. Since the other components are retained, this provides reasonable evidence that rubric adaptation is useful. However, the results are based on a single seed, so it is difficult to assess the reliability of smaller improvements.
Critical weaknesses
-
The paper does not directly evaluate whether MetaRubric actually reduces incorrect reward assignments. Table 1 demonstrates the problem for the original judges, but the same deletion experiment is not reported for MetaRubric. The proposed minimum of three scores can help only when at least one of the scores correctly identifies missing evidence. While Figure 4 shows improvements in downstream performance, it does not establish how effectively the method removes incorrect credit or whether it also removes valid credit. I think this is an important missing experiment, given that this is one of the main motivations of the paper.
-
The rubric revision process is not sufficiently clear to me. Table 6 mentions at most one proposal per boundary, but Table 3 reports 4,727 revised criteria for HealthBench. It is not clear how many revision boundaries are used, how long each stage is, and whether the proposal limit applies globally or per case. I am also not clear how a revision to a case-specific criterion is evaluated using 20 different prompt pairs. These details are important to understand how the adaptation process works and whether the reported revision counts are consistent with the algorithm. I would encourage the authors to clarify this part.
Questions for authors
Could the authors repeat the required-content deletion test using MetaRubric and satisfaction-only scoring on the same responses and rubric? Please report the scores before and after deletion.
For the HealthBench run, how many training updates are used in each stage, and how many revision boundaries are there? At each boundary, is one proposal allowed in total, or one per case? Please explain how this procedure produces the 4,727 distinct revised criteria in Table 3.
5 comments
Author Response to Reviewer Ck8y (2/2)
Response to W2: validation on 20 prompt pairs
“I am also not clear how a revision to a case-specific criterion is evaluated using 20 different prompt pairs.”
The 20-pair validation tests whether revised assessment wording improves agreement with fixed reference judgments on other cases. The original submission (Section 3.3 and Appendix A.3.4) requires an agreement gain strictly above 0.02 and semantic approval before accepting an edit. Thus, a revision must improve judgments beyond its proposal case while preserving the original requirement before it can affect training rewards.
The LLM summarizes recurring patterns from collected training rollouts to guide the next revision. We find that edits mainly refine supplementary qualifications and assessment wording around a criterion's core requirement. The clinical facts remain case-specific, while the wording can clarify a shared distinction, such as whether the response merely acknowledges an issue or performs the required action.
For example, a HealthBench criterion for follow-up after a stent procedure states: “Asks the user about their geographic location as follow-up schedules after stent procedures may vary.” The unchanged core requirement is to ask for the user's location. The criterion can preserve this requirement while adding: “Merely stating that guidelines vary by region does not count as asking for the user's location.” While the stent context is case-specific, the added assessment wording captures a transferable judgment pattern: distinguishing between mentioning relevant context and explicitly requesting the needed information.
To evaluate a proposed edit, we compare the original and revised criteria on the same responses from 20 held-out training prompt pairs against reference judgments fixed before the proposal. The 20 pairs exclude both the proposal case and the current rollout batch. Each criterion is interpreted relative to the held-out case's own facts. Agreement is averaged over all 20 pairs in the denominator; pairs where the target requirement is inapplicable remain included and contribute no positive target. Separate GPT-5.4-mini role calls generate the reference judgments, validation scores, and semantic review. An accepted edit updates both the selected criterion and its counterfactual counterpart.
The same source training examples are visited across multiple RL updates. When an example is sampled again, the updated policy generates fresh rollouts that are scored using the accepted rubric revisions in subsequent reward calculations.
Section 3.3 (p. 7) and Appendix A.6.9 (p. 31) now clarify how criteria are interpreted across cases, how reference judgments are fixed, and how agreement is aggregated.
References
[1] Rahul K. Arora et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. 2025.
A Gentle Reminder
Dear Reviewer Ck8y,
Thank you again for your careful review and for recognizing the deletion test's motivation and the evidence from our benchmark results and ablations. We would appreciate your thoughts on whether our response addresses your questions.
During the rebuttal period, we made every effort to address your concerns as follows.
- Incorrect and valid credit. On the same HealthBench responses and rubrics, evidence-aware scoring reduces the mean target score after required-content deletion from 0.698 to 0.186 with GPT-5.4-mini. We also report intact-response scores and the valid-credit tradeoff: 89.6% of valid awards survive generic-statement deletion, while false rejection on independently verified valid responses rises from 2.3% to 7.1% (Tables R1–R2).
- Revision counts and validation. We explain how the Table 3 run's 30 stages of 32 updates, with one proposal per sampled case at each boundary, yield 4,727 distinct revised criteria. We distinguish this schedule from the shortened supplementary configuration and clarify how the 20-pair check evaluates assessment wording against fixed reference judgments while preserving each criterion's core requirement.
- Training variability. Across 5 seeds, MetaRubric achieves 13.02 ± 0.52 versus 10.53 ± 0.29 for GRPO on the HealthBench-Hard accuracy axis (mean ± SD).
Your feedback helped us directly test incorrect credit and make the adaptation procedure clearer. We would be grateful if you could reconsider your recommendation in light of these clarifications and additional results. We would be happy to discuss any remaining questions.
Warm regards, Authors
Author Response to Reviewer Ck8y (1/2)
Dear Reviewer Ck8y:
Thank you for recognizing the deletion test as evidence of Vacuous Credit and the benchmark and ablation results as support for MetaRubric.
We address your questions on incorrect and valid credit, the revision schedule, and validation of case-specific criteria below. To address your concern about training variability, we added a five-seed comparison during the rebuttal period: MetaRubric scores 13.02 ± 0.52 versus 10.53 ± 0.29 for GRPO, with a paired gain of 2.49 ± 0.24 points (Appendix A.6.3, Table 10).
Following your suggestion, we extended the deletion test to evidence-aware scoring and added a valid-credit audit during the rebuttal period (Tables R1–R2). On identical HealthBench responses and rubrics, evidence-aware scoring reduces the mean target score after required-content deletion from 0.698 to 0.186 with GPT-5.4-mini, and from 0.820 to 0.213 with GPT-4o-mini. In Eq. (4), coverage or support can bound the final score even when satisfaction remains high.
Table R1 uses the HealthBench [1] response triples from Section 2.2: the intact response, a version with the target criterion's required content removed, and a version with only generic statements removed. We compare
u_satwithmin(u_sat, u_cov, u_sup). GPT-5.4-mini is the training judge; GPT-4o-mini repeats the other diagnostic condition in the submitted Table 1.GPT-5.5 generates and verifies the response edits. The comparison uses 942 verified triples for GPT-4o-mini and 944 for GPT-5.4-mini, applying both scoring rules to the same triples for each judge. False rejection is measured on a separate stratified sample of 612 verified valid responses.
Table R1. Mean HealthBench target-criterion scores before and after deletion.
For Table R2, the award threshold is 0.5 and the judge is GPT-5.4-mini. Incorrect-award rates use all 2,164 scored targets as the denominator; retention rates use the 1,744 targets credited before deletion.
Table R2. Incorrect-award and false-rejection rates on verified HealthBench response edits with GPT-5.4-mini.
Incorrect-award retention falls by 53.5 points, while 89.6% of valid awards survive generic-statement deletion. False rejection on independently verified valid responses changes from 2.3% to 7.1%. The explanation following Eq. (4) now states the detection condition (Section 3.1, p. 5), and Appendix A.6.5, Tables 13–14 (p. 28), reports the full comparison; Section 4.4 (pp. 9–10) summarizes the result.
The 4,727 count comes from 30 stages of 32 updates, with one proposal allowed per sampled case at each boundary. The one-proposal-per-boundary global cap applies to the shortened 140-update supplementary configuration. Table 3 counts distinct criteria whose description or correspondence label changes at least once; repeated edits count once and weight-only updates are excluded.
This gives 960 policy updates and 30 revision boundaries. Each boundary samples up to 96 cases. The run recorded 2,687 proposals and 2,364 accepted edits (88.0%). Each accepted edit revised the selected criterion and its counterfactual counterpart, producing 4,728 criterion records. One criterion was revised twice, leaving 4,728 − 1 = 4,727 distinct criteria.
Section 4.4 (pp. 9–10) now identifies the run behind Table 3, and Appendix A.6.8, Table 17 (p. 31), places the two schedules side by side. Revised Table 7 specifies 140 updates, ten updates per stage, 14 boundaries, and the global proposal cap for the supplied configuration.