- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EnGRICH improves generative reward models (GRMs) using a training-time MetaCritic learned from limited human critiques. MetaCritic generates response-specific rubrics and checks critique coverage/correctness. These diagnoses provide process rewards and guide revision of failed assessments, while rubric learning extends supervision to preference-only data. Experiments with two backbones and seven reward-model benchmarks show consistent improvements. MetaCritic is not needed at inference.
Critical strengths
- Well-motivated supervision. Keypoint-level feedback addresses a real weakness of outcome-only GRM training: correct preferences can be supported by poor critiques. Reusing diagnostics for both rewards and revision is a sensible design.
- Consistent results. Table 1 shows gains over RM-NLHF in seven-benchmark averages: 66.25 → 69.22 (Qwen3-4B) and 67.21 → 70.32 (Gemma3-12B). Critique-quality and downstream evaluations provide additional support.
Critical weaknesses
- Rubric reliability remains uncertain. Keypoint-F1 cannot penalize incorrect claims outside the generated rubric or missing decisive keypoints. Figure 15 reports only 65%/71% final rubric all-keypoints-correct rates for the two backbones, based on an LLM evaluator. Independent human validation of rubric quality and coverage is missing.
- Extra computation is not controlled. EnGRICH adds rubric generation, verification, and guided revision. Figure 4 reports 197.61 s/step for Gemma3-12B versus 147.49 s/step for RM-NLHF, excluding offline initialization. Compute-matched baselines and an unguided-revision control would help isolate the contribution of the proposed mechanism.
Questions for authors
- Can the authors evaluate rubric correctness and completeness using independent human annotations across held-out domains, and quantify the impact of erroneous rubrics?
- Do the gains persist under matched GPU-hour/token budgets? Does guided revision beat an equally costly unguided revision?
2 comments
Response to Weakness 2 and Question 2
We train RM-NLHF and the unguided-revision control under the GPU budget of the submitted EnGRICH run and compare all three methods. All methods use the same training data and GRM starting checkpoint; the budget follows the RL training cost accounting in Section D.6, with offline preparation and initialization reported separately.
Unguided revision uses only the original input, the parent critique with its final choice removed, and a generic self-checking instruction; the revision prompt contains neither the rubric nor keypoint-level diagnoses. This control shares the same MetaCritic initialization and reference rubrics as full EnGRICH and retains its process reward and rubric learning. Triggering conditions, sampling settings, the one-attempt limit, and success-based replacement match the full method; neither prompt exposes the gold preference.
Table: Performance and GPU usage under matched online RL GPU budgets.
| Backbone | Method | Training updates | Online RL GPU-hours | Overall ↑ | CC-F1 ↑ | OPI ↓ |
|---|---|---|---|---|---|---|
| Qwen3-4B | RM-NLHF | 1,092 | 801.76 | 66.32 | 14.61 | 32.53 |
| Qwen3-4B | EnGRICH (unguided revision) | 955 | 801.97 | 68.57 | 17.86 | 29.59 |
| Qwen3-4B | EnGRICH | 937 | 802.32 | 69.22 | 18.01 | 28.09 |
| Gemma3-12B | RM-NLHF | 1,259 | 1,640.09 | 67.03 | 17.37 | 24.66 |
| Gemma3-12B | EnGRICH (unguided revision) | 959 | 1,638.60 | 69.64 | 18.01 | 18.86 |
| Gemma3-12B | EnGRICH | 934 | 1,640.60 | 70.32 | 19.06 | 17.14 |
EnGRICH's online RL GPU-hours are calculated from the update count and mean step time reported in the submission; its Overall and OPI are taken from the submission. For each backbone, CC-F1 follows the submission's definition and is computed on the intersection of reference-eligible HS3 examples where all methods in the table make valid, correct preference judgments. OPI uses each method's own valid, correct judgments. Offline costs are reported in our cost response to Reviewer H5Ao.
Under the matched online RL GPU budget, RM-NLHF's Overall changes by +0.07 / −0.18 percentage points and OPI by +0.12 / −0.05 percentage points relative to the submission, leaving both close to their original levels. Relative to RM-NLHF, EnGRICH scores 2.90 / 3.29 percentage points higher in Overall and 3.40 / 1.69 points higher in CC-F1, with OPI lower by 4.44 / 7.52 percentage points. Relative to unguided revision, Overall is 0.65 / 0.68 percentage points higher, CC-F1 is 0.15 / 1.05 points higher, and OPI is 1.50 / 1.72 percentage points lower. Budget matching and the unguided-revision control support the contribution of the proposed mechanism; EnGRICH's advantage cannot be explained solely by additional GPU compute during online RL.
We appreciate your careful assessment and constructive suggestions. Please let us know if any points need further clarification; we would be glad to discuss them.
We appreciate your thoughtful questions about rubric reliability and the fairness of the compute comparisons. Below, we provide independent human evaluation and comparisons under matched training GPU budgets.
Response to Weakness 1 and Question 1
Following Section D.4 of the submission, we randomly sample 100 questions each from HS3 validation and RM-Bench to evaluate post-training rubrics. Before viewing generated rubrics, three PhD annotators with relevant expertise independently propose reference evidence, retaining deduplicated items endorsed by at least two annotators. They then independently answer two blinded yes/no questions: whether each generated keypoint is correct and whether each reference item is correctly covered by the rubric. All judgments use majority voting. Per-keypoint correctness and evidence coverage are averaged per question.
For the same 200 questions, both backbones use fixed critiques from an independent frozen Qwen3-4B outcome-only GRM. The three annotators answer one additional yes/no question: whether each critique contains a substantive error not covered by the rubric. We report the proportion based on majority votes.
Table: Independent human evaluation of rubrics (%). The first three metrics report HS3 / RM-Bench (100 questions each); the last pools all 200 questions.
Human all-correct rates average 67.50% / 71.50% across sources, with RM-Bench coverage of 78.47% / 80.42%; 8.50% / 7.00% of critiques contain substantive errors not covered by the rubric. These results provide independent human evidence on the correctness and evidence coverage of the generated rubrics, and quantify the incidence of substantive errors outside rubric coverage in the audited critiques.