acceptodds
Under review as a conference paper at ICLR 2027

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

Abstract

Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside preference judgments, providing finer-grained evaluation signals. Their effectiveness depends heavily on critique reliability. However, existing GRM training typically uses final preference correctness as outcome supervision. Because the preference outcome space is highly constrained, unreliable critiques can still yield correct outcomes and thus be reinforced. Recent work leverages human critiques for process supervision, but such critiques are scarce and are often reduced to scalar rewards, leaving their fine-grained evaluative information underutilized. We argue that evaluative criteria learned from human critiques can be generalized to broader outcome-only preference data. To this end, we propose EnGRICH, a GRM training framework that pairs the GRM with a training-time MetaCritic learned from a small set of human critiques. MetaCritic constructs response-specific rubrics and uses them to evaluate the evidence coverage and correctness of generated critiques. The resulting signals provide both process rewards for fine-grained credit assignment and structured guidance for exploring better critiques. During GRM training, MetaCritic is further optimized to generalize human-grounded evaluative criteria to outcome-only data. At inference, the trained GRM operates independently. Experiments across seven reward-model benchmarks show that EnGRICH consistently improves over competitive baselines, while further analyses validate the effectiveness of its core mechanisms.

Primary area:
reinforcement learning
Keywords:
Generative Reward Models, Reinforcement Learning, Human Critiques

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_phjLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EnGRICH improves generative reward models (GRMs) using a training-time MetaCritic learned from limited human critiques. MetaCritic generates response-specific rubrics and checks critique coverage/correctness. These diagnoses provide process rewards and guide revision of failed assessments, while rubric learning extends supervision to preference-only data. Experiments with two backbones and seven reward-model benchmarks show consistent improvements. MetaCritic is not needed at inference.

Critical strengths
  1. Well-motivated supervision. Keypoint-level feedback addresses a real weakness of outcome-only GRM training: correct preferences can be supported by poor critiques. Reusing diagnostics for both rewards and revision is a sensible design.
  2. Consistent results. Table 1 shows gains over RM-NLHF in seven-benchmark averages: 66.25 → 69.22 (Qwen3-4B) and 67.21 → 70.32 (Gemma3-12B). Critique-quality and downstream evaluations provide additional support.
Critical weaknesses
  1. Rubric reliability remains uncertain. Keypoint-F1 cannot penalize incorrect claims outside the generated rubric or missing decisive keypoints. Figure 15 reports only 65%/71% final rubric all-keypoints-correct rates for the two backbones, based on an LLM evaluator. Independent human validation of rubric quality and coverage is missing.
  2. Extra computation is not controlled. EnGRICH adds rubric generation, verification, and guided revision. Figure 4 reports 197.61 s/step for Gemma3-12B versus 147.49 s/step for RM-NLHF, excluding offline initialization. Compute-matched baselines and an unguided-revision control would help isolate the contribution of the proposed mechanism.
Questions for authors
  1. Can the authors evaluate rubric correctness and completeness using independent human annotations across held-out domains, and quantify the impact of erroneous rubrics?
  2. Do the gains persist under matched GPU-hour/token budgets? Does guided revision beat an equally costly unguided revision?
2 comments

We appreciate your thoughtful questions about rubric reliability and the fairness of the compute comparisons. Below, we provide independent human evaluation and comparisons under matched training GPU budgets.

Response to Weakness 1 and Question 1

Following Section D.4 of the submission, we randomly sample 100 questions each from HS3 validation and RM-Bench to evaluate post-training rubrics. Before viewing generated rubrics, three PhD annotators with relevant expertise independently propose reference evidence, retaining deduplicated items endorsed by at least two annotators. They then independently answer two blinded yes/no questions: whether each generated keypoint is correct and whether each reference item is correctly covered by the rubric. All judgments use majority voting. Per-keypoint correctness and evidence coverage are averaged per question.

For the same 200 questions, both backbones use fixed critiques from an independent frozen Qwen3-4B outcome-only GRM. The three annotators answer one additional yes/no question: whether each critique contains a substantive error not covered by the rubric. We report the proportion based on majority votes.

Table: Independent human evaluation of rubrics (%). The first three metrics report HS3 / RM-Bench (100 questions each); the last pools all 200 questions.

Rubric backbonePer-keypoint correctness (question-averaged)Evidence coverageAll-correct rubric rateCritiques with uncovered errors
Qwen3-4B88.92 / 87.1382.28 / 78.4774.00 / 61.008.50
Gemma3-12B91.08 / 89.2384.63 / 80.4276.00 / 67.007.00

Human all-correct rates average 67.50% / 71.50% across sources, with RM-Bench coverage of 78.47% / 80.42%; 8.50% / 7.00% of critiques contain substantive errors not covered by the rubric. These results provide independent human evidence on the correctness and evidence coverage of the generated rubrics, and quantify the incidence of substantive errors outside rubric coverage in the audited critiques.

Response to Weakness 2 and Question 2

We train RM-NLHF and the unguided-revision control under the GPU budget of the submitted EnGRICH run and compare all three methods. All methods use the same training data and GRM starting checkpoint; the budget follows the RL training cost accounting in Section D.6, with offline preparation and initialization reported separately.

Unguided revision uses only the original input, the parent critique with its final choice removed, and a generic self-checking instruction; the revision prompt contains neither the rubric nor keypoint-level diagnoses. This control shares the same MetaCritic initialization and reference rubrics as full EnGRICH and retains its process reward and rubric learning. Triggering conditions, sampling settings, the one-attempt limit, and success-based replacement match the full method; neither prompt exposes the gold preference.

Table: Performance and GPU usage under matched online RL GPU budgets.

BackboneMethodTraining updatesOnline RL GPU-hoursOverall ↑CC-F1 ↑OPI ↓
Qwen3-4BRM-NLHF1,092801.7666.3214.6132.53
Qwen3-4BEnGRICH (unguided revision)955801.9768.5717.8629.59
Qwen3-4BEnGRICH937802.3269.2218.0128.09
Gemma3-12BRM-NLHF1,2591,640.0967.0317.3724.66
Gemma3-12BEnGRICH (unguided revision)9591,638.6069.6418.0118.86
Gemma3-12BEnGRICH9341,640.6070.3219.0617.14

EnGRICH's online RL GPU-hours are calculated from the update count and mean step time reported in the submission; its Overall and OPI are taken from the submission. For each backbone, CC-F1 follows the submission's definition and is computed on the intersection of reference-eligible HS3 examples where all methods in the table make valid, correct preference judgments. OPI uses each method's own valid, correct judgments. Offline costs are reported in our cost response to Reviewer H5Ao.

Under the matched online RL GPU budget, RM-NLHF's Overall changes by +0.07 / −0.18 percentage points and OPI by +0.12 / −0.05 percentage points relative to the submission, leaving both close to their original levels. Relative to RM-NLHF, EnGRICH scores 2.90 / 3.29 percentage points higher in Overall and 3.40 / 1.69 points higher in CC-F1, with OPI lower by 4.44 / 7.52 percentage points. Relative to unguided revision, Overall is 0.65 / 0.68 percentage points higher, CC-F1 is 0.15 / 1.05 points higher, and OPI is 1.50 / 1.72 percentage points lower. Budget matching and the unguided-revision control support the contribution of the proposed mechanism; EnGRICH's advantage cannot be explained solely by additional GPU compute during online RL.

We appreciate your careful assessment and constructive suggestions. Please let us know if any points need further clarification; we would be glad to discuss them.

Reviewer_H5AoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

EnGRICH trains generative reward models using limited human critiques and a larger set of outcome-only preference data. Its training-time MetaCritic generates response-specific rubrics and labels whether model critiques correctly cover, omit, or misstate each keypoint. These diagnoses provide both process rewards and guidance for revising assessments in all-incorrect rollout groups. The rubric generator is updated online, while the verifier remains frozen. Only the GRM is required at inference.

The paper reports improvements across seven reward-model benchmarks, two backbones, critique-quality evaluations, and downstream policy optimization. Relative to RM-NLHF, the main additions are explicit rubric-based diagnosis, online rubric optimization, and diagnosis-guided revision.

Critical strengths
  • The paper addresses an important limitation of outcome-only supervision: correct preference predictions can be supported by unreliable critiques.

  • The structured diagnoses serve concrete training functions, supporting both reward construction and revision.

  • Evaluation extends beyond preference accuracy to critique quality and downstream policy optimization. The reported improvements across two backbones provide encouraging evidence for the overall framework.

  • MetaCritic is used only during training, keeping deployment independent of the auxiliary modules.

Critical weaknesses
  • The contribution of diagnostic guidance is not isolated from additional generation and selection. Removing guided exploration also removes an extra generation opportunity and the rule that retains a revision only when its prediction is correct. The ablation therefore supports the revision procedure as a whole, but does not establish whether keypoint-level diagnoses outperform an additional independent sample or ordinary self-revision under the same acceptance rule.

  • The comparison with RM-NLHF does not establish an advantage under matched resource budgets. The methods largely share the underlying preference data, which is a useful control. However, EnGRICH uses different auxiliary supervision and additional computation. For Gemma3-12B, its reported online update time is approximately 34% higher than RM-NLHF, excluding offline annotation and initialization. Without matched-compute comparisons, the results cannot fully attribute the improvement to the proposed supervision mechanism.

  • Evidence for generalizing human-grounded evaluative criteria remains partial. On outcome-only data, rubric learning is supervised by rankings derived from preference correctness. This objective could improve discrimination without necessarily preserving complete human-grounded evidence coverage. The rubric-quality evaluation provides relevant evidence, but measures keypoint correctness rather than omitted evidence. The claim would be better supported by independent evaluations of both correctness and coverage before and after online training.

  • The incremental contribution over closely related approaches needs sharper empirical support. Human-critique supervision and its extension to outcome-only data are already central to RM-NLHF, while rationale matching and learned rubrics appear in other related work. Direct comparisons with R-Align or Rubric-ARM, or functional controls capturing their relevant mechanisms, would clarify the contribution. Reporting variability across training runs would also strengthen confidence in the performance differences and ablation results.

Questions for authors
  • Under the same extra-generation budget and success-based replacement rule, how does diagnosis-guided revision compare with independent resampling, ordinary self-revision, and revision using rubrics without diagnostic labels?

  • Does EnGRICH retain its advantage over RM-NLHF at matched total training compute? What are the respective offline annotation and auxiliary-initialization costs?

  • Does online rubric optimization improve both keypoint correctness and coverage on independently evaluated, out-of-domain examples?

  • Would adding the same revision procedure to RM-NLHF recover a substantial fraction of the reported gain?

6 comments

Thank you for your detailed review and suggestions for additional controls. We address diagnostic guidance, training costs, rubric generalization, and the distinction from related work below.

Response to Weakness 1 and Question 1

We freeze the same pre-online-training GRM and post-SFT MetaCritic and pair all strategies on 500 preselected failure groups per backbone (K=8; all assessments format-valid and preference-incorrect). The three revision strategies share a parent critique preselected uniformly at random, with its final choice removed; the two rubric-based strategies share the same rubric. All four strategies use identical sampling and length limits, at most one extra generation per group, and the success-based replacement rule in Section 3.4 of the submission. Inputs omit gold preferences and explicit signals about whether the original preference judgment was correct.

Single-attempt recovery rate = successful groups / 500 × 100%. A group succeeds when its extra output is both format-valid and preference-correct.

Table: Paired single-attempt recovery rates, computed on 500 shared failure groups per backbone. All scores are percentages.

Additional-generation strategyQwen3-4B recoveryGemma3-12B recovery
Independent resampling18.2019.60
Unguided revision20.6022.00
Rubric-only revision27.4029.60
Diagnosis-guided revision (EnGRICH)32.6035.40

Diagnosis-guided revision improves recovery by 14.40 / 15.80 percentage points over independent resampling, 12.00 / 13.40 over unguided revision, and 5.20 / 5.80 over rubric-only revision on Qwen3-4B / Gemma3-12B. The latter differences correspond to 26 / 29 net additional recoveries on the same 500 groups. Thus, under the same extra generation opportunity and success-based replacement rule, EnGRICH's diagnosis-guided revision is the most effective of the four evaluated strategies for single-attempt recovery by the frozen GRM.

Response to Weakness 2 and Question 2

We train RM-NLHF under the GPU budget of the submitted EnGRICH run, compare their performance, and report offline preparation, auxiliary initialization, and teacher API costs separately. Both methods use the same training data and GRM starting checkpoint; the budget follows the RL training cost accounting in Section D.6.

Table: Costs and performance under matched online RL GPU budgets.

BackboneMethodOffline preparation / initialization (GPU-hours)Offline teacher API (USD)Online RL GPU-hoursTraining updatesOverall ↑CC-F1 ↑OPI ↓
Qwen3-4BRM-NLHF29.45 / 43.9668.81801.761,09266.3214.9132.53
Qwen3-4BEnGRICH31.84 / 64.27137.33802.3293769.2217.9528.09
Gemma3-12BRM-NLHF55.76 / 56.3873.471,640.091,25967.0317.5124.66
Gemma3-12BEnGRICH59.68 / 96.47141.001,640.6093470.3219.0517.14

EnGRICH's online RL GPU-hours are calculated from the update count and mean step time reported in the submission. Its Overall and OPI are taken from the submission. For each backbone, CC-F1 follows the submission's definition and is computed on the intersection of reference-eligible HS3 examples where all methods in the table make valid, correct preference judgments.

EnGRICH's higher offline teacher API cost reflects its more detailed supervision: reference rubrics derived from human critiques and keypoint-level MATCH / MISSING / INCORRECT labels. These targets initialize the rubric generator and verifier, respectively, supporting process rewards and diagnosis-guided revision.

Under the matched online RL GPU budget, RM-NLHF's Overall changes by +0.07 / −0.18 percentage points and OPI by +0.12 / −0.05 percentage points relative to the submission, leaving both close to their original levels. EnGRICH still scores 2.90 / 3.29 percentage points higher in Overall and 3.04 / 1.54 points higher in CC-F1, with OPI lower by 4.44 / 7.52 percentage points. Thus, EnGRICH's advantage cannot be explained solely by additional GPU compute during online RL.

Response to Weakness 3 and Question 3

Following the sampling and deduplication protocol in Section D.4 of the submission, we conduct a paired evaluation on 100 randomly sampled questions from each of HS3 validation and RM-Bench. Initial is the rubric generator after SFT with 12K human critiques; Final is the same module after online training. Inputs, references, and decoding settings are shared across backbones and stages.

Before viewing generated rubrics, three PhD annotators with relevant expertise independently propose reference evidence. After deduplication, items endorsed as valid by at least two annotators form a fixed reference set. They then independently answer two yes/no questions, blind to model and stage:

  1. Per-keypoint correctness: Is this generated keypoint correct given the task and candidate responses?
  2. Evidence coverage: Is this independent reference item correctly covered by the rubric?

Each item requires at least two yes votes, without adjudication. We average per-question fractions over generated keypoints (correctness) or fixed reference items (coverage). Nonempty rubrics with every keypoint judged correct count toward the all-correct rate, matching Figure 15's Keypoint Correctness.

Table: Human evaluation before and after online training (%). Each cell reports HS3 / RM-Bench, with 100 questions per source.

BackboneStagePer-keypoint correctness (question-averaged)Evidence coverageAll-correct rubric rate
Qwen3-4BInitial79.83 / 77.4265.73 / 61.3830.00 / 35.00
Qwen3-4BFinal88.92 / 87.1382.28 / 78.4774.00 / 61.00
Gemma3-12BInitial83.27 / 80.1869.12 / 64.3738.00 / 34.00
Gemma3-12BFinal91.08 / 89.2384.63 / 80.4276.00 / 67.00

On RM-Bench, coverage increases by 17.09 / 16.05 percentage points and per-keypoint correctness by 9.71 / 9.05 percentage points for Qwen3-4B / Gemma3-12B. Both metrics also improve on HS3, and all-correct rates rise on both sources. Coverage against the independent references captures more valid evidence, complementing Figure 15's evaluation of generated-keypoint correctness. Improved correctness and coverage across both sources support cross-dataset transfer of human-grounded evaluative evidence after online optimization.

Reviewer_eHoNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a limitation in GRMs trained solely on final preference labels: they may assign high rewards to critiques that are factually incorrect or lack proper justification. To mitigate this, the authors propose EnGRICH. This framework initializes a MetaCritic using a small set of preference data that includes human-provided critiques. A rubric generator constructs evaluation keypoints for each response pair, and a verifier assesses whether the GRM critique covers these keypoints correctly.

These point-wise diagnostic assessments serve two purposes: they are aggregated into process-based rewards, and they are used to refine critiques that were initially flawed. Furthermore, the rubric generator continues to learn during training, allowing it to generalize human evaluation standards to datasets that contain only preference labels. MetaCritic is employed exclusively during the training phase; at deployment, the system reverts to a standalone GRM. EnGRICH outperforms established baselines across two model backbones and seven reward-model benchmarks, achieving overall scores of 69.22 and 70.32, respectively. It consistently yields improvements in both critique quality and downstream policy optimization.

Critical strengths
  1. The method is practically designed. MetaCritic's keypoint-level diagnoses are used for both process credit assignment and guided exploration. Moreover, MetaCritic is used only during training and introduces no additional deployment-time modules for the final GRM.
  2. The empirical evaluation is comprehensive. EnGRICH outperforms the controlled baselines across all seven main benchmarks with both backbones. The ablation studies, direct critique evaluation, and downstream policy-optimization results also show broadly consistent improvements.
Critical weaknesses
  1. The method relies heavily on an external teacher. Instead of directly using human critiques, the paper employs Qwen3.7-Max to convert them into reference rubrics and verifier labels, but does not validate the accuracy of these synthetic annotations.
  2. Guided exploration generates an additional revision for all-failed groups, increasing computational cost, while improving Overall performance by only about one percentage point on both backbones relative to w/o Guided Exploration.
Questions for authors

See Weaknesses.

4 comments

Thank you for your careful review and constructive feedback on the reliability of teacher-generated supervision and the contribution of guided exploration. We address these points below.

Response to Weakness 1

We independently audit the reference keypoints and verifier-initialization relation labels produced by Qwen3.7-Max. We randomly select 100 training inputs from the human-critique subset DHD_H. For each backbone, we select one verifier-initialization record per input, with multiple keypoint labels per record. Three PhD annotators with relevant expertise independently complete two selection tasks:

  1. Reference keypoints: yes/no. Is the keypoint supported by the source human critique and correct given the task and candidate responses?
  2. Relation labels: three choices. Given the task, a rubric produced by the initialized rubric generator, and the initial GRM critique, assign each keypoint MATCH (complete and correct coverage), MISSING (omitted or incompletely covered), or INCORRECT (an explicit conflict), following Figure 7 in the submission. Teacher labels are hidden from annotators.

Table: Human audit of teacher-generated supervision (%).

Human-audit metricHuman-audit result
Source-supported and correct reference keypoints94.83
Qwen3-4B initialization records: teacher-label macro-F189.46
Gemma3-12B initialization records: teacher-label macro-F190.31

For the first metric, at least two yes votes establish a source-supported, correct keypoint; correct-keypoint fractions are computed per question and averaged across the 100 inputs. Teacher-label macro-F1 pools all keypoints from each backbone's 100 records. Against each annotator's choices, we compute the unweighted mean of the three class F1 scores, then average macro-F1 across the three annotators.

These results support the reliability of both forms of MetaCritic initialization supervision: teacher-generated keypoints are largely source-faithful and correct, and relation labels agree closely with independent human judgments. We will include this human-audit experiment and its results in the revised submission.

Response to Weakness 2

We introduce guided exploration to make fuller use of rubrics and their keypoint-level diagnoses: the same information used to compute process rewards also tells the GRM what is missing or incorrect in its critique. Section 3.4 of the submission explains that when all assessments in a group are format-valid but preference-incorrect, rewards and group-relative advantages remain zero even with the process reward. We therefore use the diagnoses to guide at most one revision in each all-failed group; Proposition A.2 in Section A.4 proves that a successful revision restores nonzero group-relative advantages.

We summarize the ablation results from Table 2 and Table 8 in the submission.

BackboneΔOverallΔRB2ΔCC-F1ΔOPI
Qwen3-4B+0.95+4.47+0.87−2.39
Gemma3-12B+1.02+4.15+1.14−2.78

Each Δ is full EnGRICH minus w/o Guided Exploration. Overall, RB2, and OPI differences are in percentage points; CC-F1 differences are in score points.

Guided exploration improves most benchmarks, with particularly large gains on commonly used reward-model benchmarks such as RB2. RB2 improves by 4.47 / 4.15 percentage points on Qwen3-4B / Gemma3-12B. Six of seven benchmarks improve on each backbone, with Overall gains of 0.95 / 1.02 percentage points.

Guided exploration also improves critique quality. CC-F1 increases by 0.87 / 1.14 points, and OPI decreases by 2.39 / 2.78 percentage points, corresponding to 7.84% / 13.96% relative reductions from the variant without guided exploration. Critiques thus agree more closely with human reference evidence, while low-quality critiques account for a smaller fraction of correct judgments.

Guided exploration controls additional computation. Revision reuses existing rubrics and cached keypoint diagnoses and is attempted at most once per group, only when all initial assessments are format-valid but preference-incorrect (Section A.5).

These results support the contribution of guided exploration: making further use of keypoint-level diagnoses to improve both preference judgment and critique quality.

Thank you again for the time and care you have devoted to our paper. We would be happy to discuss any further questions.

Thank you for the detailed response and the additional human audit, which largely address my concerns. I will maintain my positive rating 3.

Thank you for taking the time to review our rebuttal! We are glad that our responses have addressed your concerns. Your thoughtful and constructive feedback has helped us further strengthen our revised manuscript!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.