acceptodds
Under review as a conference paper at ICLR 2027

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

Abstract

Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).

Primary area:
reinforcement learning
Keywords:
rubrics, checklists, reinforcement learning, reward hacking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iXx9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper argues that rubric-based RL often optimizes recall through rubric coverage, without adequately penalizing unnecessary or incorrect content, creating reward hacking behaviors

  • The authors divide responses into sentence-like units and estimate precision using rubric-based necessity credits and LLM judgments of factual correctness. They introduce the FORK to balance precision and recall accordingly

  • Experiments compare FORK against standard rubric-RL, OnlineRubrics, universal rubrics, and holistic preference rewards. Results show higher F1 and GPT-5.4-judged win rates against the baseline methods

Critical strengths
  • Creating absence-based rubrics to balance precision and recall is difficult, and the paper innovatively introduces unit-level judgments to address this problem through structured necessity and factuality checks on responses from a stronger judge. This is an intuitive and practical approach that help address the recall-focused optimizations in rubric-RL

  • Solid experiment and ablations results across healthcare and science domains, supported by precision, recall, and win rates judged by a stronger LLM.

Critical weaknesses
  • While the authors demonstrate improvements in F1 under a separate evaluation grader, FORK's optimizes an objective built from closely related components, making it less clear how much these gains reflect quality improvements rather than alignment between the reward shaping and eval metrics. More qualitative analysis of the where the improvement comes against baseline methods (OnlineRubrics, Holistic Preference) from would strengthen author's argument

  • Related to the first point, win rate provides a quality metric less directly tied to reward shaping, but the rubric-free GPT-5.4 judge may miss task-specific requirements in these specialized domains. A rubric-informed preference evaluation or any audit of the judge’s ability to evaluate those tasks would help clarify

Questions for authors

As commented in the weakness section, I'm wondering if there's any more qualitative analysis against baseline methods (OnlineRubrics, Holistic Preference) on where the improvements come from, and whether the win rate holds for a rubric-informed judge.

1 comment

We thank the reviewer for the constructive and detailed comments.

Q1. I'm wondering if there's any more qualitative analysis against baselines (OnlineRubrics, Holistic Preference) on where the improvements come from.

A. OnlineRubrics slightly improves the rubric coverage, but since it is still a recall-based objective, the response contains unnecessary content and are verbose. We also observe the same phenomenon as the standard coverage reward where the model first generates an initial response, add another response, and repeat this process. Holistic Preference is not verbose, but the response shows low rubric coverage and are less specific. Compared to these two baselines, FORK mostly produces necessary contents and are less verbose, while retaining the rubric coverage. We add model responses of OnlineRubrics and Holistic Preference in pages 18-20 in Appendix D.1. The response of FORK is already provided in page 15.

Q2. Does the win rate hold for a rubric-informed judge?

A. We adapt the rubric-free judge prompt to include rubric coverage as one of the evaluation criteria. Specifically, the judge prompt originally considers five dimensions (completeness, factual correctness, conciseness, relevance, and safety) and we replace the completeness dimension into the rubric coverage. The following table shows that FORK still outperforms baselines in terms of rubric-informed win rate against the base policy.

MethodRubric-informed win rate
Qwen2.5-7B-InstructN/A
Standard14.9
OnlineRubrics43.5
Universal Rubrics34.5
Reweighted Absence-Based Rubrics29.8
Holistic Preference79.3
Generic Criteria73.0
Offline Criteria68.1
Online Criteria58.4
FORK (β=1)78.7
FORK (β=1.25)79.8
Reviewer_jeqHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a failure mode of rubric-based reinforcement learning: optimizing rubric coverage alone can produce increasingly verbose responses with lower content precision. The proposed method, FORK, decomposes responses into content units, allocates credit through rubric–unit matches, and combines estimated necessity with factual correctness to measure precision. It then combines precision and rubric coverage through an Fβ training reward. Experiments on medical and scientific tasks show that this approach can mitigate degradation under coverage-based rewards. In particular, the medical results demonstrate improved precision and shorter responses while largely preserving coverage, although coverage–precision trade-offs remain on some tasks.

Critical strengths
  1. The paper investigates a reward imbalance observed during actual training. Its reinforcement learning experiments demonstrate that increasing rubric coverage can coincide with declining content precision and substantially longer responses, and evaluate a practical mitigation. This gives the findings direct relevance to policy training.

  2. The main experiments demonstrate meaningful improvements. On the medical task, FORK (β=1.25) improves precision from 25.5 under the standard rubric reward to 49.0 while largely preserving coverage (35.7→35.8) and substantially reducing response length. Thus, the gains cannot be explained entirely by sacrificing coverage to produce shorter answers.

  3. The evaluation includes several relevant baseline families and comparisons of precision estimators, reward combinations, and parameter choices. These experiments provide substantial initial evidence for jointly optimizing coverage and precision, going beyond a proof of concept for a single reward formula.

Critical weaknesses
  1. The independent contribution of the credit-allocation mechanism is not fully isolated. Existing ablations support precision-aware rewards, but the complete method simultaneously introduces necessity assessment, factual checking, and constraints on verbosity. Comparisons against a simple length penalty at similar response lengths, and against necessity-only or factuality-only rewards, would clarify the gains attributable to the proposed mechanism.

  2. Necessity estimation depends on rubric completeness. Under the definition in the main text, a content unit matching no rubric receives no credit, even if it contains useful supplementary information. The deletion experiment primarily checks whether rubric coverage is preserved, which cannot reliably detect the loss of valuable information outside the rubric. Reliability under incomplete rubrics therefore remains insufficiently established.

  3. Preference evaluation is sensitive to the evaluation prompt. An alternative prompt substantially changes some win rates and relative rankings; for example, the medical win rate of Offline Criteria changes from 20.7% to 88.6%. Moreover, methods are primarily compared separately against the initial model, leaving FORK’s direct preference advantage over strong baselines less clear.

  4. Cross-task results reveal a coverage cost that needs further explanation. On ResearchQA, coverage decreases from 64.4 under the standard rubric reward to 58.9. Although preference win rates improve, it remains unclear whether the omitted content is redundant or important. The scientific evaluation also excludes some strong baselines included in the medical experiments, limiting conclusions about cross-domain advantages.

Questions for authors
  1. At comparable response lengths, does FORK retain an advantage over the original rubric reward with a length penalty? How much of the improvement remains when adding only necessity or factual correctness to the reward?

  2. When rubrics deliberately omit important information, does the credit-allocation mechanism penalize that information? Could controlled rubric-removal experiments, supported by independent human judgments, assess this risk?

  3. Could the authors report the complete baseline comparison under the alternative evaluation prompts and directly compare FORK against strong baselines such as Holistic Preference and Generic Criteria? Do these comparisons support the same method ranking?

  4. Which information accounts for the reduced coverage on ResearchQA? Is it primarily redundant detail or information important to answering the question? At comparable coverage, does FORK retain a precision or preference advantage?

2 comments

We thank the reviewer for the constructive and detailed comments.

Q1. At comparable response lengths, does FORK retain an advantage over the original rubric reward with a length penalty? How much of the improvement remains when adding only necessity or factual correctness to the reward?

A. To add length penalty term in the standard coverage reward, we adapt L1-Max [1] as follows: r(y,R,ngold)=Coverage(y,R)⋅clip⁡(α⋅(ngold−ny)+δ,0,1)r(y, \mathcal{R}, n_{\mathrm{gold}}) = \mathrm{Coverage}(y, \mathcal{R}) \cdot \operatorname{clip}(\alpha \cdot (n_{\mathrm{gold}} - n_y) + \delta, 0, 1) where α\alpha controls the penalty for length violations. Following the original work [1], we use α\alpha=0.0003 and δ\delta=0.5, and sample ngoldn_{\mathrm{gold}} from U(nmin,nmax)U(n_{\mathrm{min}}, n_{\mathrm{max}}) where nminn_{\mathrm{min}} and nmaxn_{\mathrm{max}} are 1st and 99th percentile of number of tokens of base policy responses, respectively. The following table shows that the length penalty maintains the response length while improves the rubric coverage, but it does not improve precision (i.e., necessity and correctness) compared to the base policy since there is no explicit training signal, leading to nearly 50% win rate against the base policy. In contrast, FORK improves precision as well, leading to higher win rate.

MethodRecallPrecisionF1Win RateLength
Qwen2.5-7B-Instruct22.646.626.3N/A2,432
Standard35.725.526.43.19,195
Standard w/ length penalty32.546.934.854.12,227
FORK (β=1)32.553.237.568.03,533

Next, when we ablate necessity or factual correctness term from FORK, precision is significantly reduced and the response gets verbose, leading to a very low overall win rate, which supports the effectiveness of our method.

MethodRecallPrecisionF1Win RateLength
FORK (β=1.25)35.849.038.655.44,392
FORK (β=1.25) w/o necessity36.124.826.22.19,300
FORK (β=1.25) w/o correctness27.026.720.76.18,267

Q2. When rubrics deliberately omit important information, does the credit-allocation mechanism penalize that information?

A. Yes, FORK reward discourages generating contents absent from the rubric set. However, following the definition of rubrics in the previous works [1,2], we presume that the rubric has a comprehensive coverage, and we manually confirm that most rubrics that we used during training and testing are complete. Although studying the influence of rubric incompleteness is highly important, we would like to clarify that it is an independent research topic, thus out of scope from our work.

[1] Rubrics as rewards: Reinforcement learning beyond verifiable domains. Gunjal et al., 2026.
[2] Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. Li et al., 2026.

Q3. Could the authors report the complete baseline comparison under the alternative evaluation prompts and directly compare FORK against strong baselines? Do these comparisons support the same method ranking?

A. We provide rubric-free evaluation results of four additional rubric-coverage reward baselines using a different judge prompt in the following table. Compared to the results reported in Table 4, rubric-coverage rewards consistently underperform those. The rankings are mostly preserved except Offline Criteria on the RubricHub-Medical test set as the reviewer pointed out.

MethodRubricHub-MedicalHealthBench
Standard17.716.7
OnlineRubrics51.964.1
Universal Rubrics48.245.2
Reweighted Absence-Based Rubrics34.735.4
FORK (β=1)84.888.6

We also compare FORK against Holistic Preference and Generic Criteria in the following table. The results consistently indicate the overall preference on our method.

Win Rate of FORK (β=1)
vs. Holistic Preference53.9
vs. Generic Criteria71.1

Q4. Which information accounts for the reduced coverage on ResearchQA? At comparable coverage, does FORK retain a precision or preference advantage? The science domain also excludes some strong baselines.

A. We sample 100 ResearchQA questions in the test set which contains total 764 rubric items and collect 209 rubric items where Standard scores higher rubric satisfaction than FORK. (Note that rubric satisfaction in ResearchQA is scored on a 5-point scale, not binary.) We let Claude Opus 5.5 to decide whether FORK misses simple details or important information compared to Standard. The result indicates that 69.7% of the cases miss simple details, while the remaining 30.3% miss important information.

We also compare FORK and Standard using full evaluation metrics on ResearchQA. The result shows that advantage in precision, F1 score, response length, and overall preference are retained.

MethodRecallPrecisionF1Win RateLength
Standard64.425.234.316.17,520
FORK (β=1.25)58.936.042.167.14,095

Lastly, we additionally report performance of two stronger baselines (Holistic Preference and Generic Criteria) on the RubricHub-Science test set. FORK achieves the highest F1 score and the second best win rate.

MethodRecallPrecisionF1Win RateLength
Qwen2.5-7B-Instruct27.341.829.6N/A2,673
Holistic Preference27.941.130.166.33,124
Generic Criteria30.945.433.152.93,226
FORK (β=1.25)30.445.133.461.53,038
Reviewer_V759OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies reward hacking in rubric-based reinforcement learning. The authors observe that conventional rubric-coverage rewards primarily optimize recall: they reward the inclusion of desired content but do not adequately penalize unnecessary or factually incorrect additions. To address this imbalance, they propose FORK, which combines rubric recall with a sentence-level precision objective through an Fβ score. Precision measures whether response units are both necessary and factually correct, with necessity estimated through rubric-based credit allocation and an adaptive threshold. Experiments in medical and scientific domains demonstrate improved response precision and preference scores, alongside reduced verbosity.

Overall, I find the paper’s central observation particularly insightful and the problem highly important.

Critical strengths

An insightful framing of rubric-based reward hacking. Interpreting conventional rubric coverage as a recall objective provides a clear explanation for why models can improve rubric scores while producing increasingly verbose or lower-quality responses. This perspective offers a useful foundation for designing better training objectives.

A well-motivated precision objective. Requiring generated content to be both necessary and factually correct directly addresses aspects of response quality that coverage alone does not capture. The combination of precision and recall is intuitive and aligns well with the goal of producing complete yet focused answers.

Encouraging empirical results. FORK substantially reduces verbosity and improves precision relative to conventional coverage rewards. For example, on RubricHub-Medical, the β = 1.25 configuration reduces average response length from 9,195 to 4,392 characters while maintaining essentially the same rubric recall. Improvements in preference scores on external benchmarks further support the practical promise of the approach.

An important problem with broader relevance. As rubric-based rewards are applied to open-ended tasks, distinguishing coverage from the quality of all generated content becomes increasingly important. I consider this paper’s formulation a valuable contribution even beyond the specific implementation evaluated here.

Critical weaknesses

Missing simple baselines that directly address verbosity. The most important omission is a rubric-coverage reward combined with a length penalty. Such a baseline could discourage excessive content without requiring sentence-level decomposition, rubric-to-unit matching, threshold selection, and factuality assessment. Comparing against this alternative across several penalty strengths would help establish whether FORK achieves a better balance between coverage and response quality.

Limited evidence across policy models. The experiments use Qwen2.5-7B-Instruct as the policy model. Although evaluation spans two domains and includes external benchmarks, it remains unclear whether the same benefits hold for other model families and sizes, particularly stronger models with different verbosity and factuality characteristics.

The advantage over the strongest alternative is less pronounced. FORK shows large improvements over conventional coverage rewards, but its preference-score gains over Holistic Preference are considerably smaller. Its simultaneous improvements in recall are valuable, yet a more focused comparison would help clarify when the additional complexity of FORK is warranted.

The computational trade-off is insufficiently characterized. FORK involves several grading operations beyond standard rubric evaluation. The paper would benefit from reporting the additional grading cost and training time, allowing readers to assess the practical value of its quality improvements.

Questions for authors

How does FORK compare with rubric-coverage rewards augmented by a length penalty? When varying the penalty strength, can this simpler baseline achieve a comparable balance between recall, response length, and overall quality?

Do the observed improvements extend to other policy model families or sizes? In particular, would FORK provide similar benefits for stronger models that exhibit less severe verbosity under coverage-based training?

Compared with Holistic Preference, in which settings does FORK offer the clearest advantage? Could you provide a direct comparison of their responses to better characterize the differences in completeness, correctness, and unnecessary content?

What additional grading cost and training time does FORK require relative to standard coverage rewards and Holistic Preference? Which parts of the evaluation pipeline account for most of this overhead?

How does the necessity mechanism handle useful content that is absent from an incomplete rubric? Since units with no rubric credit cannot exceed a positive necessity threshold, could the method discourage relevant explanations or caveats that the rubric fails to anticipate?

2 comments

We thank the reviewer for the constructive and detailed comments.

Q1. How does FORK compare with rubric-coverage rewards augmented by a length penalty?

A. To add length penalty term in the standard coverage reward, we adapt L1-Max [1] as follows: r(y,R,ngold)=Coverage(y,R)⋅clip⁡(α⋅(ngold−ny)+δ,0,1)r(y, \mathcal{R}, n_{\mathrm{gold}}) = \mathrm{Coverage}(y, \mathcal{R}) \cdot \operatorname{clip}(\alpha \cdot (n_{\mathrm{gold}} - n_y) + \delta, 0, 1) where α\alpha controls the penalty for length violations. Following the original work [1], we use α\alpha=0.0003 and δ\delta=0.5, and sample ngoldn_{\mathrm{gold}} from U(nmin,nmax)U(n_{\mathrm{min}}, n_{\mathrm{max}}) where nminn_{\mathrm{min}} and nmaxn_{\mathrm{max}} are 1st and 99th percentile of number of tokens of base policy responses, respectively. The following table shows that the length penalty maintains the response length while improves the rubric coverage, but it does not improve precision (i.e., necessity and correctness) compared to the base policy since there is no explicit training signal, leading to nearly 50% win rate against the base policy. In contrast, FORK improves precision as well, leading to higher win rate.

MethodRecallPrecisionF1Win RateLength
Qwen2.5-7B-Instruct22.646.626.3N/A2,432
Standard35.725.526.43.19,195
Standard w/ length penalty32.546.934.854.12,227
FORK (β=1)32.553.237.568.03,533

[1] L1: Controlling how long a reasoning model thinks with reinforcement learning. Aggarwal et al., 2025.

Q2. Would FORK provide similar benefits for stronger models that exhibit less severe verbosity under coverage-based training?

A. Yes. We use Qwen3.5-4B as a base policy, which produces less verbose response under standard coverage reward-based training (Post-trained Qwen2.5-7B-Instruct produces 3.8x longer response compared to the base policy, whereas post-trained Qwen3.5-4B produces 2.1x longer response). Compared to the standard coverage reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, leading to significantly higher win rate. The following table shows the results on the RubricHub-Medical test set.

MethodRecallPrecisionF1Win RateLength
Qwen3.5-4B41.151.342.8N/A4,013
Standard50.535.839.115.28,406
FORK (β=1.25)47.051.146.556.94,676

Q3. Compared with Holistic Preference, in which settings does FORK offer the clearest advantage?

A. Following the reviewer's suggestion, we compare FORK and Holistic Preference in terms of rubric coverage (i.e., completeness), factual correctness, and necessity. The following table shows that FORK provides great advantages in completeness and necessity, but not in correctness.

MethodCoverageCorrectnessNecessity
Holistic Preference27.588.450.4
FORK (β=1)32.578.062.7
FORK (β=1.25)35.878.957.8

Q4. What additional grading cost and training time does FORK require relative to standard coverage rewards and Holistic Preference? Which parts of the evaluation pipeline account for most of this overhead?

A. If we give the reward systems the same model responses, FORK requires 2.5x longer grading time than the standard coverage reward. However, FORK requires only 1.2x longer training time since the standard coverage reward makes the policy model to generate much longer responses. Holistic Preference shows the fastest grading and training time since it only requires an LLM judge to decide which response is preferred.

MethodTraining time / step (s)Grading time / response (s)
Standard5170.11
Holistic Preference2530.06
FORK (β=1.25)6080.27

In addition, when we analyze the evaluation pipeline of FORK, 45% of its grading time is spent to measure rubric coverage, 33% to measure necessity, and 22% to measure correctness.

Q5. How does the necessity mechanism handle useful content that is absent from an incomplete rubric?

A. This is a great question, and FORK reward discourages generating contents absent from the rubric set. However, following the definition of rubrics in the previous works [1,2], we presume that the rubric has a comprehensive coverage, and we manually confirm that most rubrics that we used during training and testing are complete. Although studying the influence of rubric incompleteness is highly important, we would like to clarify that it is an independent research topic, thus out of scope from our work.

[1] Rubrics as rewards: Reinforcement learning beyond verifiable domains. Gunjal et al., 2026.
[2] Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. Li et al., 2026.

Thank you for the detailed response. The rebuttal has addressed most of my concerns. I will maintain my positive assessment and keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.