acceptodds
Under review as a conference paper at ICLR 2027

Conformal Rubrics: Calibrating Rubric Rewards with Human Score for Policy Learning

Abstract

Rubrics provide explicit criteria, such as correctness and helpfulness, for evaluating language model responses and constructing rewards for policy learning. Recent methods improve these rewards by refining the criteria or adjusting how their scores are combined. However, specifying what to evaluate does not ensure that a judge model assigns scores consistent with human judgments. Human ratings provide a reference for correcting these scores, but collecting ratings for every new response during policy training would be costly. We therefore propose Conformal Rubrics, which use a fixed human-rated dataset to calibrate rubric rewards for downstream policy training. The key challenge is estimating how humans would score responses on new tasks, where queries differ from those in the annotated dataset. We train a predictor on the source ratings and use its prediction errors to construct a conformal interval to include possible human scores for each rubric with a pre-specified coverage level. We then calibrate judge scores against these intervals by clipping scores outside them to the nearest boundary and retaining those inside. The corrected scores are then combined into rewards using existing rubric-based methods. Experiments show that calibration with the human-rated dataset HelpSteer reduces judges' rubric scoring errors and improves policy performance on different datasets, such as HelpSteer2, WildChat, and AlpacaEval-2. Our code is available at anonymous GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, Rubrics, Data fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2HvUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Conformal Rubrics, which uses a fixed human-rated dataset to calibrate rubric rewards for reinforcement learning. The method trains a human-score predictor, constructs score intervals using residual quantiles and quantile regression, and clips judge scores outside these intervals to the nearest boundary before policy training. The paper analyzes conditions for coverage and reduced scoring error, and evaluates cross-dataset calibration and downstream reinforcement learning. Calibration reduces errors against human ratings and improves policy preference win rates under general rubrics, while benefits under the original criteria of specialized benchmarks are inconsistent.

Critical strengths
  1. The paper addresses a practical problem. Reusing existing human ratings to correct training rewards reduces the need for continual annotation. The method preserves the existing judge and reward aggregation procedure, making it modular.

  2. Improvements in score calibration are directly supported by human labels. In Table 2, QR calibration reduces average MSE and MAE for all three scorers; for example, DeepSeek-V4-Flash's average MAE decreases from 0.8157 to 0.5901. Reverse-transfer and same-domain experiments provide further evidence of effective score correction.

  3. The paper includes actual reinforcement learning experiments with three policy models and three reward aggregation methods, alongside training-judge variations and results from two evaluation judges. These experiments demonstrate practical training benefits of the complete calibration pipeline under the general criteria tested.

Critical weaknesses
  1. The method's independent contribution is not sufficiently separated from additional human supervision. The proposed pipeline trains an auxiliary predictor using HelpSteer labels, whereas the main baselines do not receive corresponding additional calibration with these labels during the experiments. The main results therefore combine the effects of additional supervision and interval clipping. Comparisons among FW, QR, and mixed intervals help distinguish interval designs, but omit direct predictor rewards, simple predictor–judge score combinations, and existing human-based calibration methods. Since related work already studies human-score prediction and interval-based score correction, this gap also limits the assessment of incremental novelty. The effect of annotation-set size is not analyzed.

  2. Coverage properties of the final mixed intervals are insufficiently established. Equation (6) fits quantile models on the calibration set, and Equation (7) mixes their outputs with fixed-width intervals without subsequent independent conformal calibration. The standard fixed-width coverage guarantee does not automatically extend to this mixture. Although Theorem 1 characterizes a coverage lower bound, it contains error terms that are neither estimated nor controlled. Empirical coverage and interval width are also absent, leaving it unclear whether the specified coverage level is achieved in practice.

  3. Calibration effectiveness on newly generated policy responses is not directly validated. Experiments against human ratings use existing dataset responses, whereas downstream policies are primarily assessed by LLM judges. As the policy changes, its response distribution may depart from the source data used to fit the calibrator. The experiments do not establish whether the fixed calibrator continues to reduce human-score errors on these responses or whether human evaluations support the policy preference gains.

  4. Benefits on specialized tasks have a significant limitation. In Table 5, Llama-3.1-8B with Raw + Ours achieves a 56.84% average win rate under the general training rubrics, but its original-rubric scores decrease on all five specialized benchmarks, with the average falling from 42.31 to 41.12. The authors acknowledge this limitation, but it still weakens the method's practical value for specialized tasks and calls for an explanation of the conflicting evaluations.

Questions for authors
  1. With the same human-rated data, how does the proposed method compare with direct predictor rewards, simple predictor–judge score combinations, and interval-midpoint scores? Could you report both scoring errors and downstream training results, and examine how their relative performance changes with the amount of human-rated data?

  2. Do the final mixed intervals have a coverage guarantee beyond Theorem 1's bound involving unknown error terms? Could you report empirical coverage and average width for FW, QR, and mixed intervals on independent same-distribution data and HelpSteer2, and compare against an independently conformalized alternative?

  3. Could a small human-annotated sample of responses from different training stages, used only for evaluation, establish calibration's effect on scoring error, interval coverage, and human preferences for the final policy?

  4. What explains the improved general-rubric win rates but reduced original-rubric scores on specialized tasks? Could representative failure cases distinguish domain-transfer errors in the calibrator from limitations of the general criteria in representing specialized task requirements?

Reviewer_d3XoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Conformal Rubrics, a way to calibrate rubric rewards for downstream policy training using fixed human-rated dataset, without collecting new human labels during training. For each rubric, a human-score predictor ff (a frozen FsfairX-LLaMA3-RM-v0.1) is fit on HelpSteer dataset. Residuals on a calibration split define specific fixed-width (FW) split-conformal interval ∣f(z)−q^,f(z)+q^∣|f(z) - \hat{q}, f(z) + \hat{q}| (Section 4.3) and also to anticipate the errors of ff in query-response pairs, the paper employs quantile-regression (QR) to estimate bounds conditional on each of it (Section 4.3). QR heads trained on the same calibration residuals give input-dependent bounds. During GRPO, a judge score outside the interval is clipped to the nearest endpoint before the usual rubric aggregation (raw, importance-weighted, or dropout; Table 4). Theorem 1 lower-bounds coverage under shift by 1−α−ηλ(z)−2Δ(z)1 - \alpha - \eta_\lambda (z) - 2 \Delta(z) (Equation 14) and theorem 2 gives the condition under which clipping reduces expected absolute error (Equation 15). Subsequently, calibration with HelpSteer reduces average MSE against HelpSteer2 human scores by 23-52% for DeepSeek-V4-Flash, Gemini 3.1 Flash-Lite, and ArmoRM (Table 2). In GRPO with a GPT-4o training judge on 3 rubrics, calibrated policies win at scale of 56-66% in pairwise comparisons against their uncalibrated counterparts on HelpSteer2-test, WildChat, and AlpacaEval-2. Although, nn 5 expert-domain benchmark, the scores under the datasets' original rubric change only marginally and inconsistently (Table 5).

Critical strengths

S1. Novel practical settings

  • The paper targets a real gap, which is prior calibration work corrects judge scores for evaluation, whereas this work studies reusing a fixed human-rated dataset (HelpSteer) to correct rewards during policy training on new queries and responses, without additional annotating phase (lowering the cost of human-grounded rubric rewards).

S2. Plug-in, judge- and aggregator-agnostic design

  • Intervals depend only on human data and the query-response pair, so the same heads plug into any judge and aggregator (raw, importance-based, dropout).

S3. Hones report with the negative result + consistent improvements on the primary evaluation

  • MSE reduction against human scores holds for the 3 different scorers, inlcuding a trained reward model (Table 2), and in both transfer directions (Appendix C.2).

  • In RL, the calibrated variants wins 57-66% of pairwise comparisons across 3 policies x 3 aggregators x 3 benchmarks (Table 4), and the effect holds across training judges (Figure 4)

Critical weaknesses

W1. Problem with the calibrated scoring

  • Calibrated score employs center of f(z)f(z) to project the judge score onto its interval (with α=0.2\alpha=0.2, the interval is relatively narrow), so whenever the judge disagrees with ff, the reward is largely determined by ff

  • Neither Table 2 nor Table 6 reports the MAE/MSE of f(z)f(z) alone, or of simple recalibrators fitted on the same HelpSteer data

W2. Missing RL's baseline experiment

  • RL experiments doesn't have baseline that utilizes f(z)f(z) as the reward or combination of both judge and ff.

  • The 56-66% (what I said in paper summary) win rates may reflect adding a HelpSteer RM, without no relation with the clipping approach

W3. The predictor and 2 of the scorers have been trained on HelpSteer

  • ArmoRN as one of the judges that being calibrated, was itself trained on HelpSteer-family data?

  • the same problem also upon the its reward-model backbone (FsfairX-LLaMA3-RM), is HelpSteer-family data contaminate its training process?

  • Both of problems above can affect the interpretation of Table 2

W4. RL claims lack statistical and experimental support

  • No seed, variance, and hyperparameters are given

  • Pairwise judge prompt forbids ties, "...Ties are not allowed..." (Appendix D.2) and doesn't mention randomizing analysis eventhough HelpSteer human scores are known to correlate with verbosity.

Questions for authors

Q1. Can you please add to Tables 2/6 the error of f(z) alone and linear/isotonic recalibration?

Q2. In RL's experimental results, can you please add GRPO with f(z)f(z) as the reward and with a judge/f average?

Q3. . Why are the QR heads trained on the calibration split instead of a separate split followed by CQR-style conformalization? Does the λ=0.3\lambda=0.3 interval used in RL retain any finite-sample guarantee?

Q4. Please report all GRPO hyperparameters, the number of seeds, the number of evaluation prompts per benchmark, and confidence intervals for the win rates.

Q5. Was the response order randomized, or were both orders evaluated? How were order-inconsistent verdicts handled? How do the results change if ties are allowed?

Q6. Please report response lengths per method and length-controlled win rates. How do 57–66% win rates on the trained rubrics square with flat or negative changes on the original rubrics?

Q7. Was HelpSteer or HelpSteer2 used to train FsfairX-LLaMA3-RM-v0.1 or ArmoRM model? Does it affect the Table 2 interpretation?

Reviewer_rqbGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper calibrates rubric-based RL rewards using a fixed set of human ratings, so no new annotation is needed during training. The method works in three steps:

Human-score predictor. For each rubric, a predictor fkf_k (an MLP head on a frozen reward-model backbone) is trained on HelpSteer human ratings. Interval. An interval is built around fkf_k by mixing two pieces, using λ=0.3\lambda = 0.3 and α=0.2\alpha = 0.2 in the RL runs: a fixed-width split-conformal residual quantile; quantile-regression residual bounds that vary with the input. Clipping. During GRPO, the judge’s per-rubric score is clipped to this interval before the existing aggregation method (uniform, importance-weighted or rubric dropout) combines the rubrics.

The theory has two parts. Theorem 1 bounds coverage under distribution shift by 1−α−ηλ−2Δ1-\alpha-\eta_\lambda-2\Delta. Theorem 2 gives the condition under which clipping lowers expected absolute error.

Validation:

Score error (HelpSteer -> HelpSteer2). Clipping lowers average MSE by 40–52% (QR) across two LLM judges and ArmoRM. General-domain RL. Three policies are trained on helpfulness, correctness and coherence. Head-to-head against their uncalibrated versions, the calibrated policies win 57–66% of comparisons on HelpSteer2-test, WildChat and AlpacaEval-2. Expert-domain RL. Win rates on the trained rubrics also improve across five medicine and science benchmarks. On those benchmarks’ own rubrics, however, the changes are marginal or negative.

Critical strengths
  • Judge–human disagreement feeding straight into RL rewards is a real problem. Clipping to an interval learned from humans is cheap, works with any judge, and works with any aggregation scheme. It plugs into existing rubric-RL pipelines without new annotation, which makes it practical to adopt.
  • The error reductions hold across three scorers and in both transfer directions (Tables 2, 6 and 8), with a same-domain control (Table 10). This is the most convincing evidence that the intervals contain information about human scores that the judge lacks.
Critical weaknesses
  • The gains may come from the predictor itself rather than from conformal clipping. The paper never reports how well fkf_k does on its own. The calibrated errors are nearly the same whichever judge is calibrated. In Table 10, calibrated MSE is about 0.60–0.67 for every scorer, although raw MSE ranges from 1.14 to 1.58. Table 2 shows the same pattern. That suggests the clipped score is mostly fkf_k, and the judge contributes little. If so, the method amounts to “use a HelpSteer-trained reward model instead of the judge”. The conformal machinery, and the claim that it corrects judge bias, would then be incidental. Missing baselines, for both score error and RL: fkf_k alone as the reward; an average of judge and fkf_k; isotonic or linear regression from judge scores to human scores on HelpSteer; the interval midpoint from Sheng et al. (2025). Also missing: empirical coverage on HelpSteer2, interval widths, and the fraction of scores that get clipped. Without these, the paper’s central attribution is untested.

  • The downstream evidence relies on the same kind of judge the paper argues is unreliable. All policy comparisons use LLM judges (Gemini 3.1 Pro, DeepSeek-V4-Pro), and mostly on the same three rubrics used for training. There is no human evaluation of the trained policies, and no analysis of response length or style. A predictor trained on HelpSteer could shift style in ways LLM judges reward. On the one more independent signal, the benchmarks’ own expert rubrics, the effect is marginal or negative. For example, it is worse on all five benchmarks for Llama with the uniform aggregation. The policy is also optimized against a reward that partly comes from a fixed reward model. That creates its own hacking risk, which the theory does not cover, because the policy’s outputs drift away from the HelpSteer distribution. There are no seeds, confidence intervals or significance tests for the win rates. The coverage guarantee also doesn’t hold for the configuration actually used. The QR heads are trained on the calibration split without a separate conformalization step (unlike CQR), so the mixed interval at λ=0.3\lambda = 0.3 has no finite-sample guarantee. Theorem 1 is conditional on true quantiles, and the paper itself says Δ\Delta and η\eta are not guaranteed to be small. Theorem 2 is an exact algebraic identity. Neither theorem gives the coverage guarantee that the “Conformal” in the title implies.

Questions for authors
  • What are the MSE and MAE of fkf_k alone on HelpSteer2? What happens in RL when the reward is fkf_k alone, and when it is a simple judge+fkf_k average? If either matches Ours, the contribution is the reward model, not the calibration.
  • What are the empirical coverage, average interval width (on the 0–4 scale), and fraction of clipped scores on HelpSteer2? And during RL, for each rubric, as training goes on?
  • Is there any human evaluation of the trained policies, even a small one? How do response lengths change between Raw and Raw+Ours?
  • How many seeds were run per RL configuration, and what is the variance of the win rates? Please also give the training details missing from the paper (steps, group size GG, KL coefficient β\beta, dataset sizes).
  • Does fkf_k get exploited over training? For example, does fkf_k keep rising while the held-out LLM-judge or human scores level off?
  • Table 8 uses different labels (UW, RQ, RQUW28) from the other tables (FW, QR, Mixed28).
  • ArmoRM’s Raw MSE on the same HelpSteer1 test split differs between Table 8 (1.1366) and Table 10 (1.1404).
  • The metrics in Table 3 aren’t defined. Its “Expected Bias Reduction” seems to equal the MAE reduction in Table 2.
  • Figures 3 and 4 have no numeric values.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.