acceptodds
Under review as a conference paper at ICLR 2027

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Abstract

Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose **Uncertainty-aware Rubric Weighting (URW)**, a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal -values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment. Code is available at https://anonymous.4open.science/r/URW-27E4.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Model, Rubric Evaluation, Uncertainty Modeling, Conformal Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Mnm8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates uncertainty-aware weighting for rubric-based LLM evaluation. The authors argue that conventional uniform aggregation overlooks differences in the reliability of individual rubric judgments, potentially introducing noise into evaluation scores and downstream reinforcement learning rewards.

To address this issue, the authors propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that evaluates each rubric criterion using multiple paraphrased verification templates. The resulting judgments are aggregated through majority voting, while split-conformal calibration is used to assign criterion-level weights based on template agreement.

The experiments evaluate URW across five preference benchmarks, multiple verifier models, and different rubric sources. The authors also investigate the effects of template count and apply URW to downstream GRPO training, reporting improvements in preference accuracy and policy performance over several weighting baselines.

Critical strengths

The problem is practically relevant, and the proposed method is simple and supported by relatively broad empirical evaluation.

Accounting for uncertainty in individual rubric judgments is a meaningful direction, particularly when rubric-based rewards are used for open-ended reinforcement learning. URW provides a straightforward approach that can be integrated with existing black-box verifiers without additional model training. The paper evaluates several verifier families and benchmarks, includes ablation and generalization studies, and further investigates downstream RL performance. These experiments provide useful empirical evidence that adjusting rubric aggregation can improve evaluation and policy optimization under the tested settings.

Critical weaknesses

1. The research motivation and core assumptions are not sufficiently justified.

The paper largely characterizes existing rubric-based evaluation as relying on uniform weighting, overlooking prior work on non-uniform and adaptive rubric weighting. It also argues that high-temperature sampling deviates from standard verification protocols and is incompatible with downstream RL algorithms requiring stable rewards. These claims are insufficiently supported. Deterministic decoding does not necessarily improve judgment accuracy, and stochastic rewards are not inherently incompatible with reinforcement learning.

More fundamentally, URW assumes that agreement across multiple verification templates reflects judgment reliability. However, consistency does not necessarily imply correctness. For difficult rubrics or weaker judges, multiple templates may consistently produce the same incorrect judgment, which URW would still assign a high weight. Such systematic errors could potentially be amplified during downstream RL training. Notably, the theoretical guarantee does not ensure that incorrect judgments receive low weights. Without directly establishing the relationship between template agreement and actual judgment accuracy, the central motivation for uncertainty-aware weighting remains insufficiently validated.

2. The experimental design and evaluation protocols are insufficient to support the conclusions.

The paper selectively evaluates subsets of widely used benchmarks and deviates from their standard evaluation protocols. In particular, RewardBench 2 officially adopts Best-of-4 accuracy, whereas the paper evaluates only Precise-IF and converts 160 instances into 480 pairwise comparisons. Pairwise accuracy and Best-of-4 accuracy measure different outcomes and are not directly comparable. Similarly, RM-Bench is restricted to the Chat/Normal setting, omitting the more challenging Hard setting. These choices limit the comprehensiveness and comparability of the reported results. Furthermore, the released anonymous repository does not include the actual generated rubrics used in these evaluations, making exact reproduction difficult.

The paper also lacks sufficiently strong comparisons under matched inference budgets. Its template-count analysis only considers N=1–6 for URW, without systematically comparing against straightforward averaging of multiple verifier judgments. As N increases, averaging may provide increasingly stable estimates while preserving continuous judgment information, whereas URW relies on hard majority voting. It remains unclear whether URW retains its advantage at larger inference budgets. Moreover, the method requires approximately six times as many verifier calls as single-template evaluation, yet downstream RL experiments match training steps rather than total computational cost. Without compute-matched comparisons and efficiency measurements, the practical advantages of URW remain uncertain.

Questions for authors
  1. Judgment Consistency and Correctness: URW assumes that greater agreement across verification templates indicates more reliable judgments. Can the authors directly validate this relationship using criterion-level accuracy, particularly for difficult rubrics and weaker judges? How frequently do highly consistent but incorrect judgments occur?

  2. Calibration Generalizability: The theoretical guarantee relies on exchangeability between calibration and test data, although their rubrics come from different sources. How robust is URW to distribution shifts or more difficult tasks? If recalibration is necessary, how can reliable criterion-level labels be obtained without substantial annotation costs?

  3. Benchmark Protocols and Reproducibility: Why does the paper use pairwise accuracy instead of the official Best-of-4 metric for RewardBench 2, and restrict RM-Bench to Chat/Normal? Do the improvements persist under the standard evaluation protocols? Can the authors also release the generated rubrics used in the experiments?

  4. Scaling and Computational Efficiency: The paper only studies N up to 6. Can the authors compare URW with straightforward averaging over multiple verifier judgments as N increases? Given the additional verifier calls, does URW still outperform simpler baselines under matched computational budgets?

  5. Deterministic Decoding Assumptions: The paper claims that high-temperature sampling is incompatible with downstream RL. What evidence supports this claim? Have the authors compared temperature-zero template ensembling with stochastic sampling under matched settings?

  6. Comparison with Existing Weighting Methods: Prior studies have already explored non-uniform and adaptive rubric weighting. How does URW differ from these approaches, and why are representative methods not included as baselines?

Reviewer_eBYZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Authors study how the judgement uncertainty of a rubric should be accounted for in rubric-based evaluation and reinforcement learning. Authors propose to generate paraphrases of a rubric using templates, and then quantify verifier uncertainty across templates. Authors use the probability of the true label as the weight for the rubric. Empirical evaluations are run on RubricBench, ResearchQA, Rm-Bench, RewardBench 1/2. Rubrics are generated with RubricARM-8B or sourced from the original benchmark. The proposed weighting method attains better preference accuracy on these benchmarks than baseline methods do. It is also shown that this method improves on downstream RLVR. Components and hyperparameters are validated with ablation study.

Critical strengths

Authors are motivated by a key limitation in the current practice: uniform weighting is quite common, but it is certainly limiting as not all rubrics are equally useful, and are not equally reliable. Therefore, improvements on this popular but simplistic practice can have a potential to have a wide impact, in particular given how rubric-based evaluations and RL are common today.

Also, authors establishes a connection between rubric-based evaluation and conformal predictions. Since both are well-established fields with rich literature, this connection can lead to a productive collaboration between two fields, borrowing practical and theoretical tools from each other. This would be considered a high impact to the research community.

Critical weaknesses

Authors only consider the uncertainty of each rubric, and the utility of each rubric is not accounted for. There is a rich literature showing that uniform weighting of quality dimensions is suboptimal, and that non-uniform weighting is beneficial. In particular, in Rubric-based RL literature, Rubric-as-a-reward https://arxiv.org/abs/2507.17746 demonstrated that learning per-rubric weighting is useful, and furthermore, LLM-based aggregation works even stronger.

Secondly, I find the methodology to have a worrying conceptual flaw. The proposed method quantifies the uncertainty across prompt templates, but what's needed is the uncertainty across responses. Two can be arbitrarily different from each other. Possibly, one can make an argument that the uncertainty across rubric paraphrases can be a good proxy for uncertainty across responses, but this has to be validated.

Questions for authors

In line 162, prior method son response-wise uncertainties are discussed. Aren't these useful alternatives/baselines for the proposed method? It sounds more desirable to account for uncertainties specific to the response, so that a rubric will be discounted only when it is uncertain on a specific response, rather than always discounting for all responses.

For Table 1: As mentioned in the weaknesses, there are prior work (such as Gunjal et al, Rubric-as-reward) which use non-uniform weighting based on utility. How does the proposed method compare against methods which account for the rubric utility?

Reviewer_euugOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at how binary rubric judgments get combined into one response-level score, which is usually a uniform average. The authors argue this treats every criterion as equally reliable. They propose URW, a training-free method that runs each criterion through 6 paraphrased judgment templates at temperature zero and averages the votes into a soft score. Using a held-out calibration set of about 2k criterion-level labels from RubricARROW-Judge-SFT, it turns that soft score into a split-conformal p-value. The p-value becomes the criterion’s weight, and the weights are normalized within each response. The theory shows that a correct voted judgment rarely gets a weight below α. Experiments cover five pairwise preference benchmarks, three API verifiers, smaller Qwen3/GLM4 verifiers, human and LLM-generated rubrics, and GRPO training with Qwen3-8B and GLM4-9B policies. URW beats uniform weighting and the uncertainty baselines across the board, with large gains on Arena-Hard and GPQA-Diamond after RL.

Critical strengths

The work targets the aggregation step of rubric-based evaluation, which most rubric work leaves at uniform weights. It gives a per-criterion uncertainty signal that works with a single deterministic black-box verifier and shows that the choice of aggregation rule, not just rubric quality or verifier strength, has a measurable effect on downstream RL with rubric rewards. It is also a training-free framework.

Critical weaknesses

The main claim is that URW weights reflect how uncertain each rubric judgment is, but I don’t think the paper actually shows this. The weight depends only on how much the 6 rephrased templates agree, so it captures sensitivity to wording, not whether the judgment is right. A verifier that’s consistently wrong across all phrasings still gets full weight. There is also no simple baseline with majority vote weighted by agreement level, with no calibration. The Beta-Binomial weights by the satisfaction probability rather than certainty, and SCP uses a hard cutoff. Furthermore, the ablation shows full URW jumping well above both components (e.g., ResearchQA with DeepSeek: 0.645 and 0.648 vs. 0.698), and the paper doesn’t explain why.

Questions for authors

See weaknesses.

Reviewer_vaL4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes URW, Uncertainty-aware Rubric Weighting, a training-free method for LLM evaluation. Most of the recent rubric-based evaluations consider each criterion as binary with uniform weights. This suffers from variation in LLM judge reliability. This paper proposes a deterministic evaluation using 6 paraphrase prompts for each criterion with split conformal prediction.

The paper evaluates URW on five different preference benchmarks using multiple verifiers, including DeepSeek V4 and Qwen 3.8 Max. The evaluation shows consistent improvements on these verifiers compared to uniform aggregation.

Critical strengths

The paper addresses an important problem of current rubric-based LLM evaluation, which considers all the criterion equally. The proposed method, URW, is training-free and shows consistent improvements across all 15 benchmarks and verifier combinations. According to Table 1, the performance improvement on DeepSeek V4 Pro from 62.5% -> 75.9% (~13.4 pp) on RewardBench 2 is quite significant. The RL experiments also show significant gains in Table 5, i.e, Arena-Hard gains ~14pp and 18pp for the two evaluation methods.

Critical weaknesses

Even though the performance improvement was significant, I still feel the algorithmic novelty is limited. The paper uses existing techniques, i.e., prompt ensembling and conformal calibration, and positions them well in this paper. The paper uses 6 verifier evaluations per criterion, but how the number 6 was decided is unclear; is there any ablation for the same? This also introduces extra computational cost, which is not properly reported in the paper.

The proposed method has a strong dependency on the rubric quality. On RubricBench for DeepSeek, the accuracy drops from 80.2% -> 51.5%, which also indicates there is a substantial sensitivity to the rubric quality.

Questions for authors
  1. Request authors to report the end-to-end computational latency of URW compared to uniform aggregation and more ablation on the number of paraphrase prompts.
  2. Do the reported RL improvements remain consistent across multiple training seeds?
  3. Theorem 3.1 controls the probability of the p-value and assigns it to the correct judgment. Request the author to clarify how this guarantee relates to the reliability of the weight used in the final judgment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.