acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Rubric Partition Sensitivity in LLM Judges

Abstract

Equivalent rubric content need not produce equivalent LLM-judge scores when its criteria are grouped differently. Raw differences confound this interface effect with repeat variability, score discreteness, and the number of partitions tested. We introduce an estimator-matched conditional randomization calibration and apply it to 780 items in seven dataset cohorts, five corpus families, and 96,120 successful calls across DeepSeek v4 Pro, Qwen3.8 27B, and Selene-1-Mini. Direct grouped scoring has calibrated drift ratios CDR = 1.259, 1.239, and 2.355 (95% intervals reported in that model order). Item-level excess remains positive after 10% within-cohort trimming for all three models, and no single item changes the corresponding pooled direct-excess estimate by more than 10%. Post-hoc mechanism diagnostics show signed residual centralization across the score range, additional multi-atom contextual residual on common support, and conditional associations with block size and atom disagreement. A partition-independent atom-fixed control has no comparable positive score-unit excess for any model. The holistic single-block contrast is model-dependent, underscoring that the result is a measurement property of the scoring interface rather than a correction coefficient that transfers unchanged across evaluators.

Primary area:
datasets and benchmarks
Keywords:
LLM judges, rubric evaluation, prompt sensitivity, conditional randomization, measurement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ACCbOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether LLM-judge scores change when the same rubric criteria are only grouped into blocks differently. It introduces a permutation-calibrated ratio (CDR) and finds that partition-induced dispersion modestly exceeds repeat noise for all three judges.

Critical strengths
  1. The studied problem is interesting and common in rubric-based LLM evaluation.
  2. The proposed metric (CDR) is a meaningful diagnostic that reveals partition sensitivity in current rubric-based judges.
  3. The statistical analysis is careful, with controls, robustness checks, and replication.
Critical weaknesses
  1. The paper provides a diagnostic metric but no remedy, and does not test which scoring interface works better.
  2. The effect is small (0.03–0.08 points), and the paper does not show that it changes rankings or downstream decisions.
  3. CDR only reflects a judge's consistency across equivalent interfaces and is not validated against human ratings, so it says nothing about judge accuracy.
  4. Only three judge models are evaluated, which is too few to show how CDR varies with model family and size.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ACCb

hank you for your positive comments and your clear list of weaknesses. We respond to each one below.

W1. No remedy, and no test of which interface works better.

We now give both (Sec. 7, App. A.8).

A remedy. First, run the calibration to see whether a judge is sensitive to grouping, and by how many points. If it is, average its scores over several boundaries. This removes the grouping effect, and on our human-labeled datasets it does not hurt agreement with human ratings. It matches or beats the typical single boundary in 13 of 15 cases (App. Table 15). Which interface is better. We compared every boundary, the boundary average, scoring each criterion separately, and one holistic score against human labels. No single option is best for every judge. Scoring each criterion separately helps Selene on GSM8K but hurts DeepSeek on HelpSteer2. For this reason we recommend calibrating each judge rather than prescribing one interface. W2. The effect is small and may not change decisions.

The 0.03 to 0.08 values are a lower bound. After removing repeat noise, the spread across partitions is 0.14, 0.17 and 0.19 points (Sec. 5.1, Table 5). Changing the boundary moves 22%, 25% and 19% of items by half a point or more. The shuffled baseline gives 15%, 18% and 2%. Pairwise order flips are above baseline for all three judges (App. Table 11). Overall rankings stay mostly intact, and we now say so. The effect matters most where scores meet a cutoff, as in filtering, top-k selection or rewards.

W3. CDR measures consistency, not accuracy.

We agree, and the paper now says this directly (Sec. 7). We think this is exactly why it should be measured on its own. A judge can match human ratings well on average and still give the same answer a different score depending on how the rubric is grouped. Agreement studies do not reveal this. Our new human-label analysis shows that the choice of boundary rarely changes agreement with humans, and that averaging over boundaries removes the grouping effect at no cost (App. A.8).

W4. Only three judges.

We agree that this is a limitation and state it in Sec. 6. The three judges are quite different: a large reasoning model, an open-weight 27B model and a small dedicated evaluator, with both deterministic and sampled decoding. The new experiments show that they differ in kind, not only in degree. Two are sensitive mainly to grouping, and one is sensitive to the whole layout. This is why the contribution is a method that can be rerun on any judge, rather than a single correction factor. The full pipeline is released for this purpose.

Thank you again. Your comments led to the effect-size, decision and human-agreement analyses now in the paper.

Thanks for the authors' response. Considering the rebuttal and the other reviewers' comments, I will maintain my original score.

Reviewer_kN7NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an LLM judge gives the same total score when the same rubric criteria are grouped differently. Criteria, weights and the 0–5 scale stay fixed, and only the boundary between two scored blocks moves. Each split is shown in both block orders with four repeats, and the block scores are combined into a weighted total. To separate real split effects from the judge's run-to-run randomness, the observed spread across splits is compared with the spread after randomly shuffling split labels, reported as a ratio (CDR, where 1 means no effect) and as a difference in score points. Across 780 items from five datasets and three judges (DeepSeek v4 Pro, Qwen3.8 27B, Selene-1-Mini), the spread exceeds the shuffled baseline for every judge (CDR = 1.259, 1.239, 2.355). A control that scores each criterion separately stays near baseline, a single holistic score shifts totals in judge-dependent directions, the first-shown block gets a small boost, and post-hoc analyses link block-level score shifts to block size and disagreement among criteria.

Critical strengths
  1. A clean, well-controlled experiment, reported honestly. Only the boundary moves; criteria text, order, weights, scale and output format stay fixed, both block orders are used, and the baseline comes from shuffling split labels within each item. The authors also report where there is no effect (MT-Bench CDR 1.008 and 1.004 for DeepSeek v4 Pro and Qwen3.8 27B, Table 3) and label the mechanism analyses as post-hoc. The core claim, that moving the boundary changes the score beyond run-to-run noise for all three judges (p < 10−410^{-4}, Section 5.1), is convincing.

  2. Useful practical findings. Collapsing all criteria into one holistic score shifts totals by −0.103, −0.198 and +0.264 points for the three judges (Table 4), so this common shortcut is not neutral and its direction depends on the judge. The block shown first also gains 0.031–0.055 points (Table 7). These findings support the paper's advice to record the criteria, the grouping shown to the judge and the aggregation code (Section 7).

Critical weaknesses
  1. CDR shows that the effect is detectable, not how large it is, so the comparisons across judges do not hold. CDR divides the observed spread by the spread expected from noise alone. For the same real effect, a less noisy judge or more repeats gives a larger CDR, because the noise baseline shrinks while the effect does not. The paper's own numbers show this. DeepSeek v4 Pro and Selene-1-Mini have almost the same observed spread (0.143 vs 0.139, Table 2), but Selene's repeated calls rarely differ (exact-match 0.830–0.942 vs 0.194–0.859, Table 8), so its noise baseline is about half as large (0.060 vs 0.114) and its CDR nearly doubles (2.355 vs 1.259). So the statements that Selene shows "the largest calibrated excess" (Section 5.1) and that CDR is a "comparable" diagnostic (Section 7) do not follow, and the decoding settings that set this noise level are not reported. Whether the effect matters is also open: the excess is 0.029–0.079 points on a 0–5 scale, and nothing tests whether it changes rankings, agreement with humans or rewards (Sections 6–7). Both points can be addressed from the existing scores (Questions 1–2).

  2. The design cannot separate an effect of grouping from sensitivity to any change in the prompt. Moving the boundary also changes the prompt text, and LLMs, judges included, react to prompt edits that should not matter (Sclar et al., 2023; Stureborg et al., 2024). The atom-fixed control cannot rule this out: its prompt is identical across split labels (Section 3.3), so it stays at baseline by construction. It confirms that the test works, not that per-criterion scoring "removes" the effect (Section 5.3). Even so, Qwen3.8 27B's control falls below 1 (0.915 [0.837, 0.995], Table 4), which is unexpected with identical prompts and independent calls. The post-hoc patterns in Section 5.5 are suggestive, but they are tied to dataset (four-criterion blocks exist only in HelpSteer2). A cheap control that keeps the split fixed and changes only cosmetics would settle this (Question 3).

Questions for authors
  1. How large is the effect in score points? [Requested analysis] Please report, per judge and dataset, the size of the real split effect in score points with run-to-run noise removed (e.g., a variance-component estimate), with confidence intervals; this needs no new calls. If it is non-negligible and preserves your ordering of the judges, my first concern is resolved; if it is small or reorders them, the cross-judge claims should be dropped.

  2. Does the split ever change a decision? [Requested analysis] Using the existing scores, how often does the ranking of two items (or of two responses to the same prompt) flip between splits, compared with the same rate after shuffling split labels? Flip rates clearly above that baseline would show the effect matters and would raise my score; rates near it would mean the effect is detectable but negligible, and the paper should say so.

  3. Grouping, or any prompt change? [Requested analysis] Please keep one split fixed and vary only cosmetics (block names, separators or output-field order), with the same number of variants, orders and repeats, and compute CDR the same way. One or two datasets with Selene-1-Mini would suffice. A cosmetic CDR near 1 would show the effect is about grouping and would raise my score; a value similar to the boundary sweep would mean the paper measures general prompt sensitivity and would lower it.

  4. How were the calls made? [Clarification] For each judge, please report temperature, top-p, seed, thinking or reasoning settings, serving stack and dates; whether the four repeats were independent requests; whether call order was randomized; whether prompt caching was on; and how failed or unparseable calls were handled (Section 4.2 keeps "one valid observation per pre-specified cell"). Please also explain why Qwen3.8 27B's atom-fixed CDR falls below 1. Independent, randomized calls would support the test's validity; any dependence between calls would lower my soundness score.

  5. Readability [Suggestion — does not affect my rating] The paper is hard to follow in places: coined terms such as "signed centralization", "contextual residual on common support" and "canonical block identity" are used without plain definitions. A short worked example that follows one item through the design would help readers.

Score criteria. I would raise my rating to 3 if the revision reports the effect in score points and tones down the cross-judge claims (Question 1), shows that a cosmetic-only change stays near baseline (Question 3), and either shows that splits change rankings (Question 2) or limits the claims to a detectable but small effect. I would lower it if cosmetic changes produce a similar effect or if the calls turn out not to be independent.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kN7N

Thank you for a very helpful review. We used your conditions as our checklist, and committed each new protocol before running it.

Q1. How large is the effect?

You are right: CDR shows the effect exists, not how big it is, and less noise raises it. We keep CDR as the test and now also report effect size in points (Sec. 5.1, Table 5, App. Table 10), subtracting each item's repeat noise from its variance across boundaries.

With noise removed, the spread across partitions is:

DeepSeek v4 Pro: 0.139 [0.102, 0.171]

Qwen3.8-27B: 0.172 [0.134, 0.205]

Selene-1-Mini: 0.193 [0.181, 0.206]

On a 0 to 5 scale this is not negligible. The excess we first reported (0.03 to 0.08) understates it, because noise and partition effects add as squares. In points, the judges are closer than their CDRs suggest: only DeepSeek and Selene differ significantly (−0.055 [−0.092, −0.021]). As you suspected, Selene's high CDR mostly reflects its low noise, and we removed the wording that compared CDR across judges.

Q2. Does the split change decisions?

Yes, though modestly. Against the shuffled baseline (App. Table 11):

Pair order flips (observed vs. baseline): DeepSeek 5.6% vs. 5.3% (p = .04), Qwen 9.3% vs. 8.4% (p < .001), Selene 4.7% vs. 4.3% (p = .03). Items moving by 0.5 point or more: DeepSeek 22% vs. 15%, Qwen 25% vs. 18%, Selene 19% vs. 2%. Flips are above baseline for all three judges, but by under one point, so overall rankings are largely preserved. The larger effect is on individual items, and Selene's top 20% set changes about twice as often as baseline. Sec. 7 now says the effect matters most at cutoffs: filtering, top-k selection, rewards. (Few prompts have two responses, so we used item pairs within each dataset.)

Q3. Grouping, or any prompt change?

We ran your control on two datasets for all three judges (new Sec. 5.5, App. Tables 12 to 14). The boundary was fixed and variants, orders and repeats matched the sweep; only headers, bullets, separators, a blank line and JSON key order changed. A repeat of the boundary sweep ran in the same randomized batch.

Spread when moving the boundary vs. when changing only formatting, with the difference in variance:

DeepSeek: 0.185 vs. 0.089; difference 0.026 [0.015, 0.040]

Qwen: 0.278 vs. 0.199; difference 0.038 [0.015, 0.059]

Selene: 0.212 vs. 0.281; difference −0.034 [−0.048, −0.021]

For DeepSeek and Qwen, the boundary matters clearly more than formatting; on UltraFeedback their formatting-only CDR is 1.05 and 1.10. For them, the main result is about grouping.

Selene is different, and the paper says so. Changing one formatting factor at a time (App. Table 13) shows that swapping the field order in the JSON example moves its scores more than moving the boundary, while a blank line moves them less. The same prompt run twice gives nearly identical scores, so this is a stable trait of Selene, not noise. Formatting never changes between boundaries in the main sweep, so it cannot affect the main estimate. For both kinds of judge, our advice is the same: fix the full prompt layout, record it and calibrate it.

We also extended the sweep to three blocks. Selene's CDR stays high (2.08 on HelpSteer2, 2.34 on UltraFeedback; App. Table 14).

You are also right that the atom-fixed control, with an identical prompt across labels, checks the calibration rather than showing the effect is "removed". We retitled Sec. 5.3.

Q4. How were the calls made?

Sec. 4.2 now gives the settings:

DeepSeek: temperature 0, with reasoning.

Selene: temperature 0, with a fixed system message restating the output format.

Qwen: thinking off and the server's default sampling, so its outputs vary between calls.

No seeds or caching were used. The null is calibrated to each judge's own noise, so Qwen's noise makes the test harder to pass, not easier.

All calls were shuffled before sending; each repeat is a separate request, and failed calls were retried with the same prompt until one valid answer was kept. We checked what the test needs (App. A.7):

calls for the same boundary are not grouped in time (p = .34, .71, .71); retries are unrelated to the boundary; the atom-fixed prompt is identical across labels for every item. Qwen's atom-fixed CDR below 1 comes from the two datasets with only two boundaries (0.73, 0.84); its excess in points is −0.007 [−0.024, 0.010]. With unequal variances, the test rejects at most 3.2% of the time at the 5% level.

Readability. App. A.9 now walks one item through the full design and defines our terms in plain words.

Your conditions, in short: (1) effect in points, cross-judge claims toned down; (2) flips above baseline, effect concentrated at cutoffs; (3) formatting stays near baseline for two of three judges, and we explain the third; (4) calls independent and randomized. Thank you again; your suggestions made the paper much stronger.

Reviewer_wMbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how rubric partitioning affects the scores assigned by large language model judges. By changing the boundaries between criterion groups while keeping the criterion content, order, and weights fixed, the paper introduces the Calibrated Drift Ratio (CDR). This metric uses an estimator-matched randomization test to distinguish partition-related score variation from variability caused by repeated evaluations.

The experiments cover five dataset families, 780 items, and three LLM judges, involving 96,120 successful scoring calls. The three models achieve CDR values of 1.259, 1.239, and 2.355, all significantly above the matched null. The atom-fixed scoring control does not exhibit comparable positive excess dispersion. The paper also examines the relationships between score variation, display order, block size, and disagreement among criteria.

Critical strengths

1.Relevant research problem. The paper examines how rubric organization affects the stability of LLM-based evaluation. Even when criteria and their weights remain unchanged, different groupings may produce different scores. This issue is relevant to the reliability of automated evaluation systems.

2.Useful statistical measurement approach. The paper employs an estimator-matched conditional randomization test that accounts for repeated scoring variability, score discreteness, and the number of available partitions. Compared with directly examining raw score standard deviations, this approach provides a more clearly defined statistical baseline.

3.Comprehensive experimental design. The experiments cover multiple datasets and LLM judges, with atom-fixed scoring, holistic scoring, and display-order controls. Independent cohorts, outlier sensitivity analyses, and statistical tests provide additional evidence for the stability of the main findings.

Critical weaknesses

1.The incremental contribution over existing rubric stability research remains insufficiently clear.

Hong et al. (2026, From Rubrics to Reliable Scores) have investigated scoring stability under semantically equivalent rubric perturbations, while Xu et al. (2026, Am I More Pointwise or Pairwise?) have examined position biases arising from score-option and criterion ordering. This paper further isolates changes in the boundary between two contiguous criterion groups while keeping the criteria, weights, and number of groups fixed, and uses a matched randomization test to calibrate score dispersion. This design provides a more specific measurement of partition sensitivity. However, it remains unclear what additional diagnostic information CDR provides beyond existing stability analyses and whether this information can improve practical evaluation decisions.

2.The statistically significant partition sensitivity is not sufficiently connected to practical evaluation outcomes.

Section 5.1 reports CDR values significantly above the matched null for all three models, but the corresponding excess dispersion values are only 0.029, 0.036, and 0.079 points on a 0–5 scale. These results demonstrate measurable additional variation associated with rubric partitioning, but they do not establish whether such variation is sufficient to affect candidate response rankings, model comparisons, or score-based data selection. The paper also acknowledges the absence of ranking and downstream training experiments in its Limitations section. Therefore, despite the statistical significance of the findings, their practical importance for evaluation decisions remains uncertain.

3.The experimental scope and mechanism analysis remain limited.

The primary experiments focus on two contiguous criterion groups and examine score variation by moving the boundary between them. Practical rubrics may involve more groups, non-contiguous criterion combinations, or hierarchical structures, which are not covered by the current experiments. Section 5.5 relates scoring residuals to score scale, block size, and disagreement among criteria, but these findings are mainly based on post-hoc associations, and some block sizes are constrained by dataset structure. The results offer possible explanations for partition sensitivity but do not establish its underlying mechanisms.

Questions for authors

Compared with Hong et al. (2026, From Rubrics to Reliable Scores) and Xu et al. (2026, Am I More Pointwise or Pairwise?), what substantive additional information does CDR provide for measuring and diagnosing scoring instability?

Section 5.1 reports excess dispersion values of 0.029–0.079 points. Are these variations sufficient to change candidate response rankings, model comparisons, or data selection decisions in practical evaluation settings?

The randomization test in Section 3.2 relies on the exchangeability of partition boundary labels. To what extent does this assumption remain valid when different boundaries produce groups with different sizes and potentially different scoring variances?

The main experiments focus on two contiguous criterion groups. Does CDR remain applicable to rubrics with more groups or more complex structures? Do the block-size and criterion-disagreement effects observed in Section 5.5 remain consistent in such settings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wMbc

Thank you for your careful reading and useful questions. Our answers follow.

Q1. What does CDR add beyond Hong et al. and Xu et al.?

Both papers show that how a rubric is presented affects scores. CDR adds three things.

A fair baseline. Raw spread across conditions mixes the real effect with repeat noise, the discrete score scale and the number of conditions compared. CDR recomputes the same statistic with the labels shuffled, so results are comparable across datasets. This matters in practice. Qwen has the largest raw spread (0.190), but most of it is noise, and its CDR is the lowest of the three. A raw stability analysis would rank the judges wrongly. A way to separate causes. In a new control (Sec. 5.5), we kept one boundary fixed and changed only formatting. For DeepSeek and Qwen, moving the boundary changes scores clearly more than reformatting does. Selene reacts to formatting as well, mainly to the order of fields in the output format. Stability analyses alone cannot tell these two kinds of judge apart. A practical check. A user can run a small boundary sweep on a new judge, see whether grouping matters beyond noise, and read off the effect in score points. Q2. Is the effect large enough to change rankings or data selection?

We added both an effect size and decision-level measures (Sec. 5.1, Sec. 7, App. Tables 10 and 11).

Effect size. After removing repeat noise, the spread across partitions is 0.14, 0.17 and 0.19 points. The 0.03 to 0.08 values you mention are a lower bound on it. Decisions. Changing the boundary moves 22%, 25% and 19% of items by half a point or more. The shuffled baseline gives 15%, 18% and 2%. Pairwise order flips are above baseline for all three judges. For Selene, the top 20% selected set changes about twice as often as the baseline. Overall rankings stay largely intact, and we now say so. The effect matters most where scores meet a cutoff, as in filtering, top-k selection or rewards.

Q3. Does exchangeability hold when boundaries differ in block size and variance?

We tested this directly (App. A.7). Starting from the real data, we made all boundary means equal and let each boundary keep its own noise. We also inflated one boundary's noise by up to three times. Over 1,000 runs per setting, the pooled test rejected at most 3.2% of the time at the 5% level. Unequal variances therefore make the test slightly conservative, not too lenient.

Q4. More groups and more complex rubrics.

We extended the sweep to every contiguous split into three blocks. The effect remains: Selene's CDR is 2.08 on HelpSteer2 and 2.34 on UltraFeedback (p < .001; Sec. 5.5, App. Table 14). The method needs no change for more blocks. Non-contiguous and nested groupings are outside this study, and Sec. 6 says so.

On the mechanism analysis. We agree that it is post-hoc and that block size depends on the dataset. The paper presents these results as associations, not causes. We moved the supporting checks to App. A.10, including the leave-one-family-out fits and the subsets in which every family contributes every block size. On those subsets, the block-size effect stays positive but smaller.

Thank you again for these questions. They led directly to the new formatting control, the decision analysis and the simulation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers for their time and their careful, specific reviews. We were glad that the reviewers found the problem relevant (wMbc, ACCb), the matched null a sound baseline (all three), and the experiment clean and honestly reported (kN7N).

The reviews raised four shared questions: how large the effect is, whether it changes decisions, whether it comes from grouping or from any prompt change, and how the calls were made. To answer them we ran new experiments (about 70,000 new judge calls) and new analyses of the existing records. For each new experiment, we fixed and committed the protocol and decision rule before making the first call. The definition of CDR and the main results are unchanged.

What we added

Effect size in score points (Sec. 5.1, Table 5). After removing repeat noise, the spread across partitions is 0.14, 0.17 and 0.19 points for DeepSeek v4 Pro, Qwen3.8-27B and Selene-1-Mini. The excess we originally reported (0.03 to 0.08) is a lower bound on this spread. Effect on decisions (Sec. 7, App. Table 11). Changing the boundary moves 22%, 25% and 19% of items by half a point or more. The shuffled baseline gives 15%, 18% and 2%. Pairwise order flips are also above baseline for all three models. Overall rankings stay mostly intact, so the cost is felt mainly where scores are compared with a cutoff. Grouping vs. formatting (new Sec. 5.5, App. Tables 12 to 14). We fixed one boundary and changed only headers, bullets, separators, spacing and JSON key order. For DeepSeek and Qwen, moving the boundary changes scores clearly more than reformatting does. Selene reacts to formatting as well, mainly to the order of fields in the output format. We also tested splits into three blocks, and the effect remains. How the calls were made (Sec. 4.2, App. A.7). We give decoding settings for each model. No caching or fixed seeds were used, and the order of calls was randomized. We checked the assumptions behind the shuffle test directly, and a simulation shows that the test does not over-reject. Agreement with human labels (App. A.8), plus a worked example and plain definitions of our terms (App. A.9). These results support the paper's main point. The same rubric content, grouped differently, gives measurably different scores, and a matched null is needed to measure this fairly. The new control also shows something practical: some judges are sensitive mainly to how criteria are grouped, while others are sensitive to the whole prompt layout. The calibration can tell the two apart.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.