acceptodds
Under review as a conference paper at ICLR 2027

JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

Abstract

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_V5aSOpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

LLMs are subjective and hence can prefer different responses in pairwise comparison. This paper studies how judges compare responses across attributes like clarity, correct and detail and prioritization. The paper finds that judges frequently agree on attribute judgements but overall judgement differs. They recalibrate attribute weights based on reference labels to improve agreements with held out set.

Critical strengths
  1. This paper addresses an important question as LLM judges are increasingly used in various evaluations. The study approach contains both a LLM-judge set and a human set, that indicates how both agree more on attribute different but less on overall preferences, which is an insightful finding.
  2. The reweighing experiment is a good method for steering LLM Judge to have less implicit bias based on the attributes they emphasize more or less on.
Critical weaknesses
  1. There should be more thorough data cleaning/categorization. For instance, a lot of the ground truth preferences in some datasets (e.g. RewardBench, Nectar) are themselves based on LLM-judges (which had specific prompts based on their category). While the paper is seeking to cover as many datasets as possible, it’s useful to distinguish human-labels from LLM-labels (which makes the study kind of circular – LLM judge predicting the preference of another LLM Judge).
  2. (Not strictly a weakness but an area for improvement) I think the current flow of the paper reads like – we find that LLM-Judge agree on attributes not on the prioritization and we train models to recalibrate their prioritization and they predict the ground truth labels better. This is fine but I think the broader impact of this work is that many evals treat LLM-Judges as interchangeable black boxes and if the authors can give some recommendations regarding how to better calibrate the judges in unseen evals (say GDPVal-AA, ICLR 2026), it can make the paper a lot more impactful.
Questions for authors
  1. Can the authors make a comparison with this paper “RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards” (ICLR, 2026) and “Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions” (COLM, 2025)? The concept of perception in this paper seems very similar to principles and values in these papers.
  2. The paper finds 87 attributes but it’s not clear how those attributes were found. Is it based on prompting llm to extract attributes from judgement rationales? How is data cleaning and deduplication handled?
  3. For LLM-judge, a critical aspect on which aspects they focus more on depends on the prompt and hence it could significantly influence the aspects that influence overall preference. How is that controlled for in this study?
7 comments

(1/2)

Dear Reviewer V5aS,

Thank you for the detailed review. All supporting numbers are included below (1/2).

[W1] Distinguishing human labels from LLM labels; is the study circular?

We agree that human and LLM labels should be clearly separated. Table 5 marks each of the 17 sources as human-labeled (9), AI-labeled (2: Nectar and UltraFeedback) or benchmark-labeled (6). Benchmark labels are set by how the benchmark is built: JudgeBench pairs a response whose final answer matches the ground truth with one that does not. You are right that RewardBench is mixed. Its code, math and safety subsets are defined by correctness (e.g., correct vs. buggy code, a refusal vs. a dangerous completion), while its chat subsets contrast responses of stronger and weaker models, some selected with GPT-4 ratings. We will mark this in Table 5. Reweighting gains by label type:

Label typeSourcesMean reweighting gain
Human9+4.7 pp
AI2+6.2 pp
Benchmark6+6.4 pp

The gains hold for human and benchmark labels, so they do not depend on LLM-produced labels. Our other findings, hidden consensus and judge-specific priorities (Sections 3–4), compare judges with each other and use no reference labels. We will report all adaptation results by label type and state each source's labeling protocol in Table 5.

[W2] Recommendations for calibrating judges on unseen evaluations (e.g., GDPVal-AA, ICLR 2026).

Thank you for this suggestion. We ran a small study on GDPval-AA. GDPval-AA grades open-ended professional work with a panel of three judges (Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash), one judge per comparison. On 44 tasks (one per occupation), we took the deliverables of two public models, GPT-6 Astra and Qwen3.8-27B, and asked all three judges for an overall verdict and for a verdict (A, B or same) on every criterion of the task's own rubric (2,031 criteria). Our 87 attributes were designed for and validated on single-turn text responses, the setting of all 17 sources in the paper. GDPval deliverables are files produced by multi-step agentic work, which we have not validated them on, so in this first study we used the rubric criteria as attributes.

Hidden consensus also holds on agentic tasks. The judges often disagree on the overall winner but agree on the criteria:

Two judges (J1–J3: the three panel judges)J1–J2J1–J3J2–J3
Same overall verdict61%55%70%
Same criterion verdict, all pairs88%85%84%
Same criterion verdict, pairs where the overall verdicts differ84%81%84%
Criteria on which the two judges pick opposite sides1.0%1.1%0.6%

The judges' priorities differ. Astra's win rate ranges widely across judges. Combining each judge's criterion verdicts with one shared set of weights (the rubric's point values) halves this range:

Astra's win rate over Qwen3.8-27BJ1J2J3Range
Overall verdicts100%61%55%45 points
Criterion verdicts with shared weights70%48%50%23 points
Overall verdict differs from the judge's own weighted criteria34%36%25%

So judging at the criterion level is more consistent across judges and easier to inspect than overall verdicts. The last row also shows its current limit: each judge's overall verdict departs from its own criterion verdicts on 25–36% of the pairs, so the task rubrics alone do not explain what each judge prefers. Rubric criteria are also specific to each task, so weights cannot be fitted across tasks as in our reweighting.

Based on this, we recommend to (1) ask judges for criterion-level verdicts rather than only an overall verdict, and report both; and (2) combine them with shared, explicit weights. Full calibration needs attributes that are shared across tasks. Extending and validating our attributes for agentic outputs would allow each judge's priorities to be measured and fitted to a small set of expert labels (128–256 suffice; Figure 9). We will add this study to the appendix and discuss this extension as future work.

(2/2)

[Q1] Comparison with RLBFF (ICLR 2026) and Values in the Wild (COLM 2025).

Thank you for pointing us to these two papers; both are closely related to our work. The main difference lies in what we mean by perception: how a judge sees the differences between two responses. For each attribute, the judge says which response has more of it, without saying which response is better.

RLBFF. RLBFF's principles and our attributes both break response quality into specific criteria. A principle, however, is checked on a single response (satisfied or not). Our perception is comparative and non-evaluative: it records which of two responses shows more of a quality. This lets us separate perceiving a difference from deciding how much it matters (prioritization).

Values in the Wild. This work finds the values an assistant expresses in real conversations, which can shift with context. These values describe what a model cares about, so they are closer to our priorities than to our perception. They are also observed while the model responds to users, whereas we study a model evaluating two responses.

We will add both papers to the "LLM-as-a-Judge and subjectivity" part of Related Work.

[Q2] How were the 87 attributes found? How were cleaning and deduplication handled?

Thank you for the question. We built the attributes in two steps, starting from dimensions that prior work derived from data and then checking them against real response pairs:

  1. Literature-guided candidates. We started from the quality dimensions used in prior evaluation work, all cited in the paper: FLASK's fine-grained skills; the attributes rated in HelpSteer/HelpSteer2 and UltraFeedback; G-Eval's evaluation dimensions; helpfulness and harmlessness (Bai et al., 2022) and length effects (Dubois et al., 2024); and the custom and data-discovered criteria of Prometheus, VibeCheck and WIMHF. We grouped these dimensions into ten families (e.g., correctness, reasoning, information, style, epistemic, social, alignment) and split each family into specific attributes that can be judged separately. This gave 81 attributes.
  2. Real-case human check. On the SubjectiveSet–Human cases, we checked whether these attributes capture what actually distinguishes the two responses, and added 6 qualities that were missing (elaboration, difficulty, formatting, forcefulness, caution framing, phrasing originality), giving 87.

Deduplication. We avoided exact duplicates while writing the attributes rather than removing them afterwards. Because each family was divided top-down, each attribute was written alongside its neighbours, with a definition and a boundary that names the closest neighbours and states how they differ. For example, verbosity is "raw amount only; whether the length is justified is conciseness", while conciseness is "economy of expression given the content; a long response covering much ground can still be concise". Related attributes are still correlated; we kept them on purpose so that the analysis does not miss a quality (see our response to Reviewer QkH1, W2). We will describe the construction in Appendix A.3 and reorganize Table 8 into the ten families plus one group for the six added attributes.

[Q3] Prompts influence what a judge focuses on. How is this controlled?

We agree that the prompt shapes which aspects a judge focuses on. We handle this in two ways.

Same prompts for all judges. All judges receive exactly the same neutral prompts. The overall-judgment prompt names no attributes, and each attribute prompt asks only which response has more of one quality, one attribute per call. (The one exception is Mistral-Small-24B, whose chat template adds a default system prompt; it is flagged in Table 7.) Differences in fitted priorities therefore come from the models themselves.

Prompt steering is measured. Our rubric experiment (Section 4, Exp. 3) changes the prompt to steer a judge's focus: the rubric tells the judge which attributes matter most. For the rebuttal we extended this with rubrics of 3 to 12 criteria and two dedicated rubric-construction methods (see our response to Reviewer QkH1, Q5). Prompting moves accuracy by a few points; reweighting the priorities directly gives larger and more consistent gains.

Thank you again for the review. We would be happy to answer any further questions.

Sincerely,
The Authors of Submission 14498

Reviewer_V5aSOpenReview ↗

Reviewer response

Hi thanks for the experiments and response!

A few follow up questions:

W2: I think a major contribution of the work is to say that the identified 87 attributes are somewhat generalizable to new tasks. Otherwise, the story is more like this paper found a bunch of attributes that works on the original tasks that they were labelled with as well as select Reward Model benchmarks (which is more obvious and less interesting). Therefore, in the GDPVal-AA expts (or some other OOD dataset using LLM judges), I would much rather see how the aspects used to grade the comparable documents (GPT, Gemini and Claude all accept document input in their API in a pretty straightforward manner). If not using the 87 attributes, I think the strength of the contribution is much less. It will be great to come up with a conclusion such as GPT judge prioritizes aspect A, B and C while Gemini prioritize D,E and F. The current experiment frankly doesn't add much to the story if you're just using the supplied rubric criteria (and I would recommend not having them in the paper, since it kind of distracts from the main contribution). Also understand that the remaining time for discussion is very limited, so if running the experiment is not feasible, I would appreciate some guidance on how others can utilize your findings when using LLM Judges in their own tasks (e.g. GDPVal-AA).

Q1: Based on your response to Q2, I am now less sure of the novelty of this work since it seems to be more like a work that consolidates different aspects from prior works with a limited set of new attributes and then use unconditional pairwise judgements to calculate weights for different attributes in specific tasks. Specifically, I want to understand how this work builds upon and differentiates from prior related works. For instance, while it's true that RLBFF only evaluates a single response, it also gives a scalar score per aspect (e.g. Correctness). Hence, if one response is say 5 on correctness and another is 4 on correctness, the first response can be said to be more factually correct. Similarly for Values in the Wild. Can you do more detailed comparisons with those works as well as ArmoRM (Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts at NAACL 2025)? This can help me make a more confident understanding of the actual contributions of the work, with a particularly focus on what readers can learn from this paper that they wouldn't learn from reading those papers.

Reviewer_pMKROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces JudgeProfile to study LLM judge subjectivity by separating perception and prioritization, showing that judges often agree on response attributes but differ in their priorities, which can be adjusted through reweighting.

Critical strengths
  1. JudgeProfile separates perception from prioritization, providing a useful way to study why LLM judges can agree on response attributes but choose different winners.

  2. Reweighting improves average agreement with reference labels across 21 judges without updating the LLM. It also outperforms fine-tuning when training labels are limited.

Critical weaknesses
  1. The main analysis keeps only attribute judgments that are consistent across response orders. Without this filtering, attribute agreement is 76.5%, compared with 76.7% for overall preferences, which limits the claim that perception is more consistent.

  2. Agreement on the observed attributes does not rule out differences in unresolved or unmeasured attributes. Overall judgments are also collected only once, so the contribution of run-to-run variation remains unclear.

  3. The largest gain comes from an RM-Bench subset where reference labels consistently favor plain responses over elaborate ones. This may reward reliance on style rather than correctness, while average performance declines on JudgeBench.

Questions for authors
  1. The main finding of hidden consensus relies on filtering attribute judgments based on response-order consistency. How should this finding be interpreted when attribute agreement is no higher than overall agreement without this filtering?

  2. Could the authors evaluate how much of the disagreement between judges comes from differences in prioritization rather than normal run-to-run variation or unmeasured attributes?

  3. Could the authors further examine whether reweighting learns dataset-specific style patterns rather than improving judge reliability? In particular, why does reweighting reduce performance on JudgeBench?

4 comments

(1/2)

Dear Reviewer pMKR,

Thank you for the detailed review. Below we answer each question together with the related weakness, and include all supporting numbers.

[Q1 / W1] How should hidden consensus be read when, without order filtering, attribute agreement (76.5%) is no higher than overall agreement (76.7%)?

Thank you for the careful reading. You are right that over all pairs, without filtering, the two rates are the same. Hidden consensus also holds in the unfiltered numbers. Hidden consensus is about judges that choose different winners, so their overall agreement on those pairs is 0%; the question is how far their attribute judgments agree beyond that. The 76.5% vs. 76.7% comparison also includes the many pairs where both judges pick the same winner. On different-winner pairs, attribute agreement stays well above 0 with or without the filter:

Two judges choosing different winners (19 judges)Overall agreementAttribute agreement
With order filtering0%85.9%
Without filtering0%69.5%

The filter keeps an attribute judgment only if the judge gives the same answer after the two responses are swapped. It thus keeps attribute differences clear enough not to depend on presentation order and drops slight ones. Two responses differ only slightly on many of the 87 attributes, so the filter focuses the comparison on the clear differences. On these clear differences, judges agree much more on attributes than on overall preference: 90.7% vs. 77.1% on the same items (Table 13). Humans show the same pattern without any filtering: 92.3% agreement on attribute judgments but 62.1% on overall preferences (Table 1). For judges and humans alike, it is easier to agree on which response has more of a quality than on which response is better. The unfiltered numbers are already in Appendix B (Tables 9 and 13); we will state this reading explicitly there.

[Q2 / W2] How much of the disagreement comes from prioritization rather than run-to-run variation or unmeasured attributes?

Thank you for suggesting this check; we ran it. Reweighting changes only the priorities; each judge keeps its own attribute judgments. Its gain therefore measures what prioritization alone accounts for. To compare this with run-to-run variation, we collected each overall verdict three more times with sampling (temperature 0.5) for three judges, giving four runs per pair. We used SubjectiveSet–LLM, where judges disagree most and run-to-run variation should be largest. Accuracies are on its 2,684 test pairs, where reweighting can be evaluated; the SD across runs uses all 10,604 of its non-tie pairs:

Accuracy (%)Run 1 (original)Run 2Run 3Run 4MeanSD across runsReweightedGain (this subset)Gain in paper
Qwen3.5-27B53.955.955.454.054.80.661.4+6.6+3.0
Qwen3.5-9B47.749.251.051.149.71.557.7+8.0+3.7
Gemma-4-31B56.055.754.755.455.50.363.9+8.5+3.9
All 21 judges+9.8+5.5

Run 1 is the original greedy run; Runs 2–4 are the new sampled runs. "Gain in paper" is the per-judge macro gain over the 17 sources (Table 36).

Even on these least stable pairs, run-to-run variation is small (SD 0.3–1.5 pp). The gains from changing priorities are several times larger, so run-to-run noise is unlikely to explain them. We agree that unmeasured attributes can also contribute: a judge's attribute judgments predict its own overall choice with 82.2% accuracy, and the rest may reflect qualities outside the list (see our response to Reviewer QkH1, W2).

(2/2)

[Q3 / W3] Does reweighting learn dataset-specific style patterns rather than improving judge reliability? Why does JudgeBench decline?

Thank you for raising this. Two things need checking: that the gains come from what judges perceive rather than from a style shortcut, and that they do not cost reliability. We address them in turn.

RM-Bench. Our 0:2 pairing (App. F.2) contrasts a plain chosen response with an elaborate, formatted rejected one, so a style shortcut would be easiest here. Removing every style-related attribute still keeps over 94% of the gain:

RM-Bench, mean over 21 judgesAccuracy (%)
Unadapted judge65.1
Reweighting, all 87 attributes94.4
Reweighting, without style / format / content / information attributes92.8

The judges perceive differences beyond formatting, and reweighting makes use of them.

JudgeBench. JudgeBench is small (154 test pairs) and mostly tests correctness, and 10 of the 19 judges already exceed 80%. The average decline comes from these strong judges; weaker judges gain substantially:

JudgeBench, 19 judgesUnadaptedReweighted
5 weakest judges (unadapted < 65%), mean58.268.7
10 strongest judges (unadapted ≥ 80%), mean86.876.6

This follows from what reweighting uses. It ignores the judge's overall verdict and predicts the label from the 87 attribute judgments alone, with one fixed set of weights per source. A weaker judge's overall verdict is often wrong, and its attribute judgments (e.g., which final answer is more correct) are more reliable, so reweighting helps. A strong judge's verdict is right about 87% of the time; on hard reasoning problems it draws on many problem-specific cues that a fixed linear combination of attributes cannot fully match, so accuracy drops. This reflects the limitation of the linear form acknowledged in Section 6. The same pattern appears across all sources (see our response to Reviewer QkH1, Q3): gains average +11.0 pp for judge–source cells starting below 60% accuracy and +0.2 pp for those starting above 80%. We will add both analyses to Appendix F.

Thank you again for the review. We would be happy to discuss further.

Sincerely,
The Authors of Submission 14498

Reviewer_pMKROpenReview ↗

Reviewer Response

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. The responses have adequately addressed my main questions, and I will maintain my original rating.

Thank you again for the thoughtful responses.

Dear Reviewer pMKR,

Thank you for reading our responses and for your careful review. Your questions on run-to-run variation and JudgeBench led to analyses that make the paper more precise.

Sincerely,
The Authors of Submission 14498

Reviewer_QkH1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the subjectivity of LLM-as-a-Judge, where LLMs show different preferences over response pairs, and introduces JudgeProfile, a framework that separates perception from prioritization. It reports hidden consensus in perception and characterizes judges' prioritization as attribute weights, and further contributes the SubjectiveSet dataset and a reweighting method that adjusts judges' prioritization to target preferences.

Critical strengths
  1. The separation of perception and prioritization presents a quantitative way to investigate the overall judgments from LLM, and the attributes are explainable.
  2. The reweighting method shows a lightweight way to adjust LLMs' judgments, without fine-tuning their base model.
Critical weaknesses
  1. The paper treats dataset-level preference labels as if they reflected a coherent “evaluation standard” that reweighting can recover. In practice, human labels aggregate heterogeneous annotator biases, and AI labels encode one model’s fixed preferences. The AI-labeled sources appear to contain a single annotation outcome rather than multiple labels obtained by injecting different preferences into the same data. This limits the support for the paper’s claim that reweighting adapts a judge to a target evaluation standard: without multiple preference labels over the same data, it is difficult to distinguish adaptation to a specified standard from fitting the default preference of a particular AI annotator.
  2. The 87 attributes form the foundation of both perception and prioritization, yet the paper does not explain how they were derived, whether they contain substantial redundancy that would shrink the effective dimensionality below 87, or how well they cover the dimensions that shape LLM judgments.
  3. Although reweighting improves agreement, it changes the inference cost from a single LLM interaction to 87 attribute-level interactions (174 calls when both presentation orders are counted), and the paper does not discuss this cost.
Questions for authors
  1. The paper does not explain how the 87 attributes were originally derived. Could the authors clarify the construction process for this attribute set?
  2. The paper does not specify whether the AI annotator was instructed to follow an explicit preference during labeling. Could the authors clarify the annotation protocol for the AI-labeled sources (e.g., Nectar, UltraFeedback)?
  3. Table 37 shows highly heterogeneous gains: seven sources are relatively small (<= +2.02) and two are negative, while the macro average is driven by RM-Bench, MT-Human, and Nectar. Could the authors explain what distinguishes sources with large gains from those with small or negative gains—e.g., label source, task type, attribute coverage, or the strength of the linear attribute–label relationship?
  4. The sample-efficiency experiment (Figure 9) uses 9 judges, and the controlled comparisons (Tables 2 and 3) use 4 judges, but the paper does not explain how these subsets were selected from the 21 judges. Could the authors list the judges used in each experiment and describe the selection criteria?
  5. The six rubric criteria are selected as the attributes with the largest absolute weights from the reweighting fit. This selection is optimized for reweighting's prediction objective rather than for rubric construction, and rubric prompting inherits these attribute names without richer descriptions or examples. Could this asymmetry disadvantage the rubric baseline? Would the comparison in Table 3 change if the rubric criteria were constructed according to the PReMISE or CalibratedRubric pipeline, or if the number of criteria were varied?
8 comments

(1/2)

Dear Reviewer QkH1,

Thank you for the detailed review. We answer each question together with the related weakness.

[Q1] How were the 87 attributes derived?

Thank you for the question. We wanted a broad list of fine-grained qualities on which two responses can differ, built in two steps:

  1. Literature-guided candidates. We started from the quality dimensions used in prior evaluation work, all cited in the paper: FLASK's fine-grained skills; the attributes rated in HelpSteer/HelpSteer2 and UltraFeedback; G-Eval's evaluation dimensions; helpfulness and harmlessness (Bai et al., 2022) and length effects (Dubois et al., 2024); and the custom and data-discovered criteria of Prometheus, VibeCheck and WIMHF.

    We grouped these dimensions into ten families (e.g., correctness, reasoning, style, social) and split each family into specific attributes that can be judged separately, each with a definition and a boundary against its neighbours (see W2). This gave 81 candidates.

  2. Real-case human check. On the SubjectiveSet–Human cases, we checked whether these attributes capture what actually distinguishes the two responses, and added 6 qualities that were missing (elaboration, difficulty, formatting, forcefulness, caution framing, phrasing originality). This gives the final 87.

We will revise Appendix A.3 to describe this process and reorganize Table 8 into the ten literature-guided families plus one group for the six added attributes.

[W2] Are the 87 attributes redundant? Do they cover what shapes judgments?

Thank you for raising this; both properties matter for attribute inventories.

Redundancy. Correlation between attributes does exist. Our aim was to make each attribute distinguishable and, on that basis, to cover the important qualities. Each attribute has a definition and a boundary that names its closest neighbours and says how they differ. For example:

  • verbosity: "Raw amount only. Whether the length is justified is conciseness."
  • conciseness: "Economy of expression given the content. Not raw shortness (verbosity): a long response covering much ground can still be concise."

No two attributes are duplicates: for each judge, the Pearson correlation between two attributes' judgments over all response pairs is at most r = 0.83–0.90.

Redundancy does not change our findings. We removed correlated attributes until no two exceed r = 0.5 (averaged over judges), which leaves 44. With these 44, hidden consensus is unchanged (83.0% vs. 83.8% with all 87, over all 21 judges) and reweighting still improves all 21 judges (+4.5 vs. +5.5 pp).

Coverage. The list starts from the literature and was checked on real cases (Q1). For the data studied in the paper, two results speak to coverage:

  • A judge's 87 attribute judgments predict its own overall choice with 82.2% five-fold accuracy using a linear model (Table 19). The rest reflects qualities outside the list, non-linear trade-offs or noise.
  • The attributes are already sufficient to steer judges: reweighting improves all 21 judges (Figure 8).

We will add this discussion to Appendix A.3.

[Q2] Annotation protocol of AI-labeled sources.

Thank you for asking; this matters for how these labels should be read. The two AI-labeled sources were annotated as follows:

  • UltraFeedback. GPT-4 used a fine-grained rubric (instruction following, truthfulness, honesty, helpfulness); we pair the best completion with a random lower-scored one.
  • Nectar. GPT-4 ranked seven responses under a helpful-and-harmless instruction; we pair rank 1 with a random one from ranks 2–7.

We will state this in Table 5.

[W1] Do dataset labels define a target standard, or just one annotator's default preference?

Thank you for this thoughtful point. We agree that labels from a single AI annotator carry that model's preferences, so gains on such labels alone could reflect one annotator's default. Our results do not depend only on such labels: 15 of the 17 sources are labeled in other ways.

  • Human labels (9 sources). Every source is labeled by many people, and in several each item combines several people's judgments (HelpSteer2: at least three annotators per response; MT-Bench Human: several experts per pair; SHP: votes of many community users).
  • Benchmark labels (6 sources), set by how the benchmark is built rather than by an annotator's preference. JudgeBench pairs a response whose final answer matches the ground truth with one that does not; RewardBench includes, e.g., correct vs. buggy code.
  • AI labels (2 sources), each from a single model (GPT-4; Q2).

Reweighting improves agreement on each kind of label:

Label typeSourcesMean reweighting gain
Human9+4.7 pp
Benchmark6+6.4 pp
AI (GPT-4)2+6.2 pp

The mean gain is positive on all 9 human-labeled sources, so reweighting adapts to targets beyond one AI annotator's preference.

(2/2)

[Q3] What distinguishes sources with large vs. small or negative gains?

Thank you for the question. Two factors matter: how much room the unadapted judge leaves, and whether a linear combination of attribute judgments can capture the labels.

Large gains appear where both hold. In the three sources with the largest gains, unadapted judges reach only 62–65%, while a single attribute predicts the labels much better:

SourceUnadapted judgeBest single attributeGain
RM-Bench65.1formatting, 98.7% (less formatted wins)+29.4
MT-Human62.1reasoning completeness, 85.5%+16.5
Nectar64.5conceptual complexity, 83.7%+10.6

The judges perceive these qualities but weigh them differently from the labels, and reweighting adjusts these priorities. (On RM-Bench, over 94% of the gain remains after removing all style attributes; see our response to Reviewer pMKR.)

Small or negative gains appear where the judge is already strong. Across all 357 judge–source cells:

Unadapted accuracy< 60%60–70%70–80%≥ 80%
Mean gain (pp)+11.0+4.8+2.9+0.2

Reweighting replaces the judge's verdict with a linear combination of its attribute judgments. When a strong judge's verdict already captures more than its attribute judgments do, as on RewardBench and JudgeBench, there is little to gain and some accuracy can be lost. This is the limitation of the linear form noted in Section 6.

[Q4] Which judges are used in Figure 9 and Tables 2–3, and why?

The judge sets follow the purpose of each experiment:

  • Figure 9 (sample efficiency, 9 judges): DeepSeek-R1-Distill-32B, Gemma-3-27B, Gemma-4-26B-A4B, Gemma-4-31B, Mistral-Small-24B, Muse-Glimmer-30B, Qwen3.5-27B, Qwen3.6-27B and Llama-4-Scout (MoE, 17B active). This analysis asks how many labels are needed to adapt a judge in practice, so we used mid-sized judges (24–32B), in the 10–100B range commonly used for dedicated judge models (e.g., JudgeLM-13B and JudgeLM-33B; Zhu et al., 2025).
  • Tables 2–3 (controlled comparisons, 4 judges): Qwen3.5-4B, 9B and 27B form a size ladder within one family, and Llama-4-Scout adds a different family and architecture. These comparisons need per-source LoRA fine-tuning at every label budget plus a full rubric run per judge, so we kept the set small while still varying size and family.

We will list these judge sets and the reasons for them in Table 4.

[Q5] Would the comparison change with a dedicated rubric-construction method, or with a different number of criteria?

Thank you for the suggestion; we ran both. Our rubric criteria give each attribute's full definition, direction and importance rank. We added rubrics with 3 and 12 criteria, and two 6-criterion rubrics built from the training split in the style of PReMISE (criteria discovered from labeled pairs, filtered for order-swap reliability, ranked by preference fit) and CalibratedRubric (same candidates, filtered by cross-judge measurability, assembled for coverage). All use the same 11,765 test pairs:

Accuracy (%)Rubric k=3 / 6 / 12PReMISE (6)CalibratedRubric (6)Reweighting k=3 / 6 / 12
Qwen3.5-4B66.88 / 66.83 / 66.2965.8765.5567.56 / 68.58 / 69.15
Qwen3.5-9B67.94 / 68.14 / 68.4768.7068.5969.69 / 70.94 / 71.25
Qwen3.5-27B73.43 / 73.33 / 72.9871.5271.5071.94 / 73.26 / 74.21

Neither change alters the comparison. Rubrics built with these methods perform about the same as ours or slightly worse, and adding criteria moves rubric accuracy by less than 1 pp, while reweighting keeps improving as attributes are added. The best reweighting setting beats every rubric variant on all three judges. A rubric tells the judge what to look at, but the judge still decides how to trade the criteria off; reweighting sets that trade-off directly.

[W3] Inference cost.

We agree that cost matters. Eliciting all 87 attributes in both orders takes 174 calls per pair; this full set is meant for analysis, where coverage matters most. To steer a judge in practice, the set should be reduced to the attributes that matter for the target, which differ from one dataset to another. Table 3 therefore also reports reweighting with 6 and 12 attributes, selected once on the target's training split: 12 attributes keep 91–98% of the full gain with 24 instead of 174 calls per pair. Each call is also short: the judge compares a single quality without extended reasoning and outputs about 20 tokens, so this setting is practical to use. We will add this cost discussion to the appendix.

Thank you again for the review. We are happy to answer any further questions.

Sincerely,
The Authors of Submission 14498

Thank you for the detailed response. Most of our concerns have been addressed, but I have a follow-up on W1: could the authors label the same dataset using different AI models, or the same model instructed to adopt explicitly different preferences, and test whether the proposed method yields separate gains on each target rather than merely fitting one shared preference?

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the feedback. We have uploaded a revised manuscript. To make the changes easy to locate, we have marked them in blue throughout the manuscript. The main updates are as follows.

  • Attribute construction (QkH1 and V5aS). Section 2 explains how we derived the 87 attributes. Table 1 summarizes the ten attribute families, and Appendix A.3 provides the full inventory.
  • Novelty and practical implications (QkH1 and V5aS). Section 5 profiles individual judges on chat responses and GDPval-AA documents. The conclusion explains how benchmark developers and RL trainers can use these profiles to assess candidate judges. Section 6 clarifies how JudgeProfile differs from RLBFF, ArmoRM, and Values in the Wild by separating what judges perceive from what they prioritize. Conclusion section is also rewritten based on the revisions.
  • Additional analyses (all three reviewers). Appendix F reports the RM-Bench ablation and JudgeBench breakdown requested by pMKR, the rubric comparisons and the results across target annotators requested by QkH1, and reference-label types discussed with V5aS.

For the final version. Several writing suggestions came up near the end of the discussion. We appreciate the reviewers’ feedback and will use it to improve the clarity of both the main text and the appendix in the final version. We will make the main takeaway of each group of experiments more explicit to improve readability (QkH1). We will also build on the practical guidance by discussing criterion-specific prompts, judge ensembles, and how to match human annotators’ preferences (V5aS).

Sincerely,
The Authors of Submission 14498

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.