- Rating
- 4(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
LLMs are subjective and hence can prefer different responses in pairwise comparison. This paper studies how judges compare responses across attributes like clarity, correct and detail and prioritization. The paper finds that judges frequently agree on attribute judgements but overall judgement differs. They recalibrate attribute weights based on reference labels to improve agreements with held out set.
Critical strengths
- This paper addresses an important question as LLM judges are increasingly used in various evaluations. The study approach contains both a LLM-judge set and a human set, that indicates how both agree more on attribute different but less on overall preferences, which is an insightful finding.
- The reweighing experiment is a good method for steering LLM Judge to have less implicit bias based on the attributes they emphasize more or less on.
Critical weaknesses
- There should be more thorough data cleaning/categorization. For instance, a lot of the ground truth preferences in some datasets (e.g. RewardBench, Nectar) are themselves based on LLM-judges (which had specific prompts based on their category). While the paper is seeking to cover as many datasets as possible, it’s useful to distinguish human-labels from LLM-labels (which makes the study kind of circular – LLM judge predicting the preference of another LLM Judge).
- (Not strictly a weakness but an area for improvement) I think the current flow of the paper reads like – we find that LLM-Judge agree on attributes not on the prioritization and we train models to recalibrate their prioritization and they predict the ground truth labels better. This is fine but I think the broader impact of this work is that many evals treat LLM-Judges as interchangeable black boxes and if the authors can give some recommendations regarding how to better calibrate the judges in unseen evals (say GDPVal-AA, ICLR 2026), it can make the paper a lot more impactful.
Questions for authors
- Can the authors make a comparison with this paper “RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards” (ICLR, 2026) and “Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions” (COLM, 2025)? The concept of perception in this paper seems very similar to principles and values in these papers.
- The paper finds 87 attributes but it’s not clear how those attributes were found. Is it based on prompting llm to extract attributes from judgement rationales? How is data cleaning and deduplication handled?
- For LLM-judge, a critical aspect on which aspects they focus more on depends on the prompt and hence it could significantly influence the aspects that influence overall preference. How is that controlled for in this study?
7 comments
(2/2)
[Q1] Comparison with RLBFF (ICLR 2026) and Values in the Wild (COLM 2025).
Thank you for pointing us to these two papers; both are closely related to our work. The main difference lies in what we mean by perception: how a judge sees the differences between two responses. For each attribute, the judge says which response has more of it, without saying which response is better.
RLBFF. RLBFF's principles and our attributes both break response quality into specific criteria. A principle, however, is checked on a single response (satisfied or not). Our perception is comparative and non-evaluative: it records which of two responses shows more of a quality. This lets us separate perceiving a difference from deciding how much it matters (prioritization).
Values in the Wild. This work finds the values an assistant expresses in real conversations, which can shift with context. These values describe what a model cares about, so they are closer to our priorities than to our perception. They are also observed while the model responds to users, whereas we study a model evaluating two responses.
We will add both papers to the "LLM-as-a-Judge and subjectivity" part of Related Work.
[Q2] How were the 87 attributes found? How were cleaning and deduplication handled?
Thank you for the question. We built the attributes in two steps, starting from dimensions that prior work derived from data and then checking them against real response pairs:
- Literature-guided candidates. We started from the quality dimensions used in prior evaluation work, all cited in the paper: FLASK's fine-grained skills; the attributes rated in HelpSteer/HelpSteer2 and UltraFeedback; G-Eval's evaluation dimensions; helpfulness and harmlessness (Bai et al., 2022) and length effects (Dubois et al., 2024); and the custom and data-discovered criteria of Prometheus, VibeCheck and WIMHF. We grouped these dimensions into ten families (e.g., correctness, reasoning, information, style, epistemic, social, alignment) and split each family into specific attributes that can be judged separately. This gave 81 attributes.
- Real-case human check. On the SubjectiveSet–Human cases, we checked whether these attributes capture what actually distinguishes the two responses, and added 6 qualities that were missing (elaboration, difficulty, formatting, forcefulness, caution framing, phrasing originality), giving 87.
Deduplication. We avoided exact duplicates while writing the attributes rather than removing them afterwards. Because each family was divided top-down, each attribute was written alongside its neighbours, with a definition and a boundary that names the closest neighbours and states how they differ. For example, verbosity is "raw amount only; whether the length is justified is conciseness", while conciseness is "economy of expression given the content; a long response covering much ground can still be concise". Related attributes are still correlated; we kept them on purpose so that the analysis does not miss a quality (see our response to Reviewer QkH1, W2). We will describe the construction in Appendix A.3 and reorganize Table 8 into the ten families plus one group for the six added attributes.
[Q3] Prompts influence what a judge focuses on. How is this controlled?
We agree that the prompt shapes which aspects a judge focuses on. We handle this in two ways.
Same prompts for all judges. All judges receive exactly the same neutral prompts. The overall-judgment prompt names no attributes, and each attribute prompt asks only which response has more of one quality, one attribute per call. (The one exception is Mistral-Small-24B, whose chat template adds a default system prompt; it is flagged in Table 7.) Differences in fitted priorities therefore come from the models themselves.
Prompt steering is measured. Our rubric experiment (Section 4, Exp. 3) changes the prompt to steer a judge's focus: the rubric tells the judge which attributes matter most. For the rebuttal we extended this with rubrics of 3 to 12 criteria and two dedicated rubric-construction methods (see our response to Reviewer QkH1, Q5). Prompting moves accuracy by a few points; reweighting the priorities directly gives larger and more consistent gains.
Thank you again for the review. We would be happy to answer any further questions.
Sincerely,
The Authors of Submission 14498
Reviewer response
Hi thanks for the experiments and response!
A few follow up questions:
W2: I think a major contribution of the work is to say that the identified 87 attributes are somewhat generalizable to new tasks. Otherwise, the story is more like this paper found a bunch of attributes that works on the original tasks that they were labelled with as well as select Reward Model benchmarks (which is more obvious and less interesting). Therefore, in the GDPVal-AA expts (or some other OOD dataset using LLM judges), I would much rather see how the aspects used to grade the comparable documents (GPT, Gemini and Claude all accept document input in their API in a pretty straightforward manner). If not using the 87 attributes, I think the strength of the contribution is much less. It will be great to come up with a conclusion such as GPT judge prioritizes aspect A, B and C while Gemini prioritize D,E and F. The current experiment frankly doesn't add much to the story if you're just using the supplied rubric criteria (and I would recommend not having them in the paper, since it kind of distracts from the main contribution). Also understand that the remaining time for discussion is very limited, so if running the experiment is not feasible, I would appreciate some guidance on how others can utilize your findings when using LLM Judges in their own tasks (e.g. GDPVal-AA).
Q1: Based on your response to Q2, I am now less sure of the novelty of this work since it seems to be more like a work that consolidates different aspects from prior works with a limited set of new attributes and then use unconditional pairwise judgements to calculate weights for different attributes in specific tasks. Specifically, I want to understand how this work builds upon and differentiates from prior related works. For instance, while it's true that RLBFF only evaluates a single response, it also gives a scalar score per aspect (e.g. Correctness). Hence, if one response is say 5 on correctness and another is 4 on correctness, the first response can be said to be more factually correct. Similarly for Values in the Wild. Can you do more detailed comparisons with those works as well as ArmoRM (Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts at NAACL 2025)? This can help me make a more confident understanding of the actual contributions of the work, with a particularly focus on what readers can learn from this paper that they wouldn't learn from reading those papers.
(1/2)
Dear Reviewer V5aS,
Thank you for the detailed review. All supporting numbers are included below (1/2).
[W1] Distinguishing human labels from LLM labels; is the study circular?
We agree that human and LLM labels should be clearly separated. Table 5 marks each of the 17 sources as human-labeled (9), AI-labeled (2: Nectar and UltraFeedback) or benchmark-labeled (6). Benchmark labels are set by how the benchmark is built: JudgeBench pairs a response whose final answer matches the ground truth with one that does not. You are right that RewardBench is mixed. Its code, math and safety subsets are defined by correctness (e.g., correct vs. buggy code, a refusal vs. a dangerous completion), while its chat subsets contrast responses of stronger and weaker models, some selected with GPT-4 ratings. We will mark this in Table 5. Reweighting gains by label type:
The gains hold for human and benchmark labels, so they do not depend on LLM-produced labels. Our other findings, hidden consensus and judge-specific priorities (Sections 3–4), compare judges with each other and use no reference labels. We will report all adaptation results by label type and state each source's labeling protocol in Table 5.
[W2] Recommendations for calibrating judges on unseen evaluations (e.g., GDPVal-AA, ICLR 2026).
Thank you for this suggestion. We ran a small study on GDPval-AA. GDPval-AA grades open-ended professional work with a panel of three judges (Claude Opus 5, GPT-5.6 Sol and Gemini 3.8 Flash), one judge per comparison. On 44 tasks (one per occupation), we took the deliverables of two public models, GPT-6 Astra and Qwen3.8-27B, and asked all three judges for an overall verdict and for a verdict (A, B or same) on every criterion of the task's own rubric (2,031 criteria). Our 87 attributes were designed for and validated on single-turn text responses, the setting of all 17 sources in the paper. GDPval deliverables are files produced by multi-step agentic work, which we have not validated them on, so in this first study we used the rubric criteria as attributes.
Hidden consensus also holds on agentic tasks. The judges often disagree on the overall winner but agree on the criteria:
The judges' priorities differ. Astra's win rate ranges widely across judges. Combining each judge's criterion verdicts with one shared set of weights (the rubric's point values) halves this range:
So judging at the criterion level is more consistent across judges and easier to inspect than overall verdicts. The last row also shows its current limit: each judge's overall verdict departs from its own criterion verdicts on 25–36% of the pairs, so the task rubrics alone do not explain what each judge prefers. Rubric criteria are also specific to each task, so weights cannot be fitted across tasks as in our reweighting.
Based on this, we recommend to (1) ask judges for criterion-level verdicts rather than only an overall verdict, and report both; and (2) combine them with shared, explicit weights. Full calibration needs attributes that are shared across tasks. Extending and validating our attributes for agentic outputs would allow each judge's priorities to be measured and fitted to a small set of expert labels (128–256 suffice; Figure 9). We will add this study to the appendix and discuss this extension as future work.