- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates whether LLM judges issue definitive verdicts when the evaluation criteria do not establish a unique winner. To study this issue, the authors introduce TradeoffBench, a controlled benchmark that tests compliance with an explicit decision rule using verified criterion-level comparisons, while varying input format, candidate order, and reasoning mode. The paper also proposes an identified-first evaluation framework and analyzes human criterion scores from MultiPref and HUMAINE to examine how score sources and explicitly specified trade-offs affect decision coverage and agreement with individual human raters.
Critical strengths
The paper addresses an important question concerning the reliability of rubric-based LLM evaluation. TradeoffBench offers a formal characterization of unsupported verdicts and a controlled setting for testing whether judges adhere to explicit decision rules, providing a useful diagnostic framework.
Critical weaknesses
- Limited realism and practical validation of TradeoffBench.
TradeoffBench relies on simple arithmetic and related tasks with five mechanically verifiable binary criteria, which differ substantially from realistic LLM evaluation settings. Moreover, the primary experiment replaces response text with anonymous criterion-comparison vectors. While this design effectively isolates rule-following failures, it remains unclear whether the observed violations persist when judges evaluate open-ended responses. The subsequent analyses of MultiPref and HUMAINE rely on human-provided criterion scores rather than directly evaluating LLM judge behavior. Consequently, the practical relevance of the benchmark's findings to real-world LLM judging remains insufficiently established.
- Limited model coverage and uncertainty about the generality of the observed failures.
The main audit evaluates six open-weight models ranging from 7B to 32B parameters, primarily in non-reasoning configurations. In contrast, the reasoning-enabled follow-up reports only two violations across 9,720 conflict judgments, suggesting that the observed failures may be largely attributable to instruction-following limitations in specific model configurations rather than a pervasive issue in LLM judging. Furthermore, the absence of frontier models such as GPT, Claude, and Gemini makes it difficult to assess whether these failures persist in stronger, widely used LLM judges. This limits the practical significance and generalizability of the findings.
Questions for authors
-
Could the authors evaluate TradeoffBench's central hypothesis in more realistic settings, such as open-ended question answering or writing comparisons with human-verified conflicting criterion judgments?
-
Could the authors extend the audit to frontier LLM judges, such as GPT, Claude, and Gemini, under their standard evaluation configurations?
4 comments
Frontier model coverage and the scope of the result
W2 and Q2. A frontier judge with reasoning enabled. We added GPT-6.1-sol on the same 800 raw TradeoffBench presentations under two conditions: 200 verified conflict pairs, 100 dominance pairs, and 100 equality pairs, each in both orders. Both runs use the Responses API, medium reasoning effort, Standard service, and a 2,048-token cap including reasoning.
| Trade-offs | Strict verdicts on verified conflicts | Of these, reported conflict | Unstated premise | Correct controls |
|---|---|---|---|---|
| Explicit deferral rule | 0/400 | — | — | 400/400 |
| Unspecified | 176/400 (44.0%) | 176/176 | 119/176 | 400/400 |
The frontier judge follows the explicit rule on every verified conflict. Under the same rule, reasoning-enabled Qwen3-8B and Qwen3.8-27B produce 84/400 and 31/400 strict verdicts on verified conflicts, with 396/400 and 400/400 controls correct. These raw-text judgments jointly test component assessment and aggregation.
With trade-offs unspecified, GPT-6.1-sol reports all 400 conflicts yet resolves 176; 153 strict verdicts follow component reports matching the complete verified vector. Of the 176 completions, 52 state an added premise, 119 leave it unstated, and five are ambiguous. Within the unstated group, 52 final answers contain only components and a verdict. GPT-6.1-sol follows the explicit rule but supplies undeclared completions when trade-offs are unspecified.
All 1,600 outputs are valid and complete, with one generation per presentation and no retries. The explicit condition returns only a verdict; the unspecified condition requests components and permits explanations. The decision instruction and output format change together. The strict-verdict rate is 44.0% (task-cluster bootstrap 95% CI [37.6, 50.7]); the unstated-premise rate is 29.8% of verified-conflict presentations ([24.3, 35.6]; Appendix F.1).
Open-ended and open-weight evaluations. GPT-6.1-sol also evaluates the 73 four-rater-consensus MultiPref pairs described in our first comment: 581/584 requests return valid, complete outputs, with three service rejections unresolved. Qwen3.5-9B repeats the design with thinking enabled, model-card sampling settings, and an 8,192-token output budget. One pair associated with the hosted-service safety rejection is not generated locally, leaving 72 pairs and eight planned presentations ungenerated, without replacement.
All 47 auditable Qwen3.5-9B strict conflict verdicts under unspecified trade-offs state a priority or trade-off: three with assessed components and 44 with fixed components. Under the explicit rule, no strict verdict occurs over a reported conflict. Primary parsing resolves 481/576 outputs; literal-field sensitivity resolves 542/576 without changing these strict conflict counts. The remaining 34 outputs, including one capped response, remain unresolved (Appendix F.3). This comparison documents explicit completion and compliant deferral on conflicts.
Scope. The frontier extension covers one hosted model on verified synthetic tasks and open-ended MultiPref responses. Successful rule execution coexists with silent completion across the tested instruction conditions.
Thank you for addressing my concerns, I am satisfied with the response and will raise my score accordingly.
Thank you for raising the score
Thank you for your positive feedback and for reconsidering your assessment. We sincerely appreciate your time and thoughtful review.
Real responses with human-verified component conflicts
Thank you. The new MultiPref audit includes silent completions whose reported components exactly match a four-rater human reference.
W1 and Q1. Realistic inputs and human conflict judgments. We use all 73 pairs in the 700-pair MultiPref audit sample for which four distinct recorded raters agree on the complete helpfulness, truthfulness, and harmlessness sign vector: 23 conflicts, 23 dominance cases, and 27 equalities. The model reads the original request and complete responses in both orders.
The design crosses model-assessed versus fixed human-reference components with unspecified trade-offs versus an explicit strict Pareto rule. Human labels are withheld in assessed conditions. Fixed-component conditions isolate aggregation while retaining actual response text. All conditions allow explanations, equality, and deferral.
Silent and explicit completion. With unspecified trade-offs, GPT-6.1-sol gives 36 strict verdicts over its own reported conflicts: 29 state a premise, six leave it unstated, and one is ambiguous. Two unstated-premise outputs match all four raters' complete component sign vector. A stated natural-language priority, outweighing relation, or criterion-specific reason for the overall choice counts as explicit; numerical weights are unnecessary.
With fixed human-reference components, both conditions preserve all 100 dominance/equality controls, while strict conflict verdicts change from 36 to zero under the explicit rule. All 36 fixed-component completions state a premise. Control-verdict agreement in the assessed-component conditions is 62/100 versus 50/100, reflecting changes in both assessment and aggregation.
Component assessment under the explicit rule. On the 23 four-rater-consensus conflict pairs, model-assessed explicit-rule judgments give:
Counts use valid outputs under each audit's primary parsing policy; planned presentations include both orders. Rule mismatches are measured against each output's reported components. GPT-6.1-sol has one service rejection. Qwen3.5-9B generates 44 outputs: 15 remain unresolved under primary parsing and two planned presentations are ungenerated. Appendix F.2 reports parsing sensitivity and identifies the parent audit for each model.
Every strict verdict in this table follows reported dominance while the human reference is a conflict. These are component disagreements, not aggregation violations: identification checks support relative to the measured components.
The human annotations also separate components from overall preferences. Among the 23 unanimous-sign conflict pairs, seven have opposing strict overall labels; four retain opposing labels among raters with identical full ordinal components. This establishes non-uniqueness in the recorded mapping.
Sec. 3.5 presents both response-text settings; our second comment reports the complementary TradeoffBench test with mechanically verified components. Appendices F.2–F.3 give protocols and full counts. The stratified samples and existing multi-rater annotations support case-level analysis, not deployment-frequency estimates.