acceptodds
Under review as a conference paper at ICLR 2027

When Rubrics Do Not Decide: Silent Value Completion in LLM Evaluation

Abstract

LLM judges are often given rubrics that score responses on several criteria without specifying how conflicts between criteria should be resolved. When criteria conflict, even exact scores may not determine a unique winner. A strict verdict in such a case, therefore, requires an additional trade-off premise not supplied by the rubric. We call an undeclared resolution of this kind silent value completion. We introduce TradeoffBench to test for this failure while separating rule execution from score estimation. Six tested open-weight judges in non-reasoning configurations pass every dominance and equality control, yet select unsupported winners on 9%-81% of 1,345 conflicting pairs. Among the 577 model-pair cases that retain the same winner after display reversal, every selected response passes more criteria, despite an explicit ban on majority voting. In contrast, reasoning-enabled configurations of three models produce only 2 unsupported winners in 9,720 conflict judgments, almost always deferring when the rubric does not determine a winner. Such deferral respects the rubric, but the comparison remains unresolved. Resolving it requires additional trade-off premises, whose resulting verdicts may still disagree with the intended rater.We therefore propose identified-first evaluation, which first identifies strict verdicts supported by every allowed weighting, then makes any added trade-off premises explicit, and separately measures disagreement with the intended rater. We next use human criterion scores to examine the consequences of these added decision premises without confounding them with LLM judge behavior. In HUMAINE, declared trade-offs increase coverage by 2.5% points over strict Pareto dominance, while added decisions disagree more often with strict rater preferences than Pareto decisions (3.9% vs. 0.9%). In MultiPref, with the target and decision rule fixed, replacing raters' own scores with peer means increases disagreement from 0.6% to 28.8% on the same 2,707 jointly decided comparisons. Together, our benchmark and framework show that evaluating an LLM judge requires separating rubric support, explicit trade-off completion, and agreement with the intended rater.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM judges, silent value completion, TradeoffBench, identified-first evaluation, necessary preference, strict Pareto dominance, explicit trade-off completion, target-risk auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YzfgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates whether LLM judges issue definitive verdicts when the evaluation criteria do not establish a unique winner. To study this issue, the authors introduce TradeoffBench, a controlled benchmark that tests compliance with an explicit decision rule using verified criterion-level comparisons, while varying input format, candidate order, and reasoning mode. The paper also proposes an identified-first evaluation framework and analyzes human criterion scores from MultiPref and HUMAINE to examine how score sources and explicitly specified trade-offs affect decision coverage and agreement with individual human raters.

Critical strengths

The paper addresses an important question concerning the reliability of rubric-based LLM evaluation. TradeoffBench offers a formal characterization of unsupported verdicts and a controlled setting for testing whether judges adhere to explicit decision rules, providing a useful diagnostic framework.

Critical weaknesses
  1. Limited realism and practical validation of TradeoffBench.

TradeoffBench relies on simple arithmetic and related tasks with five mechanically verifiable binary criteria, which differ substantially from realistic LLM evaluation settings. Moreover, the primary experiment replaces response text with anonymous criterion-comparison vectors. While this design effectively isolates rule-following failures, it remains unclear whether the observed violations persist when judges evaluate open-ended responses. The subsequent analyses of MultiPref and HUMAINE rely on human-provided criterion scores rather than directly evaluating LLM judge behavior. Consequently, the practical relevance of the benchmark's findings to real-world LLM judging remains insufficiently established.

  1. Limited model coverage and uncertainty about the generality of the observed failures.

The main audit evaluates six open-weight models ranging from 7B to 32B parameters, primarily in non-reasoning configurations. In contrast, the reasoning-enabled follow-up reports only two violations across 9,720 conflict judgments, suggesting that the observed failures may be largely attributable to instruction-following limitations in specific model configurations rather than a pervasive issue in LLM judging. Furthermore, the absence of frontier models such as GPT, Claude, and Gemini makes it difficult to assess whether these failures persist in stronger, widely used LLM judges. This limits the practical significance and generalizability of the findings.

Questions for authors
  1. Could the authors evaluate TradeoffBench's central hypothesis in more realistic settings, such as open-ended question answering or writing comparisons with human-verified conflicting criterion judgments?

  2. Could the authors extend the audit to frontier LLM judges, such as GPT, Claude, and Gemini, under their standard evaluation configurations?

4 comments
AuthorsOpenReview ↗

Real responses with human-verified component conflicts

Thank you. The new MultiPref audit includes silent completions whose reported components exactly match a four-rater human reference.

W1 and Q1. Realistic inputs and human conflict judgments. We use all 73 pairs in the 700-pair MultiPref audit sample for which four distinct recorded raters agree on the complete helpfulness, truthfulness, and harmlessness sign vector: 23 conflicts, 23 dominance cases, and 27 equalities. The model reads the original request and complete responses in both orders.

The design crosses model-assessed versus fixed human-reference components with unspecified trade-offs versus an explicit strict Pareto rule. Human labels are withheld in assessed conditions. Fixed-component conditions isolate aggregation while retaining actual response text. All conditions allow explanations, equality, and deferral.

Silent and explicit completion. With unspecified trade-offs, GPT-6.1-sol gives 36 strict verdicts over its own reported conflicts: 29 state a premise, six leave it unstated, and one is ambiguous. Two unstated-premise outputs match all four raters' complete component sign vector. A stated natural-language priority, outweighing relation, or criterion-specific reason for the overall choice counts as explicit; numerical weights are unnecessary.

With fixed human-reference components, both conditions preserve all 100 dominance/equality controls, while strict conflict verdicts change from 36 to zero under the explicit rule. All 36 fixed-component completions state a premise. Control-verdict agreement in the assessed-component conditions is 62/100 versus 50/100, reflecting changes in both assessment and aggregation.

Component assessment under the explicit rule. On the 23 four-rater-consensus conflict pairs, model-assessed explicit-rule judgments give:

JudgeValid / plannedStrict verdictsReported conflictsEQUALRule mismatches
GPT-6.1-sol45/46162900
Qwen3.5-9B29/4625400
Qwen3.8-27B46/46351010
Qwen3-8B46/46252100

Counts use valid outputs under each audit's primary parsing policy; planned presentations include both orders. Rule mismatches are measured against each output's reported components. GPT-6.1-sol has one service rejection. Qwen3.5-9B generates 44 outputs: 15 remain unresolved under primary parsing and two planned presentations are ungenerated. Appendix F.2 reports parsing sensitivity and identifies the parent audit for each model.

Every strict verdict in this table follows reported dominance while the human reference is a conflict. These are component disagreements, not aggregation violations: identification checks support relative to the measured components.

The human annotations also separate components from overall preferences. Among the 23 unanimous-sign conflict pairs, seven have opposing strict overall labels; four retain opposing labels among raters with identical full ordinal components. This establishes non-uniqueness in the recorded mapping.

Sec. 3.5 presents both response-text settings; our second comment reports the complementary TradeoffBench test with mechanically verified components. Appendices F.2–F.3 give protocols and full counts. The stratified samples and existing multi-rater annotations support case-level analysis, not deployment-frequency estimates.

AuthorsOpenReview ↗

Frontier model coverage and the scope of the result

W2 and Q2. A frontier judge with reasoning enabled. We added GPT-6.1-sol on the same 800 raw TradeoffBench presentations under two conditions: 200 verified conflict pairs, 100 dominance pairs, and 100 equality pairs, each in both orders. Both runs use the Responses API, medium reasoning effort, Standard service, and a 2,048-token cap including reasoning.

Trade-offsStrict verdicts on verified conflictsOf these, reported conflictUnstated premiseCorrect controls
Explicit deferral rule0/400——400/400
Unspecified176/400 (44.0%)176/176119/176400/400

The frontier judge follows the explicit rule on every verified conflict. Under the same rule, reasoning-enabled Qwen3-8B and Qwen3.8-27B produce 84/400 and 31/400 strict verdicts on verified conflicts, with 396/400 and 400/400 controls correct. These raw-text judgments jointly test component assessment and aggregation.

With trade-offs unspecified, GPT-6.1-sol reports all 400 conflicts yet resolves 176; 153 strict verdicts follow component reports matching the complete verified vector. Of the 176 completions, 52 state an added premise, 119 leave it unstated, and five are ambiguous. Within the unstated group, 52 final answers contain only components and a verdict. GPT-6.1-sol follows the explicit rule but supplies undeclared completions when trade-offs are unspecified.

All 1,600 outputs are valid and complete, with one generation per presentation and no retries. The explicit condition returns only a verdict; the unspecified condition requests components and permits explanations. The decision instruction and output format change together. The strict-verdict rate is 44.0% (task-cluster bootstrap 95% CI [37.6, 50.7]); the unstated-premise rate is 29.8% of verified-conflict presentations ([24.3, 35.6]; Appendix F.1).

Open-ended and open-weight evaluations. GPT-6.1-sol also evaluates the 73 four-rater-consensus MultiPref pairs described in our first comment: 581/584 requests return valid, complete outputs, with three service rejections unresolved. Qwen3.5-9B repeats the design with thinking enabled, model-card sampling settings, and an 8,192-token output budget. One pair associated with the hosted-service safety rejection is not generated locally, leaving 72 pairs and eight planned presentations ungenerated, without replacement.

All 47 auditable Qwen3.5-9B strict conflict verdicts under unspecified trade-offs state a priority or trade-off: three with assessed components and 44 with fixed components. Under the explicit rule, no strict verdict occurs over a reported conflict. Primary parsing resolves 481/576 outputs; literal-field sensitivity resolves 542/576 without changing these strict conflict counts. The remaining 34 outputs, including one capped response, remain unresolved (Appendix F.3). This comparison documents explicit completion and compliant deferral on conflicts.

Scope. The frontier extension covers one hosted model on verified synthetic tasks and open-ended MultiPref responses. Successful rule execution coexists with silent completion across the tested instruction conditions.

Thank you for addressing my concerns, I am satisfied with the response and will raise my score accordingly.

AuthorsOpenReview ↗

Thank you for raising the score

Thank you for your positive feedback and for reconsidering your assessment. We sincerely appreciate your time and thoughtful review.

Reviewer_zJpNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates silent value completion in LLM judges and proposes identified-first evaluation to separate rubric-supported verdicts, explicit trade-off assumptions, and agreement with the intended rater.

Critical strengths
  1. The paper usefully separates whether a verdict follows from the rubric from whether it matches human preferences, highlighting a limitation of agreement-based evaluation.

  2. The analyses on MultiPref and HUMAINE provide useful evidence that correct rule execution does not guarantee agreement with human preferences, and that explicit trade-offs introduce a trade-off between coverage and disagreement.

Critical weaknesses
  1. The paper is difficult to follow, as it covers several aspects, including rule-following failures, human preference analyses, and risk calibration, without a sufficiently clear central message. The authors should clarify the main contribution and better connect the different analyses to the paper's core argument.

  2. The main controlled experiments rely on applying prescribed rules to anonymous criterion vectors, and reasoning-enabled configurations almost eliminate rule violations. It remains unclear how frequently this failure occurs in realistic, open-ended LLM evaluation.

  3. The human-data studies use human-provided criterion scores rather than LLM-generated scores. They therefore do not establish whether the proposed workflow improves practical judge reliability over a reasoning-enabled baseline.

Questions for authors
  1. Do reasoning-enabled judges still exhibit silent value completion when evaluating actual response text under rubrics with unspecified trade-offs, rather than classifying exact vectors?

  2. How should admissible weight sets be elicited or justified for the intended raters, rather than chosen as illustrative specifications?

5 comments
AuthorsOpenReview ↗

Central contribution and silent completion on actual text

Thank you. We have clarified the central contribution: distinguishing specification support, explicit completion, and target agreement, with separate evidence for each.


W1. Central message. TradeoffBench fixes component evidence and the rule to make unsupported verdicts checkable. MultiPref holds the rule and intended rater fixed while changing the component source. HUMAINE declares additional trade-offs and measures the resulting coverage and strict-direction risk. Identified-first evaluation connects these studies by computing what the current specification supports, recording any premises needed to resolve any remaining conflicts, and auditing the resulting decisions against the intended rater. Correct deferral is successful rule execution; it leaves the trade-off unresolved.


W2 and Q1. Unspecified trade-offs on actual responses. GPT-6.1-sol reads the same 800 raw TradeoffBench presentations under two conditions: 200 verified conflict pairs, 100 dominance pairs, and 100 equality pairs, each in both orders. Medium reasoning effort and inference parameters are identical.

Trade-offsStrict verdicts on verified conflictsOf these, reported conflictUnstated premiseCorrect controls
Explicit deferral rule0/400——400/400
Unspecified176/400 (44.0%)176/176119/176400/400

With trade-offs unspecified, the judge reports all 400 conflicts in its own components. 153 of the 176 strict verdicts follow component reports matching the complete verified vector, so those completions cannot be attributed to component mismeasurement.

The full-answer disclosure audit finds 52 explicit premises, 119 unstated premises, and five ambiguous cases. A stated priority, outweighing relation, or criterion-specific reason linking the overall choice to a criterion counts as explicit; numerical weights are unnecessary. Silent value completion occurs on 29.8% of verified-conflict presentations (95% CI [24.3, 35.6]). Within this group, 52 outputs contain only components and a verdict; 49 match the complete verified vector.

All 1,600 outputs are valid and complete. The explicit condition is tag-only; the unspecified condition requests components and allows explanations. The decision instruction and output format therefore change together. Appendix F.1 reports the protocol, task-cluster intervals, and disclosure categories.

Open-ended evidence. The MultiPref audit uses 73 actual response pairs with four-rater agreement on the complete helpfulness, truthfulness, and harmlessness sign vector. Four conditions cross model-assessed versus fixed human-reference components with unspecified trade-offs versus the explicit strict Pareto rule; all allow explanations and deferral. In the assessed, unspecified condition, 36 strict verdicts over reported conflicts comprise 29 explicit premises, six unstated premises, and one ambiguous case. Two unstated-premise outputs match the complete human reference.

On 31 presentations with identical assessed conflict vectors across the two instruction conditions, strict verdicts change from 30 to zero. Control-verdict agreement remains 100/100 in both fixed-component conditions; in the assessed-component conditions, it is 62/100 versus 50/100. The fixed-component comparison isolates aggregation, while the assessed comparison also changes component assessment (Appendix F.2). This establishes silent completion on open-ended text alongside successful rule execution; the stratified samples do not estimate deployment prevalence. Accordingly, we have updated the manuscript, especially the main contribution and better connected the different analyses to the paper's core argument.

AuthorsOpenReview ↗

Measured components and eliciting admissible weights

W3. Practical reliability with LLM-generated components. On the 700-pair open-ended audit, deterministic identification changes all seven Qwen3-8B strict conflict verdicts to deferral. Reported components and all other valid native verdicts are unchanged. Qwen3.8-27B's already compliant verdicts remain unchanged. Six of the seven changes require brace-only normalization; strict-format parsing retains one. The check takes O(d)O(d) operations and requires no additional generation after component assessment.

Identification checks support for the reported components. We assess component validity and agreement with the intended rater separately (Sec. 7; Appendix F.4).

We added confirmation comparisons of component-only assessment followed by deterministic identification against reasoning-enabled joint components-plus-verdict baselines. Prompts are fixed using separate development data; confirmation prompts are disjoint from development.

Confirmation judgeJoint balanced component agreementComponent-onlyDifference in pp [95% CI]
GPT-6.1-sol, 120 pairs51.65%52.24%+0.59 [-5.20, 7.59]
Qwen3.8-27B, 200 pairs50.70%50.50%-0.19 [-4.40, 3.73]

Both orders are evaluated. Balanced component agreement is the mean recall across nine criterion-by-reference-sign cells, with unresolved outputs counted as incorrect. Intervals use paired crossed prompt/evaluator bootstrap resampling. Appendix F.4 reports output validity and all agreement metrics. These comparisons do not establish improved component or target agreement. For GPT-6.1-sol, strict-target releases/errors are 153/55 for joint assessment and 153/56 for component-only assessment.


Q2. Eliciting admissible weight sets. We implemented elicitation from each intended rater's historical component comparisons and overall choices (Sec. 2.1). Starting from normalized weights with a declared lower bound, a strict reference preference adds yiw⊤x(i)≥10−6y_i w^\top x^{(i)} \geq 10^{-6}, retaining all weights satisfying the constraints. Tied targets consume a query but impose no directional constraint. Empty intersections trigger fallback to the initial set, with the incompatibility and all queried constraints recorded.

The primary retrospective MultiPref analysis uses at most four earlier mixed-sign comparisons per rater and a weight lower bound of .01. Earlier and later records are disjoint in time and prompt. It acquires 180 labels, 164 strict. The resulting sets are nonempty for 78/80 raters, including 14 with no directional query information; two infeasible sets trigger fallback. On 4,847 later records, the policy adds 110 decisions (+2.269 coverage points) over the same own-component Pareto rule.

The complete policy has 39 errors among 3,375 released strict targets (1.156%), versus Pareto's 23/3,269 (0.704%). The added layer has 16/106 errors (15.1%). Elicitation therefore expands coverage here at higher observed disagreement. The added decisions require separate target-risk validation. Appendix F.5 reports query-budget sensitivity and feasibility checks.

MultiPref uses reference comparisons to constrain admissible weights. HUMAINE evaluates declared specifications. The intended stakeholders specify the criteria, scales, and noncompensatory requirements.

AuthorsOpenReview ↗

Rebuttal Discussion: Follow-Up on Any Remaining Concerns

Dear Reviewer zJpN,

As the rebuttal discussion closes very soon (18 Nov AOE), we would appreciate hearing your thoughts on our response and whether it has addressed your concerns. Please feel free to let us know if any points would benefit from further clarification. We have also updated the manuscript accordingly.

Thank you for your time and consideration.

Reviewer_X447OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper suggests that when rubric criteria conflict, an LLM judge that issues a strict verdict must add a trade-off premise the rubric does not supply and calls this undeclared resolution silent value completion. This paper introduces TradeoffBench, which separates rule execution from score estimation, and finds that six open-weight judges in non-reasoning configurations pass every dominance and equality control yet select unsupported winners on 9%-81% of 1,345 conflicting pairs, with all 577 candidate-consistent choices favoring the criterion majority despite an explicit ban. However, reasoning-enabled configurations of three models produce only 2 unsupported winners in 9,720 conflict judgments. This paper also proposes identified-first evaluation, which identifies strict verdicts supported by every allowed weighting. Using human criterion scores, they show that in HUMAINE declared trade-offs raise coverage by 2.5% points over strict Pareto with higher disagreement on added decisions (3.9% vs. 0.9%), and in MultiPref replacing raters’ own scores with peer means raises disagreement from 0.6% to 28.8% on 2,707 jointly decided comparisons.

Critical strengths
  1. TradeoffBench makes unsupported verdicts directly checkable using exact scores and an explicit deferral rule. This separates rule execution from score estimation, giving a known identification boundary without another evaluator as ground truth. This supports the notion that human agreement alone cannot rule out silent value completion.
  2. The MultiPref and HUMAINE studies show that correct rule execution does not ensure rater agreement.
Critical weaknesses
  1. TradeoffBench uses synthetic verifiable tasks, 5 binary criteria, and 6 open-weight checkpoints (7B to 32B), with no larger or proprietary judges and no open-ended rubrics. Add at least one open-ended rubric setting and stronger judges for evaluation.
  2. Reasoning-enabled configurations almost eliminate violations (2 of 9,720), but that audit uses anonymous vector encodings rather than raw response text. So it is unclear whether the failure persists for capable judges in realistic inputs. It would be good to run reasoning-enabled judges on the raw evidence conditions.
  3. Since the component source and averaging change jointly, it makes sense to add a single-peer component condition to separate source from averaging. This could perhaps explain the large gap in disagreements in Multiperf.
  4. The fixed-rule coverage gain is 2.485 percentage points and the added layer’s observed risk (3.873%) exceeds the 2.5% tolerance and is not separately certified. Moreover, under Holm testing the selected-policy coverage difference disappears and independence is assumed without participant identifiers. It would be good show layer-specific bounds and dependence-aware sensitivity analysis.
  5. The paper is very hard to read as one has to jump to appendix way too often to understand the paper.
Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Actual responses, open-ended rubrics, and stronger judges

Thank you for these suggestions. We added reasoning-enabled evaluations on raw response text and an open-ended audit with a hosted frontier judge (Sec. 3.5).


W2. Reasoning-enabled judges on raw evidence. A strict conflict verdict selects a winner despite conflicting criteria and no specified trade-off rule. We evaluated 400 TradeoffBench pairs (200 conflicts, 100 dominance cases, 100 equalities), each in both orders (800 presentations). Under the explicit deferral rule:

Reasoning-enabled judgeStrict verdicts on verified conflictsCorrect dominance/equality controls
Qwen3-8B84/400 (21.00%)396/400
Qwen3.8-27B31/400 (7.75%)400/400
GPT-6.1-sol0/400 (0.00%)400/400

Without specified trade-offs, GPT-6.1-sol selects winners in 176/400 verified conflicts, despite identifying the conflicting criteria itself. Of these, 153 fully match the verified component sign vector. The disclosure audit identifies 52 explicit premises, 119 unstated premises, and five ambiguous cases. Thus, silent value completion occurs in 29.8% (119/400) of verified-conflict presentations; 52 contain only component scores and a forced verdict.

Both conditions preserve all 400 controls and use identical inference parameters. However, decision instructions and output formats change together. Raw-text judgments also combine component assessment and aggregation, so comparisons with the anonymous-vector audit do not isolate input format (Appendix F.1).


W1. Open-ended rubrics and stronger judges. We extended the evaluation to real human preference responses from MultiPref. GPT-6.1-sol assessed helpfulness, truthfulness, and harmlessness across all 73 qualifying pairs from the 700-pair audit sample, with unanimous four-rater agreement on the complete component sign vector (23 conflicts, 23 dominance cases, 27 equalities).

We crossed model-assessed versus fixed human-reference components with unspecified trade-offs versus the explicit strict Pareto rule. All conditions permit explanations, equality, and deferral. Counts cover both presentation orders; conflicts are defined by reported components in assessed conditions and supplied components in fixed conditions.

ComponentsTrade-offsValid / plannedConflictsStrict on conflictsUnstated premise
AssessedUnspecified145/14637366
AssessedExplicit rule145/1463800
FixedUnspecified145/14645360
FixedExplicit rule146/1464600

GPT-6.1-sol used medium reasoning effort, producing 581 valid outputs with three unresolved service rejections. The 36 assessed-component strict verdicts comprise 29 explicit premises, six unstated premises, and one ambiguous case. Two unstated-premise cases fully match all four human raters' component judgments, showing that silent completion can occur even when the criteria are correctly assessed.

Control agreement remains 100/100 in each fixed-component condition, versus 62/100 (unspecified) and 50/100 (explicit rule) with assessed components, where component-grading errors also affect verdicts. These rates describe the stratified sample (Appendix F.2).


W5. Readability. We have updated the manuscript accordingly. For instance, Sec. 3.5 and Table 3 now present the response-text evidence in the main paper. The Introduction and Sec. 2 follow three central pillars: specification support, explicit completion, and target agreement. Detailed protocols, parsing checks, and full accounting remain in Appendix F.

AuthorsOpenReview ↗

Component source, layer risk, and calibration

W3. Single-peer ablation. To distinguish peer substitution from averaging effects, we added a single-peer condition selected by a fixed hash, independently of preference outcomes. The intended rater and strict Pareto rule remain fixed. All conditions use the same 2,175 released strict targets.

Component sourceDirectional errorsStrict-direction risk
Own12/2,1750.55%
Single peer651/2,17529.93%
Peer mean590/2,17527.13%

Replacing the intended rater's scores with a single peer's increases risk by 29.38 percentage points (three-contrast-adjusted crossed-bootstrap interval [24.75, 34.10]). Thus, substantial disagreement occurs even without averaging. Peer mean minus single peer is -2.80 points (interval [-5.59, 0.18]), providing no conclusive evidence of an additional averaging effect. The original two-condition analysis uses 2,707 shared strict targets (Sec. 4; Appendix C.2).


W4a. Layer-specific bounds. The fixed symmetric partial specification adds 756 releases across 30,425 records, increasing coverage by 2.485 percentage points. To determine whether the aggregate risk conceals higher-risk marginal decisions, we separately report the evaluation layers:

Evaluation layerErrors / released strict targetsObserved riskSimultaneous 95% UCB
Pareto core196/22,4370.874%1.016%
Added layer28/7233.873%5.700%
Complete partial core224/23,1600.967%1.114%

The added layer alone exceeds the 2.5% risk threshold (3.873% observed) and is not separately qualified. However, the complete policy achieves 0.967% observed risk, with a 1.114% upper confidence bound. These descriptive one-sided Clopper–Pearson bounds use Bonferroni correction across three layers and assume independent comparisons. Complete-policy qualification relies on the original seven-candidate calibration using earlier releases, not these later evaluation bounds (Sec. 5.2; Appendix D.3).


W4b. Multiplicity and dependence sensitivity. We tested whether the coverage advantage survives multiple-testing corrections and reduced effective information. A supplementary nested design retains Pareto in the partial-value and safety-protective libraries to isolate fallback effects. Both designs use the same seven candidate tests, risk tolerance, and family error level.

Effective-information fractionBonferroni: non-nestedBonferroni: nestedHolm: non-nestedHolm: nested
1.000+2.485+2.48500
0.075+2.485+2.485+2.485+2.485
0.050-12.6970+2.485+2.485
0.0250000

Entries show partial-value minus Pareto-first coverage, in percentage points.

Across all tested fractions from 0.75 through 0.075, the coverage advantage remains +2.485 points under both corrections and designs. At nominal information, Holm selects E.25 across all libraries, eliminating the selected-policy coverage difference. At 0.05, the negative non-nested Bonferroni result arises from different fallback selections (E.50 versus Pareto); the nested design instead selects Pareto, eliminating this difference. At 0.025, all differences vanish. The fixed-rule coverage gain remains +2.485 points, but the selected-policy advantage depends on correction and effective information (Appendix D.4).

Finally, the five metric rows per conversation are already combined into one comparison. However, participant linkage across conversations is unavailable, preventing cluster-level adjustment. Effective-information sensitivity does not replace this adjustment; qualification remains conditional on independent comparisons and population transport.

AuthorsOpenReview ↗

Rebuttal Discussion: Follow-Up on Any Remaining Concerns

Dear Reviewer X447,

As the rebuttal discussion closes very soon (18 Nov AOE), we would appreciate hearing your thoughts on our response and whether it has addressed your concerns. Please feel free to let us know if any points would benefit from further clarification. We have also updated the manuscript accordingly.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary and Core Takeaways

We thank all reviewers (X447, zJpN, Yzfg) for the thorough discussion. Yzfg has confirmed satisfaction and indicated a score raise, and every requested experiment from X447 and zJpN is now complete and integrated into the revised manuscript (Section 3.5, Table 3, Appendices C.1, D.3–D.4, F.1–F.5).

Why this paper merits acceptance at ICLR 2027:
Current LLM-as-a-judge evaluations conflate three separate requirements:

  • rubric entailment,
  • explicit trade-off premises, and
  • agreement with the intended human rater.

Our paper isolates and empirically proves each failure, and provides a clean, auditable benchmark (TradeoffBench) and protocol (identified-first evaluation) to control for them.


1. Why "just prompt a reasoning judge explicitly" does not solve the problem (zJpN, X447-W2):

  • Explicit rules still fail in open-weight reasoning judges on real text: Under the explicit deferral rule on raw TradeoffBench, reasoning-enabled Qwen3-8B and Qwen3.8-27B still violate the rule on 21.00% (84/400) and 7.75% (31/400) of conflicts (§3.5). Deterministic O(d) identification eliminates these violations completely (7/7 Qwen3-8B violations fixed on the 700-pair audit, App. F.4).
  • Frontier compliance on strict Pareto leaves real judging unsolved: While GPT-6.1-sol obeys a strict-Pareto ban (0/400), forcing Pareto deferral leaves 22%–30% of comparisons undecided. In standard rubric judging where trade-offs are left open, GPT-6.1-sol still issues unstated strict verdicts on 29.8% (119/400) of TradeoffBench conflicts and 6/36 on MultiPref (Table 3, Table 31). Nor can an LLM prompt enforce global polyhedral trade-offs across 30,000+ pairs, which our protocol computes exactly (+2.485 pp coverage on HUMAINE, Table 4).

2. Why auditability and target-risk qualification are indispensable (X447-W3/W4, zJpN-Q2):
Even 0% rule violations do not prevent downstream failure:

  • Component-source risk persists without averaging (X447-W3): Our new single-peer ablation on 2,175 shared targets (Own 0.55% vs. Single peer 29.93%, +29.38 pp, Table 15) proves the MultiPref disagreement gap is intrinsic to component substitution, not peer averaging.
  • Every added trade-off carries measurable risk (X447-W4, zJpN-Q2): Our layer audit exposes 4.4× higher error on HUMAINE's added layer (3.873% vs. 0.874%, Table 20), shows primary selections survive effective-sample reductions down to q∗=0.0612q^* = 0.0612 (alongside transparent Holm sensitivity, Tables 22–24), proves retrospective weight elicitation (+2.269 pp, 15.1% added-layer error, Table 35) still requires target-risk auditing, and isolates 54.00% of directional errors in 8.32% of the workload (Section 5.3).

We have restructured the main paper around these three boundaries (Section 2, Section 3.5, Table 3). Together, these findings establish a foundational evaluation gap and a validated auditing protocol for reliable LLM judging.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.