acceptodds
Under review as a conference paper at ICLR 2027

Benchmark Reliability: A controlled comparison of human preference, rubric scoring, and LLM judges on EnterpriseVAL

Abstract

LLM judges are validated by their agreement with expert ratings, a practice that treats expert reliability as a fixed property of the raters. We show that it depends substantially on what those raters are asked. On EnterpriseVAL, a benchmark of expert-authored finance and accounting tasks, we hold tasks, outputs and rubrics fixed and vary only what the judgement asks for: blinded preference between an LLM output and the expert reference deliverable; a score on each criterion of the task rubric; and a binary critical-failure gate, where a single disqualifying error fails the output. Expert agreement rises as the claim under judgement becomes more specific: Krippendorff’s α = 0.20 on preference, α = 0.55 on criterion grading, Cohen’s κ = 0.90 on the critical-failure gate. Two consequences follow for validation: a judge’s agreement with experts is uninterpretable without the expert–expert baseline on the same instrument, and preference can favour outputs the gate disqualifies. Reliability trades against discriminative power. The critical-failure gate, our most reliable instrument, separates the evaluated LLMs least, because a criterion every LLM passes, or every LLM fails, carries no comparative information. Reliability and discriminative power are separate claims about an instrument, and a benchmark tuned to maximise the first can end up measuring nothing about the LLMs it exists to compare. Where a single wrong value disqualifies a deliverable, categorical correctness belongs in a benchmark’s results as a claim of its own, reported separately from graded and preference measures.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, evaluation methodology, inter-rater reliability, benchmark, expert evaluation, measurement validity, rubric-based evaluation, human agreement, domain-specific benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TyA4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies a problem in LLM-as-a-Judge evaluation: a judge's agreement with experts should be interpreted in light of the reliability of the measurement instrument that produced the human ratings. To investigate this, the authors introduce EnterpriseVAL, a benchmark of complex finance and accounting tasks, and conduct three studies comparing human preference, expert rubric scoring, critical-failure gates, and an automated LLM judge. The results show differences in expert agreement across evaluation instruments, while higher reliability does not necessarily lead to stronger discriminative power.

Critical strengths
  1. The paper studies the reliability of evaluation instruments themselves, which is an interesting and practically important problem.
  2. EnterpriseVAL is a valuable contribution, with high-complexity enterprise tasks and detailed expert-authored rubrics and reference deliverables.
Critical weaknesses
  1. The writing is sometimes difficult to follow, especially in the Introduction. For example, in L40, what exactly does "the variance the judge is being asked to reproduce" refer to? Is it variation across outputs, annotator heterogeneity, or measurement error? Similarly, what does "the same structure" in L54 refer to? I also found the flow of the Introduction somewhat confusing. For example, in L56–L63, my understanding is that the authors want to discuss how benchmark evaluation is moving beyond deliverable quality toward a more detailed view. However, this discussion mainly relies on a single paper that is still under review at TMLR, where GDPval is only one of the case studies. Moreover, much of the paragraph discusses how length prompt in GDPval affects model performance, which is not directly related to the evaluation problem studied in this paper. I think the authors should clarify the connections between these ideas and better organize the paper and improve the overall readability.

  2. The paper's positioning and discussion of related work are not sufficiently clear. The paper does not have a separate Related Work section, and most related work is discussed in the Introduction. I think this is not necessarily a problem, but the current discussion lacks a clear overview of the relevant research directions. In particular, it is not clear what existing work has already addressed, what challenges remain, and what specific gaps this paper aims to fill. E.g. the advantages of EnterpriseVAL's design could be better explained, such as comparing elicitation formats on the same outputs, and why these features are important for the research question.

  3. Some of the main claims are not sufficiently supported by the experiments.

    3.1. The paper emphasizes that only the "form of the judgement" varies across the studies (e.g., L16–17 and L93–96), but this does not seem to be strictly controlled. For example, the caption of Figure 2 (L289) states that all panels use the same 125 outputs. However, the reported results for Study 1 appear to be based on all 150 comparison events involving 6 LLMs, while Studies 2 and 3 use only 125 outputs from 5 LLMs. This suggests that even the set of evaluated outputs may not be identical across the reported comparisons.

    3.2. In L361–362, the paper states that "a judge-measured difference is better read as a lower bound than an estimate." I don't understand how this conclusion follows from the results. The experiments suggest that using an LLM Judge may change the measured differences between LLMs. However, treating judge-measured differences as lower bounds would require evidence that the judge systematically underestimates these differences, which has not been established. Moreover, the fact that one test is statistically significant while another is not does not imply that the estimated effects are significantly different. I think this claim is a bit strong given the current evidence and needs further justification.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

response to reviewer TyA4

We thank the reviewer for specific, actionable comments, and for the positive assessment of soundness and of EnterpriseVAL as a contribution. We answer each point below.

W1. Writing, especially the Introduction. The reviewer asks what two phrases referred to.

"The variance the judge is being asked to reproduce" (L40) referred to disagreement between experts: the part of the variation in expert ratings that reflects differences between raters rather than differences between outputs. The revision replaces the phrase with a concrete example. A judge that agrees with experts 70% of the time is doing well if experts agree with each other 70% of the time, and poorly if it turns out experts agree 95% of the time (Section 1).

"The same structure" (L54) referred to a trade-off. Krishnappa et al. (2026) find that an automatic metric can separate genuine answers from random ones yet fail to rank systems; in our study, the most reliable human format separated the LLMs least. The revision states this parallel explicitly (Sections 2 and 6.2).

The paragraph at L56–L63, built on Hua et al. and on GDPval's prompt-length experiment, is removed, since its connection to how judgements are elicited was indirect. Throughout, sentences are shorter, terms are defined at first use, and each statistic is described in plain terms (Section 4).

W2. Positioning and related work. A dedicated Related Work section (Section 2) covers three lines of work: expert-rubric benchmarks for professional work, validation of LLM judges against human agreement, and the relation between agreement, accuracy and discrimination. Table 1 compares EnterpriseVAL with seven related benchmarks, with detail in Appendix A. The gap is stated directly: several benchmarks report expert-expert agreement, but each on one format and with its own statistic, so none shows whether that baseline depends on the format.

EnterpriseVAL's design is built to fill that gap. Eliciting all three formats on the same outputs, with the same tasks and rubrics, rules out the material being judged as the source of differences in agreement. Designated critical criteria let correctness be reported separately from graded and preference scores. And deliverables in native file formats let a failure be located at a specific value. The Discussion sets out, for each line of work, what was known, what this paper adds, and what follows in practice (Section 6).

3.1. Identical outputs and "only the form varies". Both observations are correct for the submitted version. Study 1 included a sixth LLM rated by a separate group of evaluators, a point our analysis plan had flagged in advance. The revision analyses Study 1 on the same five LLMs as Studies 2 and 3, so every comparison now uses the same 125 outputs (For Study 1, this means 250 ratings from 12 evaluators). Preference agreement is α = 0.27 on this set. The LLMs still differ on preference, and on two presentation dimensions (Style and Format) but not on Accuracy or Professional Judgement; Structure no longer reaches significance. The sentences at L16–17 and L93–96 are rewritten: tasks, outputs and rubrics are held fixed, and the text now states that the rater groups differ too (Sections 1 and 7).

3.2. Judge-measured differences. We agree that comparing a significant with a non-significant test does not establish a difference, and have replaced that interpretation with a direct test (Appendix F.4). The judge scores every LLM higher than the grader panel by a similar amount, and the offset does not differ across LLMs (p = 0.23). Its per-LLM means are as spread out as the panel's: the slope of judge means on panel means is 1.06 (95% interval 0.46 to 1.55). There is therefore no evidence that the judge underestimates differences, and the "lower bound" sentence is removed. The revision describes the judge's failure to separate the LLMs as inconclusive rather than as evidence that they are equal (Section 5.2).

Reviewer_dSEYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses a gap in validating LLM judges against expert agreement: the consistency of the expert judgments used as the reference standard is rarely examined closely. To study this, the authors present EnterpriseVAL, an expert-written benchmark of financial and accounting tasks, and run three studies across 25 tasks. In Study 1, experts compare anonymized LLM outputs with expert-written reference outputs, which measures both their preferences and how well the experts agree with one another. In Study 2, experts score each output against rubric items, and a critical-failure gate removes any output that fails a pre-specified essential criterion. In Study 3, the authors compare an LLM judge's scores and gate decisions, using the same rubric, against the expert panel's results.

The experiments show that inter-expert agreement changes substantially depending on the evaluation method, and that preferred outputs can still contain serious errors. Based on this, the authors propose using inter-expert agreement as the baseline for judge validation, and reporting preferences, rubric scores, and the presence of critical errors separately.

Critical strengths

This paper has the following strengths:

  • The study uses complex financial and accounting tasks written and reviewed by experts, and it applies several evaluation methods to the same tasks and outputs. That makes it useful for seeing how much the results depend on the evaluation method itself.
  • It considers expert-to-expert agreement when assessing LLM judges, and it analyzes critical errors separately from overall preference or average scores. Those errors could otherwise be hidden by the averages, so this is a practical contribution.
Critical weaknesses

This paper has the following weakness points:

  • Overall readability is poor. It is unclear what each experiment tested and what claims its results support. Experimental results and interpretation are mixed together; for example, key comparison results first appear in the Discussion. I think it would be better to describe each study’s setup and results together. The figures also lack the information needed to interpret the results.
  • Some of the claims seems over-claimed. See the questions below.
Questions for authors

About the over-claiming, I have the following views. Please answer whether I understand correctly or not.

  • The claim that this is a controlled comparison that changes only the evaluation method is not supported by the design. As noted in Section 3, Study 2 uses a different group of evaluators from Study 1, and these evaluators already know the task context. So it is hard to separate the effect of the evaluation method from the effects of evaluator group and task familiarity in the observed agreement differences. In addition, overall preference judgments and the satisfaction of critical criteria concern different objects of judgment. The study does have the advantage of comparing the same outputs, but that alone does not support the conclusion that “making questions more specific increases expert agreement.”
  • The interpretation that the study demonstrates a trade-off between agreement and the ability to distinguish models is overclaimed. The observation that 35 of 50 critical criteria gave identical results for all models is evidence that these criteria have limited power to distinguish the models tested. However, all models receiving the same verdict and evaluators agreeing on the same outputs are different properties, so calling the criteria “perfectly reliable” is an overclaim.
  • The Reproducibility Statement says decoding settings are provided, but the actual temperature and top-p values cannot be found in the main text or the appendix. The number of repeated generations per model and the procedure for selecting generated outputs are also unclear. The paper states that the judge evaluated each output only once, and it acknowledges that this prevents estimating variability across repeated judgments. The authors should report the actual generation and evaluation settings and the number of repetitions. If stochastic sampling was used, they should also show that the main results hold across repeated runs.
1 comment
AuthorsOpenReview ↗

Response to reviewer dSEY

We thank the reviewer for a careful reading. The reviewer asked whether each concern about over-claiming is understood correctly, and we answer each directly below.

Readability. The paper is restructured so that each claim appears with its evidence. A new Experimental design section states the four research questions and how each is measured (Section 4). Results are organised by those questions, each subsection headed by its finding (Section 5), and the Discussion interprets without introducing new numbers (Section 6). Figure captions now explain how to read each panel. We organise Results by question rather than by study because each finding draws on more than one study; the setup of all three studies is summarised together (Section 4, Table 2).

  1. Controlled comparison: partly correct. The comparison holds tasks, outputs and rubrics fixed, which rules out the material being judged as the source of the differences in agreement. The reviewer is right that it does not isolate the format itself: the rater groups differ, graders held task context, and preference and critical criteria judge different objects. The revision therefore presents the difference across formats as an observed ordering, not as the effect of specificity alone, and states the confound in Sections 1, 5.1 and 7. The claim that specificity matters now rests on evidence where the raters are fixed. Within a task the same two graders score every criterion, so differences between criteria cannot reflect the rater group; criteria that bundle several requirements draw lower agreement with the grader pair held fixed (Appendix E.6). Agreement also did not rise where a grader had written the rubric (Appendix E.5).

  2. "Perfectly reliable" and the trade-off: correct. Identical verdicts across LLMs and agreement between raters are different properties, and the phrase is removed. The revision measures how the two relate instead. Critical criteria that give the same verdict for every LLM are also easier to agree on: 98.3% exact agreement, against 92.0% on the 15 that separate the LLMs. On the 11 tasks where the LLMs' verdicts differ, gate agreement stays high (κ = 0.82, against 0.90 overall), so uniform verdicts do not explain the gate's agreement on their own. The trade-off is now stated as a pattern observed in this study: each format's separation of the LLMs, measured by Kendall's W, falls from 0.13 to 0.11 to 0.07 as agreement rises, but the intervals overlap (Section 5.4, Figure 2a). We do not present it as a general law.

  3. Settings and repeated runs: correct for the submitted version. Full generation settings are now in SI 1, Section 6. Each LLM produced one output per task, which is the one evaluated; none was selected from several attempts. Temperature and top-p were provider defaults, not configurable through the access routes used. Settings common to all LLMs, such as tools, prompt structure and dates, are in Table 12; each LLM's version, access route and reasoning setting are in Table 13.

For the judge, a five-run screening preceded Study 3 (Section 5.2, Figure 3b, Appendix F.1). Three candidate judges each scored all 125 outputs five times with the same prompt and settings. Claude Opus 4.8, the judge used, gave the same gate verdict in all five runs on 96% of outputs (κ between runs 0.96; α on criterion scores 0.94), and its agreement with the panel varied by about 0.03 in κ across runs, so a repeated Study 3 run would change the reported figures little. The screening used an earlier version of the judge's rubric files, so it is reported for consistency only. [Judge decoding settings: confirm they are stated in Appendix F.1 or SI 1 before posting.]

Each LLM output is a single sample. Expert re-grading of a second sample was outside the scope of this revision, and Section 7 notes it.

Reviewer_UXoEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This papers holds 125 LLM outputs, 5 LLMs and 25 finance/accounting tasks, and their expert written rubrics fixed and elicits judgement three ways: blinded pairwise preference against expert preference, per-criterion rubric scoring, and a binary critical failure gate. Expert agreement rises across all three (α 0.2 to α 0.55 to κ 0.9). An LLM judge even reaches κ0.73 on the gate against the panel, below the panel's own 0.9, and compresses between-LLM differences. The gate discriminates the least between LLMs. Essentially, hte paper argues that judge-vs-human agreement is unintepretable without the human-human baseline, and that correctness should be reported separately from graded and preference scores.

Critical strengths
  1. The matched design is the right choice and presents an innovative approach to the problem. Most judge-validation work reports judge-human agreement with no human-human baseline on the same instrument.

  2. The reliability vs. discrimination trade-offs is well demonstrated by Table D3. Out of all, 26/50 critical criteria never fail, 9/50 always fail, leaving 15 that carry the right signal.

Critical weaknesses
  1. Rater qualification is not present: there's a no information on selection, sub-domain match, and calibration for all raters. The "average 22 years' experience" is stated for authors and not the 15 evaluators or 8 graders. At ~20 rating per evaluator and α = 0.2, one or two raters that are not properly trained or sourced could produce the result.

  2. Likewise, rater pool is not held constant across instruments. study ones 15 evaluators; study 2 uses 8 graders. The paper does not state whether grades were the task authors or the five-phase reviewers of the tasks they graded. If graders authored or reviewed the tasks they graded, this presents an obvious bias, as kappa = 0.9 would measure shared rubric construction instead.

Questions for authors
  1. Were Study 2 graders authors or five-phase reviewers of the tasks they graded? Report κ for graders with no prior involvement.

  2. For both rater groups: count, selection criteria, years of experience, sub-domain match to task, calibration protocol, and leave-one-rater-out α for Study 1.

  3. What is the LLM output generation protocol?

  4. Can the 25-task core be provided as anonymised supplementary material now?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UXoE

We thank the reviewer for a precise review and for recognising the value of the matched design. Each weakness and question is answered below, with new analyses where useful.

W1 and Q2. Rater qualifications, and whether one or two raters could drive the result. The "22 years" figure describes the full pool of practitioners who authored the 81 tasks. The eight who authored the 25 tasks in this study average 16.9 years. The revision reports both, and gives separate figures for each rater group (Section 4; SI 1, Tables 2–3): Study 1 evaluators, 12, mean 16.0 years (range 5–27); Study 2 graders, 8, mean 14.9 years (range 5–32). All hold a professional accounting qualification (CPA or national equivalent) and work across seven countries. SI 1 gives each rater's primary sub-domain (Tables 6 and 11) and each grader's task assignments by sub-domain (Table 9). Study 1 evaluators completed calibration on five anchor tasks, with a pass threshold of Krippendorff's α ≥ 0.60 set in a pre-specified analysis plan; all passed. Study 2 graders attended an onboarding session with mock tasks, without a formal pass threshold.

The result does not rest on one or two raters. Leaving out each of the 12 evaluators in turn keeps preference α between 0.20 and 0.34 (full value 0.27; Appendix D.1–D.2). Evaluators vary in how often they prefer the LLM, from 6% to 72% of their comparisons, and leaving out the most favourable lowers the win rate from 35% to 29%; no single evaluator changes the conclusion that experts prefer the reference. Agreement on the five dimension ratings, the analysis plan's primary measure, is α = 0.22 (Appendix D.3). Low agreement on preference is spread across the evaluator pool rather than produced by one or two raters.

W2 and Q1. Graders' relationship to the tasks they graded. Graders scored LLM outputs only. Five-phase reviewers were drawn from a separate pool, so no grader had reviewed a task they graded. On 9 of the 25 tasks one grader had written the rubric, on 11 one grader stood in for an absent author, and on 5 neither grader had an authoring role (Section 4; SI 1, Table 9).

This allows the reviewer's hypothesis to be tested directly. If shared rubric construction produced the high agreement, pairs including the author should agree most. They do not (Appendix E.5). Pairs including the author reached gate κ = 0.83 and 76% criterion agreement; pairs including a stand-in, κ = 0.89 and 81%. For graders with no prior involvement in the task, gate agreement was complete (κ = 1.00 on 25 outputs; 71.5% criterion agreement), though that group is small. A second test does not depend on group sizes: within a task the same two graders score every criterion, so comparing criteria within tasks holds the grader pair fixed. Differences between tasks, and so between grader pairs, account for only 15% of the variation in agreement, and within tasks, criteria that bundle several requirements draw lower agreement (−0.048 per standard deviation, p < 0.001, errors clustered by task; Appendix E.6). Section 7 states that the rater groups for the two human studies differ by design.

Q3. Generation protocol. Each LLM produced one output per task from the prompt and input files, and that output is the one evaluated. Temperature and top-p were provider defaults, not configurable through the access routes used. SI 1, Section 6 gives the settings common to all LLMs, including tools, prompt structure, failure handling and generation dates (21 July to 17 August 2026), and each LLM's version identifier, access route and reasoning setting (Tables 12–13).

Q4. The 25-task core. The supplementary material at submission contained the full rating data for all three studies. The revision adds the tasks themselves as SI 3: prompts, input files, reference deliverables and rubrics for all 25 tasks, under CC BY-NC-ND. The rating data, now including the judge screening runs, is SI 2.

On the judge. The reviewer's summary mentions compression of between-LLM differences. A direct test, prompted by Reviewer TyA4, shows the judge raises every LLM's score by a similar amount and preserves their spread (Appendix F.4), so the revision reports the judge's lower separation as inconclusive rather than as compression.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.