- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies a problem in LLM-as-a-Judge evaluation: a judge's agreement with experts should be interpreted in light of the reliability of the measurement instrument that produced the human ratings. To investigate this, the authors introduce EnterpriseVAL, a benchmark of complex finance and accounting tasks, and conduct three studies comparing human preference, expert rubric scoring, critical-failure gates, and an automated LLM judge. The results show differences in expert agreement across evaluation instruments, while higher reliability does not necessarily lead to stronger discriminative power.
Critical strengths
- The paper studies the reliability of evaluation instruments themselves, which is an interesting and practically important problem.
- EnterpriseVAL is a valuable contribution, with high-complexity enterprise tasks and detailed expert-authored rubrics and reference deliverables.
Critical weaknesses
-
The writing is sometimes difficult to follow, especially in the Introduction. For example, in L40, what exactly does "the variance the judge is being asked to reproduce" refer to? Is it variation across outputs, annotator heterogeneity, or measurement error? Similarly, what does "the same structure" in L54 refer to? I also found the flow of the Introduction somewhat confusing. For example, in L56–L63, my understanding is that the authors want to discuss how benchmark evaluation is moving beyond deliverable quality toward a more detailed view. However, this discussion mainly relies on a single paper that is still under review at TMLR, where GDPval is only one of the case studies. Moreover, much of the paragraph discusses how length prompt in GDPval affects model performance, which is not directly related to the evaluation problem studied in this paper. I think the authors should clarify the connections between these ideas and better organize the paper and improve the overall readability.
-
The paper's positioning and discussion of related work are not sufficiently clear. The paper does not have a separate Related Work section, and most related work is discussed in the Introduction. I think this is not necessarily a problem, but the current discussion lacks a clear overview of the relevant research directions. In particular, it is not clear what existing work has already addressed, what challenges remain, and what specific gaps this paper aims to fill. E.g. the advantages of EnterpriseVAL's design could be better explained, such as comparing elicitation formats on the same outputs, and why these features are important for the research question.
-
Some of the main claims are not sufficiently supported by the experiments.
3.1. The paper emphasizes that only the "form of the judgement" varies across the studies (e.g., L16–17 and L93–96), but this does not seem to be strictly controlled. For example, the caption of Figure 2 (L289) states that all panels use the same 125 outputs. However, the reported results for Study 1 appear to be based on all 150 comparison events involving 6 LLMs, while Studies 2 and 3 use only 125 outputs from 5 LLMs. This suggests that even the set of evaluated outputs may not be identical across the reported comparisons.
3.2. In L361–362, the paper states that "a judge-measured difference is better read as a lower bound than an estimate." I don't understand how this conclusion follows from the results. The experiments suggest that using an LLM Judge may change the measured differences between LLMs. However, treating judge-measured differences as lower bounds would require evidence that the judge systematically underestimates these differences, which has not been established. Moreover, the fact that one test is statistically significant while another is not does not imply that the estimated effects are significantly different. I think this claim is a bit strong given the current evidence and needs further justification.
Questions for authors
See Weaknesses.
response to reviewer TyA4
We thank the reviewer for specific, actionable comments, and for the positive assessment of soundness and of EnterpriseVAL as a contribution. We answer each point below.
W1. Writing, especially the Introduction. The reviewer asks what two phrases referred to.
"The variance the judge is being asked to reproduce" (L40) referred to disagreement between experts: the part of the variation in expert ratings that reflects differences between raters rather than differences between outputs. The revision replaces the phrase with a concrete example. A judge that agrees with experts 70% of the time is doing well if experts agree with each other 70% of the time, and poorly if it turns out experts agree 95% of the time (Section 1).
"The same structure" (L54) referred to a trade-off. Krishnappa et al. (2026) find that an automatic metric can separate genuine answers from random ones yet fail to rank systems; in our study, the most reliable human format separated the LLMs least. The revision states this parallel explicitly (Sections 2 and 6.2).
The paragraph at L56–L63, built on Hua et al. and on GDPval's prompt-length experiment, is removed, since its connection to how judgements are elicited was indirect. Throughout, sentences are shorter, terms are defined at first use, and each statistic is described in plain terms (Section 4).
W2. Positioning and related work. A dedicated Related Work section (Section 2) covers three lines of work: expert-rubric benchmarks for professional work, validation of LLM judges against human agreement, and the relation between agreement, accuracy and discrimination. Table 1 compares EnterpriseVAL with seven related benchmarks, with detail in Appendix A. The gap is stated directly: several benchmarks report expert-expert agreement, but each on one format and with its own statistic, so none shows whether that baseline depends on the format.
EnterpriseVAL's design is built to fill that gap. Eliciting all three formats on the same outputs, with the same tasks and rubrics, rules out the material being judged as the source of differences in agreement. Designated critical criteria let correctness be reported separately from graded and preference scores. And deliverables in native file formats let a failure be located at a specific value. The Discussion sets out, for each line of work, what was known, what this paper adds, and what follows in practice (Section 6).
3.1. Identical outputs and "only the form varies". Both observations are correct for the submitted version. Study 1 included a sixth LLM rated by a separate group of evaluators, a point our analysis plan had flagged in advance. The revision analyses Study 1 on the same five LLMs as Studies 2 and 3, so every comparison now uses the same 125 outputs (For Study 1, this means 250 ratings from 12 evaluators). Preference agreement is α = 0.27 on this set. The LLMs still differ on preference, and on two presentation dimensions (Style and Format) but not on Accuracy or Professional Judgement; Structure no longer reaches significance. The sentences at L16–17 and L93–96 are rewritten: tasks, outputs and rubrics are held fixed, and the text now states that the rater groups differ too (Sections 1 and 7).
3.2. Judge-measured differences. We agree that comparing a significant with a non-significant test does not establish a difference, and have replaced that interpretation with a direct test (Appendix F.4). The judge scores every LLM higher than the grader panel by a similar amount, and the offset does not differ across LLMs (p = 0.23). Its per-LLM means are as spread out as the panel's: the slope of judge means on panel means is 1.06 (95% interval 0.46 to 1.55). There is therefore no evidence that the judge underestimates differences, and the "lower bound" sentence is removed. The revision describes the judge's failure to separate the LLMs as inconclusive rather than as evidence that they are equal (Section 5.2).