acceptodds
Withdrawn from ICLR 2027

Quantifying Consensus: Overcoming Exact Match Limitations in LLM-as-a-Judge Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As Large Language Models (LLMs) increasingly serve as autonomous evaluators for complex outputs (LLM-as-a-Judge), the demand for fine-grained evaluation on granular scales (e.g., 1-5 or 1-10) has grown. Yet, this paradigm shift has exposed a critical vulnerability: we are evaluating the alignment of the next-generation artificial judges using legacy metrics that were never designed for this task. Current evaluation methods primarily rely on rank correlation coefficients (e.g., Spearman’s ) or traditional Inter-Annotator Agreement (IAA) metrics like Cohen’s Kappa. Both approaches are fundamentally limited for modern LLM evaluation: rank metrics ignore absolute calibration, while standard IAA metrics demand exact numerical consensus and are highly susceptible to statistical deflation under skewed distributions. To address these gaps, we introduce Tolerance-Weighted Agreement (TWA, ), an absolute alignment metric natively designed for the LLM-as-a-Judge paradigm. By using graded tolerance weights to absorb minor subjective variance, TWA isolates true semantic consensus between artificial judges and human evaluators without suffering from statistical deflation. We validate the metric across an In-Context Learning (ICL) classification explainability dataset and the evaluation sets of established benchmarks (BiGGen-Bench, SummEval). The results demonstrate TWA's robustness in quantifying Human-LLM alignment across diverse tasks. Furthermore, it provides a highly interpretable alternative to complex metrics like Gwet’s AC2 and naturally scales to measure Inter-Judge Agreement directly between competing LLMs.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-Judge, Evaluation Metrics, Inter-Annotator Agreement, Human Alignment, Statistical Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Dy12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Tolerance-Weighted Agreement (TWA), which measures absolute agreement between human and LLM ratings using distance-based partial credit derived from a discretized Gaussian distribution. It evaluates TWA on three datasets and proposes thresholds for interpreting evaluator alignment.

Critical strengths

TWA provides a simple, interpretable summary of absolute rating agreement that distinguishes small deviations from large errors. The experiments across three datasets offer useful illustrations of how different agreement measures characterize evaluator behavior.

Critical weaknesses
  1. The comparisons do not establish TWA’s claimed advantages. Absolute agreement and rank correlation serve different purposes. Using Exact Match and MAE as reference points favors absolute agreement by construction. The experiments show that TWA captures score proximity, but provide limited independent evidence that it produces more useful evaluator assessments than existing measures.

  2. The weighting and alignment thresholds lack sufficient justification. Gaussian discretization specifies the weights, but does not establish that they appropriately represent acceptable disagreement. The 0.65/0.75 thresholds also lack independent validation. Furthermore, the claimed variance guarantee does not hold generally: an error distribution with 80% exact agreement, 10% errors of +3, and 10% errors of −3 yields TWA approximately 0.803, while the standard deviation of the signed errors is approximately 1.342.

Questions for authors
  1. What independent evidence shows that TWA provides more useful evaluator assessments than existing measures, beyond reflecting a different emphasis on absolute agreement rather than rank correlation?

  2. What supports the Gaussian weights and the 0.65/0.75 thresholds as meaningful alignment criteria? Under what assumptions does the claimed variance bound hold, given the counterexample above?

  3. How are multiple human ratings handled in each dataset: aggregated into one reference score, or compared individually with the LLM and then averaged?

  4. How are the tolerance weights and alignment thresholds validated across rating scales? Specifically, were G-Eval’s 1–3 fluency scores made comparable with SummEval’s 1–5 human scores before computing TWA?

Reviewer_3TMuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focused on the metrics for evaluating LLMs as judges for tasks involving fine-grained scales (e.g., 1--5). Specifically, it underlines two failure modes when using existing metrics: (1) correlation coefficients fail to capture absolute score calibration, (2) standard chance-adjusted IAA metrics (like Cohen's K) fail when output distributions are skewed. To resolve these, the paper introduces Tolerance-Weighted Agreement (TWA) metric that discretizes (maps Likert scale values to dynamic weights) a continuous standard normal distribution (e.g., W(0) = 1.0, W(1) ~ 0.631, W(2) ~ 0.158...) to absorb any human subjective variance. The metric is validated across various benchmarks and dimensions to quantify Human-LLM, Human-Human, and LLM-LLM alignments.

Critical strengths
  • Motivation for the problem is strong and this work clearly underlines why existing metrics fail for evaluating modern LLM-as-a-judge settings.
  • Grounding in standard normal distribution makes TWA weights parameter-free and this work empirically (through H-H, H-LLM, LLM-LLM settings) shows that TWA scores represent semantic consensus (rather than models guessing).
Critical weaknesses
  1. TWA is task-agnostic (or rubric agnostic) when applying weight decay. So, for tasks where difference of 1 in scale represents an error (e.g., miscounting elements) rather than human subjectivity/noise, TWA ends up assigning higher partial credit disregarding facts (specific to the task).
  2. The paper acknowledges that TWA assumes zero-centered disagreement in its limitations. But given that end-users of TWA will not be able to differentiate calibration bias from evaluation noise, this aspect needs to be elaborated and discussed in the main paper.
Questions for authors

Questions:

  1. For highly objective tasks, where difference of 1 in scale is an error (not noise), how should σ\sigma be adjusted?
Reviewer_Btd6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Tolerance-Weighted Agreement (TWA), a metric for measuring agreement between LLM judges and human evaluators.

The main idea is that exact match is often too strict. If a human gives a 4 and the LLM gives a 5, that should count as a smaller disagreement than a human 1 against an LLM 5. TWA gives:

  • Full credit for an exact match
  • Partial credit for a one-point difference
  • Less credit for a two-point difference
  • Almost no credit for large differences

TWA is a simple way to summarise rating differences. However, a higher TWA score does not mean the LLM judge itself has improved. It only means the metric gives partial credit to nearby scores.

Critical strengths

Importance. Exact match can be too strict for subjective rating tasks. A one-point difference is usually less serious than a difference of three or four points, and TWA captures this distinction in a simple way.

Simplicity. TWA gives a score for each human-LLM rating pair, so it is easy to see how individual differences contribute to the final agreement score. It is also easy to calculate and can be applied to new evaluation results without a complex process.

Breadth of comparison. The paper compares TWA against a wide set of existing metrics:

  • Exact match
  • Mean Absolute Error
  • Pearson correlation
  • Rank correlation
  • Cohen's kappa
  • Weighted kappa
  • Krippendorff's alpha
  • Gwet's AC2

The evaluation covers three datasets and includes LLM-human alignment, human-human alignment, and LLM-LLM agreement.

Critical weaknesses

1. Limited novelty. TWA is similar to existing weighted-agreement methods, which also give partial credit when two ratings are close. The main difference appears to be that TWA uses Gaussian weights and does not correct for chance agreement.

2. The "Aligned" threshold ignores chance agreement. The paper treats a TWA of 0.75 or higher as "Aligned". But Table 9 shows that the average score after random shuffling is already above 0.75 in 12 out of 43 comparisons. For example, the observed TWA for ICL Conciseness is 0.918, while the random baseline is already 0.862. A TWA above 0.75 therefore does not always show meaningful agreement.

3. The claim linking TWA to error standard deviation does not hold in general. The paper says that a TWA of at least 0.75 means the standard deviation of rating errors is approximately 0.85 or less. This is not true for every error distribution. If 75% of ratings match exactly and 25% differ by four points, TWA is still approximately 0.75, but the error standard deviation is about 1.7.

4. Unclear scale handling for G-Eval Fluency. G-Eval Fluency scores use a 1-3 scale, while the human ratings use a 1-5 scale. The paper does not clearly explain whether the scores were converted to a common scale before calculating TWA. If the original scales were compared directly, the low TWA may mainly reflect the scale mismatch rather than poor LLM judgement.

5. Inconsistencies between text and figures.

  • Figure 15 appears to show TWA increasing as kurtosis increases, but the text says TWA "plunges rapidly". The figure and its explanation disagree.
  • The weight formula in Section 3 differs from the formula shown in Figure 3. The two give different weights for the same rating difference.

6. The "indistinguishable from a human" claim is not supported. Similar agreement scores do not prove that the LLM and humans are statistically equivalent. A proper equivalence or non-inferiority test would be needed.

7. Different is not the same as better. The paper shows that TWA produces different results from other metrics, but that does not make them better results. The Gaussian weights and the 0.65 / 0.75 thresholds are chosen by the authors and are not validated against an independent measure of judge quality.

Questions for authors
  1. How should readers interpret an "Aligned" result when the random baseline is already above 0.75? Can the authors provide a chance-corrected score, or define thresholds relative to the random baseline?

  2. Under what assumptions does a TWA of at least 0.75 imply a low error standard deviation? Can the authors revise this claim and explain how it handles the counterexample above?

  3. Were the G-Eval Fluency scores converted from the 1-3 scale to the human 1-5 scale? If so, what exact conversion was used, and was it applied before or after rounding?

  4. Which exact weight formula was used in each experiment? Can the authors explain the difference between Section 3 and Figure 3, and correct the contradiction between Figure 15 and its description?

  5. What statistical test supports the claim that the LLM is indistinguishable from a human evaluator? Can the authors provide an equivalence or non-inferiority analysis with a clearly justified margin?

  6. What external evidence shows that the chosen Gaussian weights and the 0.65 / 0.75 thresholds reflect useful judge quality? Can the authors validate them against expert decisions, system rankings, or another independent measure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.