acceptodds
Withdrawn from ICLR 2027

Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM judges are increasingly used to assess whether AI outputs meet workplace requirements, but agreement on response rankings does not establish agreement on acceptance rates or occupational aggregates. We introduce O*NET-BENCH, an audit suite derived from an existing survey of 45,796 worker ratings, and evaluate 33 pre-existing judge configurations across six model families on 4,501 test ratings. Twenty-five configurations achieve tie-aware pair accuracy of at least 0.60, although a train-fitted response-only TF–IDF baseline nearly matches the strongest judge. Despite this ordering agreement, judges estimate that 3.0%–97.9% of responses are acceptable, compared with 61.1% for occupation-matched workers. In one fine-tuned lineage, changing from pointwise scoring to a bundled few-shot/listwise protocol improves response ordering while reducing agreement with worker means at the task and occupation levels; this reversal replicates on a task- and worker-disjoint validation split under prespecified criteria. Cross-validated calibration largely removes mean bias, but calibrated scores explain at most 8.5% of individual worker-rating variance. Prediction-assisted estimation yields at most small precision gains at the studied label budgets. These results show that ranking agreement alone is insufficient for occupational measurement. Judges should be validated against the acceptance rates and aggregates their scores will be used to estimate.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, evaluation validity, occupational AI measurement, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CJy6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper claims that while LLM Judge can have good ranking capability, it fails predicting with the correct score scales, which leads to low agreement acceptance rates or occupational aggregates. To support this claim, the paper introduce O*NET-BENCH, which is an audit suite derived from an existing survey of 45,796 worker ratings, and evaluate 33 judge configurations on the benchmark. The paper also shows that supervised finetuning doesn't solve the score scale issue but improves the models's ranking capability. To solve the score scale issue, the paper proposes to calibrate the score distribution. Overall, the paper's main claim is ranking agreement alone is insufficient for occupational measurement, while acceptance rates and aggregates are also important metrics.

Critical strengths

The paper addresses an important question in using LLM Judges for occupational ai measurement.

All the claims in the paper are backed by comprehensive evaluations.

Critical weaknesses

The paper focuses on occupantional AI measurement, which makes the impact of this paper very limited. It also remains unclear whether the findings generalize to other evaluation settings.

As the author mentioned, the proposed O*NET-BENCH comes from a survey of 45,796 worker ratings, so the dataset itself has inconsistency. In other words, there are misalignment between the data samples, which could also explain the performance gap of the model on this benchmark. Repeated ratings of the same responses are needed to justify the low individual R² and aggregate correlations of the benchmark.

To understand whether or not LLM without calibration can be good at occupational ai measurement, why don't you treat it as a binary classification problem, where the model outputs only qualified or disqualified. Would the model still performs bad on acceptance rates or occupational aggregate?

Questions for authors

Are there other tasks where the score scale matters in LLM judge? It would be benefitial to list them in the paper to strengthen the generalizability of this paper.

How sensitive are LLM Judge performance to the rubrics in prompt? Can the given prompt accurately reflect the same scoring standard in the survey?

Reviewer_djFQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audit 33 LLM-judge configs on an existing dataset of worker ratings for O*NET derived tasks (Mertens et al.) and show that judges agree somehow with workers on the ordering of responses but not on the score level, so acceptance rates and occupation level aggregates change a lot depending on the judge (from 3% to 98% vs 61% for workers). They also show a reversal in one fine-tuned model where the listwise protocol improve ordering but make aggregate agreement worse, and that calibration and PPI give only small gains. At the end authors propose a reporting standard.

Critical strengths

Large and careful empirical work, many configs and some controls. Stats are done with care, clustered CIs, preregistered replication on validation split. The topic is relevant since LLM judges are used more and more for evaluations related to work and occupations.

Critical weaknesses
  1. The novelty is limited. The main message (ranking agreement tell nothing about the level of the scores) is basically rank invariance, the authors themselves show it in few lines in sec 4. This is well known in psychometrics and statistics since long time, and the same is true for standard correlations (Pearson/Spearman) which are also blind to scale. So the paper is more an empirical confirmation of something expected than a new insight.

  2. The motivation is not well supported. The paper claim that "scale-blind selection" is a common practice, but no concrete example is given of a work that select a judge only by ordering and then use it to report absolute acceptance rates. The sentence in intro about this has no citations at all. Looking at the occupational benchmarks they cite, they don't seem to do this, e.g. GDPval validate its grader w/ pairwise agreement but it also report pairwise win rates (so it is consistent), and ProfBench use F1 on absolute labels + a bias index, which the paper itself mention in related work.

  3. Only one dataset, with one rating per response and very low reliability of the human aggregates (split-half Spearman-Brown reliability approx 0.09 at occupation level). With this level of noise is hard to say how much of the disagreement at task/occupation level come from the judge and how much from the workers. Authors acknowledge it, but the claims about aggregates in abstract and intro are still quite strong.

  4. The most interesting result (ordering better but aggregates worse) is only in one fine-tuned lineage (2 checkpoints, not independent), and the protocol change mix together presentation, exemplars and output schema, plus the model was fine tuned on pointwise format. So is not clear what cause it and if it generalizes. In the other matched pairs there is no clear effect.

Questions for authors
  • Can you give concrete examples of published works that do scale-blind selection and then report absolute rates?
  • Why not use agreement measures that take scale into account (e.g. ICC, Lin's CCC) together w/ bias and SD ratio?
Reviewer_AZ64OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits whether response-level ranking agreement validates LLM judges for occupational acceptance rates and aggregates. Using an existing worker-rating corpus, it compares 33 judge configurations and studies calibration, prediction-assisted estimation, and a prespecified protocol reversal in one fine-tuned lineage. The results show that ordering, score levels, and occupational aggregates require distinct validation.

Critical strengths

S1. The paper connects a familiar ordinal-versus-cardinal distinction to consequential evaluation quantities. Figure 1 and Sections 5.2 and 5.5 make the implications for acceptance shares and occupational summaries concrete.

S2. The audit includes useful safeguards and disclosures, including task- and worker-disjoint splits, tie-aware metrics and coverage, train-fitted controls, and an explicit distinction between exploratory comparisons and the prespecified reversal replication.

S3. Section 5.6 usefully separates removal of mean bias from individual predictive information and threshold-probability estimation. This prevents an overly optimistic interpretation of global calibration.

Critical weaknesses

W1. Uncertainty in the occupational reference deserves more direct propagation. Section 4 reports sparse individual ratings and an occupational split-half correlation of about 0.093. The clearer-majority subset in Section 5.5 is helpful, but reporting occupational sample sizes and reference intervals alongside model disagreement would better distinguish robust aggregate discrepancies from unstable fine-grained rankings.

W2. The label-efficiency conclusions in Section 5.6 need to remain specific to the predictors and estimators studied. Weak individual prediction and the reported approximate prediction-assisted intervals do not establish a general limitation of prediction-powered inference. A clearer alignment between the target-specific variance argument, clustered estimation, and the implemented tuned estimator would strengthen this section.

Questions for authors

Q1. How do the occupational discrepancies change when reference uncertainty is propagated through the full aggregation procedure, with sample size reported for each occupation?

Q2. Does the ordering–aggregation reversal replicate in another lineage, or when the bundled protocol factors are changed separately?

Q3. Can task- or occupation-conditional acceptance calibration and a fully specified cluster-aware prediction-powered estimator improve on the global calibration results?

Q4. Building on your discussion of JuStRank, which distinguishes response-level agreement from aggregate system ranking, could you clarify how the findings in Sections 4 and 5.6 relate to human-guided ordinal calibration such as AtC and adaptive prediction-powered inference such as PPI++?

[1] PPI++: Efficient Prediction-Powered Inference

[2] AtC: Aggregate-then-Calibrate for Human-centered Assessment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.