- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper claims that while LLM Judge can have good ranking capability, it fails predicting with the correct score scales, which leads to low agreement acceptance rates or occupational aggregates. To support this claim, the paper introduce O*NET-BENCH, which is an audit suite derived from an existing survey of 45,796 worker ratings, and evaluate 33 judge configurations on the benchmark. The paper also shows that supervised finetuning doesn't solve the score scale issue but improves the models's ranking capability. To solve the score scale issue, the paper proposes to calibrate the score distribution. Overall, the paper's main claim is ranking agreement alone is insufficient for occupational measurement, while acceptance rates and aggregates are also important metrics.
Critical strengths
The paper addresses an important question in using LLM Judges for occupational ai measurement.
All the claims in the paper are backed by comprehensive evaluations.
Critical weaknesses
The paper focuses on occupantional AI measurement, which makes the impact of this paper very limited. It also remains unclear whether the findings generalize to other evaluation settings.
As the author mentioned, the proposed O*NET-BENCH comes from a survey of 45,796 worker ratings, so the dataset itself has inconsistency. In other words, there are misalignment between the data samples, which could also explain the performance gap of the model on this benchmark. Repeated ratings of the same responses are needed to justify the low individual R² and aggregate correlations of the benchmark.
To understand whether or not LLM without calibration can be good at occupational ai measurement, why don't you treat it as a binary classification problem, where the model outputs only qualified or disqualified. Would the model still performs bad on acceptance rates or occupational aggregate?
Questions for authors
Are there other tasks where the score scale matters in LLM judge? It would be benefitial to list them in the paper to strengthen the generalizability of this paper.
How sensitive are LLM Judge performance to the rubrics in prompt? Can the given prompt accurately reflect the same scoring standard in the survey?