- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper explores how best to evaluate automatic evaluation methods, for example LLM-as-judge, machine translation metrics, or reward models. Such automatic evaluations are often themselves evaluated based on metrics like correlation to a ground truth (e.g., human label).
The starting observation is that automatic evaluation methods are often judged based on correlation pooled across groups. For example, the evaluator might assign a score to several generations for a single prompt. This pooling ignores the fact that we might care about picking the best candidate within a group rather than how well a model predicts an absolute score (across groups).
The paper then describes related work, some experimental protocols, and resulting numbers from comparing different ways of evaluating automatic evaluations over datasets like PPE, WMT25 MQM, and WMT25 ESA.
Critical strengths
The research topic is interesting: what value do we leave on the table by not fully considering the dependence structure and different use-cases of automatic evaluations?
The authors have correctly identified an important area for further research. What is the proper way to statistically account for within-cluster dependence? What automatic evaluations should we use when picking the best candidate from a cluster, as opposed to assessing the overall performance of a model? The topic has many important, under-studied directions.
Critical weaknesses
The paper does not present a coherent argument for any specific conclusion about a well-identified research question. Instead, the paper describes the topic, followed by related work, and then some experimental protocols and their results. This description is disjointed and fragmented, making it difficult to follow. Most of the text describes details about what was done thus it reads like a log rather than a research paper. Consequently, it is unclear what lessons, insights, facts, or discoveries one should take away from the paper, except perhaps that it is better to consider dependence structure when analyzing hierarchical / mixed-effects data, which is a well-researched area in statistics.
To be more specific, here are the contributions listed at the end of the introduction:
- "Ordering is not choosing": This bullet comes closest to presenting a conclusion; however, the bullet's text is ambiguous: "... a statistic grouped on the decision's clusters orders evaluators better than pooling in the mean over a set's configurations, and the evaluator it selects is not reliably better than pooling's...". In other words, using the dependence structure orders evaluators better, but does not select a better evaluator.
- "What a set can show, and where the advantage falls": The paragraph for this bullet just repeats numbers from the results but gives no research questions or conclusions.
- "An estimand and its calibration": This bullet describes their evaluation metric.
- "Choosing a judge from eight labelled prompts": Here, they say that on one dataset (PPE), their tie-calibrated accuracy score gave them a confidence interval which contains zero. The research question isn't stated, and the result did not have sufficient power to show something one way or the other.
The paper needs significant revision. The experimental protocols and outcomes are potentially well-documented (there are 102 pages of appendices); however, they have not been interpreted or assembled into a coherent argument.
Questions for authors
My main questions are about the authors' interpretation of the results, and the argument they are trying to make:
- What core research question(s) did you want to ask with this work?
- What were your hypotheses about these questions?
- In your empirical results, which methods do you consider your baselines (the pooled Pearson correlation)? - Which methods do you view as the most important variants you wished to test (e.g., the utility from line 207)?