- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces Tolerance-Weighted Agreement (TWA), which measures absolute agreement between human and LLM ratings using distance-based partial credit derived from a discretized Gaussian distribution. It evaluates TWA on three datasets and proposes thresholds for interpreting evaluator alignment.
Critical strengths
TWA provides a simple, interpretable summary of absolute rating agreement that distinguishes small deviations from large errors. The experiments across three datasets offer useful illustrations of how different agreement measures characterize evaluator behavior.
Critical weaknesses
-
The comparisons do not establish TWA’s claimed advantages. Absolute agreement and rank correlation serve different purposes. Using Exact Match and MAE as reference points favors absolute agreement by construction. The experiments show that TWA captures score proximity, but provide limited independent evidence that it produces more useful evaluator assessments than existing measures.
-
The weighting and alignment thresholds lack sufficient justification. Gaussian discretization specifies the weights, but does not establish that they appropriately represent acceptable disagreement. The 0.65/0.75 thresholds also lack independent validation. Furthermore, the claimed variance guarantee does not hold generally: an error distribution with 80% exact agreement, 10% errors of +3, and 10% errors of −3 yields TWA approximately 0.803, while the standard deviation of the signed errors is approximately 1.342.
Questions for authors
-
What independent evidence shows that TWA provides more useful evaluator assessments than existing measures, beyond reflecting a different emphasis on absolute agreement rather than rank correlation?
-
What supports the Gaussian weights and the 0.65/0.75 thresholds as meaningful alignment criteria? Under what assumptions does the claimed variance bound hold, given the counterexample above?
-
How are multiple human ratings handled in each dataset: aggregated into one reference score, or compared individually with the LLM and then averaged?
-
How are the tolerance weights and alignment thresholds validated across rating scales? Specifically, were G-Eval’s 1–3 fluency scores made comparable with SummEval’s 1–5 human scores before computing TWA?