- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Alternative Annotator Test with Resampled Annotations (Altra-Test), a statistical method to determine whether an LLM-as-a-judge can replace human annotator. Notably unlike previous methods, this method does not require the same annotator across all labeled items. The paper provides experiments over simulated data and real word datasets.
Critical strengths
The paper provides a statistically sound method to estimate whether LLM can be used to replace human annotators. They provided clean proofs and simulations to show that the method is theoretically sound. Empirically, only 100 or so labeled items are necessary -- suggesting the test is cheap and feasible to run in the real world.
Critical weaknesses
The motivation that we might not have annotations from the same annotators over all the items seem somewhat contrived to me. Presumably, if I wanted to use these statistical methods, in reality I would just hire a few annotators to label a small pool of data.
Also, it is not clear to me as to the novelty of this method. It seems that the estimator already exists from classical statistics. The only novelty is in applying this to LLM-as-a-judge setting.
Questions for authors
See weaknesses. Also for the experiments, were the datasets labeled by a non-constant mix of annotators over each task?