acceptodds
Under review as a conference paper at ICLR 2027

AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators

Abstract

Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (alt-test) was introduced for dense settings where all items are annotated by the same annotators. However, in many large-scale annotation settings, items are typically annotated by different subsets of annotators. We therefore propose the Alternative Annotator Test with Resampled Annotations (AltRa-Test), a statistical test that does not require shared annotators across all items. AltRa-Test uses a bootstrap that resamples items, draws human annotations for each item, and substitutes one of them with an LLM annotation. This produces paired all-human and mixed human–LLM panels. A replacement is supported when the inter-annotator agreement (IAA) of the mixed panel is statistically non-inferior to the IAA of the all-human panel. We first establish that AltRa-Test is a consistent test of asymptotic level . We then evaluate it through finite-sample size simulations and real-world experiments in which we test 18 LLMs and 5 prompting techniques on 20 datasets spanning four modalities: text, image, audio, and video. Across these evaluations, AltRa-Test requires as few as 100 items to yield stable replacement decisions. In simulations, it dominates baselines by improving the trade-off: for any fixed rate of falsely accepting models noisier than simulated humans, , it correctly accepts more models that are less noisy. In real-world data, AltRa-Test supports replacing humans with LLMs in about half of the evaluated setups.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, annotation, annotator reliability, annotator agreement

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_AHpGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Alternative Annotator Test with Resampled Annotations (Altra-Test), a statistical method to determine whether an LLM-as-a-judge can replace human annotator. Notably unlike previous methods, this method does not require the same annotator across all labeled items. The paper provides experiments over simulated data and real word datasets.

Critical strengths

The paper provides a statistically sound method to estimate whether LLM can be used to replace human annotators. They provided clean proofs and simulations to show that the method is theoretically sound. Empirically, only 100 or so labeled items are necessary -- suggesting the test is cheap and feasible to run in the real world.

Critical weaknesses

The motivation that we might not have annotations from the same annotators over all the items seem somewhat contrived to me. Presumably, if I wanted to use these statistical methods, in reality I would just hire a few annotators to label a small pool of data.

Also, it is not clear to me as to the novelty of this method. It seems that the estimator already exists from classical statistics. The only novelty is in applying this to LLM-as-a-judge setting.

Questions for authors

See weaknesses. Also for the experiments, were the datasets labeled by a non-constant mix of annotators over each task?

Reviewer_JguiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses challenges of current statistical procedures for testing whether an LLM judge can replace a human annotator: annotator information is needed and the annotator needs to be consistent over samples. However, mostly when collecting annotations or even for existing datasets, this information is not available nor do all annotators annotate all samples; they rather annotate a subset of the samples.

As such, this paper introduces the AltRA-Test, which allows for testing whether an LLM judge can replace a human annotator for a task. This test allows for different annotators across samples, in contrast to previous work. The test first samples N items from the dataset, where for each item, k human labels (YhY_h) are sampled. Per sample, one human label is replaced by an LLM label (YmY_m). Then the Krippendorff’s α\alpha, inter-annotator agreement, is computed on YhY_h and YmY_m and the difference between them is computed to get the substitution gap. This indicates whether the LLM can actually replace the human or not. This also includes a small margin for error as a trade-off from going from costly human annotation to cheaper LLM judgments. To determine whether an LLM is actually an alternative to the LLM, a statistical non-inferiority percentile test is done to see whether the null hypothesis of a human being replaced by an LLM reduces the agreement by more than the allowed margin is rejected.

The authors show on simulations that AltRA-Test outperforms other statistical procedures it builds further on and on real-world datasets it shows that a human being replaced with an LLM is rather dataset-dependent.

Critical strengths
  • I found the paper to address an important and timely gap. As more and more researchers and practitioners have started to rely on LLM judges, it is highly essential to actually verify whether this reliance is justified or not. And current annotator procedures usually make use of multiple annotators without having everyone annotate everything. As such, I feel that the proposed test addresses this gap adequately. I also found the breadth of the experimental setup on the simulations and the wide variety of real-world datasets they apply the test on very nice. They use 20 datasets spanning across text, image, etc to showcase the utility of the test across 18 LLM judges and 5 different prompting techniques.
  • The paper was also, in my opinion, very well-written and well-presented (e.g. Figure 1 was really nice, made it really easy to follow). The authors also nicely place the work amongst relevant and similar work.
Critical weaknesses
  • For me, the biggest weakness I found was a lack of discussion surrounding the results with respect to subjectivity. The introduction acknowledges how problematic it is when minority viewpoints in subjective tasks are removed and the adaptable margin was also a nice addition for subjective tasks so I was hoping this comes back in the results section a bit stronger as well. Especially, since the authors mention that some datasets used are that of subjective tasks. For instance, an easy way to look at this would have been the effect of increasing k and seeing how this affects the subjective datasets. I also wonder if for such tasks it is a good idea to replace human annotations with LLM judges, how are we going to achieve this difference in perspectives? How does (or can) the AltRA-Test give us more insights into this? Maybe a different strategy is needed in this case?
Questions for authors
  • Do you think changing the strategy of choosing k human labels by maximizing for diversity (i.e., ensuring that minority viewpoints are also represented) gives us a better view of whether the LLM judge can replace a human?
Reviewer_F9hmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AltRA-Test to decide whether a human annotator can be replaced by an LLM annotator. The target setting is that items are labeled by different human annotators, which differs from existing work that requires a shared annotator panel. AltRA-Test uses a bootstrap approach in which each replicate resamples N items, draws three human labels per item, replaces one label with the LLM’s label, and computes Krippendorff’s α for the human panel and the mixed panel. The authors define the conditions under which the LLM can pass while preserving a consistent level of α. Both simulated and real-world studies show that LLMs pass on 35.3% of the Judge datasets under CoT, and on 47.8% when each judge uses its best prompt.

Critical strengths
  1. The paper is well motivated. Existing methods require a shared annotator panel, whereas this paper does not. Among the 20 datasets, STT applies to only one, ALT-TEST to nine, and AltRA-Test applies to all 20. The procedure works with any chance-corrected agreement measure and has an interpretable parameter.

  2. The experiments are extensive, covering both simulated and real-world studies across many datasets and LLM models. Robustness is examined with respect to sample size, panel size, and number of labels. The paper also correctly notes that passing at low human agreement is less meaningful.

Critical weaknesses
  1. I am concerned that when human agreement is low, the acceptance margin becomes lenient enough that even a random labeler could pass. The allowed drop in agreement increases as human agreement decreases. As a result, an annotator that ignores the item and labels randomly may pass when human agreement is low.

  2. Most components are inherited: mixed-panel agreement and the bootstrap come from STT, while the non-inferiority formulation comes from ALT-TEST. The new elements are per-item annotation resampling and the adaptive margin, analyzed using standard delta-method and percentile-bootstrap arguments. The paper’s contribution therefore appears to lie mainly in improved applicability and the empirical study. Further external validation on real data would strengthen the claims. For example, CIFAR-10H includes the original CIFAR-10 labels, which could be used to check whether passing judges are actually at least as accurate as humans.

Questions for authors
  1. How does the test behave in simulations where the LLM outputs the human plurality label, or where its errors are correlated with human errors? On datasets with gold labels, such as CIFAR-10H, do passing judges match or exceed average human accuracy, and do failing judges fall below it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.