Three Is Not a Crowd: A Decision Theory for Label Validity

Annotators disagree, and in many annotation tasks more than one label for the same object may be genuinely valid. Label aggregation must therefore address two questions simultaneously: which labels are sufficiently supported to be treated as valid, and how reliable is the resulting set of selections? Standard approaches do not generally provide both. Majority vote and classical latent-label models such as Dawid–Skene typically return a single aggregate label, often without an explicit mechanism for withholding a decision when evidence is weak. Distribution-preserving approaches take a different view, representing disagreement through soft labels rather than determining which individual labels have sufficient evidence to be selected as valid. More broadly, existing methods typically quantify uncertainty through global performance measures or model-dependent quantities, rather than providing an explicit error-rate guarantee over the labels ultimately selected. We recast label recovery as a multiple-testing selection problem. Each object–label pair defines a statistical hypothesis, and labels are selected only when the annotation evidence is sufficiently strong; otherwise, they remain unresolved. Using false discovery rate (FDR) control, the procedure bounds the expected proportion of invalid labels among the selected object–label pairs at a user-specified level. The resulting guarantee is therefore not merely a post-hoc evaluation metric, but an error-control criterion built directly into the recovery procedure. Our framework requires neither a predefined number of valid labels per object nor an explicit model of annotator reliability, and avoids combinatorial search over candidate label subsets. This reframing turns label aggregation from forced estimation into statistically controlled selection under disagreement.