acceptodds
Under review as a conference paper at ICLR 2027

Three Is Not a Crowd: A Decision Theory for Label Validity

Abstract

Annotators disagree, and in many annotation tasks more than one label for the same object may be genuinely valid. Label aggregation must therefore address two questions simultaneously: which labels are sufficiently supported to be treated as valid, and how reliable is the resulting set of selections? Standard approaches do not generally provide both. Majority vote and classical latent-label models such as Dawid–Skene typically return a single aggregate label, often without an explicit mechanism for withholding a decision when evidence is weak. Distribution-preserving approaches take a different view, representing disagreement through soft labels rather than determining which individual labels have sufficient evidence to be selected as valid. More broadly, existing methods typically quantify uncertainty through global performance measures or model-dependent quantities, rather than providing an explicit error-rate guarantee over the labels ultimately selected. We recast label recovery as a multiple-testing selection problem. Each object–label pair defines a statistical hypothesis, and labels are selected only when the annotation evidence is sufficiently strong; otherwise, they remain unresolved. Using false discovery rate (FDR) control, the procedure bounds the expected proportion of invalid labels among the selected object–label pairs at a user-specified level. The resulting guarantee is therefore not merely a post-hoc evaluation metric, but an error-control criterion built directly into the recovery procedure. Our framework requires neither a predefined number of valid labels per object nor an explicit model of annotator reliability, and avoids combinatorial search over candidate label subsets. This reframing turns label aggregation from forced estimation into statistically controlled selection under disagreement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-label Annotation, False Discovery Rate, Multiple Testing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8bifOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper reframes multi-label crowd annotation as a multiple-testing problem with FDR control. Rather than forcing a single label per object (Dawid-Skene, majority vote), the method selects a variable-size label set (or abstains when evidence is weak) while bounding the fraction of invalid selections at a user-specified level qq. A three-step pipeline screens null-enriched pairs via Poisson thinning, fits a truncated Poisson model to debias null intensities, and applies weighted Benjamini-Hochberg on cross-fitted p-values. Experiments on synthetic and six real crowdsourced datasets show BH/WBH hold FDR at target while standard baselines exceed it.

Critical strengths
  • S1: Code is available, facilitating the reproducibility of the proposed approach.
  • S2: The experimental setup is extremely clear.
  • S3: Reframing label aggregation from estimation to statistically controlled selection is novel and well-motivated.
Critical weaknesses
  • W1: Some important related works are missing [1,2]. They are studing when majority vote (and its variants) are enough to achieve results that are comparable to an oracle MAP.
  • W2: While lots of baselines are presented, in Section 2 you discuss about perspectivist methods but you are not adding them to the set of baselines.
  • W3: While the experimental setup is clearly described, the Results section can be improved. It is not clear how each of your results is solving one of the research questions of your paper.
  • W4: I think you should discuss [3], which in the title is stating something that is the negation of what you are stating.
  • W5: Can you priovide a runtime analysis to understand how all the methods work in terms of time? Also on a single dataset is enough.
  • W6: Figure 2 shows particular p-values on ConvAbuse (the FDR guarantee breaks). The paper flags this but does not provide a diagnostic or at least a tentative explanation.
Questions for authors
  • M1: The first citation should be a citep and not a cite.
  • M2: Lines 46 to 48. I think that a citation at that point is necessary.Same for lines 51 to 53. You refer to some methods but you are not citing them.
  • M3: Line 141 "RECOV7RY"
  • M4: Figure 1 is very small and not easy to be read.
  • M5: While in general the experimental setup is well presented (as I wrote in S2), I think that the datasets could be described better. You are introducing them but never explaining them.

Additionally, consider as questions what is written in the Critical Weaknesses form.

4 comments
AuthorsOpenReview ↗

## Reply to Reviewer 8bif (part I)

Thank you for the detailed review and for identifying the specific comparisons and presentation changes that would improve the paper. Below, we address each of the issues raised and outline the corresponding revisions to the manuscript.

W1. We could not locate the full references for [1] and [2] in the review. Could you please provide their bibliographic details so that we can appropriately cite and discuss them in our revised manuscript? We will cite and discuss them. Our work is complementary in two ways:

  • (i) Even oracle MAP has limits. Prop. 1 is a lower bound for any estimator, oracle MAP included. Where majority ≈\approx MAP, both still cannot recover minority-endorsed labels from 3–10 votes, so matching MAP does not guarantee reliable labels.
  • (ii) Neither rule targets an error rate. In our simulations, MAP-decoded DS/GLAD/MACE/Spectral exceed q=0.2q=0.2 on HS-Brexit and WebCrowd. Majority is safe where gold tracks the vote, but loses power elsewhere (0.240.24 on MAESTRO). Our method adds an explicit error target to this picture.

W2 — NUTMEG and VariErr perspectivist baselines not applicable to our cohort:

  • The standard distribution-preserving rule, which keeps a label if its soft-label mass S/M≥τS/M \geq \tau. Because it has no error target, τ\tau can only be tuned with gold, we omit in our simulation.
  • NUTMEG needs annotator demographics, and VariErr needs extra validity judgments. Our six tables have neither, which is why §1.2 calls these methods complementary.
  • Revision: Soft-label rules with τ∈{1/4,1/3,1/2}\tau \in \{1/4, 1/3, 1/2\} added to Fig. 4 and the real-data table.

W3 — Results vs. research questions. Sec. 4 will open each paragraph with its question and close with a one-line answer:

  • Q1 — Is the fitted null calibrated? Super-uniform on 5/6 tables. The ConvAbuse warning traces to a detectable degenerate screen and is fixed (Fig. 1).
  • Q2 — Synthetic truth. BH/WBH meet qq under false plurality and grouped dependence. High-power majority and objectwise latent models exceed qq; conservative ones lose power. The new robustness table is added (Fig. 3).
  • Q3 — Real tables. BH/WBH meet qq on 6/6 tables, versus 2–3/6 for worker models (Fig. 2, Fig. 4).

W4. As we can not see the references [3] but we guess you mentioned about the work "Majority-of-Three is Optimal." The two claims are compatible because they are about different things:

  • [3]: PAC learning. A majority of three independent consistent classifiers is an optimal learner for a single, realizable target.
  • Ours: Three is the annotation depth per object, and the target is a set of valid labels, some endorsed only by a minority. Prop. 1 shows that in this setting three votes generally cannot certify the full set.

We will cite [3] in the introduction and make this distinction explicit.

W5 — Runtime. One grouped power-law table (N=250N=250, and 6,039 pairs, mean depth 106). Median of 3 runs on the same machine.

MethodTime (s)
Majority / single-label majority<0.001<0.001
Obj-DS / Obj-GLAD0.0140.014 / 0.0280.028
Worker-level DS+GLAD0.100.10
Obj-MACE0.710.71
Multi-valid powerset DS+GLAD (∣Z∣≤3\lvert Z\rvert \leq 3)1.021.02
Spectral DS1.131.13
Ours (BH/WBH, 5-fold screen + fit + test)3.093.09
  • All methods run in seconds. Ours takes 0.6 s on ConvAbuse (6,240 pairs).
  • Our cost is 5 small GLM fits, linear in the number of pairs, with no enumeration of label subsets.
  • The multi-valid DS/GLAD baselines grow as ∑s≤S(Kis)\sum_{s \leq S} \binom{K_i}{s} states per object and must be truncated when KiK_i is large.
  • The gold-free fallback with nested CV costs about 19 s on ConvAbuse.

W6 — Why ConvAbuse deviates. ConvAbuse is the only dataset where null pp-values deviate from super-uniformity, which is expected for two reasons. First, with only 2–7 annotators per instance, its label counts cannot reliably approximate a Poisson distribution, violating the annotation-depth requirements for our consistency theorem (Thm. 5). Second, because its gold labels derive solely from majority vote (App. D), the "null" set includes minority-endorsed labels that naturally yield small pp-values. This underscores a broader scarcity in the field: real datasets offering valid multi-labels, ground truth, and sufficient annotation depth per sample remain exceptionally rare.

AuthorsOpenReview ↗

Reply to Reviewer 8bif (part II)

Minor points — all fixed in the revision.

  • M1: \cite → \citep; we will check all citation commands.
  • M2:
    • Lines 46–48 (majority and latent-class aggregation): Dawid & Skene 1979; Whitehill et al. 2009; Raykar et al. 2010; Zhang et al. 2016; Sheng & Zhang 2019 (survey).
    • Lines 51–53 (multi-label crowd methods using factorization, structural restriction or a bound on the number of valid labels): Duan et al. 2014; Zhang & Wu 2018; Yin et al. 2021; Xia et al. 2024; Wang et al. 2025.
  • M3: "Recov7ry" → "Recovery".
  • M4: Fig. 1 widened to full width with larger fonts, and the depth panels moved to a second row.
  • M5: Thank you for pointing this out. We agree that the dataset descriptions were too brief. We have expanded Appendix D with a dedicated table and descriptions covering each dataset’s domain, object and label definitions, annotation protocol, gold-label construction, and relevant limitations.

We appreciate these concrete requests; they improve the comparison and make the scope of the conclusions much clearer. We hope the clarified contribution and targeted revisions will support a higher assessment.

Thanks for the detailed response . It covers all the points I raised in my review. I am increasing my score.

Thank you for your thoughtful feedback. We're glad our rebuttal addressed all your concerns, and we appreciate your decision to increase the score.

Reviewer_BdjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

When annotators disagree on the same object, the paper argues that this does not necessarily mean that some annotations are incorrect, as a single object may genuinely have multiple valid labels. Therefore, it reformulates multi-annotator label recovery as a problem of multiple hypothesis testing and statistical selection.

Critical strengths

Reformulating multi-annotator label recovery as a problem of multiple hypothesis testing and statistical selection is insightful, and the method has a clear statistical foundation.

Critical weaknesses

The theoretical FDR control in this paper relies on statistical assumptions, including null hypothesis calibration, and the real-data experiments do not sufficiently demonstrate that these theoretical guarantees hold reliably in practice. Specifically, the paper finds that the ConvAbuse dataset exhibits anti-conservative null p-values among the six real-world datasets, indicating that the method does not satisfy the key calibration requirements across all datasets.

Questions for authors

See Critical Weaknesses.

4 comments
AuthorsOpenReview ↗

## Reply to Reviewer BdjK

We thank Reviewer BdjK for the thoughtful and constructive review, particularly for recognizing our reformulation as “insightful” with “a clear statistical foundation.” We appreciate your concern regarding practical FDR control and ConvAbuse calibration, which we address below by clarifying our theoretical guarantees and empirical findings. We hope these explanations resolve your concerns.

1. The framework is designed to need fewer assumptions, and can adapt. We move label recovery from point estimation to a decision-theoretic problem. Rather than an oversimplified annotation model (signal plus noise) that breaks under group-dependent annotator behavior, we only need a model of workers’ error behavior, not their full reliability. The null is not restricted to Poisson: any distribution that better describes the data can be used, including a worker-level null when enough individual annotations are available. We present Poisson as the simplest and most natural instantiation; a dataset like ConvAbuse is exactly where a richer null should be used.

2. Why ConvAbuse deviates, and why we expected it. Across all synthetic and real datasets, ConvAbuse is the only one whose null p-values are not super-uniform. Each ConvAbuse object has only 2–7 annotators, so summed label counts can hardly follow Poisson behavior. Also ConvAbuse is the only dataset whose gold labels are the majority vote (App. D). So its “null” pairs include labels a minority endorsed, which naturally get small p-values. The Poisson approximation, and the identifiability condition of our consistency result (Thm. 5), require enough annotations per object, which ConvAbuse does not provide. This also reflects a broader challenge for the field: real datasets that combine multiple valid labels, ground truth and enough annotators per object are rare.

3. The FDR target was met on every real table, including ConvAbuse. The anti-conservative left tail in Fig. 2 did not translate into an FDR violation: on ConvAbuse, BH’s realized FDP is 0.012 at q = 0.2 (App. E), and BH and weighted BH stay at or below q on all six real tables (Fig. 4). Fig. 2 is a stricter, pair-level calibration check; what it flags on ConvAbuse is a calibration warning, not a failure of error control.

We hope this direct separation of a theoretical guarantee, a diagnostic failure, and descriptive empirical performance addresses the soundness concern and warrants reconsideration.

Dear Reviewer BdjK,

Thank you for your review. We have submitted our rebuttal and would greatly appreciate any feedback you may have.

As the discussion period ends tomorrow, we would be grateful if you could share your thoughts before the deadline. Please let us know if you have any remaining questions or concerns.

Thank you again for your time and consideration.

Thank you to the authors for their detailed responses and clarifications. I maintain my original rating.

Thank you for engaging with our rebuttal. Since you feel the concerns are resolved, is there a remaining issue keeping your score unchanged? If any part still reads as unclear, a pointer to it would help us improve the paper. Thank you for your time.

Reviewer_K6qZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies valid-label selection from sparse multi-label crowd annotations using a multiple-testing framework. The authors estimate null support distributions through lower-tail screening, Poisson thinning, truncated likelihood estimation, and cross-fitting, then apply BH or weighted BH to control the false discovery rate (FDR). The method is evaluated on synthetic and real-world annotation datasets.

Critical strengths
  1. The paper considers a meaningful problem where multiple labels may be valid for the same object. FDR-controlled selection offers an alternative to conventional label aggregation.

  2. The proposed framework does not require annotator identities or explicit reliability models, which may be useful when only aggregated annotations are available.

Critical weaknesses

1. Limited methodological novelty and unclear contribution to crowdsourcing.

The proposed method operates entirely on aggregated label-support counts, making the problem closer to noisy multi-label selection than conventional crowdsourcing inference. Most methodological components, including Poisson thinning, truncated likelihood estimation, cross-fitting, and weighted BH, are based on established statistical techniques. While their combination may be useful, I am not convinced that the paper sufficiently demonstrates what new challenges arise from the crowdsourcing setting and how the proposed framework specifically addresses them.

2. Restrictive assumptions on the annotation process.

Although the paper motivates its approach by discussing the restrictive assumptions of traditional crowd models, its own theoretical results rely on Poisson null distributions and several independence assumptions. These assumptions may not hold in realistic crowdsourcing settings, where annotation errors can be heterogeneous, instance-dependent, and correlated across annotators or labels. While Eq. (2) allows the null intensity to depend on object–label covariates, it does not necessarily capture annotator-specific instance-dependent noise or dependencies across labels. The authors should better justify these assumptions and clarify which types of crowdsourcing noise are covered by their framework.

Questions for authors
  1. Methodological contribution. Which parts of the proposed theory or algorithm specifically address challenges unique to crowdsourcing, beyond applying existing multiple-testing techniques to aggregated noisy multi-label data?

  2. Modeling assumptions. How are the Poisson and independence assumptions justified under realistic crowdsourcing mechanisms involving annotator-specific, instance-dependent, or correlated errors? How sensitive is the method to violations of these assumptions?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer K6qZ

Thank you for the precise questions. We explain what is new and why it improves on prior works.

W1. The building blocks are established; Our novelty lies in the problem formulation and the way these tools are combined to solve it. Standard crowd models do parameter estimation: fit an annotator model, then report a MAP or majority label per object. We treat this problem as a joint estimation-and-selection decision problem: a label is reported only when evidence supports it, with errors controlled over reported decisions. Four features of real annotator behavior, assumed away by existing models, drive this:

  1. Annotators are neither individual nor independent. They cluster by social/demographic group with correlated errors, so per-annotator confusion matrices with conditional independence are misspecified. Using support counts deliberately avoids this; it is not a simplification. Although our model is rich enough to consider annotator-levels models.
  2. Several labels can be valid, and how many is unknown. Most models force one label or fix the number in advance; we need neither.
  3. Evidence strength varies across objects. Two-step pipelines give one global error rate, yet dense objects deserve more confidence than sparse ones. Our inference depends object's support.
  4. The method can abstain. Existing methods always output a label or use ad hoc cut-offs (e.g. "75% agreement"); ours abstains via the error-controlled rule itself.

Each component answers one challenge: Poisson (or any arbitrry distribution) thinning models how much evidence each object has (3); the truncated likelihood handles the unknown label count (2); cross-fitting keeps selection valid although the same data fit the model; weighted BH turns per-object evidence into principled abstention (4). We will make this explicit. Inference is designed around how annotators actually behave, not an idealized model.

W2. Real errors are indeed heterogeneous, instance-dependent and correlated. We aim for fewer and weaker assumptions than prior models, not none:

  • Not tied to Poisson. Only a null model yielding valid p-values is needed. Poisson is the natural default for aggregated counts, with covariate-dependent intensity via Eq. (2).
  • Extensible. Annotator-object, overdispersed or multivariate nulls plug in when data allow; cross-fitting and weighted BH are unchanged.
  • Weaker than Dawid–Skene, which needs identical worker behavior across objects, conditionally independent workers and one true label. Group-level heterogeneity is absorbed into our aggregate intensity.

We will distinguish covered noise from noise requiring a richer null.

Q1 . Thinning, truncated likelihoods and (weighted) BH are classical; we contribute a decision-theoretic paradigm replacing forced label recovery:

  1. Joint estimation and selection: report only supported labels, else abstain; prior works have only data-free thresholds.
  2. No trusted nulls needed: conformal, learn-then-test and PAC-labeling need labeled reference data. In crowds validity is never observed, so we build null evidence from the mixed table itself, under minimal assumptions.
  3. Few votes, plural truth: with 3–10 votes per object, Prop. 1 shows any estimator, even oracle MAP, needs depth ~(wΔ)⁻² to recover minority-endorsed labels, so forced estimation fails.
  4. Finite-sample FDR control over reported labels without labeled data, which no existing crowd label-recovery method offers.
  5. Generality: annotator-object noise models when per-annotator data suffice.
  6. A bridge to machine annotation: LLMs can support human annotation without pooling with human votes.

Q2. The method does not depend on Poisson; it is the most intuitive model of human error, and the null can be chosen from dataset metadata. Poisson is justified by this result (to be added to the appendix):

New Prop. Under H0, let annotators select label k independently with arbitrary annotator-specific, instance-dependent probabilities p_ijk, and let λ_ik = Σ_j p_ijk. Then:

  • (a) P(S_ik ≥ s) ≤ P(Pois(λ_ik) ≥ s) for every s ≥ λ_ik+1 ; the Poisson p-value at the true mean is super-uniform in its far tail.
  • (b) For every t, |P(p̂≤t) − t| ≤ d_TV ≤ max_j p_ijk.

Annotator-specific, instance-dependent noise acts only through λ_ik (estimated by Eq. 2), so aggregate counts suffice without an annotator model. Only dependence between annotators breaks the Poisson law.

Real data. Fig. 2 shows super-uniform null p-values, as predicted, on all datasets but ConvAbuse, so the model captures their nuances. ConvAbuse deviates as expected: only 2–7 expert annotations per object drive the left-tail departure.

We hope these clarifications help convey the crowd-specific statistical challenges and the scope of our current guarantee, and would be grateful if the reviewers could reconsider their soundness and overall assessments.

Note: To save space, we have omitted LaTeX formatting.

Thank you for the detailed response. You've addressed all the concerns I raised in my review, so I've decided to increase my score.

Thanks for your careful feedback. We’re pleased our rebuttal resolved all your concerns, and we appreciate you raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General author comment

We thank the reviewers for their careful and constructive feedback. Reviewer BdjK described our reframing of label recovery as “insightful” and noted its “clear statistical foundation”; Reviewer 8bif called it “novel and well-motivated”; and Reviewer K6qZ highlighted the value of not requiring annotator identities or explicit reliability models.

The reviewers’ concerns focus on three issues—methodological novelty, modeling assumptions, and ConvAbuse calibration—which we address directly.

  • First, our main contribution is a new decision-theoretic paradigm for label recovery, rather than simply an application of existing multiple-testing techniques. Traditional aggregation methods typically formulate label recovery as a point-estimation problem: they specify a generative signal-and-noise model, estimate its parameters, and recover labels through majority voting or MAP inference. Such approaches often rely on assumptions about annotator reliability and conditional independence, while the final recovered labels are determined by the fitted model without explicitly controlling the proportion of erroneous selections. Crucially, fitting a model and decoding its predictions does not itself guarantee that the reported labels are sufficiently supported by the annotation evidence. Our framework fundamentally changes this formulation by integrating statistical evidence, error control, and label selection into a single decision-making procedure. Each object–label pair becomes a hypothesis, and a label is selected only when the evidence is sufficiently strong relative to a calibrated null distribution; otherwise, the procedure abstains. The methodological contribution lies in making this possible without gold-labeled null examples, predefined label-set sizes, or explicit annotator-reliability models, through a carefully constructed combination of lower-tail screening, selection-adjusted null estimation, cross-fitting, and FDR-controlled selection.

  • Second, our assumptions are more targeted than those required to specify a complete annotator-level generative model. Rather than modeling every aspect of annotator behavior, we model the support distribution of invalid labels, allowing systematic instance- and label-dependent heterogeneity through the null intensity. Our guarantees require appropriate null calibration and dependence conditions, which we explicitly distinguish from unrestricted correlated errors.

  • Third, ConvAbuse represents a foreseeable boundary case, not a general failure of the proposed framework. Its exceptionally sparse annotations (3.3 per object on average) and majority-derived gold labels make null estimation and calibration assessment particularly challenging. Although its null p-value diagnostic exhibits anti-conservatism, the observed gold-relative FDP is only 0.012 at (q=0.20), substantially below the nominal threshold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.