- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper reframes multi-label crowd annotation as a multiple-testing problem with FDR control. Rather than forcing a single label per object (Dawid-Skene, majority vote), the method selects a variable-size label set (or abstains when evidence is weak) while bounding the fraction of invalid selections at a user-specified level . A three-step pipeline screens null-enriched pairs via Poisson thinning, fits a truncated Poisson model to debias null intensities, and applies weighted Benjamini-Hochberg on cross-fitted p-values. Experiments on synthetic and six real crowdsourced datasets show BH/WBH hold FDR at target while standard baselines exceed it.
Critical strengths
- S1: Code is available, facilitating the reproducibility of the proposed approach.
- S2: The experimental setup is extremely clear.
- S3: Reframing label aggregation from estimation to statistically controlled selection is novel and well-motivated.
Critical weaknesses
- W1: Some important related works are missing [1,2]. They are studing when majority vote (and its variants) are enough to achieve results that are comparable to an oracle MAP.
- W2: While lots of baselines are presented, in Section 2 you discuss about perspectivist methods but you are not adding them to the set of baselines.
- W3: While the experimental setup is clearly described, the Results section can be improved. It is not clear how each of your results is solving one of the research questions of your paper.
- W4: I think you should discuss [3], which in the title is stating something that is the negation of what you are stating.
- W5: Can you priovide a runtime analysis to understand how all the methods work in terms of time? Also on a single dataset is enough.
- W6: Figure 2 shows particular p-values on ConvAbuse (the FDR guarantee breaks). The paper flags this but does not provide a diagnostic or at least a tentative explanation.
Questions for authors
- M1: The first citation should be a
citepand not acite. - M2: Lines 46 to 48. I think that a citation at that point is necessary.Same for lines 51 to 53. You refer to some methods but you are not citing them.
- M3: Line 141 "RECOV7RY"
- M4: Figure 1 is very small and not easy to be read.
- M5: While in general the experimental setup is well presented (as I wrote in S2), I think that the datasets could be described better. You are introducing them but never explaining them.
Additionally, consider as questions what is written in the Critical Weaknesses form.
4 comments
Reply to Reviewer 8bif (part II)
Minor points — all fixed in the revision.
- M1:
\cite→\citep; we will check all citation commands. - M2:
- Lines 46–48 (majority and latent-class aggregation): Dawid & Skene 1979; Whitehill et al. 2009; Raykar et al. 2010; Zhang et al. 2016; Sheng & Zhang 2019 (survey).
- Lines 51–53 (multi-label crowd methods using factorization, structural restriction or a bound on the number of valid labels): Duan et al. 2014; Zhang & Wu 2018; Yin et al. 2021; Xia et al. 2024; Wang et al. 2025.
- M3: "Recov7ry" → "Recovery".
- M4: Fig. 1 widened to full width with larger fonts, and the depth panels moved to a second row.
- M5: Thank you for pointing this out. We agree that the dataset descriptions were too brief. We have expanded Appendix D with a dedicated table and descriptions covering each dataset’s domain, object and label definitions, annotation protocol, gold-label construction, and relevant limitations.
We appreciate these concrete requests; they improve the comparison and make the scope of the conclusions much clearer. We hope the clarified contribution and targeted revisions will support a higher assessment.
Thanks for the detailed response . It covers all the points I raised in my review. I am increasing my score.
Thank you for your thoughtful feedback. We're glad our rebuttal addressed all your concerns, and we appreciate your decision to increase the score.
## Reply to Reviewer 8bif (part I)
Thank you for the detailed review and for identifying the specific comparisons and presentation changes that would improve the paper. Below, we address each of the issues raised and outline the corresponding revisions to the manuscript.
W1. We could not locate the full references for [1] and [2] in the review. Could you please provide their bibliographic details so that we can appropriately cite and discuss them in our revised manuscript? We will cite and discuss them. Our work is complementary in two ways:
W2 — NUTMEG and VariErr perspectivist baselines not applicable to our cohort:
W3 — Results vs. research questions. Sec. 4 will open each paragraph with its question and close with a one-line answer:
W4. As we can not see the references [3] but we guess you mentioned about the work "Majority-of-Three is Optimal." The two claims are compatible because they are about different things:
We will cite [3] in the introduction and make this distinction explicit.
W5 — Runtime. One grouped power-law table (N=250, and 6,039 pairs, mean depth 106). Median of 3 runs on the same machine.
W6 — Why ConvAbuse deviates. ConvAbuse is the only dataset where null p-values deviate from super-uniformity, which is expected for two reasons. First, with only 2–7 annotators per instance, its label counts cannot reliably approximate a Poisson distribution, violating the annotation-depth requirements for our consistency theorem (Thm. 5). Second, because its gold labels derive solely from majority vote (App. D), the "null" set includes minority-endorsed labels that naturally yield small p-values. This underscores a broader scarcity in the field: real datasets offering valid multi-labels, ground truth, and sufficient annotation depth per sample remain exceptionally rare.