- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents the first study of calibration in crowdsourcing label aggregation. The authors show that standard aggregators are systematically miscalibrated in opposite directions: Majority Vote is underconfident while model-based methods (Dawid-Skene, GLAD, CrowdFM) are overconfident. Theorem 1 proves that a weighted majority vote can only be asymptotically calibrated if the weighted crowd accuracy converges to perfection — an extremely restrictive condition that explains the observed miscalibration. As a practical fix, Algorithm 1 proposes "Soup+TS": averaging complementary aggregators (MV + DS + target) before applying temperature scaling. Experiments on 39 human datasets and 2 LLM-crowd datasets show Soup+TS improves calibration.
Critical strengths
- S1: You are working on an interesting problem that was not studied before.
- S2: You are testing your approach on lots of datasets with different characteristics.
- S3: The experimental setup is clearly described.
Critical weaknesses
- W1: Code is not available, impacting the reproducibility of the proposed approach.
- W2: Main concern is regarding Theorem 1, your main contribution. You say that calibration requires the crowd to be asymptotically perfect. This is a strong statement, but unsurprising. An imperfect classifier cannot produce perfectly calibrated probabilities.
- W3: For most of the 39 datasets, the "ground truth" is itself estimated via majority vote or expert consensus. Evaluating MV's calibration against MV-derived gold labels introduces systematic bias.
- W4 :The Related Work Section is half a page. Since you have some space, consider adding some related works.
- W5: ECE is known to be sensitive to binning choices (number of bins, equal-width vs. equal-mass). The paper never specifies the binning strategy.
- W6: Why uniform weights? Why specifically MV + DS? No ablation over mixing weights, no comparison with a learned convex combination, no sensitivity to the choice of reference set. A simple grid search or cross-validated weighting would be an improvement to test.
Questions for authors
- M1: Lines 37-38 Consider adding a citation to support your statement. Same for Lines 46-47.
- M2: You start talking about calibration but you are never explaining what it is. While it is true that most of the people know it, not all of them are aware of the meaning of this word. Also a small sentence is enough.
- M3: Some typos can be found in the paper: "Hoefdding inequalitie", "topk class sens”.
- M4: Figure 3 is very small and not easy to be read.
Consider also answering the Weaknesses.
2 comments
Thanks for the response. I am keeping my score unchanged.
We thank the reviewer for their constructive comments and for recognizing the importance of the problem and the diversity of our benchmarks.
For W2, We say that an imperfect classifier can actually be perfectly calibrated. A simple example is a classifier that always predicts the class proportions. It is perfectly calibrated but can have poor accuracy.
Our result characterizes when weighted majority votes are asymptotically calibrated under the assumptions of G. It does not establish a general relationship between perfect calibration and perfect accuracy.
W3 We agree that evaluating Majority Vote against labels obtained through Majority Vote can introduce bias. We will better document the origin of the ground-truth labels and report separate results excluding datasets with MV-derived reference labels.
W4 We will extend this section, particularly regarding calibration and human disagreement.
W5 We will clarify our experimental protocol and complement ECE with the Brier score, which does not depend on binning.
for W6: We chose MV and DS because they empirically exhibit opposite calibration behaviors. We used uniform weights for simplicity and to avoid introducing additional parameters. We agree that learning a convex combination and studying different reference aggregators are natural extensions. We will investigate the sensitivity of our method to these choices.