acceptodds
Under review as a conference paper at ICLR 2027

On the Calibration of Crowdsourcing Aggregators

Abstract

Calibration has been extensively studied for classifiers, but remains largely unexplored for crowdsourcing aggregation methods, which are becoming increasingly popular for making probabilistic decisions. To the best of our knowledge, we provide the first theoretical and empirical study of calibration in label aggregation. To our great surprise, we find that widely used aggregation methods can be severely miscalibrated. We prove that the Bayesian aggregator is perfectly calibrated under its data-generating model. We also show that exact calibration of weighted majority votes requires highly restrictive conditions, which helps explain why Majority Vote is generally miscalibrated in realistic crowds. Empirically, we study calibration across a large collection of human crowdsourcing datasets and LLM-based crowds. We observe, in both cases, systematic miscalibration for aggregator models such as Majority Vote, Dawid–Skene, GLAD, and CrowdFM. Finally, we study post-hoc calibration in crowdsourcing. We propose an aggregation of aggregators approach: for any aggregation function , we introduce a transformation that first combines its predictive distribution with those of complementary (under- and overconfident) aggregators, before applying Temperature Scaling. This simple and interpretable transformation makes Temperature Scaling more effective than when applied directly to , improves calibration, especially in the high-redundancy regime, where there are many annotations per item, while preserving or even improving accuracy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Calibration, Crowdsourcing, Statistics, Classification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kCVrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the first study of calibration in crowdsourcing label aggregation. The authors show that standard aggregators are systematically miscalibrated in opposite directions: Majority Vote is underconfident while model-based methods (Dawid-Skene, GLAD, CrowdFM) are overconfident. Theorem 1 proves that a weighted majority vote can only be asymptotically calibrated if the weighted crowd accuracy converges to perfection — an extremely restrictive condition that explains the observed miscalibration. As a practical fix, Algorithm 1 proposes "Soup+TS": averaging complementary aggregators (MV + DS + target) before applying temperature scaling. Experiments on 39 human datasets and 2 LLM-crowd datasets show Soup+TS improves calibration.

Critical strengths
  • S1: You are working on an interesting problem that was not studied before.
  • S2: You are testing your approach on lots of datasets with different characteristics.
  • S3: The experimental setup is clearly described.
Critical weaknesses
  • W1: Code is not available, impacting the reproducibility of the proposed approach.
  • W2: Main concern is regarding Theorem 1, your main contribution. You say that calibration requires the crowd to be asymptotically perfect. This is a strong statement, but unsurprising. An imperfect classifier cannot produce perfectly calibrated probabilities.
  • W3: For most of the 39 datasets, the "ground truth" is itself estimated via majority vote or expert consensus. Evaluating MV's calibration against MV-derived gold labels introduces systematic bias.
  • W4 :The Related Work Section is half a page. Since you have some space, consider adding some related works.
  • W5: ECE is known to be sensitive to binning choices (number of bins, equal-width vs. equal-mass). The paper never specifies the binning strategy.
  • W6: Why uniform 1/31/3 weights? Why specifically MV + DS? No ablation over mixing weights, no comparison with a learned convex combination, no sensitivity to the choice of reference set. A simple grid search or cross-validated weighting would be an improvement to test.
Questions for authors
  • M1: Lines 37-38 Consider adding a citation to support your statement. Same for Lines 46-47.
  • M2: You start talking about calibration but you are never explaining what it is. While it is true that most of the people know it, not all of them are aware of the meaning of this word. Also a small sentence is enough.
  • M3: Some typos can be found in the paper: "Hoefdding inequalitie", "topk class sens”.
  • M4: Figure 3 is very small and not easy to be read.

Consider also answering the Weaknesses.

2 comments

We thank the reviewer for their constructive comments and for recognizing the importance of the problem and the diversity of our benchmarks.

For W2, We say that an imperfect classifier can actually be perfectly calibrated. A simple example is a classifier that always predicts the class proportions. It is perfectly calibrated but can have poor accuracy.

Our result characterizes when weighted majority votes are asymptotically calibrated under the assumptions of G\mathcal G. It does not establish a general relationship between perfect calibration and perfect accuracy.

W3 We agree that evaluating Majority Vote against labels obtained through Majority Vote can introduce bias. We will better document the origin of the ground-truth labels and report separate results excluding datasets with MV-derived reference labels.

W4 We will extend this section, particularly regarding calibration and human disagreement.

W5 We will clarify our experimental protocol and complement ECE with the Brier score, which does not depend on binning.

for W6: We chose MV and DS because they empirically exhibit opposite calibration behaviors. We used uniform weights for simplicity and to avoid introducing additional parameters. We agree that learning a convex combination and studying different reference aggregators are natural extensions. We will investigate the sensitivity of our method to these choices.

Thanks for the response. I am keeping my score unchanged.

Reviewer_zb2cOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the problem of the calibration of a classifier by crowdsourcing. Given the labels provided by some finite set of annotators, the problem is to compute a probability distribution on the labels. The authors prove that the Bayesian vote is calibrated (Proposition 1) and that any calibration based on weighted Majority vote requires a perfect crowd (in the limit of a large number of annotators, Theorem 1 and Corollary 1). The experiments confirm that no aggregation method achieves perfect calibration.

Critical strengths

None

Critical weaknesses

The theoretical framework of the paper is not properly defined. In fact, the considered problem of calibration looks infeasible unless 1. the distribution is known (that's the Bayesian vote, which is impractical) or 2. the crowd is perfect so that the label distribution can be derived from the frequency of labels, by the law of large numbers. It is then not surprising that no calibration method works, without any further assumption on the annotators.

The experimental part is not well described. In particular, it is unclear which ground-truth distribution of the labels is considered.

Questions for authors

Please explain what is the ground-truth label distribution and how it can be derived in practice. If there is uncertainty on the true label, then it seems that the distribution can only be given by annotators. Can the problem be formulated as the discrepancy between two independent sets of annotators? I don't see how this is described by your theoretical model.

2 comments

We thank the reviewer for their comments. We agree that our problem setting needs clarification.

Regarding the Ground-truth labels and calibration discussion: Once again, we follow the classical crowdsourcing setting of Dawid and Skene (1979). Each item has a true label (Y), and annotators provide noisy observations of this label. We are not trying to estimate the distribution of annotators' opinions, but rather the uncertainty about the true label.

As in standard classification, we do not need to know the true posterior distribution to evaluate calibration. We only need reference labels. For instance, if an aggregator predicts the answers to multiple-choice questions with 80% confidence, we expect 80% of these predictions to be correct.

For theorem 1. I think there is a miss understanding. We do not claim that calibration is infeasible in general. Our result concerns weighted majority votes under the assumptions of G\mathcal G. Consider the classical Condorcet setting with independent (given YY) annotators having 70% accuracy. As the crowd grows, Majority Vote becomes perfectly accurate so P(Y_MV\neq Y) to 1 as R goes to \infty, while its confidence ECE converges to alpha>0 (a proof of this new statement is done in the appendix). This is precisely our point: good accuracy does not imply good calibration.

Moreover, we can improve calibration without knowing the underlying distribution by using a set of reference labels, as we do in Section 5.

The reviewer's suggestion of comparing two independent crowds is interesting, especially for subjective tasks. However, this addresses a different problem from ours. We will make this distinction clearer in the revised manuscript.

Reviewer_zb2cOpenReview ↗

Reply

Thanks for the response. Unfortunately, it does not address my concerns. In particular, I still don't see what is meant by the "uncertainty" on the true label and the "confidence" of the annotator, and how these are evaluated in practice.

Reviewer_tkLFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the calibration of class probability distributions derived from aggregation methods on crowdsourced annotations on a sample-level.

When studying the calibration of different aggregation methods, the authors find that common methods such as majority vote (underconfident) and Dawid-Skene (overconfident) are heavily miscalibrated.

To improve the lack of calibration, the authors introduce a transformation where a target aggregator’s probabilities are first recalibrated by other aggregators with similar calibration profiles, this leads to a soup aggregation. After that, temperature scaling is applied to this soup aggregation. The results indicate that this method outperforms directly applying temperature scaling.

Critical strengths
  • The paper studies an interesting topic, how to get well-calibrated probability estimates from aggregation methods is essential. I really like the idea of the transformation where the target aggregator is, in a way, pre-calibrated by aggregators with similar calibration profiles.
  • The breadth of the experiments is really nice. The authors use 40 datasets and 11 models from 3 model families.
Critical weaknesses
  • My main concern for the paper is its novelty. While the authors say that they are one of the firsts to investigate calibration with crowdsourcing aggregators, I wonder how different it is from e.g., [1]. Can the authors elaborate a bit more on the differences? They also e.g., show the miscalibration of majority vote. This paper is not mentioned in the Related Work, while (in my opinion) highly relevant, so I would definitely wonder if another round is needed there to add more similar work.
  • As we are discussing crowdsourced annotations, I was hoping to see a discussion regarding its relation to subjectivity. Our field has recognized that differences in crowd labels are not noise, rather signal. So how does this notion change calibration? What reinforces this tension in the paper is that the tasks used are also subjective. For instance, the authors use a toxicity dataset, where differences in annotations are not noise. I would have thus loved to see this interweaved in the storyline for a fuller picture.
  • As ECE is shown to be sensitive to binning, would it be also possible to have another calibration metric, e.g., the Brier score?

[1] Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop Measuring Calibration When Humans Disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Questions for authors
  • I think I have missed this but can the authors point me to the section in the paper that discusses how the soup is exactly constructed?
2 comments

First of all, we thank the reviewer for his time and involving. If you agree let first answer the weakness.

For W1: The paper from Baan et al. (2022) studies the calibration of classifiers evaluated against majority-vote labels. Our setting is different: we study the calibration of the aggregators themselves with respect to a latent ground-truth label. We also provide necessary and sufficient conditions for the asymptotic calibration of weighted majority votes and introduce a calibration method. We have added this reference and clarified our contributions.

For W2 and the subjectivity. We agree that disagreement does not always reflect annotation noise. Our framework assumes a latent ground-truth label, as in DS. From our point of view, our setting is particularly relevant when labels can be independently verified, as for instance in railway defect detection or other infrastructure failure detection applications. We now clarify this assumption and leave ground-truth uncertainty for future work. We have checked and a big part of the datasets have nonambiguous ground truth labels.

W3: Regarding the Brier score. We have added the multiclass Brier score to complement ECE. For instance, on RTE with LLM annotators, MV and DS obtain Brier scores of 0.14 and 0.18, respectively.

Then for Soup, the idea is to make a meta aggregation (a majority vote aggregation in the space of distribution). So The Soup is simply the average of the target aggregator, MV, and DS:

[ Q^{\mathrm{Soup},\phi} =\frac{Q^\phi+Q^{\mathrm{MV}}+Q^{\mathrm{DS}}}{3}. ]

Temperature scaling is then fitted on this distribution. We have clarified its construction in Algorithm 1.

Thank you for your response. My concerns however still remain so I will keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.