acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Learning from Crowds

Abstract

We study the learning from multiple annotations (crowds) problem in classification settings and propose a contrastive learning approach, generalizing its (un)supervised counterparts. Based on _soft labels_, which are normalized histograms of annotations over classes, we exploit the disagreement signal in the annotations to drive the shaping of the representation space. Our key contribution is to write the common (un)supervised contrastive loss functions as a cross-entropy and generalize the target distribution to be the inverted distance over the soft labels. Blurred class information allows to perform easier hard mining without the need for large batch sizes, which we support theoretically and empirically. We provide a theoretical relationship between the risks of the contrastive stage and the downstream linear classification that also uses soft labels. Empirically, we experiment with real-world datasets of various flavors (image classification, sleep staging) to demonstrate the effectiveness of the proposed method.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multiple annotations, crowdsourcing, contrastive learning, soft labels

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6WpvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a contrastive learning method that uses soft labels, which are naturally derived from multiple labels obtained by multiple annotators. Specifically, the proposed method extends the supervised contrastive learning method (SupCon), which uses hard labels, to soft-label settings, where the similarity between embedded instances is measured based on their soft labels. The proposed method is a simple but reasonable extension. By using soft labels, the proposed method can leverage more supervised information than hard labels and works especially well when the batch size is small. Experiments on five image datasets and one time-series dataset often demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper is generally well-written and easy to follow. The proposed method is a simple but reasonable extension of the supervised contrastive learning method (SupCon) and is easy to understand. Learning from soft labels may become an increasingly important research topic in the era of generative AI, since generative AIs can generate soft labels.
  • The effectiveness of using soft labels in contrastive learning is empirically shown using image and time-series real-world datasets, although its effect is limited in some image datasets (LableMe and ImageNet-15-N). The results showing that soft labels are particularly effective with small batch sizes are convincing.
Critical weaknesses
  • I have concerns about the lack of discussion for related work and the novelty of the proposed method. Although this paper discusses soft label learning in the context of crowdsourcing or multiple annotators, there have been many studies that use soft labels for learning beyond these contexts, such as knowledge distillation and soft label (confidence)-based learning [a]. In particular, some studies already use soft labels in contrastive learning to measure the similarity between embeddings and their soft labels [b] like the proposed method. Therefore, the positioning of this work within existing studies is insufficiently discussed, and its novelty relative to prior work remains unclear.

[a] de Vries, Sjoerd, and Dirk Thierens. "Learning with confidence: training better classifiers from soft labels." Machine learning (2025). [b] Kim, Jaewon, et al. "Generalized supervised contrastive learning." arXiv preprint arXiv:2206.00384 (2022).

Questions for authors
  • Please see the comments in the Critical Weaknesses section. My main concerns are the insufficient discussion of related work and the lack of novelty of the proposed method.
  • In (a) and (b) of Figure 3, SupCon outperformed the proposed method when the batch size is not small. Would the effectiveness of the proposed method diminish as the batch size increases?
Reviewer_Qk3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Softened Disagreement-Supervised Contrastive Learning (SDSL), a contrastive learning method for classification with multiple, potentially conflicting annotations. Instead of relying on hard labels or discarding label information, SDSL uses soft labels (normalized histograms of annotator votes) to capture disagreement and uncertainty. It defines a target distribution based on the inverse (L1) distance between soft labels, allowing instances to be pulled closer or pushed apart to varying degrees when learning the representation space. The authors theoretically relate the contrastive learning risk to the risk of a downstream linear classifier and empirically demonstrate competitive or better performance than existing methods across several real-world image classification and sleep-staging datasets. SDSL also reduces dependence on large batch sizes for effective contrastive learning, potentially lowering memory and training costs, while avoiding the need for data augmentation.

Critical strengths
  • This work shows promising potential for tasks where obtaining sufficient labeled data is challenging, or where annotator disagreement reflects genuine ambiguity and multiple labels may be valid, making a single gold-standard label difficult to define.

  • It is valuable that the authors relate the SDSL contrastive objective to the risk of a downstream linear classifier. This provides theoretical motivation beyond empirical performance and strengthens the method’s potential for practical real-world applications.

  • The results are promising, as they report competitive performance against existing methods across multiple settings.

Critical weaknesses
  • Although the paper claims benefits such as reduced dependence on large batch sizes and no need for data augmentation, it is not always clear how much each design choice contributes to the observed improvements over existing methods. More targeted ablation studies or controlled comparisons could strengthen these claims.

  • Section 4.1 argues that SDSL is less sensitive to batch size than SupCon. However, the methods differ in their supervision: SDSL uses annotation-derived soft labels, while SupCon receives gold labels. The downstream classifiers are also trained using different targets. This comparison is informative about the practical value of the proposed training setup, but it does not establish that the improvement specifically comes from the contrastive objective or from easier hard mining. The advantage could arise partly from the additional distributional information retained in the soft labels, rather than from the proposed loss itself.

  • The real-world results are encouraging, but the experiments do not sufficiently isolate the contributions of the method’s key design choices, such as: how the number of annotations per instance affects performance, the degree and nature of annotator disagreement affect the effectiveness of SDSL, the choice of L1 distance, compared with alternative distance measures, influences performance.

  • The experiments cover CIFAR-10 variants and sleep staging, which provide some diversity but do not establish performance across a broad range of domains

Questions for authors
  • Why was inverse normalized L1L_1 distance selected to construct the target distribution? Have alternative distance metrics or similarity functions been evaluated?

  • How does performance change as annotation disagreement increases, and can the model distinguish genuine ambiguity from annotation errors?

  • Can SDSL account for annotators with different levels of expertise or systematic biases, rather than treating vote distributions as equally reliable?

  • Can you provide implementation details, including the hardware used for the experiments (e.g., GPU model and number of GPUs)?

Reviewer_cTMTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents softened disagreement-supervised contrastive learning (SDSL) for classification with crowdsourced annotations. It aggregates multiple annotations into a label distribution for each instance and uses a simple similarity measure between these distributions to construct soft contrastive targets, extending the target definitions used in SimCLR and SupCon. The work studies label-distribution-guided contrastive learning as a standalone mechanism, accompanied by theoretical analysis supporting the proposed objective. Experiments on two synthetic datasets and three real-world datasets demonstrate the effectiveness of this simple approach relative to existing, more complex methods for learning from crowds.

Critical strengths
  1. A simple and interpretable formulation. SDSL converts crowdsourced label distributions into soft pairwise similarities that guide contrastive learning. This provides a direct extension of familiar contrastive objectives and makes the role of annotation disagreement easy to understand. The resulting simplicity is consequential because the method achieves strong results without requiring the complexity of competing crowdsourcing approaches.

  2. A focused theoretical and empirical investigation. The paper isolates label-distribution-guided contrastive learning as a mechanism for dedicated study, providing theoretical analysis and experiments on two synthetic and three real-world datasets. Although the underlying idea appears within CrowdCons, this focused investigation helps establish its effectiveness as a standalone approach. The reported performance against more complex methods supports the practical value of the contribution.

  3. Clear motivation and positioning. The introduction and related work provide an accessible account of the motivation for learning from multiple annotations and the development of existing approaches. This presentation helps readers understand the problem and locate the paper within the broader literature, making its methodological choices and intended contribution easier to assess.

Critical weaknesses
  1. Imprecise characterization of the closest prior work. The description of CrowdCons does not adequately acknowledge its joint use of annotations through label aggregation and similarity-based supervision. It also characterizes the annotator-specific losses as uniformly weighted, whereas Equation (9) weights their averages by the number of positive samples. These descriptions understate the conceptual overlap between the methods and weaken the paper’s justification of its contribution relative to its closest predecessor.

  2. The contribution and terminology could be positioned more precisely. I value the paper’s focused investigation of this simple mechanism, particularly its theoretical analysis and empirical validation. Since label-distribution-based contrastive supervision already appears within CrowdCons, the contribution would be clearer if framed around the direct formulation and deeper understanding provided here. The name “softened disagreement-supervised contrastive learning” also invokes a broader notion of disagreement than the method specifically models. Describing the approach as label-distribution-guided supervised contrastive learning would more precisely reflect its mechanism and clarify its relationship to prior work, from my perspective.

  3. Some mathematical concerns. Provided in Questions For Authors as follows.

Questions for authors
  1. Proposition 1. Could you clarify the key equalities in its proof, particularly Equations (20)–(21) and (28)–(29)? How is kbk_b defined when the corresponding soft label is not one-hot, and why does conditioning on that soft label yield the class prototype Wg[kb,:]W_g[k_b,:]? Likewise, what justifies converting the expectation weighted by the normalized pairwise similarities αx[b]\alpha_x[b] into the soft-label-weighted classification expression?

  2. Gradient interpretation and hard mining. Equation (9) can be rewritten as

∂Li∂zi=1τ∑j≠i(pij−αij)zj.\frac{\partial L_i}{\partial z_i} = \frac{1}{\tau}\sum_{j\ne i}(p_{ij}-\alpha_{ij})z_j.

Thus, SupCon negatives still contribute through pijp_{ij}, even when their target weights are zero. Could you reconcile this with the statement that no direct gradient information is obtained from different-class instances, and clarify what the analysis establishes about SDSL’s reduced batch-size dependence?

  1. Proposition 2. Is the variance bound in Equation (13) intended to hold conditional on the sampled inputs? As written, the variance includes randomness in XX. For example, consider equally likely inputs with binary posteriors (1,0)(1,0) and (1/2,1/2)(1/2,1/2), with annotations sampled independently conditional on each input. For fixed NN, the estimator’s variance approaches 1/(16N)1/(16N) as annotations per instance increase, whereas the stated bound approaches zero. Should the proposition include an input-sampling variance term or specify a conditional formulation?

Comments on the paper

1 comment
Reviewer_cTMTOpenReview ↗

Authors' responses have addressed my concerns

Thank you for the clarifications. Your explanation of kbk_b as the latent class of the instance clears up my notation question about Proposition 1. In Q2, I used pijp_{ij} for the softmax weight you denote by wikw_{ik} in Eq. (9), with jj indexing the comparison sample. I now understand what you meant by a “direct” contribution through the target weight.

To clarify where 1/(16N)1/(16N) came from in Q3, I chose the binary case (K=2K=2) and assumed that each input independently has posterior (1,0)(1,0) or (1/2,1/2)(1/2,1/2) with equal probability. Fix NN and let the number of annotations per input grow. The estimated error for an input then converges to 00 for the first type and 1/21/2 for the second. If MM of the NN sampled inputs are of the second type, the overall estimate converges to M/(2N)M/(2N). Since M∼Binomial(N,1/2)M \sim \mathrm{Binomial}(N,1/2), we have Var(M)=N/4\mathrm{Var}(M)=N/4, giving Var(M/(2N))=1/(16N)\mathrm{Var}(M/(2N))=1/(16N). This was the variance I had in mind across fresh draws of the NN inputs.

I appreciate your explanation of the two sampling steps. Your responses have addressed my concerns, and I have no further questions. I will raise my soundness score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.