- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a contrastive learning method that uses soft labels, which are naturally derived from multiple labels obtained by multiple annotators. Specifically, the proposed method extends the supervised contrastive learning method (SupCon), which uses hard labels, to soft-label settings, where the similarity between embedded instances is measured based on their soft labels. The proposed method is a simple but reasonable extension. By using soft labels, the proposed method can leverage more supervised information than hard labels and works especially well when the batch size is small. Experiments on five image datasets and one time-series dataset often demonstrate the effectiveness of the proposed method.
Critical strengths
- The paper is generally well-written and easy to follow. The proposed method is a simple but reasonable extension of the supervised contrastive learning method (SupCon) and is easy to understand. Learning from soft labels may become an increasingly important research topic in the era of generative AI, since generative AIs can generate soft labels.
- The effectiveness of using soft labels in contrastive learning is empirically shown using image and time-series real-world datasets, although its effect is limited in some image datasets (LableMe and ImageNet-15-N). The results showing that soft labels are particularly effective with small batch sizes are convincing.
Critical weaknesses
- I have concerns about the lack of discussion for related work and the novelty of the proposed method. Although this paper discusses soft label learning in the context of crowdsourcing or multiple annotators, there have been many studies that use soft labels for learning beyond these contexts, such as knowledge distillation and soft label (confidence)-based learning [a]. In particular, some studies already use soft labels in contrastive learning to measure the similarity between embeddings and their soft labels [b] like the proposed method. Therefore, the positioning of this work within existing studies is insufficiently discussed, and its novelty relative to prior work remains unclear.
[a] de Vries, Sjoerd, and Dirk Thierens. "Learning with confidence: training better classifiers from soft labels." Machine learning (2025). [b] Kim, Jaewon, et al. "Generalized supervised contrastive learning." arXiv preprint arXiv:2206.00384 (2022).
Questions for authors
- Please see the comments in the Critical Weaknesses section. My main concerns are the insufficient discussion of related work and the lack of novelty of the proposed method.
- In (a) and (b) of Figure 3, SupCon outperformed the proposed method when the batch size is not small. Would the effectiveness of the proposed method diminish as the batch size increases?
Authors' responses have addressed my concerns
Thank you for the clarifications. Your explanation of kb as the latent class of the instance clears up my notation question about Proposition 1. In Q2, I used pij for the softmax weight you denote by wik in Eq. (9), with j indexing the comparison sample. I now understand what you meant by a “direct” contribution through the target weight.
To clarify where 1/(16N) came from in Q3, I chose the binary case (K=2) and assumed that each input independently has posterior (1,0) or (1/2,1/2) with equal probability. Fix N and let the number of annotations per input grow. The estimated error for an input then converges to 0 for the first type and 1/2 for the second. If M of the N sampled inputs are of the second type, the overall estimate converges to M/(2N). Since M∼Binomial(N,1/2), we have Var(M)=N/4, giving Var(M/(2N))=1/(16N). This was the variance I had in mind across fresh draws of the N inputs.
I appreciate your explanation of the two sampling steps. Your responses have addressed my concerns, and I have no further questions. I will raise my soundness score to 3.