- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 1
Read review
Paper summary
Authors modify SupCon's loss to be label noise-aware by weighting per-anchor loss terms where the weights are derived from similarities (inner product) of softmaxed predictions of a model trained on the noisy data in a normal way. Experiments with synthetic and real world datasets consistently show improvement over other methods even if it's mostly around +0.5% in absolute terms.
Critical strengths
-
Proposition is easy to understand and implement: a loss weighting modification per anchor where weights are based on a trusted network's softmaxed outputs' similarities.
- why it is consequential to your overall assessment: it is a new*, interesting way to address the problem (*: analogues may be found in combating label noise in supervised normal learning, e.g., Patrini et al.'s CVPR 2017 paper "Making Deep Neural Networks Robust to Label Noise" or the like but its adaptation to its contrastive part is nontrivial and authors make the distinction between samplewise and pairwise very clear.)
-
Experiments cover a good range: synthetic and real-world datasets, and quite a few other recent methods from the literature.
- why it is consequential to your overall assessment: it's good to show emprical support for the proposition.
Critical weaknesses
-
Section 4.1, i.e., the derivation of is arguably the most important part of the paper, yet it is not justified sufficiently in my opinion (the trust it is given might be too risky). What if overfits the noise? What if it underfits the clean samples? Training on the noisy data in a normal way makes it fallible for these questions. Yes, the emprical support suggests it's fine but what effect does it have on those results? Line 803 mentions 15 epochs: how did the authors choose this, for example? Similarly, 's design is explained for its properties but it is not clear why one should not do it any other feasible way but have instead of , for example: any specific constructive reason? Furthermore, in either case, if I need to first disjointly train a model on the noisy data in a traditional way (i.e., no contrasts), why should I not use label noise-tolerant techniques while I'm at it and be done with it? For example, one can integrate Zhang et al.'s ICLR 2021 paper "Learning with Feature-Dependent Label Noise: A Progressive Approach" into that first network (and their algorithm is amenable for that) and they report 74.02% for Clothing1M and 85.28% for Food-101N highly comparable with your results but without 2-stage training or any augmentation or contrastive learning. Note that I'm not arguing in a general sense why one should prefer traditional learning versus contrastive; my argument is due to your pipeline having 2 stages already (actually 3 stages if we count the linear classifier at the end) and why one should not take a shortcut at the first stage which does no contrastive learning.
- why it is consequential to your overall assessment: the disjoint nature with a normal way of training a model on the noisy data seems risky and unjustified in that one may desert contrastive learning at that point
-
While the experiments cover a wide range, all of them are about image classification. Yes, the vast majority of the literature only ever experiments with images and they are important but it does not mean they should be the only ones. Even a synthetic experiment on, say, audio signals or tabular data or ... could help see the applicability of the method. After all, there is nothing visual-specific in the proposition. Note that one does not have to do augmentation in supervised contrastive learning: one has label information now; augmentation is mainly a leftover from its self-supervised counterpart where it is the main ingredient.
- why it is consequential to your overall assessment: label noise concerns datasets other than of image nature and even 1 synthetic experiment on that front could help but is missing.
Questions for authors
-
Why do the authors keep the augmentation (same question could be asked to the SupCon paper): one already has label information (noisy or not does not matter as you are dealing with that and SupCon assumes the noise away)? It is needed and the main driving point in, say, SimCLR and the like as they don't have labels and self-supervise but I don't see the need here. Would also increase applicability domain of your method to not-so-easily-augmentable types of data, e.g., tabular, time series.
-
Line 294: by "unit weight", you mean unit weight before normalization by , right? Or you manually force them to be 1 for augmented siblings?
-
What is a typical range of values for in your experience in the experiments (with )?
-
The generalization accuracy improvements in Table 1 seem to be mostly around +0.5%; authors say in Line 365 these are consistent improvements and in fact they are, but do the authors think +0.5% is significant of an improvement? (I'm not implying it isn't but rather seeking an explanation more than stating the improvement itself from the authors.)
-
On the flip side, there does not seem to be a mention of Limitations: are there any particular dataset or some other configurations where you see the proposed model not prevail?
-
Line 489 mentions "proof drafting" but there is no proof in the paper (Appendix included), which is fine of course but why mention that then?
-
Figure 2's caption reads "converts endpoint posteriors into pairwise semantic support": what do the words "endpoint" and "semantic" mean here? Isn't this saying "derives weights of the loss using predictions of a noisily-trained model"? I think there is use of loaded words for relatively simple phenomena throughout the text that could be revisited.