- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In learning with label noise, the high-loss training examples can be mislabeled or correct but hard examples. The authors claim that Cross-entropy (CE), which depends only on the probability assigned to that label, cannot distinguish these hard confident examples. And cross-entropy does not describe how strongly the model favors another class. To address this, the authors propose a sample-weighting method that combines model confidence and the agreement with the observed label. The proposed method includes the ratio of the observed-class probability to the maximum probability, which can be used to reduce the influence of examples that have inconsistent observed labels with predictions. Experiments are conducted on CIFAR-N and Clothing1M to verify the proposed method.
Critical strengths
- The motivation of the paper is clear. Relying only on cross-entropy loss cannot hard confident examples and examples with incorrect labels. The authors consider both the model’s confidence in the observed label and its confidence in its preferred prediction.
- The proposed method is simple and straightforward.
- The authors conduct experiments on CIFAR-N and Clothing1M to verify the proposed method.
- The authors provide visualizations of easy and hard examples in the appendix, which can enable readers to understand the easy and hard examples intuitively.
Critical weaknesses
- The authors do not provide ablation studies to verify the effectiveness of key components, such as the three weightings and the ratio p_i/q_i.
- The research in this paper is directly related to the research direction for hard confident examples. But the manuscript neither sufficiently discusses this direction nor includes relevant papers that study hard confident examples.
- Experiments on CIFAR-N only include the baseline CE, and other baselines are missing.
Questions for authors
Please see the critical weaknesses.