- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes two loss functions to train neural networks, such that less (or no) emphasis is given to data points having a low probability score for their correct class label. The idea is that these samples are likely to be mislabeled and should thus simply be skipped to avoid interfering with the training. The first loss, called blurry loss, multiplies the crosse-entropy by a power of the probability in question, the second, called piecewise-zero loss, zeroes-out the loss under a fixed probability threshold. Results measuring mislabeled detection capabilities are provided through F1-scores, that seem on-par or marginally better than other methods.
Critical strengths
None
Critical weaknesses
-
The proposed losses are very similar to GCE, yet no acknowledgment nor contrast of that similarity is made. Essentially, the gradient of GCE is the cross-entropy loss multiplied by , and the proposed BL is the cross-entropy loss multiplied by that factor, hence BL's gradient will be similar but with the disadvantage of being positive for small values of , as mentioned. Further, the GCE paper recommends using the truncated GCE, which sets the loss to a constant below a given threshold. This is exactly what the proposed PZ loss does, with the constant being zero, but the derivative will be exactly the same. Therefore, there is mathematically no difference in model learning between PZ and what the GCE paper recommends. The provided results likely do not use the truncated GCE, which explains the small differences in performance. If there is any significant difference or benefit of the proposed losses (why two of them, by the way?) over GCE, that I might have missed, they should be heavily emphasized and discussed.
-
The reported performances only report mislabel detections, using the F1 score (and balanced accuracy sometimes, still for mislabel detection). The goal of the model eventually is to classify images, thus classification accuracy tables should be reported as well. Otherwise, it is impossible to know if the model is still able to learn anything useful for its main task. In the same vein, how to distinguish hard samples from bad labels, as the title of the paper suggests? Alternatively, if mislabel detection is really the topic of the paper, then probably some mislabeled images detected in existing datasets should be exhibited, to show the value of the tool.
-
The reported performances for the proposed losses seem heavily cherry-picked by selecting the best hyper parameters on the test sets of interest (no clear information on that topic is available but that is what it feels like when reading the results section), while other performances are likely obtained with the methods as is. This is not good practice and should be avoided at all cost. Also, most reported benefits are really marginal, and do not convincingly show the superiority of the proposed losses.
Questions for authors
Is there any significant advantage or difference between the proposed losses and (the truncated) GCE?
With the proposed losses, how well does the model perform in classification accuracy, compared to the other methods?
What unique hyper parameter should be chosen and used throughout all comparisons (as done for the other methods), and thus how would that impact the results?
Let me note that, unless I am proven plain wrong on multiple points, there is little chance I change my rating given the criticality of the weaknesses that I spotted. These questions are meant to help authors revise in depth their paper for submission for other (and as advice lower-tier) venues.
4 comments
#2
The reported performances only report mislabel detections, using the F1 score (and balanced accuracy sometimes, still for mislabel detection). The goal of the model eventually is to classify images, thus classification accuracy tables should be reported as well. Otherwise, it is impossible to know if the model is still able to learn anything useful for its main task. In the same vein, how to distinguish hard samples from bad labels, as the title of the paper suggests? Alternatively, if mislabel detection is really the topic of the paper, then probably some mislabeled images detected in existing datasets should be exhibited, to show the value of the tool.
We would like to clarify that the primary objective of our work is to improve label-error detection, rather than directly optimize downstream classification accuracy. Our proposed loss functions are designed to train models that are more resistant to fitting erroneous labels, thereby improving the effectiveness of existing label-error detection frameworks such as Confident Learning and AUM. The identified errors can subsequently be reviewed, corrected, or removed to produce cleaner training datasets. The benefits of reducing label errors for downstream model performance and generalization are already established in the literature, as discussed in our Introduction. Consequently, we evaluate methods primarily using label error detection metrics, including F1 score, Balanced Accuracy, Precision and Recall, since these directly measure the intended purpose of our approach. Classification accuracy, while relevant, measures a different objective and does not necessarily reflect how effectively a model can detect label errors.
#3
The reported performances for the proposed losses seem heavily cherry-picked by selecting the best hyper parameters on the test sets of interest (no clear information on that topic is available but that is what it feels like when reading the results section), while other performances are likely obtained with the methods as is. This is not good practice and should be avoided at all cost. Also, most reported benefits are really marginal, and do not convincingly show the superiority of the proposed losses.
For all competing loss functions, we use the hyperparameter settings recommended in their respective original publications, which were identified through the author's own experimental evaluations and hyperparameter optimization. Methods such as ANL-CE and ANL-FL involve multiple hyperparameters, all with dataset-specific settings, for which we adopt the recommended values from the original work. Thus, the competing methods are evaluated using the configurations previously identified as most effective on each dataset.
For our proposed loss functions, we similarly perform a hyperparameter search and report the best-performing settings for each experimental condition. These help to show what parameter values lead to the best performance in different situations. However, our improvements are not restricted to these individually optimized values. As demonstrated by the hyperparameter sensitivity analyses in Appendix A.2 (Fig. 7 and 8), our proposed losses maintain strong detection performance across a broad range of parameter values. This indicates that the observed improvements are not attributed to selecting a narrow, favorable hyperparameter configuration. Appendix Figure 6, 7 and 8 show the results of our loss functions outperforming the competing loss functions over a wide range of parameters on both our main experiment, and the large-scale dataset experiments.
We look forward to your response. Thank you.
Paper Revisions Posted
Dear Reviewer Jnfz, we have now revised our paper. Changed text is coloured green for ease of review.
As stated in the general comment to all reviewers, our most important revision was reframing PZ Loss as a general loss-zeroing mechanism that can be applied to any loss function, not just CE. This revision applied throughout the paper, but most prominently in the methods section (see paragraph on Piecewise-zero Loss starting on line 220). Additionally, the relationship of PZ to the truncated Lq loss from GCE is addressed in the background section (see paragraph starting on line 174). We hope these revisions address item #1 (discussed above in our prior comment to you).
Please let us know if you have other concerns. We look forward to your response. Thank you.
Thank you
I thank the authors for providing answers to reviewers' comment and a revised manuscript. However, I will not revise my rating.
The proposed loss mechanism is simply too similar to the truncated Lq loss from GCE, and, despite the revisions made, I do not think this can appropriately be accepted at ICLR without much more convincing evidence of the difference and/or of the superiority of the approach.
Regarding classification accuracy, the authors clarify that the goal is mislabel detection, but once this detection is done and the dataset is cleaned, it will be used for image classification. I believe my comment remains valid and unresolved. Likewise, the authors did not comment on "how to distinguish hard samples from bad labels, as the title of the paper suggests?", nor shown detected mislabeled images.
Regarding the cherry-picking of the hyperparameters on the test set, the answer of the authors do not address my comment, thus I remain with doubts about the process conducted.
Dear Reviewer Jnfz, thank you for your detailed reading and review of our paper. We are happy to address your concerns, point by point, below.
#1
While both BL and GCE share the motivation of reducing the influence of potentially mislabeled samples, we would like to clarify an important mathematical distinction between Blurry Loss and GCE. GCE is defined as LGCE=(1−pq)/q, with parameter q. Its gradient is ∂LGCE/∂p=−pq−1, equivalent to the Cross-Entropy (CE) gradient scaled by pq, while BL is defined as LBL=−pγlog(p), with gradient ∂LBL/∂p=−pγ−1(1+γlogp). Thus, BL does not simply scale CE by pγ;l it introduces an additional factor, 1+γlogp, which fundamentally changes its behavior. In particular, while GCE always maintains a negative gradient with respect to the assigned label probability, BL exhibits a positive gradient when p<e−1/γ. This is an intentional design feature rather than a disadvantage: for samples with sufficiently low confidence in their assigned labels, BL actively discourages fitting those labels, rather than merely reducing their influence. This distinction is particularly relevant for label-error detection, where preventing the model from fitting incorrectly labeled samples is central to maintaining their detectability. This mechanism is additionally verified to work as intended in Figure 3 and Figure 4, evidenced by the distribution of gradients between clean and corrupted samples.
Regarding Piecewise-zero Loss, we acknowledge its close relationship to Truncated L_q loss presented in GCE's paper, wherein both loss functions are designed to produce zero gradients below a certain threshold. However, the Truncated L_q loss is only implemented to work with GCE (a blend of CE and MSE), with no supporting evidence as to its methodology working, or indication of application to loss function other than GCE. In contrast, PZ is designed as a mechanism to be applied to any loss function, evidenced by our experiments in Figure 3 and 4, identifying that our mechanism of zero the loss below a certain threshold for a loss function is able to make those sample ignored by the model and the generalize enough to produce correct predicted probabilities for those samples. We will be clarifying our presentation of PZ to position it as a methodology that can be applied to various loss functions rather than just CE. As evidenced by our analysis of how predicted probabilities and gradients change over training with this mechanism, we can present PZ as a fundamental method applicable to any loss.