acceptodds
Under review as a conference paper at ICLR 2027

Hard Samples, Bad Labels: Robust Loss Functions That Know When to Back Off

Abstract

Incorrectly labelled training data are frustratingly ubiquitous in both benchmark and specially curated datasets. Such mislabelling clearly adversely affects the performance and generalizability of models trained through supervised learning on the associated datasets. Frameworks for detecting label errors typically require well-trained / well-generalized models; however, at the same time most frameworks rely on training these models on corrupt data, which clearly has the effect of reducing model generalizability and subsequent effectiveness in error detection — unless a training scheme robust to label errors is employed. We propose two novel loss functions, Blurry Loss and Piecewise-zero Loss, that enhance robustness to label errors by de-weighting or disregarding difficult-to-classify samples, which are likely to be erroneous. These loss functions leverage the idea that mislabelled examples typically appear as outliers to their as-labelled class, being difficult to classify, and should contribute less to the learning signal. Comprehensive experiments on a variety of both artificially corrupted and real-world datasets demonstrate that the proposed loss functions outperform state-of-the-art robust loss functions in nearly all cases, achieving superior F1 and Balanced Accuracy scores for error detection. Further analyses through ablation studies offer insights to confirm the mechanism through which these loss functions operate, and demonstrate their broad applicability to cases of both uniform and non-uniform corruption, and with different label error detection frameworks. By using these robust loss functions, machine learning practitioners can more effectively identify, prune, or correct errors in their training data. Code, including a working demonstration Jupyter Notebook, is available at https://anonymous.4open.science/r/Robust_Loss.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Label errors, label noise, loss function, robust loss function, learning with noisy labels

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JnfzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes two loss functions to train neural networks, such that less (or no) emphasis is given to data points having a low probability score for their correct class label. The idea is that these samples are likely to be mislabeled and should thus simply be skipped to avoid interfering with the training. The first loss, called blurry loss, multiplies the crosse-entropy by a power of the probability in question, the second, called piecewise-zero loss, zeroes-out the loss under a fixed probability threshold. Results measuring mislabeled detection capabilities are provided through F1-scores, that seem on-par or marginally better than other methods.

Critical strengths

None

Critical weaknesses
  1. The proposed losses are very similar to GCE, yet no acknowledgment nor contrast of that similarity is made. Essentially, the gradient of GCE is the cross-entropy loss multiplied by pγp^\gamma, and the proposed BL is the cross-entropy loss multiplied by that factor, hence BL's gradient will be similar but with the disadvantage of being positive for small values of pp, as mentioned. Further, the GCE paper recommends using the truncated GCE, which sets the loss to a constant below a given threshold. This is exactly what the proposed PZ loss does, with the constant being zero, but the derivative will be exactly the same. Therefore, there is mathematically no difference in model learning between PZ and what the GCE paper recommends. The provided results likely do not use the truncated GCE, which explains the small differences in performance. If there is any significant difference or benefit of the proposed losses (why two of them, by the way?) over GCE, that I might have missed, they should be heavily emphasized and discussed.

  2. The reported performances only report mislabel detections, using the F1 score (and balanced accuracy sometimes, still for mislabel detection). The goal of the model eventually is to classify images, thus classification accuracy tables should be reported as well. Otherwise, it is impossible to know if the model is still able to learn anything useful for its main task. In the same vein, how to distinguish hard samples from bad labels, as the title of the paper suggests? Alternatively, if mislabel detection is really the topic of the paper, then probably some mislabeled images detected in existing datasets should be exhibited, to show the value of the tool.

  3. The reported performances for the proposed losses seem heavily cherry-picked by selecting the best hyper parameters on the test sets of interest (no clear information on that topic is available but that is what it feels like when reading the results section), while other performances are likely obtained with the methods as is. This is not good practice and should be avoided at all cost. Also, most reported benefits are really marginal, and do not convincingly show the superiority of the proposed losses.

Questions for authors

Is there any significant advantage or difference between the proposed losses and (the truncated) GCE?

With the proposed losses, how well does the model perform in classification accuracy, compared to the other methods?

What unique hyper parameter should be chosen and used throughout all comparisons (as done for the other methods), and thus how would that impact the results?

Let me note that, unless I am proven plain wrong on multiple points, there is little chance I change my rating given the criticality of the weaknesses that I spotted. These questions are meant to help authors revise in depth their paper for submission for other (and as advice lower-tier) venues.

4 comments

Dear Reviewer Jnfz, thank you for your detailed reading and review of our paper. We are happy to address your concerns, point by point, below.

#1

The proposed losses are very similar to GCE, yet no acknowledgment nor contrast of that similarity is made. Essentially, the gradient of GCE is the cross-entropy loss multiplied by , and the proposed BL is the cross-entropy loss multiplied by that factor, hence BL's gradient will be similar but with the disadvantage of being positive for small values of , as mentioned. Further, the GCE paper recommends using the truncated GCE, which sets the loss to a constant below a given threshold. This is exactly what the proposed PZ loss does, with the constant being zero, but the derivative will be exactly the same. Therefore, there is mathematically no difference in model learning between PZ and what the GCE paper recommends. The provided results likely do not use the truncated GCE, which explains the small differences in performance. If there is any significant difference or benefit of the proposed losses (why two of them, by the way?) over GCE, that I might have missed, they should be heavily emphasized and discussed.

While both BL and GCE share the motivation of reducing the influence of potentially mislabeled samples, we would like to clarify an important mathematical distinction between Blurry Loss and GCE. GCE is defined as LGCE=(1−pq)/qL_{\mathrm{GCE}}=(1-p^q)/q, with parameter qq. Its gradient is ∂LGCE/∂p=−pq−1\partial L_{\mathrm{GCE}}/\partial p=-p^{q-1}, equivalent to the Cross-Entropy (CE) gradient scaled by pqp^q, while BL is defined as LBL=−pγlog⁡(p)L_{\mathrm{BL}}=-p^\gamma\log(p), with gradient ∂LBL/∂p=−pγ−1(1+γlog⁡p)\partial L_{\mathrm{BL}}/\partial p=-p^{\gamma-1}(1+\gamma\log p). Thus, BL does not simply scale CE by pγp^\gamma;l it introduces an additional factor, 1+γlog⁡p1+\gamma\log p, which fundamentally changes its behavior. In particular, while GCE always maintains a negative gradient with respect to the assigned label probability, BL exhibits a positive gradient when p<e−1/γp<e^{-1/\gamma}. This is an intentional design feature rather than a disadvantage: for samples with sufficiently low confidence in their assigned labels, BL actively discourages fitting those labels, rather than merely reducing their influence. This distinction is particularly relevant for label-error detection, where preventing the model from fitting incorrectly labeled samples is central to maintaining their detectability. This mechanism is additionally verified to work as intended in Figure 3 and Figure 4, evidenced by the distribution of gradients between clean and corrupted samples.

Regarding Piecewise-zero Loss, we acknowledge its close relationship to Truncated L_q loss presented in GCE's paper, wherein both loss functions are designed to produce zero gradients below a certain threshold. However, the Truncated L_q loss is only implemented to work with GCE (a blend of CE and MSE), with no supporting evidence as to its methodology working, or indication of application to loss function other than GCE. In contrast, PZ is designed as a mechanism to be applied to any loss function, evidenced by our experiments in Figure 3 and 4, identifying that our mechanism of zero the loss below a certain threshold for a loss function is able to make those sample ignored by the model and the generalize enough to produce correct predicted probabilities for those samples. We will be clarifying our presentation of PZ to position it as a methodology that can be applied to various loss functions rather than just CE. As evidenced by our analysis of how predicted probabilities and gradients change over training with this mechanism, we can present PZ as a fundamental method applicable to any loss.

#2

The reported performances only report mislabel detections, using the F1 score (and balanced accuracy sometimes, still for mislabel detection). The goal of the model eventually is to classify images, thus classification accuracy tables should be reported as well. Otherwise, it is impossible to know if the model is still able to learn anything useful for its main task. In the same vein, how to distinguish hard samples from bad labels, as the title of the paper suggests? Alternatively, if mislabel detection is really the topic of the paper, then probably some mislabeled images detected in existing datasets should be exhibited, to show the value of the tool.

We would like to clarify that the primary objective of our work is to improve label-error detection, rather than directly optimize downstream classification accuracy. Our proposed loss functions are designed to train models that are more resistant to fitting erroneous labels, thereby improving the effectiveness of existing label-error detection frameworks such as Confident Learning and AUM. The identified errors can subsequently be reviewed, corrected, or removed to produce cleaner training datasets. The benefits of reducing label errors for downstream model performance and generalization are already established in the literature, as discussed in our Introduction. Consequently, we evaluate methods primarily using label error detection metrics, including F1 score, Balanced Accuracy, Precision and Recall, since these directly measure the intended purpose of our approach. Classification accuracy, while relevant, measures a different objective and does not necessarily reflect how effectively a model can detect label errors.

#3

The reported performances for the proposed losses seem heavily cherry-picked by selecting the best hyper parameters on the test sets of interest (no clear information on that topic is available but that is what it feels like when reading the results section), while other performances are likely obtained with the methods as is. This is not good practice and should be avoided at all cost. Also, most reported benefits are really marginal, and do not convincingly show the superiority of the proposed losses.

For all competing loss functions, we use the hyperparameter settings recommended in their respective original publications, which were identified through the author's own experimental evaluations and hyperparameter optimization. Methods such as ANL-CE and ANL-FL involve multiple hyperparameters, all with dataset-specific settings, for which we adopt the recommended values from the original work. Thus, the competing methods are evaluated using the configurations previously identified as most effective on each dataset.

For our proposed loss functions, we similarly perform a hyperparameter search and report the best-performing settings for each experimental condition. These help to show what parameter values lead to the best performance in different situations. However, our improvements are not restricted to these individually optimized values. As demonstrated by the hyperparameter sensitivity analyses in Appendix A.2 (Fig. 7 and 8), our proposed losses maintain strong detection performance across a broad range of parameter values. This indicates that the observed improvements are not attributed to selecting a narrow, favorable hyperparameter configuration. Appendix Figure 6, 7 and 8 show the results of our loss functions outperforming the competing loss functions over a wide range of parameters on both our main experiment, and the large-scale dataset experiments.

We look forward to your response. Thank you.

AuthorsOpenReview ↗

Paper Revisions Posted

Dear Reviewer Jnfz, we have now revised our paper. Changed text is coloured green for ease of review.

As stated in the general comment to all reviewers, our most important revision was reframing PZ Loss as a general loss-zeroing mechanism that can be applied to any loss function, not just CE. This revision applied throughout the paper, but most prominently in the methods section (see paragraph on Piecewise-zero Loss starting on line 220). Additionally, the relationship of PZ to the truncated Lq loss from GCE is addressed in the background section (see paragraph starting on line 174). We hope these revisions address item #1 (discussed above in our prior comment to you).

Please let us know if you have other concerns. We look forward to your response. Thank you.

Reviewer_JnfzOpenReview ↗

Thank you

I thank the authors for providing answers to reviewers' comment and a revised manuscript. However, I will not revise my rating.

The proposed loss mechanism is simply too similar to the truncated Lq loss from GCE, and, despite the revisions made, I do not think this can appropriately be accepted at ICLR without much more convincing evidence of the difference and/or of the superiority of the approach.

Regarding classification accuracy, the authors clarify that the goal is mislabel detection, but once this detection is done and the dataset is cleaned, it will be used for image classification. I believe my comment remains valid and unresolved. Likewise, the authors did not comment on "how to distinguish hard samples from bad labels, as the title of the paper suggests?", nor shown detected mislabeled images.

Regarding the cherry-picking of the hyperparameters on the test set, the answer of the authors do not address my comment, thus I remain with doubts about the process conducted.

Reviewer_kwFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Label noise is a common phenomenon in datasets. If a loss function is robust to label noise, it greatly benefits the generalization of the model.

  • To tackle this issue, this paper proposes two novel loss functions: Blurry Loss and Piecewise-zero Loss. They improve robustness against label errors by down-weighting or ignoring hard-to-classify samples, which are likely to be mislabeled.

  • The paper provides detailed analysis (in the method section) and verifies the effectiveness of the proposed loss functions (in the experiment section).

  • The paper is well organized. The manuscript is easy to follow throughout and reader-friendly.

Critical strengths
  • The authors conduct thorough analysis of the proposed loss functions in the methodology section and validate their effectiveness via experimental studies.
  • The manuscript has a coherent structure. Its content is highly accessible across all sections and exhibits good readability.
  • The experimental suite in this work is comprehensive.
Critical weaknesses
  • The justification for selecting hyperparameters γ and c in this paper is insufficient. How were γ and c chosen for the experimental tables?
  • Do hyperparameters γ and c depend on the intrinsic properties of the datasets?
  • The values of γ and c are inconsistent across different tables. What is the rationale behind this discrepancy?
  • The two newly proposed loss functions bear strong similarities to Focal Loss and CE Loss. The authors should further elaborate on the motivation behind their design.
  • In Table 1, GCE achieves performance closest to the two newly proposed loss functions. It is therefore recommended that GCE be included as a baseline in the remaining tables.
Questions for authors

to see above weaknesses

3 comments

Dear Reviewer kwFY, thank you for your detailed reading and review of our paper. We are happy to address your concerns, point by point, below.

#1

The justification for selecting hyperparameters γ and c in this paper is insufficient. How were γ and c chosen for the experimental tables?

We performed a grid search and reported the best performing hyperparameters, as measured by F1 score, in the associated tables. See the paragraph starting on Line 295 in Section 5, Models and Hyperparameters which describes this in more detail.

#2

Do hyperparameters γ and c depend on the intrinsic properties of the datasets?

We hypothesize that the intrinsic properties of the dataset do determine which hyperparameter settings will be best suited to the data. Although no detailed study was performed, it appears that in general, more complex / difficult / large datasets do demand greater values for γ. This effect does not seem to affect the optimal value for c (cutoff). The paragraph “Loss Function Parameters” in A.2, starting on line 698 describes this in more detail.

#3

The values of γ and c are inconsistent across different tables. What is the rationale behind this discrepancy?

For each dataset, we performed a hyperparameter search and presented results from the best-performing hyperparameter selection. Best performing hyperparameters are shown in Tables 1, 2, 3, and 4. We hope that by sharing these best-performing hyperparameters, the reader will gain some insight towards which settings to use in their own future experiments. Results covering the full range of experimented hyperparameters can be found in Appendix A.2, Figure 6, 7 and 8.

#4

The two newly proposed loss functions bear strong similarities to Focal Loss and CE Loss. The authors should further elaborate on the motivation behind their design.

Indeed, our proposed loss functions, Blurry and Piecewise-zero Loss are directly based on CE Loss (as is Focal Loss, GCE, ANL-CE and ANL-FL). Blurry loss is also directly motivated by Focal Loss but has the opposite intent: rather than focusing on difficult-to-classify examples, Blurry loss de-weights difficult-to-classify examples which are likely to be mislabelled in this context. This motivation is already briefly given in the paragraph on Line 216, Blurry Loss, in Section 4. Please let us know whether further elaboration in the paper will be required. Piecewise-zero Loss is intentionally based on CE Loss, as Cross Entropy is the gold standard loss function, on which we can best demonstrate how zeroing the gradients for low predicted probabilities significantly improves the performance over traditional CE Loss.

#5

In Table 1, GCE achieves performance closest to the two newly proposed loss functions. It is therefore recommended that GCE be included as a baseline in the remaining tables.

We feel we have already included GCE as a baseline in all relevant tables. GCE is not included in Tables 2 and 5. The results of Table 2 are presented to show that the performance gap is even greater when we take into account the cleaned test sets. The goal was not to show better performance against other loss functions, but rather to show the general way in which label error detection works with our proposed loss functions. Table 5 shows the applicability of our method to other label error detection frameworks, not to demonstrate that the proposed loss functions outperform other baseline loss functions. For comparison to other loss functions, most important is the main Table 1, and experiments on large-scale datasets (Table 4), for which we included GCE and ANL.

We look forward to your response. Thank you.

Thank you very much for your responses.

  • #1: I accept your explanation that grid‑search over γ/c is performed to pick best F1 score.
  • #2: Your current answer is a hypothesis drawn from empirical observation, without systematic experimental validation. Please explicitly state in the manuscript that this is an empirical observation / hypothesis. Avoid giving readers the impression that you have rigorously verified this correlation.
  • #3: Your explanation is reasonable. Please add a short footnote or text note in the main tables to notify readers that hyper‑parameters are dataset‑specific best values from grid‑search, and full ablation results refer to the Appendix.
  • #4: It is better to expand this motivation and conceptual comparison in the main‑text method section, not only in rebuttal.
  • #5: Now, I understand your different experimental purposes for Table2 and Table5. Even if those tables are not aimed at full comparison against baselines, it will benefit readers to include at least key baseline results (GCE) for reference. If you cannot run full GCE experiments for those tables, please add an explicit note explaining why GCE baseline is absent in Table2 and Table5 in the caption or table footnote.

I will carefully re‑assess whether to change my overall score according to your responses and other reviewers' comments.

Dear authors and other reviewers,

I have read the other reviewers’ feedback and align with the opinion of Reviewer Jnfz. The proposed loss function closely resembles the truncated Lq loss from GCE. While revisions have been submitted, I remain of the view that acceptance at ICLR would require much stronger evidence establishing the novelty and/or advantages of the method.

I will retain my initial rating.

Reviewer_uYVZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of detecting mislabeled samples in supervised training data. Specifically, the authors propose two new loss functions, Blurry Loss and Piecewise-zero Loss, that reduce the influence of samples with a low predicted probability for their given label by down-weighting or ignoring them, correspondingly. These loss functions are intended for training the classifier on which label-error detection frameworks (the experimental setup includes Confident Learning and Area Under the Margin) rely, helping it avoid overfitting to mislabeled samples. The method is evaluated on different modifications of MNIST, Fashion-MNIST and CIfar with different natural and synthetic noise as well as the large-scale web-noisy Food-101N and Clothing-1M.

Critical strengths
  • The paper is well-written and easy to follow.
  • The paper addresses the important and real problem of detection of noisy samples in the supervised datasets.
  • The experimental setup is extensive, including experiments with real human annotation errors (in addition to datasets with synthetic noise) as well as larger-scale datasets. The proposed losses are tested on two prominent and widely used noise detection frameworks.
  • All experiments are conducted with several seed values and ensure reproducibility.
Critical weaknesses
  • Novelty. My concern is the novelty of the proposed loss functions. E.g., the Piecewise zero loss very closely resembles the truncated L_q loss proposed in the same paper that introduced GCE (Zhang & Sabuncu, 2018, which the authors cite and use as a baseline), as it also yields zero gradient for samples whose predicted probability for the given label falls below a threshold, and it becomes equivalent to PZ as q → 0, when L_q reduces to CE. The warm-up is also applied there before truncation. Also, the downweighting of the low-probability samples (in Blurry loss) is the mechanism Zhang & Sabuncu describe for GCE, which puts less emphasis on the samples with weak agreement with the label. These connections should be discussed, and the truncated L_q loss should be included as a baseline.
  • Hyperparameter selection. In all experiments, the proposed losses are reported with "the best performing" HPs found with grid search. However, no selection split or validation protocol is described - e.g., which metric was used for selection? Also, my concern is practical applicability: selecting HPs by detection performance requires ground-truth noise labels, which are unavailable in the very setting the method targets. The baselines use the fixed default settings, which makes the results biased towards the proposed functions, and it's difficult to tell how much of the improvement comes from the losses themselves and how much from per-setting tuning. Apart from that, grid search is not particularly scalable and has high computational cost.
  • Figure 3 seems redundant and does not verify the mechanism. The paper presents Figure 3 as evidence that the proposed losses operate as intended; however, the plotted quantity ∂L/∂p_ỹ is a fixed function of p_ỹ for each loss by construction. Given where the corrupt samples' probabilities lie, which is already shown in Figure 4, the gradient distributions in Figure 3 follow directly from the definitions in Eqs. 4–5 and, in my understanding, could not have turned out differently.
  • Figure 4. The paper uses Figure 4 to explain how the proposed losses benefit downstream detection - but it is not stated which predictions are shown. If these are the probabilities of the training samples themselves during training, they are not what CL uses, as it relies on out-of-fold predictions from models that never trained on the sample. In that case, the in-sample separation shown in Figure 4 doesn't directly explain the improvement in CL detection.
  • It would be easier to interpret the results if Precision and Recall were reported alongside the F1 score in the main text rather than in the Appendix (and also - for all baselines, not FL only). As the authors note themselves, the F1 scores might not be representative in case of unusually high Prec or Recall and not always reflective when attempting to identify label errors, so it would be valuable to have both metrics for results interpretation.
  • Consider increasing the tables font.
Questions for authors
  • Table 4 lists different HP values for the F1 and balanced-accuracy rows of the same dataset. Does this mean the HPs were selected separately for each reported metric, on the same data on which the results are reported? How should HPs be chosen in practice, when no ground-truth noise labels are available to compute detection metrics?
  • How exactly is the proposed piecewise zero loss is different from the truncated L_q loss from Zhang & Sabuncu, 2018?
  • What exactly does Figure 3 add to the narrative?
  • Are the predicted probabilities in Figure 4 computed on the training samples (in-sample) or on the held-out folds used by CL?
5 comments

Dear Reviewer uYVZ, thank you for your detailed reading and review of our paper. We are happy to address your concerns, point by point, below.

#1

Table 4 lists different HP values for the F1 and balanced-accuracy rows of the same dataset. Does this mean the HPs were selected separately for each reported metric, on the same data on which the results are reported? How should HPs be chosen in practice, when no ground-truth noise labels are available to compute detection metrics?

Yes, results from different hyperparameter settings are shown for each metric, and on the same data. The full results over all hyperparameter values can be seen in A.2 Figures 7 and 8, where the performance is improved for any hyperparameter setting across a broad range of values. In practice, hyperparameters should be chosen based on pre-existing experiments, for example, those in this paper. Use the best performing settings for the most similar dataset to your own experiments. In general though, without any understanding of a user’s data, the selection is difficult. If one has a sense of the corruption rate, then that should also be used as a guide, with greater corruption rates requiring greater parameter settings.

#2

Hyperparameter selection. In all experiments, the proposed losses are reported with "the best performing" HPs found with grid search. However, no selection split or validation protocol is described - e.g., which metric was used for selection? Also, my concern is practical applicability: selecting HPs by detection performance requires ground-truth noise labels, which are unavailable in the very setting the method targets. The baselines use the fixed default settings, which makes the results biased towards the proposed functions, and it's difficult to tell how much of the improvement comes from the losses themselves and how much from per-setting tuning. Apart from that, grid search is not particularly scalable and has high computational cost.

A grid search over the hyperparameters was performed, using F1 score as the metric by which to determine the best performance. Although grid search is not scalable and requires ground truth, an alternative is to use similar parameter values as those from existing experiments on similar datasets (e.g., those in this paper). Note that this situation is no worse than for the baseline loss functions of GCE and ANL. Additionally, we found that the loss functions are fairly robust to the choice of hyperparameters (see Figures 7 and 8 in A.2). Regarding the bias of using the best hyperparameters for our losses, this is possibly true; however, our hyperparameter selections for the baseline losses were taken directly from their respective papers, in which similar hyperparameter searches were performed. The competing methods are not evaluated using fixed default settings. Methods such as ANL-CE and ANL-FL involve multiple hyperparameters, all with dataset-specific settings, for which we adopt the recommended values for each dataset from the original work. Thus, the competing methods are evaluated using the configurations previously identified as most effective on each dataset.

#3

How exactly is the proposed piecewise zero loss is different from the truncated L_q loss from Zhang & Sabuncu, 2018?

and

My concern is the novelty of the proposed loss functions. E.g., the Piecewise zero loss very closely resembles the truncated L_q loss proposed in the same paper that introduced GCE (Zhang & Sabuncu, 2018, which the authors cite and use as a baseline), as it also yields zero gradient for samples whose predicted probability for the given label falls below a threshold, and it becomes equivalent to PZ as q → 0, when L_q reduces to CE. The warm-up is also applied there before truncation. Also, the downweighting of the low-probability samples (in Blurry loss) is the mechanism Zhang & Sabuncu describe for GCE, which puts less emphasis on the samples with weak agreement with the label. These connections should be discussed, and the truncated L_q loss should be included as a baseline.

Regarding Piecewise-zero Loss, we acknowledge its close relationship to Truncated L_q loss presented in GCE's paper, wherein both loss functions are designed to produce zero gradients below a certain threshold. However, the Truncated L_q loss is only implemented to work with GCE (a blend of CE and MSE), with no supporting evidence as to its methodology working, or indication of application to loss function other than GCE. In contrast, PZ is designed as a mechanism that can be applied to any loss function, evidenced by our experiments in Figure 3 and 4, identifying that our mechanism of zero the loss below a certain threshold for a loss function is able to make those sample ignored by the model and the generalize enough to produce correct predicted probabilities for those samples. We will be clarifying our presentation of PZ to position it as a methodology that can be applied to various loss functions rather than just CE. As evidenced by our analysis of how predicted probabilities and gradients change over training with this mechanism, we can present PZ as a fundamental method applicable to any loss.

Regarding the downweighting of GCE and thus its similarity to BL, it is true that the loss is reduced; however, the gradients with BL may be very large and positive (rather than negative with GCE and CE), resulting in large updates that steer the model away from mislabelled samples. This region of positive gradient is a unique characteristic of BL and makes it distinctly different than GCE.

#4

What exactly does Figure 3 add to the narrative?

and

Figure 3 seems redundant and does not verify the mechanism. The paper presents Figure 3 as evidence that the proposed losses operate as intended; however, the plotted quantity ∂L/∂p_ỹ is a fixed function of p_ỹ for each loss by construction. Given where the corrupt samples' probabilities lie, which is already shown in Figure 4, the gradient distributions in Figure 3 follow directly from the definitions in Eqs. 4–5 and, in my understanding, could not have turned out differently.

Figure 3 is all about demonstrating how the proposed loss functions influence training, as directed by the gradients. While it is true that the gradients can be derived directly from the probabilities in Figure 4, that fact does not diminish the insight one can gain from Figure 3.

Figure 3 does demonstrate and verify the mechanisms of Blurry and Piecewise-zero loss by directly showing the gradients, which determine in which direction the model is steered during training, both by clean and corrupt data. Corrupt data has far weaker influence on the model training with the proposed loss functions vs with CE. With BL, the sign of the gradient is opposite (steering away from fitting to corrupt data) and with PZ, the gradients are zero or near zero for corrupt data (minimal influence).

#5

Are the predicted probabilities in Figure 4 computed on the training samples (in-sample) or on the held-out folds used by CL?

and

Figure 4. The paper uses Figure 4 to explain how the proposed losses benefit downstream detection - but it is not stated which predictions are shown. If these are the probabilities of the training samples themselves during training, they are not what CL uses, as it relies on out-of-fold predictions from models that never trained on the sample. In that case, the in-sample separation shown in Figure 4 doesn't directly explain the improvement in CL detection.

Data for Figures 3 and 4 are from averaged in-fold and held-out samples, and are averaged over all folds. Therefore, you are correct that these values are not exactly what CL uses; however, assuming that the data are well split over the five folds (a reasonable assumption in general), these results are indicative of the overall training behaviour induced by the proposed loss functions. These figures are intended to demonstrate and verify the mechanism by which the loss functions operate, rather than as direct evidence of what CL uses.

#6

It would be easier to interpret the results if Precision and Recall were reported alongside the F1 score in the main text rather than in the Appendix (and also - for all baselines, not FL only). As the authors note themselves, the F1 scores might not be representative in case of unusually high Prec or Recall and not always reflective when attempting to identify label errors, so it would be valuable to have both metrics for results interpretation.

and

Consider increasing the tables font.

We would be happy to move Precision and Recall results into the main paper and increase table increase font size if space allows.

We look forward to your response. Thank you.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers

Dear reviewers, we thank all of you for your reviews of our paper. We have responded to each of you individually, addressing all of your raised concerns. We will also be revising our paper and will indicate where each change has been made.

An important question was raised by two reviewers relating to the similarity of our Piecewise-zero (PZ) loss to the Truncated Lq loss of GCE (Zhang & Sabuncu, 2018). We have separately addressed this concern in the individual responses; however, briefly, Truncated Lq loss does use a similar mechanism and was presented only on GCE (a blend of MSE and CE loss), whereas we develop PZ in the context of standard CE loss. That said, we will reframe our presentation of PZ as a general mechanism that can be applied to any loss function, not just CE. In our paper, the PZ mechanism will be shown applied to CE loss, as this is the foundational non-robust baseline, rather than applying the mechanism to already robust loss functions (e.g., CGE, BL, ANL, etc.).

We hope these responses and planned revisions address your concerns and support a more favorable assessment of our work. If any concerns remain, please let us know and we will continue to work towards resolving them during the discussion period. Thank you again for your time and feedback!

AuthorsOpenReview ↗

Paper Revisions Posted

Dear Reviewers, we have now revised our paper. Changed text is coloured green for ease of review.

Our most important revision was reframing PZ Loss as a general loss-zeroing mechanism that can be applied to any loss function, not just CE. This revision applied throughout the paper, but most prominently in the methods section (see paragraph on Piecewise-zero Loss starting on line 220). Additionally, the relationship of PZ to the truncated Lq loss from GCE is addressed in the background section (see paragraph starting on line 174).

Other revisions were minor and are indicated separately to individual reviewers.

Please let us know if any concerns remain. Thank you.

Thank you reviewers for your time and effort during this discussion period. Your feedback will help us improve our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.