acceptodds
Under review as a conference paper at ICLR 2027

When Do Noisy-Label Corrections Help? A Counterfactual Audit

Abstract

Human annotation is imperfect, and the standard remedy corrects labels before training, yet the corrections it proposes mix repairs with damage that must be separated. Latent-shift noisy-prediction correction (LSNPC) repairs a prediction by moving latent variables rather than the input. Their reliability can be assessed before retraining by scoring each proposed correction. First, we develop an audit with label-free scores inspired by counterfactual properties; label-free refers to scoring and selection, independently of corrector training. We apply LSNPC to training labels and study latent paths that change the decoded label as counterfactual candidates. Second, we find that confidence, one candidate score, is anti-informative where corrections rarely succeed and informative where they usually do. Score informativeness varies with the recovery rate and encoder. In paired held-out diagnostic runs, label-free proximity retains about half of its clean-label-referenced separation over chance. Across our end-to-end study, a rank threshold over the scores reverts the corrections it scores lowest before retraining. At the operating coverage the thresholded stream falls at most 0.1 accuracy points below the original noisy stream in the reported means over corrector seeds. The pattern holds across the three downstream models. At wide coverage it outperforms Co-teaching, the in-training baseline, on the crowd-labelled text streams. Its margin over the original stream is widest on the image streams. At matched coverage, label-free selection performs similarly to random rejection, identifying score discrimination as a target for improvement. This study connects counterfactual properties to selective acceptance of label corrections.

Primary area:
interpretability and explainable AI
Keywords:
Noisy Label Correction, Latent Counterfactual, Algorithmic Recourse

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XULwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits LSNPC, a latent-shift method for proposing corrections to noisy training labels. It defines four per-correction scores inspired by counterfactual properties and studies whether they distinguish repairs from harmful edits. Across text and image datasets with human and injected noise, confidence changes from anti-informative to informative as the corrector's recovery rate changes. Label-free thresholding generally preserves the gains of unselective correction, but often performs close to matched random rejection. Experiments include clean-label diagnostics, downstream retraining with three fixed model types, Co-teaching comparisons, and seed and encoder analyses.

Critical strengths
  1. The paper is unusually careful about separating clean-label diagnostics from the label-free setting that would matter in practice. This makes the interpretation of the results more transparent and avoids presenting diagnostic evidence as if it were directly available at deployment time.
  2. The study covers multiple modalities, both real and injected label noise, and several downstream model types. This breadth, together with the additional transition-level and seed-level analyses, gives the audit a more comprehensive scope than a single-dataset evaluation.
Critical weaknesses
  1. The strongest selection evidence is diagnostic, not the deployment setting. Table 3 and Section 5.2 use clean-label-referenced scores on a WR/RW subset. In paired label-free runs, noise-robustness is near chance at 0.55 and proximity is 0.66. End-to-end, thresholding is often within half an accuracy point of matched random rejection.
  2. Evidence across runs is limited. Main retraining results average five corrector seeds and five downstream seeds, while ten-seed replication covers only two settings. Appendix A says the retained run archive does not record the corrector's trusted-set size.
Questions for authors
  1. Can the authors report paired uncertainty intervals for the end-to-end advantage over matched random rejection in each primary setting and downstream head?
  2. Can they verify the trusted-set size and full corrector configuration used in the archived runs, and report whether conclusions persist across more corrector seeds?
Reviewer_uQR1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when label correction is helpful and assesses its reliability. The authors propose label corrections using counterfactual properties. Based on Latent-shift noisy-prediction correction (LSNPC), four counterfactual properties of validity, proximity, noise-robustness, and plausibility are used to identify whether the labels after correction are reliable. Then the selected correction labels are used to retrain the classifier. The authors conduct experiments on text and image datasets to verify their method and mainly compare it with Co-teaching.

Critical strengths
  1. The question studied in this paper is interesting and meaningful. Label correction sometimes can also introduce incorrect labels, and it is worth studying.
  2. The authors conduct experiments on two modalities, i.e., text and image.
Critical weaknesses
  1. The presentation of this paper can be further improved. For the images in the paper, the font sizes are very small and hard to read. For the text in the paper, the paper's organization is bad and hard to follow. Besides, in the Introduction, some unusual terms were introduced, but the author did not explain their meanings.
  2. The experiment cannot validate the core method's effectiveness in a real-world setting. In the real world, the ground truth labels are unavailable, so the true label can not be used as a score reference. In Table F11, when the clean-label reference is removed, the performance degrades. Even the performance is close to the performance of random rejection, as shown in Table F9.
  3. The four scores used (Confidence, Noise-robustness, Proximity, Plausibility) lack justification. The authors do not explain why these are relevant to the reliability of label correction.
  4. The proposed methods mainly compare with co-teaching, but other relevant baselines are missing.
Questions for authors

Why does the performance of the label-free variant drop dramatically?

Can the proposed method be used in real-world settings?

Why do the authors mainly compare the methods with co-teaching and not include more baselines?

Reviewer_AJR8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the reliability of noisy-label corrections produced by LSNPC, which is a VAE that corrects a label through a shift in latent space. The paper adapts it to multiclass training-label correction and proposes an audit framework to evaluate individual correction candidates. The authors propose four counterfactual-inspired properties: confidence, noise-robustness, latent-space proximity, and plausibility. They show three results: (1) the confidence AUROC improves from 0.23 to 0.90 as the recovery rate increases; (2) scores that use clean labels are better than random rejection; (3) in the final training runs, selection without clean labels is approximately equal to random rejection. It is better than Co-teaching on text but not on CIFAR-10N.

Critical strengths
  1. The problem is well defined: a rule must accept or reject each correction before a training run. The RR/WR/RW/WW counts and the random-rejection control at the same coverage provide a protocol that other studies of label correction can use.
  2. The authors evaluated both real and synthetic label noise on text and image datasets with multiple downstream classifiers and random-rejection baselines. The authors also distinguished between diagnostic scores that require clean labels and label-free scores that can be used in practice.
Critical weaknesses
  1. The best results in Section 5.2 use the true label. In Table F11, proximity AUROC dropped from 0.819 to 0.664, and noise-robustness dropped from 0.953 to 0.547 without clean labels. Table F9 shows that label-free selection performs similarly to random rejection at matched coverage. It was also mentioned that 98% of text correction candidates receive the same top proximity score, making it difficult to rank them meaningfully.
  2. The novelty is limited because the method mainly applies existing reliability measures to LSNPC. The experiments compare mostly against Co-teaching and random rejection. They don't include stronger or more recent noisy-label correction methods such as Confident Learning.
Questions for authors
  1. What is the AUROC without clean labels for the four scores, WR against RW, on the edit set E?
  2. What is the confidence AUROC of Figure 2 on changed examples only? Unchanged WW examples may have high confidence, and their fraction increases when the recovery rate decreases.
  3. What is the reason for the differences between Tables C2, E5, and F9? Which tables give label-level accuracy? Which recipe was used for Table F9?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you all

We are sincerely grateful to all the reviewers. We submitted the work without much confidence that it might be of interest beyond our team. The thoughtful reviews encouraged us and also gave us many real insights that could help us improve the work substantially. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.