acceptodds
Under review as a conference paper at ICLR 2027

A Committee of One: Detecting and Correcting Label Noise with Test-Time Augmentation

Abstract

Label noise can substantially degrade the stability and generalization of deep classifiers. Rather than designing a noise-robust training procedure, we address the problem at the data level by identifying and correcting corrupted labels before training the final model. We propose a data-cleaning framework based on a Siamese network and test-time augmentation. The network is trained once on the noisy dataset using a joint contrastive and classification objective and is then frozen. For each training sample, we generate views from a fixed set of twelve augmentation families and measure how often their predictions disagree with the recorded label. Our key insight is that mislabeled samples are more sensitive to small input perturbations, particularly near decision boundaries, whereas clean samples with larger margins remain more stable. We use this augmentation-induced disagreement to identify suspicious samples, relabel them to the most frequently predicted class when predictions are sufficiently consistent, and remove them otherwise. We further show that the gap in expected disagreement splits into a margin term, non-negative whenever clean samples dominate in sensitivity-normalized margin, and an excess we measure to be positive. The detector is trained once and never retrained; only threshold selection trains a downstream classifier. Across CIFAR-10, CIFAR-100, and Fashion-MNIST with synthetic instance-dependent noise, our approach achieves detection AUROC values ranging from to . On real annotation noise, a standard classifier trained on our cleaned labels reaches accuracy on CIFAR-10N, compared with for the strongest of nine single-model robust-training baselines evaluated together in a published common-codebase study, while matching its performance on CIFAR-100N. On Animal-10N, it achieves , compared with for the strongest of the baselines we compare against.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
learning with noisy labels, instance-dependent label noise, label noise detection, label correction, test-time augmentation, contrastive representation learning, memorization, data cleaning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xqhgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TTACleaner, an offline label-cleaning pipeline. A shared-weight Siamese classifier is trained on noisy labels using contrastive and classification losses, with its classifier operating on normalized embeddings. A checkpoint is selected using noisy validation labels. The frozen detector classifies twelve augmentation families per training image and counts predictions that contradict the observed label. Suspicious examples are relabeled when the views sufficiently support an alternative class and discarded otherwise. A downstream classifier is then trained on the cleaned data. The analysis decomposes the clean/noisy disagreement gap into a margin term and a residual, with the former nonnegative under stochastic-dominance conditions. Experiments evaluate detection, cleaned-label composition, and downstream accuracy.

Critical strengths
  1. The method returns cleaned labels and a retained set, rather than only a final model. Table 1 reports residual noise, retention, and changes in correct labels: CIFAR-10N noise falls from 9.0% to 4.60%, and CIFAR-100N from 40.2% to 28.42%, while retained correct-label counts increase in both. These directly address the stated data-cleaning objective.

  2. Appendix F studies view count, single-pass confidence, relabeling versus discarding, and matched cleaning budgets. The same-model comparisons support benefits of multiple views in several settings. The matched clean-versus-unclean downstream experiment on CIFAR-10 IDN 40% improves accuracy from 86.93% to 91.95%. Appendix J additionally offers a threshold rule without downstream retraining and discusses its validity conditions.

Critical weaknesses

Table 3 imports reference results from another common-codebase study, whereas this pipeline uses a pretrained detector and its own downstream recipes and searches. The authors appropriately acknowledge on p. 9 that these cross-paper gaps cannot be attributed to cleaning, but this leaves the central real-data comparison without a matched end-to-end control.

The score counts disagreement with the observed label, not disagreement among views. Twelve stable predictions of the same alternative class produce the maximum suspiciousness score, so the score alone does not isolate perturbation instability. The included simple baselines are valuable but also reveal limitations: on CIFAR-100N, count AUROC is 0.840 versus 0.834 for single-pass confidence and 0.855 for mean view probability. In one synthetic ablation, removing the contrastive loss changes AUROC only from 0.9873 to 0.9865, although relabeling accuracy deteriorates. The evidence supports multi-view cleaning more strongly than the necessity of the Siamese contrastive term and discrete count. The conditional margin decomposition also does not guarantee a positive residual.

Questions for authors

Q1. On CIFAR-10N/100N, can you compare uncleaned data, single-pass confidence cleaning, mean-view-probability cleaning, and count-based cleaning under identical pretraining, downstream architecture, recipes, and search budgets? Which matched comparisons can the existing caches support? Q2. How much variation appears when detector training and cleaning, and synthetic noise realizations where applicable, are repeated? Please distinguish full-pipeline variability from variability due only to retraining the downstream classifier. Q3. Can you separate disagreement already present on the unaugmented input from disagreement introduced by augmentation, and report the contrastive term’s relabeling and downstream contribution on the main real-noise settings? This would clarify additional fragility versus a base predictor already rejecting the label.

1 comment
AuthorsOpenReview ↗

Response to Reviewer xqhg

Thank you for the review.

W1 and Q1 (matched comparison on CIFAR-10N/100N). We ran this comparison (general items 1 and 2). With the same downstream network, recipe, epochs and early stopping, cleaning gains an average of 2.73 points on CIFAR-10N and 2.59 on CIFAR-100N, winning all six seed pairs. We chose the recipe once, on the cleaned data, and tuned no run separately. The probability scores use the reported CIFAR-100N detector and CIFAR-10N detector A of general item 3.

W2 and Q3a (what the score measures, and what the views add). The score counts how many of the 12 views predict a class other than the recorded label, and this is on purpose. A cleaner has to decide whether the recorded label is wrong, so the score must look at that label. Scores that only ask whether the views disagree with each other ignore the label and reach only 0.50 to 0.60 AUROC on the synthetic settings (0.50 is chance), against 0.91 to 0.99 for the count. When all 12 views name the same other class the score is at its maximum, and such samples are mislabeled 91.4% to 99.7% of the time on CIFAR-10N, CIFAR-100N and CIFAR-10 IDN 40%. The revised abstract and introduction now say this.

To see what the views add, we use the prediction on the original, unaugmented image. It rejects the label when it predicts another class, and accepts the label otherwise. For trained-on samples, accepting a wrong label counts as memorization. We split every flagged sample on CIFAR-10 IDN 20% and 40%, Fashion-MNIST IDN 30% and CIFAR-100N this way.

  • For 94.1% to 98.9% of flags, the original prediction already rejects the label.
  • The views decide which of these rejections to trust. When the original prediction rejects the label but the views do not reach the flag threshold, the label is wrong only 30% to 50% of the time. So the count is more precise than flagging every such rejection (precision 0.969 vs 0.918, 0.959 vs 0.917, 0.957 vs 0.938, 0.788 vs 0.737).
  • The views also flag some samples whose original prediction accepts the label, 1.1% to 5.9% of all flags, and 47% to 74% of these are truly mislabeled.
  • Among samples whose original prediction accepts the label, the count separates wrong labels from clean samples better than single-pass confidence (AUROC 0.911, 0.925, 0.885, 0.747 against 0.712, 0.797, 0.749, 0.723).

On the count against mean view probability, the count has the higher AUROC on three of these four settings, and mean view probability on CIFAR-100N (0.855 vs 0.840). Downstream, mean view probability is 0.21 points higher on CIFAR-100N (60.85 vs 60.64) and the count is 0.16 higher on CIFAR-10N (92.98 vs 92.82). Our contribution is the cleaning method, and the count is its simplest score. The same 12 votes also pick the new label, and the label-free rule you noted (Appendix J) can set its threshold.

On the residual. Under Assumptions 1 and 2 the theorem proves the margin term is never negative, and it makes up 79% to 92% of the measured gap. A positive residual X means that, on average, mislabeled samples disagree more than expected for clean samples at the same margin. We measure it instead of assuming it. X is positive on all four 2,000-sample subsets in Appendix D (0.058 to 0.145), as are the residuals at all 12 thresholds per subset.

Q2 (variability). General item 3 repeats the pipeline with three new detector seeds and the thresholds and recipe fixed. On CIFAR-10N the standard deviation across detector means is 0.80 points. Across downstream seeds within a detector it averages 0.14. On CIFAR-100N the two are 0.38 and 0.19. All nine CIFAR-10N runs exceed SGN's published mean (92.06).

Q3b (the contrastive term on real noise). Matched detector comparisons with λ = 1 and λ = 0 (no contrastive loss), cleaned at the reported thresholds:

BenchmarkAUROC λ=1 / λ=0Correct labels added λ=1 / λ=0Relabel accuracy λ=1 / λ=0Downstream λ=1 / λ=0
CIFAR-10N0.931 / 0.942+1,362 / +1,6370.802 / 0.80392.98 ± 0.25 / 93.28 ± 0.05
CIFAR-100N0.845 / 0.833+1,379 / +1,5550.518 / 0.55561.04 ± 0.20 / 60.45 ± 0.16 †

Within each detector pair, only λ differs. † One single-sample class was left out to allow a stratified split (44,927 of 44,928 kept).

In the CIFAR-100N comparison above, λ = 1 has higher recall and F1 at the reported threshold and gives higher downstream accuracy on all three downstream seeds, by 0.59 points on average. On synthetic CIFAR-10 IDN 40%, λ = 1 relabels more accurately (92.7% vs 90.4%).

Reviewer_cPGiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a model-based noisy label cleaning method. It trains a Siamese classifier on the same noisy dataset, and exploits its predictions on twelve augmented views to keep, relabel, or discard the samples in the dataset. Experiments show that the method effectively identifies and modifies noisy labels for both synthetic and human annotation noise, with a downstream accuracy gain demonstrated in one matched synthetic-noise setting.

Critical strengths

Learning from noisy labels is an important problem, and using one model’s augmented-view predictions to both detect suspect labels and propose corrections is an interesting idea. The paper also provides a theoretical analysis of the method and empirical validation on synthetic and human-noise benchmarks.

Critical weaknesses
  • The practical value of the full procedure remains insufficiently established. A performance gain on cleaned dataset alone does not justify the practical value. One immediate questions is how does the Siamese detector perform under the noisy settings? Given the effort required to train this detector, if the subsequent models do not show meaningful improvements over this detector, the benefit of training a second model is unclear.
  • Moreover, the detector and final classifier differ in architecture, pretraining, and objective. Lee et al.’s (2022, they also exploit augmentation in noisy data setting) TTA-based selection results provide relevant context: selecting examples does not uniformly improve a comparable classifier. This makes it important to understand how the reported gain depends on model architecture and which types of corrected samples contribute to downstream performance.
  • The method’s operating domain remains unclear. Which types of mislabeled examples can it detect and correct, and which does it miss? The paper suggests that mislabeled examples can lie near a decision boundary and become sensitive to input perturbations. However, a clearly mislabeled example could receive the maximum score through consistently predicting an alternative class without being near a boundary; A more detailed analysis of detected and missed noisy examples is needed to clarify what the method actually detects and where it works.

[1] H. Lee, H. Lee, H. Hong, and J. Kim. “Noisy Label Classification Using Label Noise Selection with Test-Time Augmentation Cross-Entropy and NoiseMix Learning.” In Data Augmentation, Labelling, and Imperfections (DALI 2022), LNCS 13567, pp. 74–82, 2022. DOI: 10.1007/978-3-031-17027-0_8.

Questions for authors
  • On the same held-out clean test sets, how does the Siamese detector trained on noisy labels perform with direct and twelve-view-majority inference, compared with the classifier trained on its cleaned dataset?
  • Under matched architectures, pretraining, training objectives, and inference policies, how do classifiers trained on noisy versus cleaned labels compare, particularly for human noise?
  • Which corrected or discarded sample groups account for the downstream accuracy change?
  • Which types of label errors are detected, correctly relabeled, discarded, or missed (it is especially interesting in the human noise case)? Do the detected examples actually lie near decision boundaries, and what happens to hard but correctly labeled examples?
2 comments
AuthorsOpenReview ↗

Response to Reviewer cPGi

Thank you for the review and the Lee et al. reference, now cited.

W1 and Q1 (the detector alone against the downstream classifier). On the same clean test sets, here is the detector used three ways, next to the classifier trained on its cleaned data:

SettingDetector, directDetector, 12-view voteDetector, 12-view meanDownstream
CIFAR-10 IDN 20%92.3992.9893.2593.93 ± 0.39
CIFAR-10 IDN 40%87.9990.2990.2591.95 ± 0.46
Fashion-MNIST IDN 30%93.2693.2493.6593.90 ± 0.12
CIFAR-100N56.5258.7459.0860.52 ± 0.38
CIFAR-10N92.5292.6793.0493.90 ± 0.11
Animal-10N86.32n/an/a87.11 ± 0.44

In every setting the downstream mean is higher, even though the detector starts from ImageNet weights and the downstream PreActResNet trains from scratch. With one pass per image, it gains 0.64 to 4.00 points over the detector's direct mode. It is still 0.25 to 1.66 points higher than the best 12-view mode, which evaluates 12 views per image. With the raw-label control of general item 1, this shows what the second model adds. Our product is the cleaned dataset, not a model. It is built once, and any classifier can then train on it.

W2 and Q2 (Lee et al., and noisy vs cleaned labels under matched conditions).

  • Lee et al. Their score is the cross-entropy between the recorded label and the views' vote shares. On the same 12 predictions it rises exactly when our count rises, so the two rank samples in the same order. Their NoiseMix trains on mixtures of the selected clean subset and the noisy data, while we fix or remove each flagged label and return a cleaned dataset.
  • Matched conditions. General item 1 trains the same downstream network from scratch, with the same objective, recipe and inference, on raw and cleaned human-noise labels. Cleaning gains +2.73 ± 0.45 on CIFAR-10N and +2.59 ± 1.14 on CIFAR-100N. With synthetic CIFAR-10 IDN 40% (91.95 vs 86.93, Appendix F.4), cleaning wins all nine seed pairs. A second downstream network, PreActResNet-18, gives human-noise gains of +2.99 ± 0.37 and +3.73 ± 1.13, again in all six pairs.

Q3 (which groups drive the downstream change). Relabeling is the only step that can add correct labels.

  • CIFAR-10N. Of 4,505 mislabeled samples, 2,361 get the correct label, 247 get another wrong label, 172 are discarded and 1,725 are missed. 397 clean samples are flagged.
  • CIFAR-100N. Of 20,100 mislabeled samples, 4,324 get the correct label, 2,374 get another wrong label, 5,188 are discarded and 8,214 are missed. 3,203 clean samples are flagged (1,729 get a wrong label, 1,474 are discarded).
  • On every setting with reference labels, the cleaned data has more correct labels than the original.

To isolate relabeling, we also discarded every flagged sample, with the same detector, recipe and seeds. On CIFAR-10 IDN 40%, relabeling beats discarding by 0.97 points on average, in all three seed pairs. Elsewhere the difference is small and goes both ways (−0.22 on CIFAR-10N, +0.02 on CIFAR-100N, −0.39 on Fashion-MNIST IDN 40%). Relabeling always keeps more correctly labeled samples, 2,361 more on CIFAR-10N and 4,324 more on CIFAR-100N.

W3 and Q4 (which errors are detected or missed, the boundary, and hard clean samples). A mislabeled sample can get a high score in two ways. Its views may favor another class, or its predictions may change across views because it sits near a class boundary. The revised abstract and introduction name both.

  • Stable alternatives, the case you describe. When all 12 views name the same other class, the label is wrong for 97.5% of such samples on CIFAR-10N, 91.4% on CIFAR-100N and 99.7% on CIFAR-10 IDN 40%, and relabeling gives the correct label for 97.1%, 84.8% and 99.3% of them. Of the detected errors, 16.8%, 1.9% and 23.4% are of this kind.
  • Missed errors sit near the boundary. We re-scored 2,000 samples from each of CIFAR-10 IDN 20% and 40%, Fashion-MNIST IDN 40% and CIFAR-100N. We measure closeness to the boundary with the normalized margin |s|, a first-order estimate, and call a sample close when |s| is below the 25th percentile of clean kept samples. 61% to 89% of missed errors are close, against 19% to 45% of detected errors.
  • Human-noise error types. On CIFAR-10N the method detects 81.5% of errors between an animal and a vehicle class, and 54.2% of errors within animals or within vehicles. On CIFAR-100N it detects 64.1% of errors that change the superclass and 49.7% of errors within one.
  • Hard but clean samples. On CIFAR-10N, 397 of 45,495 clean samples are flagged (0.87%), most often a dog relabeled as cat (87 cases). On CIFAR-100N, 3,203 of 29,900 are flagged (10.7%), and for 54% of them the views' most-voted class is in the same superclass as the true label.

Thanks for your responses and most of my concerns are addressed. I would like to increase my score to 3.

Reviewer_XNp5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a post-hoc, data-cleaning method for classification with noisy labels. It trains a Siamese network once on the noisy dataset using joint contrastive and classification losses, freezes the model, and applies 12 test-time augmentations to each training example. Samples are scored by how often augmented views disagree with the recorded label: low-disagreement examples are retained, high-disagreement examples with a stable alternative plurality prediction are relabeled, and those with scattered predictions are discarded. The authors argue that mislabeled samples tend to have smaller sensitivity-normalized margins and thus exhibit greater augmentation-induced instability; they provide a conditional theoretical decomposition of this disagreement gap and evaluate the approach on synthetic instance-dependent noise and human-noise image benchmarks. They report improved label-cleaning metrics relative to single-pass confidence baselines and competitive downstream accuracy on CIFAR-N and Animal-10N.

Critical strengths
  1. A simple, single-model, post-hoc TTA instability score for label-error detection, paired with a practical relabel-versus-remove rule and a conditional margin-based analysis.

  2. In terms of writing,the paper is reasonably clear at the methodological level, with helpful figures and an intuitive cleaning rule.

Critical weaknesses
  1. The limited theoretical analysis (a lone theorem 1) does not establish/guarantee successful detection by the method.

2.) The method needs label-preserving transformations and that a single global threshold struggles with ambiguous but correctly labeled examples—particularly on CIFAR-100. A limitation is reliance on a useful set of transformations that genuinely preserve labels, which is more natural for images than for many other modalities. For example, the paper’s human-noise results are useful, but its evaluation remains heavily centered on CIFAR/Fashion-MNIST-scale imagery and one synthetic IDN generator.

4.) Several detector and robust-training comparisons rely on previously published results rather than a common reimplementation with matched architectures, augmentation policies, tuning budgets, and compute. These comparisons are informative but do not conclusively establish superiority.

Questions for authors

I'm afraid the literature on data-cleaning pipelines is rather mature. The following question could help me evaluate the contribution better.

  1. Can the authors compare, under a matched architecture, augmentation policy, tuning budget, and compute budget, against Confident Learning, SimiFeat, and DISC? Please also include continuous TTA baselines such as mean observed-label probability, predictive entropy, vote margin, and logit variance.
2 comments
AuthorsOpenReview ↗

Response to Reviewer XNp5

Thank you for the review. We give all four continuous TTA scores you asked for and test per-sample thresholds.

W1 (one theorem, no detection guarantee). Theorem 1 is conditional by necessity. It splits the gap in expected disagreement between mislabeled and clean samples into two exact parts. Under Assumptions 1 and 2 it proves that the first part, the margin term, is never negative, and it gives a lower bound on its size. Its converse shows that dominance (Assumption 1) is needed for any such guarantee that holds for all non-increasing responses, and dominance holds exactly on all four tested subsets. The second part, the excess, is measured instead of assumed. It is positive on all four (0.058 to 0.145), and the margin term makes up 79% to 92% of the gap.

The theorem covers detection itself. For the flag at one threshold, the gap is the share of mislabeled samples flagged minus the share of clean samples flagged. With a non-increasing fit to the clean response, Assumption 1 again makes its margin part non-negative. Lemma 5 (Appendix C.4) shows that AUROC minus 0.5 is a sum of these gaps with non-negative weights. On all four subsets the gap and its excess are positive at all 12 thresholds.

W2 (label-preserving transformations, a global threshold, scale).

  • Transformations. Our 12 views are chosen to preserve image class (Appendix E). The paper's scope is image classification. Other data types need their own label-preserving transformations, a separate research question.
  • A single global threshold. The per-sample thresholds we tested add little over the best global threshold on the reported detectors of CIFAR-10 IDN 20% and 40%, CIFAR-10N and CIFAR-100N (new Appendix J.1). We sorted samples into 20 equal-size groups by margin, view distance or their product. Grouping by margin lets samples near the boundary, where ambiguous samples tend to lie, receive a different threshold. Even with each group's threshold picked using the true labels, F1 rises by only 0.0022 to 0.0062 over the best global one. A label-free rule that sets each sample's threshold from its margin and view distance gives most samples (62% to 92%) the same threshold, and using that one value for every sample changes F1 by less than 0.002. One threshold per recorded class, also picked with the true labels, gains more (0.0037 to 0.0320 F1), and the revised Discussion names it as future work.
  • Benchmarks and noise models. CIFAR-10N and CIFAR-100N (Wei et al., ICLR 2022) are standard human-noise benchmarks with a reference label for every training image. That lets us count precision, recall and remaining noise, which measure cleaning directly. We use three kinds of noise. Instance-dependent noise at three rates on CIFAR-10, CIFAR-100 and Fashion-MNIST comes from the same generator as the IDN baselines of Table 15, so both face the same kind of noise. Symmetric 50% noise on CIFAR-10 is in Appendix H.3. Real human noise covers CIFAR-10N, CIFAR-100N and Animal-10N at its native 64×64.

W4 and Q1 (matched baselines and continuous TTA scores). The first score you list, mean observed-label probability, was already in the submitted Appendix F.3, with single-pass probability, cross-entropy and margin. We computed the other three on the same checkpoints as the count, so each comparison changes only the score. AUROC measures how well a score ranks mislabeled samples above clean ones (1 is perfect, 0.5 is chance).

  • Vote margin is within 0.008 AUROC of the count on all 11 settings, and is 0.833 against 0.840 on CIFAR-100N.
  • Vote entropy and variation ratio ignore the recorded label and reach only 0.50 to 0.60 on the synthetic settings, close to chance.
  • Predictive entropy and logit variance over the 12 views, on five detectors (CIFAR-10N uses detector A of general item 3). The entropy of the mean prediction reaches 0.51 to 0.56 on the synthetic settings, 0.72 on CIFAR-100N and 0.81 on CIFAR-10N, against 0.99, 0.84 and 0.93 for the count. Logit variance reaches at most 0.75.
  • Where we computed them, the count is never more than 0.015 AUROC behind any of these scores (mean view probability on CIFAR-100N, 0.855 vs 0.840), and the scores that ignore the recorded label fall far behind.
  • Downstream, with the same number of flags, the three flagging scores land within 0.27 points of each other on both human-noise benchmarks (general response, item 2).

Confident Learning and SimiFeat are in Table 2 with their published numbers on the same benchmarks. DISC trains its classifier end to end, so re-implementing it would compare whole training pipelines, not detection scores. The matched raw-label control (general response, item 1) isolates what our cleaning adds, and cleaning wins all six seed pairs.

Reviewer_XNp5OpenReview ↗

Rebuttal Acknowledged

I acknowledge the authors' follow-up remarks to everyone in which they report more numbers. And I am willing to raise my score from 2 to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers. The main new result is a direct test on human noise. With the same downstream classifier and recipe, our cleaned labels beat the raw labels in all six seed pairs, by 2.73 points on CIFAR-10N and 2.59 on CIFAR-100N on average. The revised PDF adds the new results to Appendices F.2, F.4, F.5, F.6, I.1 and J.1.

1. Matched control on human noise (xqhg Q1, cPGi Q2, XNp5 W4). We trained the downstream classifier on the raw CIFAR-10N and CIFAR-100N labels, with the same network, recipe, epochs, split rule, early stopping and seeds as the cleaned run, which we re-ran in the same software. Only the data differ. Cleaning changes some labels and removes some samples, in both the training and validation sets. Results are paired over three seeds. Here ± is the standard deviation over seeds, and for a gain it is that of the per-seed differences.

Raw labelsCleaned by our methodPaired gain
CIFAR-10N90.90 ± 0.2593.63 ± 0.25+2.73 ± 0.45
CIFAR-100N58.05 ± 0.9260.64 ± 0.39+2.59 ± 1.14

Cleaning helps in all six seed pairs. The re-run cleaned results are within 0.3 points of the submitted ones (93.90 and 60.52). With PreActResNet-18 as the downstream network, the gains are +2.99 ± 0.37 and +3.73 ± 1.13, again in all six pairs.

2. Flagging score at a fixed budget (xqhg Q1). We let each of three scores pick the same number of samples to flag. Every run then fixes or removes its flagged samples with the same 12-view rule, so only the choice of flags differs.

  • On CIFAR-100N the count reaches 60.64 ± 0.39, single-pass confidence 60.74 ± 0.48 and mean view probability 60.85 ± 0.22.
  • On CIFAR-10N, with detector A of item 3 at its count budget of 2,294 flags, the count reaches 92.98 ± 0.25, single-pass confidence 92.71 ± 0.18 and mean view probability 92.82 ± 0.13.

The three scores land within 0.21 points of each other on CIFAR-100N and 0.27 on CIFAR-10N. The count uses the same 12 votes that pick the new label.

3. Repeated detectors (xqhg Q2). For each human-noise benchmark we trained three new detectors with the reported settings and the same software. They differ only in the training seed. The reported detector, trained in August with older code and software, is shown for reference. We cleaned each at the reported thresholds and trained three downstream seeds with the fixed recipe.

  • Detection AUROC is 0.929 to 0.958 on CIFAR-10N and 0.845 to 0.847 on CIFAR-100N (reported detector 0.951 and 0.840).
  • Every new detector lowers the noise rate (from 9.01% to between 4.4% and 6.0% on CIFAR-10N, from 40.2% to between 27.7% and 28.4% on CIFAR-100N) and adds correct labels (1,303 to 1,953 and 614 to 1,379).
  • Downstream accuracy, three seeds per detector:
DetectorCIFAR-10NCIFAR-100N
Detector A92.98 ± 0.2560.29 ± 0.19
Detector B92.35 ± 0.1361.04 ± 0.20
Detector C93.95 ± 0.0560.81 ± 0.18
SD of the three detector means0.800.38
Mean SD over downstream seeds0.140.19
Reported detector (reference)93.63 ± 0.2560.64 ± 0.39

All nine CIFAR-10N runs score above the strongest published entry in Table 3 (SGN, 92.06 ± 0.12). Every new detector beats the raw labels of item 1 in all three downstream seeds, with mean gains of 1.45 to 3.05 points on CIFAR-10N and 2.24 to 2.98 on CIFAR-100N. On CIFAR-10N, accuracy varies more between detectors than between downstream seeds. This is because at the fixed τd = 9 the two detectors with lower AUROC flag fewer samples (2,294 and 1,982, against 4,218), so more noise stays in the data (5.8% and 6.0%, against 4.4%), and downstream accuracy follows the noise that is left. On CIFAR-100N the variation between detectors is about twice that between downstream seeds.

4. The contrastive term on human noise (xqhg Q3). Setting λ = 0 turns off only the contrastive loss. The detector is still the Siamese network of Section 3.1.

  • In the matched CIFAR-100N comparison (Appendix F.2, Table 9), λ = 1 has higher recall, F1 and AUROC, leaves less noise and gives higher downstream accuracy on all three downstream seeds (+0.59 points on average).
  • In the matched CIFAR-10N comparison, λ = 0 gives higher downstream accuracy by 0.30 points on average.
  • On synthetic CIFAR-10 IDN 40%, λ = 1 relabels more accurately (92.7% vs 90.4%, Appendix F.2).

Text changes in the revision.

  • Lee et al. (2022) is cited and positioned in Section 2 (cPGi).
  • The abstract and introduction describe the score as disagreement with the recorded label. A sample gets a high score either when its predictions change across views or when its views agree on another class (cPGi W3, xqhg W2).
  • In Section 5.3 the synthetic memorization range now ends at 0.180 (was 0.163), and the truck → automobile example sentence is removed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.