- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TTACleaner, an offline label-cleaning pipeline. A shared-weight Siamese classifier is trained on noisy labels using contrastive and classification losses, with its classifier operating on normalized embeddings. A checkpoint is selected using noisy validation labels. The frozen detector classifies twelve augmentation families per training image and counts predictions that contradict the observed label. Suspicious examples are relabeled when the views sufficiently support an alternative class and discarded otherwise. A downstream classifier is then trained on the cleaned data. The analysis decomposes the clean/noisy disagreement gap into a margin term and a residual, with the former nonnegative under stochastic-dominance conditions. Experiments evaluate detection, cleaned-label composition, and downstream accuracy.
Critical strengths
-
The method returns cleaned labels and a retained set, rather than only a final model. Table 1 reports residual noise, retention, and changes in correct labels: CIFAR-10N noise falls from 9.0% to 4.60%, and CIFAR-100N from 40.2% to 28.42%, while retained correct-label counts increase in both. These directly address the stated data-cleaning objective.
-
Appendix F studies view count, single-pass confidence, relabeling versus discarding, and matched cleaning budgets. The same-model comparisons support benefits of multiple views in several settings. The matched clean-versus-unclean downstream experiment on CIFAR-10 IDN 40% improves accuracy from 86.93% to 91.95%. Appendix J additionally offers a threshold rule without downstream retraining and discusses its validity conditions.
Critical weaknesses
Table 3 imports reference results from another common-codebase study, whereas this pipeline uses a pretrained detector and its own downstream recipes and searches. The authors appropriately acknowledge on p. 9 that these cross-paper gaps cannot be attributed to cleaning, but this leaves the central real-data comparison without a matched end-to-end control.
The score counts disagreement with the observed label, not disagreement among views. Twelve stable predictions of the same alternative class produce the maximum suspiciousness score, so the score alone does not isolate perturbation instability. The included simple baselines are valuable but also reveal limitations: on CIFAR-100N, count AUROC is 0.840 versus 0.834 for single-pass confidence and 0.855 for mean view probability. In one synthetic ablation, removing the contrastive loss changes AUROC only from 0.9873 to 0.9865, although relabeling accuracy deteriorates. The evidence supports multi-view cleaning more strongly than the necessity of the Siamese contrastive term and discrete count. The conditional margin decomposition also does not guarantee a positive residual.
Questions for authors
Q1. On CIFAR-10N/100N, can you compare uncleaned data, single-pass confidence cleaning, mean-view-probability cleaning, and count-based cleaning under identical pretraining, downstream architecture, recipes, and search budgets? Which matched comparisons can the existing caches support? Q2. How much variation appears when detector training and cleaning, and synthetic noise realizations where applicable, are repeated? Please distinguish full-pipeline variability from variability due only to retraining the downstream classifier. Q3. Can you separate disagreement already present on the unaugmented input from disagreement introduced by augmentation, and report the contrastive term’s relabeling and downstream contribution on the main real-noise settings? This would clarify additional fragility versus a base predictor already rejecting the label.
Response to Reviewer xqhg
Thank you for the review.
W1 and Q1 (matched comparison on CIFAR-10N/100N). We ran this comparison (general items 1 and 2). With the same downstream network, recipe, epochs and early stopping, cleaning gains an average of 2.73 points on CIFAR-10N and 2.59 on CIFAR-100N, winning all six seed pairs. We chose the recipe once, on the cleaned data, and tuned no run separately. The probability scores use the reported CIFAR-100N detector and CIFAR-10N detector A of general item 3.
W2 and Q3a (what the score measures, and what the views add). The score counts how many of the 12 views predict a class other than the recorded label, and this is on purpose. A cleaner has to decide whether the recorded label is wrong, so the score must look at that label. Scores that only ask whether the views disagree with each other ignore the label and reach only 0.50 to 0.60 AUROC on the synthetic settings (0.50 is chance), against 0.91 to 0.99 for the count. When all 12 views name the same other class the score is at its maximum, and such samples are mislabeled 91.4% to 99.7% of the time on CIFAR-10N, CIFAR-100N and CIFAR-10 IDN 40%. The revised abstract and introduction now say this.
To see what the views add, we use the prediction on the original, unaugmented image. It rejects the label when it predicts another class, and accepts the label otherwise. For trained-on samples, accepting a wrong label counts as memorization. We split every flagged sample on CIFAR-10 IDN 20% and 40%, Fashion-MNIST IDN 30% and CIFAR-100N this way.
On the count against mean view probability, the count has the higher AUROC on three of these four settings, and mean view probability on CIFAR-100N (0.855 vs 0.840). Downstream, mean view probability is 0.21 points higher on CIFAR-100N (60.85 vs 60.64) and the count is 0.16 higher on CIFAR-10N (92.98 vs 92.82). Our contribution is the cleaning method, and the count is its simplest score. The same 12 votes also pick the new label, and the label-free rule you noted (Appendix J) can set its threshold.
On the residual. Under Assumptions 1 and 2 the theorem proves the margin term is never negative, and it makes up 79% to 92% of the measured gap. A positive residual X means that, on average, mislabeled samples disagree more than expected for clean samples at the same margin. We measure it instead of assuming it. X is positive on all four 2,000-sample subsets in Appendix D (0.058 to 0.145), as are the residuals at all 12 thresholds per subset.
Q2 (variability). General item 3 repeats the pipeline with three new detector seeds and the thresholds and recipe fixed. On CIFAR-10N the standard deviation across detector means is 0.80 points. Across downstream seeds within a detector it averages 0.14. On CIFAR-100N the two are 0.38 and 0.19. All nine CIFAR-10N runs exceed SGN's published mean (92.06).
Q3b (the contrastive term on real noise). Matched detector comparisons with λ = 1 and λ = 0 (no contrastive loss), cleaned at the reported thresholds:
Within each detector pair, only λ differs. † One single-sample class was left out to allow a stratified split (44,927 of 44,928 kept).
In the CIFAR-100N comparison above, λ = 1 has higher recall and F1 at the reported threshold and gives higher downstream accuracy on all three downstream seeds, by 0.59 points on average. On synthetic CIFAR-10 IDN 40%, λ = 1 relabels more accurately (92.7% vs 90.4%).