acceptodds
Under review as a conference paper at ICLR 2027

Stop Replacing Noise with Noise: Two-Source Reliability Assessment for Label Correction and Sample Reweighting in Label-Noise Learning

Abstract

Refurbishment-based noisy-label learning mixes an observed label with a model-derived pseudo target, typically using one sample-wise cleanliness score to control both branches. This creates a hidden coupling: reducing trust in the observed label automatically increases trust in the pseudo target. We show that this complementarity can replace one unreliable signal with another because a pseudo target learned from corrupted supervision may reproduce the noise it is meant to correct. Our representation diagnostics provide a consistent account of this mismatch: noisy supervision redirects deeper layers more strongly, whereas shallower relations remain comparatively stable and provide information beyond the loss posterior. We therefore propose TRACE, a Two-Source Reliability Assessment framework for Label Correction and Sample Reweighting. TRACE assesses the observed label using loss fit, shallow relation stability, and prediction agreement, while separately assessing the pseudo target using model confidence. Its source-specific scores control target correction and supervision strength without assuming complementary reliability. Across synthetic and real-world noisy benchmarks, TRACE improves representative refurbishment baselines and yields more reliable pseudo supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Learning with noisy labels, Label correction, Pseudo-target reliability, Representation stability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G517OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper challenges the implicit complementarity in label refurbishment: reducing trust in the observed label automatically increases the pseudo-target weight. TRACE separately estimates observed-label and pseudo-target reliability, sets a=s_obs and b=(1−s_obs)s_pseudo, and uses them to construct a corrected target and total sample weight. Observed-label reliability combines loss, local relational stability across network depth, and dual-network prediction agreement; pseudo-target reliability is confidence-based. The interface preserves the base learner’s pseudo-target generator and is evaluated as a plug-in across several learners, synthetic noise, and real-noise datasets. Depth-wise diagnostics motivate separate assessment, and the experiments report both classification performance and pseudo-target quality.

Critical strengths

The weight b=(1−s_obs)s_pseudo combines correction need with replacement reliability and permits both signals to be unreliable. Weighting by a+b also avoids restoring full supervision after target normalization. This is a coherent interface improvement that does not require a new pseudo-label generator.

The paper reports paired gains for learners including DivideMix, RoLR, DISC, and ANNE, with extensions to CIFAR-N, WebVision, Food-101N, and Clothing1M. Table 5 includes a coupled-null control, not merely module removals, to separate observed-label refinement, pseudo-target gating, and sample weighting. The paper also discloses modest gains on difficult human noise and decreases in some global pseudo-target accuracies.

Critical weaknesses
  1. In Table 5, Last accuracy is 74.71% for RoLR, 75.86% after observed-label refinement with complementary weights retained, 76.10% after adding the pseudo-target gate, and 76.22% for full TRACE. The gating increment is therefore 0.24 percentage points in this ablation; the full 1.51-point gain should not be attributed to decoupling. Tables 1–2 also mix cited and replicated results; the base rows used for paired attribution should be explicitly identified as matched reruns.

  2. Appendix E.1 explains that Figure 1’s “Follow noisy” is computed only on genuinely corrupted observed labels, whereas “Pseudo correct” is the overall pseudo-target accuracy within the confidence stratum. Thus, 99.7% and 74.7% are not complementary quantities on the same population, and presenting their 25.0-point difference as a same-sample mismatch is potentially misleading. The low-clean+noisy subsets in Tables 6/17 are also selected separately by each method, confounding membership changes with target-quality changes. Global pseudo-target accuracy on CIFAR-100N decreases from 71.86% to 71.22%, so benefits are not uniform. The maximum-probability gate remains model-derived and cannot exclude confident errors. Common denominators, fixed evaluation sets, and explicit calibration definitions are needed to substantiate safer replacement supervision.

Questions for authors

Q1. Can you report seed-level paired differences and uncertainty for coupled null, decoupled reliability, and full TRACE across multiple existing noise settings? Please identify which base rows are matched reruns and which are contextual cross-paper references. Q2. Can Figure 1’s statistics be recomputed on the same corrupted-label population within common confidence strata, and can the methods be compared on a fixed shared low-clean set? Please report bin counts and denominators rather than directly contrasting percentages from different populations. Q3. How are calibration and high-confidence errors defined for the sharpened pseudo-target’s maximum probability? On a common set that actually requires replacement, does the gate consistently reduce confident errors relative to coupled null, particularly on CIFAR-100N?

1 comment

Dear Reviewer G517, with the discussion period ending soon, we would appreciate it if you could take a look at our responses. If anything needs further clarification, we would be happy to continue the discussion. Thank you again for your time.

Reviewer_2DsFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TRACE, which separately assesses observed-label and pseudo-target reliability and uses both label correction and sample reweighting to reduce the influence of unreliable supervision. The method combines loss fit, cross-layer relational stability, dual-network agreement, and prediction confidence while preserving the base learner’s pseudo-target generator, and is evaluated across multiple learners and noise settings.

Critical strengths

1.The paper is well written and clearly motivated , making its core ideas accessible to readers unfamiliar with the field. Its starting point is particularly interesting: under conventional coupled weighting, decreasing the weight of the observed label necessarily increases that of the pseudo target, potentially allowing errors to snowball.

2.The paper presents extensive experiments supporting the effectiveness of the proposed components , along with detailed hyperparameters and experimental settings that facilitate reproducibility.

Critical weaknesses

1.Limited theoretical justification for reliability scoring: Applying no attenuation when the two networks agree, using a fixed attenuation factor when they disagree, and estimating pseudo-target reliability through a power of the maximum predicted probability are intuitive but heuristic choices. Although ablations and sensitivity analyses support their empirical effectiveness, the theoretical justification for these specific scoring rules remains limited.

2.Some reliability gains are difficult to interpret because of differing comparison populations: Although Table 17 provides full-population results, the highlighted low-clean+noisy subsets in Table 6 are selected separately by each method, so their gains combine prediction changes with sample-selection differences.

Questions for authors

1.Could the authors provide further theoretical analysis or targeted experiments to justify the specific heuristic reliability scoring rules?

2.With reliability scores and sample weights held fixed, how do soft pseudo-targets compare with argmax-based hard pseudo-targets? Could soft pseudo-targets introduce additional incorrect supervision for samples with correct observed labels?

3.Have the authors evaluated a simpler decoupling baseline that retains the original loss-based observed-label score and adds only pseudo-target confidence gating, without structural scoring, agreement gating, or sample reweighting?

1 comment

I thank the authors for their rebuttal, which has addressed most of my concerns. I therefore maintain my positive rating.

Reviewer_nqK3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies refurbishment-based learning with noisy labels. In this setting the training target interpolates between the observed label and a model-derived pseudo target, and a single per-sample cleanliness score, typically the GMM posterior on the loss, sets the weight. The authors argue that this coupling turns reduced trust in the observed label into increased trust in the pseudo target, which can replace one error with another, and they support the argument with layer-wise representation diagnostics. TRACE scores the observed label from the loss posterior, cross-layer relation drift and agreement between two networks. It scores the pseudo target by its maximum probability and uses the two scores to form the corrected target and a per-sample weight. As a plug-in to six base learners, it reports average gains of 0.46 to 1.16 points on CIFAR-10/100 with synthetic noise, CIFAR-N, WebVision and Food-101N, a gain of 2.8 points on Clothing1M, and higher pseudo-target accuracy than RoLR...

Critical strengths
  1. The interface is simple and practical. Separating the need for correction (1 − s_obs) from the reliability of the proposed correction (s_pseudo) is easy to interpret, adds no training objective, and leaves the base learner’s pseudo-target generator unchanged. It can therefore be attached to existing refurbishment methods with little effort. This is the paper’s main merit.

  2. The evaluation is broad and extends beyond test accuracy. TRACE is combined with six base learners on synthetic, human-annotated and large-scale real-world noise, and it improves or matches its base learner in every reported paired comparison. The paper also reports pseudo-target accuracy, high-confidence errors, calibration error, reliability maps and a component ablation. These diagnostics make the paper’s claims checkable, and several of the concerns below are based on them.

Critical weaknesses
  1. The paper does not establish that the complementary coupling is the operative problem, or that decoupling the two sources is what produces its improvements. The title, abstract and stated contributions rest on this claim, so it is the main reason for my recommendation. In the component ablation (Table 5, CIFAR-100 with 50% symmetric noise), the coupled-null variant keeps the complementary pseudo weight and only replaces the loss posterior with the refined observed-label score. This alone raises RoLR from 75.68/74.71 to 76.58/75.86 (best/last). Among the reported variants, the pseudo-target gate then adds 0.31/0.24 points and sample reweighting 0.06/0.12. No standard deviations are given, and the variant that would test decoupling on its own, the gate applied to the original loss posterior, is not reported. The motivating evidence points the same way. Fig. 1(a) shows that pseudo targets are often wrong when λ is low, but that is the known confirmation-bias problem discussed below. The headline numbers (99.7%, 74.7% and 19.2%) come instead from the high-clean bins. There the coupled rule already gives most of the weight to the observed label, and nearly every mislabeled sample has a pseudo target that agrees with its wrong observed label. For such a sample, Eq. (14) cannot change the target class. Because a + b ≥ max(s_obs, s_pseudo), its weight also cannot fall below the pseudo-target confidence (max_c q_c)^ρ, so a confidently wrong pseudo target keeps nearly full weight. Neither conclusion depends on s_obs, so relation drift and the agreement gate, which act only through s_obs, cannot alter them. The failure used to motivate the paper therefore lies outside what the proposed mechanism changes directly. Finally, the evidence for more reliable pseudo supervision (Table 6, Figs. 6 and 7) compares the full method with RoLR. As the paper notes, the gate does not change how pseudo targets are constructed, so differences in pseudo-target quality reflect altered training under all components and cannot be credited to decoupling. Table 6 also evaluates each method on the subset selected by its own score, which the authors acknowledge may differ in membership. On CIFAR-100N, the dataset of Fig. 1, the reported gain is 0.70 points.

  2. The novelty is narrower than presented, which matters because both the problem and the two-source framework are claimed as contributions. The underlying phenomenon is that pseudo targets from a model trained on noisy labels often fail to correct those labels and can repeat them. This is the confirmation bias that motivated RoLR, the paper’s main base learner. Chen et al. (2023) showed, from the training dynamics of DivideMix, that pseudo-labels leave many noisy labels uncorrected and that the errors accumulate. Their Fig. 1 already tracks samples whose prediction equals the noisy label but not the ground truth. What is distinct here is the attribution of this phenomenon to the single-score interface, which is the claim questioned in the first weakness. Separating the two sources also has precedent within refurbishment-based learning. The related work states that TRACE brings this distinction into refurbishment-based classification, and the introduction lists SELFIE among methods governed by a single cleanliness score. Yet SELFIE treats small-loss samples as clean, refurbishes only samples whose predictions are consistent, and excludes the rest. It therefore already judges the two sources by different criteria and discards samples for which neither is reliable. The mechanism is also simpler than its presentation suggests. Take the loss in Eq. (15), with ε negligible, a + b ≥ w_min, and the per-batch normalization of w set aside. Each sample then contributes s_obs·CE(p, ŷ) + (1 − s_obs)·s_pseudo·CE(p, q). This is the coupled objective with the pseudo-label term multiplied by (max_c q_c)^ρ. Weighting a pseudo-label loss by model confidence is standard in semi-supervised learning, as a threshold in FixMatch and as a smooth weight in SoftMatch. What remains specific to TRACE is mainly the refined observed-label score, which accounts for most of the improvement among the variants in Table 5. The only quantitative evidence in the main text that its relation-drift component tracks label correctness at the sample level is Fig. 3. That figure uses symmetric noise and, as the authors note, shows the separation narrowing as c_loss approaches one. It does not show that relation drift identifies the mislabeled high-c_loss samples that Fig. 1 documents on CIFAR-100N.

Questions for authors
  1. Please report the variant that isolates decoupling: the pseudo-target gate applied to the original loss posterior (s_obs = c_loss and b = (1 − c_loss)·s_pseudo, without c_str or g_agr). It should be run on CIFAR-100N and on CIFAR-100 with 50% symmetric noise, with standard deviations over the three seeds. If this variant does not improve on RoLR by more than seed variation, the gains would have to be attributed to the refined observed-label score rather than to decoupling.

  2. Consider the samples behind the headline numbers in Fig. 1: mislabeled CIFAR-100N samples in the high-clean bins whose pseudo target agrees with the observed label. Please report, under RoLR and under TRACE, the fraction whose pseudo target still agrees with the wrong label at the end of training, and the weights these samples receive (λ under RoLR; a, b and a + b under TRACE). This would show whether TRACE changes the outcome for the group that motivates it. Please also define the regimes in Fig. 1(b) and the denominator of “follow noisy label”. I could not reconcile the mid-clean values in Fig. 1(b) with the per-bin curves in Fig. 1(a).

  3. Table 6 evaluates each method on the subset selected by its own score. Please report pseudo-target accuracy and the high-confidence error rate on a fixed, method-independent subset, for example all mislabeled training samples, together with the size and recall of the subset each method flags. Please also include the coupled-null and decoupled variants of Table 5, so that the effect of decoupling on pseudo-target quality can be separated from that of the refined observed-label score.

  4. The DivideMix entries for symmetric noise in Tables 1 and 4 (95.7, 94.4 and 92.9 on CIFAR-10; 76.9, 74.2 and 59.6 on CIFAR-100) coincide with the last-10-epoch accuracies reported by Li et al. (2020). The experimental setup states that each paired comparison changes only the reliability mechanism. Please therefore state, for each paired baseline, whether it was rerun with the same code, seeds and protocol as the corresponding +TRACE run, and report standard deviations for Tables 1 to 4. Please also describe how q, c_str and g_agr are obtained for base learners other than RoLR, given that Eqs. (5) and (11) and the averaging of c_str are specified for two networks.

2 comments

Dear Reviewer nqK3, as the discussion period is coming to a close, we would appreciate your thoughts on our responses. Please let us know if any concerns remain—we would be happy to discuss them further. Thank you again for your time.

Thanks for the detailed response.

I read the rebuttal and the revised manuscript. The revision absorbed essentially all of my factual criticisms: my mechanism analysis now appears in the Limitations (Appendix G), the gate-only variant I requested is in the main ablation with seeds and SDs, including the negative result on CIFAR-100N, the loss equivalence I derived is stated in the main text, baseline sourcing is disclosed with full reruns, and Figure 1 was redone with a common denominator. The causal version of the decoupling claim has been withdrawn, and what is now claimed matches the evidence. I have accordingly raised Soundness from 2 to 3.

My recommendation remains weak rejection, on grounds of contribution rather than validity. The gains are carried by the refined observed-label score (+1.15) and reweighting (+0.52), which are the least novel components, while the decoupling-specific gate is +0.24 ± 0.21 / +0.22 ± 0.21, not significant at n = 3 and negative standalone on the motivating dataset, with total gains of 0.3–1.5 points. Competently engineered and honestly reported, but incremental for this venue. I maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.