- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tries to tell which training labels are wrong by looking at the shape of a network's intermediate activations. For each unit (an image, or a latent location in segmentation) it computes five statistics of the activations: near-zero ratio, mean, standard deviation, skewness and kurtosis. For classification these are taken over space within each channel and concatenated. For segmentation they are taken across channels at each location, so each pixel gets a 5-number descriptor.
Class prototypes are moving averages of the descriptors of confidently predicted units. A unit's margin is its distance to the prototype of its observed label minus its distance to the nearest other prototype. This margin sets a detached loss weight between 0.2 and 1 (Eq. 2), and a prototype-alignment loss is added (Eq. 3). Theorem 1 and Proposition 2 give conditions under which margins separate clean from corrupted units and the prototypes stay stable.
Experiments use CIFAR-10/100 and UC Merced with 20% and 40% symmetric and asymmetric noise, PASCAL VOC with three pixel-noise types, and Clothing1M. The headline results are 93.51% on CIFAR-10 with 40% asymmetric noise, about 6 mIoU over SCE on VOC, and a corruption-detection AUROC of 0.92.
Critical strengths
The method is simple and only used in training, with a measured overhead of about 15% and none at inference. One formulation covers image-level and pixel-level noise. The weight is smooth and bounded, and the three-phase schedule is sensible. The experiments are the right kind: a controlled comparison of reliability signals, a split of weighting against alignment, a descriptor ablation and a direct detection test. The algorithm and hyperparameters are written down, and the proofs are correct.
Critical weaknesses
-
Some numbers cannot come from the method as described. In the VOC "Mean only" ablation (Table 9) the descriptor is one scalar per pixel. The paper normalizes descriptors to unit length, so every descriptor and prototype becomes +1 for post-ReLU features. All margins are then 0 and all weights 1, so this variant should equal cross-entropy at 59.72 mIoU. The table reports 64.80. On CIFAR, "Mean only" is a pooled-feature prototype and scores 88.90, while the "Feature Prototype" variant that should be the same mechanism scores 85.67.
-
The qualitative figures do not match the experiments. The segmentation examples in Figs. 2 and 6 are labeled with Oxford-IIIT Pet breeds such as "Birman" and "Beagle", and that dataset is used nowhere in the paper. In Fig. 5, the same captions with identical confidences appear over different images, for example "Dense residential, baseline 0.31, ours 0.98" on four of them. These numbers do not look like per-image model outputs.
-
Other results disagree with each other. Fig. 4 reports five seeds and the appendix three. Fig. 4(a) shows margins that look almost fully separated, while Table 7 reports an AUROC of 0.92 for the same setting. Fig. 4(c) shows prototype contamination rising from about 2.5% to 10% exactly when the method is switched on, and the paper does not explain it.
-
The gains do not follow from the weighting. With the paper's own settings, Eq. 2 gives a weight of about 0.76 at the reported mean corrupted margin of 0.314, and only for units with confidence above 0.8. In Table 4, weighting alone adds 4.46 points and the alignment loss alone adds 3.55, but together they add 11.83. No mechanism is offered for this interaction. The full model also beats the other variants on clean data (Table 1) and beats CE by 1.5 mIoU on clean VOC, which a reweighting method should not do. The paper calls the reliability signal its main contribution, and its own ablation says otherwise.
-
The comparisons are not controlled. Table 3 sets this paper's ResNet-34 runs next to numbers that appear to come from other papers with other backbones and schedules. It has no CE row, no UC Merced baselines, and shows DivideMix only at two settings. The claim of slightly exceeding DivideMix at 93.4% is a 0.11-point gap inside the method's own standard deviation. The ViT-L/8 rows have no ViT baseline, so the transfer claim is unsupported. Segmentation is compared only against generic robust losses. Most tables report no variance, and Clothing1M is a single run.
-
The novelty is narrower than presented, and the detection test is weak. The mean block of the descriptor is the ordinary pooled feature. The margin between the observed label and the nearest other class is the idea behind AUM, confidence-gated EMA prototypes appear in MoPro, and prototype-distance weighting of noisy pseudo-labels appears in ProDA. Gram-matrix detection, RoG, D2L and FINE also use class-conditional statistics of intermediate activations. None of these is discussed. In Table 7 the confidence baseline ignores the observed label, there is no logit-margin baseline, and the feature-prototype baselines use absolute distance rather than a margin, so the comparison mixes two factors.
-
The contribution of each statistic is never shown. The paper does not justify these five moments. Table 9 adds them in one fixed order, merges skewness with kurtosis, and reports accuracy rather than detection quality, so it cannot say which statistic helps find corrupted labels. On CIFAR the layer has few spatial positions, so skewness and kurtosis may be very noisy.
-
The theory assumes what it needs to show. Theorem 1 puts the descriptors of mislabeled units around their true-class centre, which is the open question for a network trained on noisy labels. The centres are treated as fixed even though they move during training, and none of the constants is estimated. Fig. 4 shows a gap in mean margins, which does not test these assumptions.
-
The protocol and coverage have gaps. The validation set used for checkpoint selection is not described, UC Merced has no stated split, and the VOC split used for reporting is unclear. The CIFAR descriptor layer is given only as the "third layer", and the set of confident units used by the alignment loss is undefined. There are nine hyperparameters and fixed phase lengths with no sensitivity study. Evaluation stops at 40% noise, with no instance-dependent noise and no CIFAR-N or Animal-10N. No code release is mentioned.
Questions for authors
- How can the VOC "Mean only" variant differ from CE, and what exact normalization is used?
- Which layer and spatial size produce the CIFAR descriptor, and why does "Mean only" (88.90) differ from the feature-prototype variant (85.67)?
- Where do Figs. 2, 5 and 6 come from, and can you replace them with real outputs from the reported experiments?
- What do the weights look like for clean and corrupted units over training, and why do weighting and the alignment loss add up to far more than their separate gains?
- Can you add CE, DivideMix, ELR+ and a reweighting method under your exact protocol, with standard deviations, and ViT baselines for segmentation?
- Can you report single-statistic, leave-one-out and all-subset results, with AUROC for each, and with skewness and kurtosis separated?
- For Table 7, which model, epoch and units were scored, and how does a logit-margin baseline compare?
- Why is the contamination in Fig. 4(c) so high right after the method starts, and were three or five seeds used?