acceptodds
Withdrawn from ICLR 2027

Activation-Distribution Prototypes for Robust Learning under Noisy Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep neural networks trained with noisy annotations may eventually memorize corrupted supervision because standard objectives treat all labels as equally reliable. We introduce a distribution-aware framework for estimating supervision reliability from intermediate network activations. Rather than representing a supervised unit only as a point in feature space, we summarize its activation behavior through sparsity, mean, standard deviation, skewness, and kurtosis. High-confidence predictions are used to construct class-specific activation-distribution prototypes. Reliability is then estimated by comparing the descriptor of each unit with the prototype of its observed label and the nearest competing prototype. When the activation distribution is more consistent with a competing class, the corresponding supervision is smoothly downweighted according to the resulting disagreement margin. This yields an online reliability signal complementary to prediction confidence, task loss, and point-wise feature representations. Experiments across image classification and semantic segmentation, synthetic and structured corruption, real-world label noise, and CNN and ViT architectures demonstrate consistent robustness gains. Direct reliability evaluation further shows that nactivation-distribution disagreement separates corrupted supervision more neffectively than confidence-, loss-, feature-prototype-, and Mahalanobis-based alternatives. These results show that class-specific activation-distribution modeling provides a practical and informative signal for learning under imperfect supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
noisy-label learning, supervision reliability, activation-distribution modeling, prototype-based learning, robust learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_myruOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tries to tell which training labels are wrong by looking at the shape of a network's intermediate activations. For each unit (an image, or a latent location in segmentation) it computes five statistics of the activations: near-zero ratio, mean, standard deviation, skewness and kurtosis. For classification these are taken over space within each channel and concatenated. For segmentation they are taken across channels at each location, so each pixel gets a 5-number descriptor.

Class prototypes are moving averages of the descriptors of confidently predicted units. A unit's margin is its distance to the prototype of its observed label minus its distance to the nearest other prototype. This margin sets a detached loss weight between 0.2 and 1 (Eq. 2), and a prototype-alignment loss is added (Eq. 3). Theorem 1 and Proposition 2 give conditions under which margins separate clean from corrupted units and the prototypes stay stable.

Experiments use CIFAR-10/100 and UC Merced with 20% and 40% symmetric and asymmetric noise, PASCAL VOC with three pixel-noise types, and Clothing1M. The headline results are 93.51% on CIFAR-10 with 40% asymmetric noise, about 6 mIoU over SCE on VOC, and a corruption-detection AUROC of 0.92.

Critical strengths

The method is simple and only used in training, with a measured overhead of about 15% and none at inference. One formulation covers image-level and pixel-level noise. The weight is smooth and bounded, and the three-phase schedule is sensible. The experiments are the right kind: a controlled comparison of reliability signals, a split of weighting against alignment, a descriptor ablation and a direct detection test. The algorithm and hyperparameters are written down, and the proofs are correct.

Critical weaknesses
  1. Some numbers cannot come from the method as described. In the VOC "Mean only" ablation (Table 9) the descriptor is one scalar per pixel. The paper normalizes descriptors to unit length, so every descriptor and prototype becomes +1 for post-ReLU features. All margins are then 0 and all weights 1, so this variant should equal cross-entropy at 59.72 mIoU. The table reports 64.80. On CIFAR, "Mean only" is a pooled-feature prototype and scores 88.90, while the "Feature Prototype" variant that should be the same mechanism scores 85.67.

  2. The qualitative figures do not match the experiments. The segmentation examples in Figs. 2 and 6 are labeled with Oxford-IIIT Pet breeds such as "Birman" and "Beagle", and that dataset is used nowhere in the paper. In Fig. 5, the same captions with identical confidences appear over different images, for example "Dense residential, baseline 0.31, ours 0.98" on four of them. These numbers do not look like per-image model outputs.

  3. Other results disagree with each other. Fig. 4 reports five seeds and the appendix three. Fig. 4(a) shows margins that look almost fully separated, while Table 7 reports an AUROC of 0.92 for the same setting. Fig. 4(c) shows prototype contamination rising from about 2.5% to 10% exactly when the method is switched on, and the paper does not explain it.

  4. The gains do not follow from the weighting. With the paper's own settings, Eq. 2 gives a weight of about 0.76 at the reported mean corrupted margin of 0.314, and only for units with confidence above 0.8. In Table 4, weighting alone adds 4.46 points and the alignment loss alone adds 3.55, but together they add 11.83. No mechanism is offered for this interaction. The full model also beats the other variants on clean data (Table 1) and beats CE by 1.5 mIoU on clean VOC, which a reweighting method should not do. The paper calls the reliability signal its main contribution, and its own ablation says otherwise.

  5. The comparisons are not controlled. Table 3 sets this paper's ResNet-34 runs next to numbers that appear to come from other papers with other backbones and schedules. It has no CE row, no UC Merced baselines, and shows DivideMix only at two settings. The claim of slightly exceeding DivideMix at 93.4% is a 0.11-point gap inside the method's own standard deviation. The ViT-L/8 rows have no ViT baseline, so the transfer claim is unsupported. Segmentation is compared only against generic robust losses. Most tables report no variance, and Clothing1M is a single run.

  6. The novelty is narrower than presented, and the detection test is weak. The mean block of the descriptor is the ordinary pooled feature. The margin between the observed label and the nearest other class is the idea behind AUM, confidence-gated EMA prototypes appear in MoPro, and prototype-distance weighting of noisy pseudo-labels appears in ProDA. Gram-matrix detection, RoG, D2L and FINE also use class-conditional statistics of intermediate activations. None of these is discussed. In Table 7 the confidence baseline ignores the observed label, there is no logit-margin baseline, and the feature-prototype baselines use absolute distance rather than a margin, so the comparison mixes two factors.

  7. The contribution of each statistic is never shown. The paper does not justify these five moments. Table 9 adds them in one fixed order, merges skewness with kurtosis, and reports accuracy rather than detection quality, so it cannot say which statistic helps find corrupted labels. On CIFAR the layer has few spatial positions, so skewness and kurtosis may be very noisy.

  8. The theory assumes what it needs to show. Theorem 1 puts the descriptors of mislabeled units around their true-class centre, which is the open question for a network trained on noisy labels. The centres are treated as fixed even though they move during training, and none of the constants is estimated. Fig. 4 shows a gap in mean margins, which does not test these assumptions.

  9. The protocol and coverage have gaps. The validation set used for checkpoint selection is not described, UC Merced has no stated split, and the VOC split used for reporting is unclear. The CIFAR descriptor layer is given only as the "third layer", and the set of confident units used by the alignment loss is undefined. There are nine hyperparameters and fixed phase lengths with no sensitivity study. Evaluation stops at 40% noise, with no instance-dependent noise and no CIFAR-N or Animal-10N. No code release is mentioned.

Questions for authors
  1. How can the VOC "Mean only" variant differ from CE, and what exact normalization is used?
  2. Which layer and spatial size produce the CIFAR descriptor, and why does "Mean only" (88.90) differ from the feature-prototype variant (85.67)?
  3. Where do Figs. 2, 5 and 6 come from, and can you replace them with real outputs from the reported experiments?
  4. What do the weights look like for clean and corrupted units over training, and why do weighting and the alignment loss add up to far more than their separate gains?
  5. Can you add CE, DivideMix, ELR+ and a reweighting method under your exact protocol, with standard deviations, and ViT baselines for segmentation?
  6. Can you report single-statistic, leave-one-out and all-subset results, with AUROC for each, and with skewness and kurtosis separated?
  7. For Table 7, which model, epoch and units were scored, and how does a logit-margin baseline compare?
  8. Why is the contamination in Fig. 4(c) so high right after the method starts, and were three or five seeds used?
Reviewer_hStXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a label noise approach for classification and semantic segmentation. The approach measures supervision reliability through a metric derived from the distance from model predictions to prototypes computed during training. The main contribution is the computation of the prototypes from summary statistics of the internal representations of the models. The authors evaluate this approach on three classification datasets and a single semantic segmentation dataset. Across the classification datasets, they explore 20% and 40% symmetric and asymmetric noise; for the segmentation task, they explore pseudo-label noise, erosion and dilation noise, and similar-class perturbation noise.

Critical strengths
  • The design of the proposed metric to identify corrupted supervision is interesting and, combined with the proposed training design, the approach provides competitive results.
  • The mathematical framework theoretically supports the claims about the proposed margin and prototype stability.
  • The paper shows that the proposed approach generalizes to the semantic segmentation task, adding value to the contribution.
Critical weaknesses
  • The descriptor construction and design should be better justified. The authors use statistical moments of the activations of the internal representations. The paper should provide a rationale for why these were chosen, as well as a study of the importance of each of them in assessing supervision reliability.
  • The suspicion score and reliability weight design should be better introduced. Why is the suspicion score defined this way, and how does it affect the final loss? The text should introduce the intuition behind how the margin affects the loss function.
  • The differences in implementation between the classification and semantic segmentation setups are briefly discussed and are slightly difficult to understand. The paper should delve into the implications of computing the summary statistics across different spatial locations in the segmentation task. This has an effect on the supervision when training in a clean setup (no noise), and it is likely to add complexities when training with label noise. The paper should include a discussion of the correlations this introduces in the supervision signal and in the computation of the prototypes, and how these should be taken into account.
  • The experimental scope of the paper is limited. The datasets used for classification are relatively simple. The noise distributions are synthetically introduced and do not reliably show that the proposed approach will generalize to other types of noise more likely to be present in real-world datasets. The authors address this with initial results on the UC Merced dataset but do not include baseline results for other approaches. This is also a domain-specific dataset that does not demonstrate broader generalization of the approach.
  • Table 2 shows "Ours" performing better than cross-entropy with 0% noise. This suggests that the proposed method is adding something beyond noise robustness to the training process, or that the baselines are not strong enough. Please consider introducing a justification or discussion of this observation in the paper.
  • Along the same lines, it would be valuable to include the no-noise results in Table 3.
  • The contribution is focused on noise detection and loss weighting. To make this a more robust contribution, the noise detection approach should be tested with other types of label-noise training strategies. The paper only evaluates standard cross-entropy-guided training, while the literature on label noise includes a diversity of loss corrections and sample-weighting approaches. This would help illustrate the generalization capabilities of the proposed noise detection approach.

Presentation weaknesses:

  • Figure 2: Review the caption for consistency with the images. The yy and y^\hat{y} in the caption do not appear in the images. These seem to correspond to "Original" and "Baseline", respectively. Please specify how the noise was introduced in the segmentation case for this example.
  • Figure 3: The caption should be self-contained. The illustration should be explained. If this is properly introduced and described, it is a valuable element in the paper, since it would help the reader understand the role of the introduced margin.
  • Figures 2 and 3 are not mentioned in the text. Figures should be introduced in the main text as a way to guide the reader and facilitate the understanding of the presented visualizations/results.
  • Subsection 3.2: gθg_{\theta} is not defined or introduced.
Questions for authors
  • How sensitive is the proposed supervision-reliability metric to the choice of summary statistics used to construct the descriptors? Are all of the selected moments necessary for the reported performance?
  • Can the authors clarify why the proposed method outperforms standard cross-entropy even in the 0% noise setting? Does the reliability weighting provide an additional regularization effect beyond robustness to label noise?
  • How do the authors expect the proposed approach to behave under more realistic or naturally occurring label noise, where the corruption pattern may not follow the synthetic noise models used in the experiments?
  • In the segmentation setting, how does aggregating statistics across spatial locations affect the reliability estimate and prototype construction, particularly when noisy labels are spatially correlated?
Reviewer_ewnAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an activation-guided statistics mechanism for noisy learning. It constructs activation descriptors from sparsity, mean, standard deviation, skewness, and kurtosis, and downweights supervision when a descriptor is closer to a competing class prototype than to the prototype of its observed label. Experiments on image classification and semantic segmentation, covering synthetic and real-world label noise as well as CNN and ViT architectures, demonstrate robustness gains. However, the evidence does not clearly establish whether these improvements can be attributed specifically to the proposed activation-distribution signal.

Critical strengths

The paper proposes a signal from activation distributions for assessing sample reliability, providing a more effective criterion than commonly used confidence and loss alternatives.

The paper provides a broad experimental evaluation spanning image classification and semantic segmentation, synthetic and real-world label noise, and CNN and ViT architectures.

Critical weaknesses

The theoretical analysis does not fully characterize the implemented method. Theorem 1 assumes separated class prototypes but does not show that activation statistics satisfy this condition better than ordinary features. Proposition 2 treats population prototypes as fixed while the feature extractor changes during training, and its EMA error bound is not connected to the normalized prototypes used for distance computation. These differences leave a gap between the theoretical analysis and the method as implemented.

Equation (2) gives full weight to samples below the confidence threshold, even when their activation descriptors favor another class. The paper does not report what fraction of corrupted samples is missed under the fixed threshold δ=0.8, or how the results change with different thresholds.

The method uses a single feature layer, but its location is not specified precisely. Appendix A.6.1 refers to the "third feature-extraction layer of the backbone" for classification and the "spatial latent representation immediately before the final prediction head" for segmentation, without identifying the exact stage or block, particularly for ViT. The authors should specify the feature source for each architecture and assess whether the results are sensitive to this choice.

Table 9 shows that progressively adding activation statistics improves performance, but it does not test skewness and kurtosis separately or compare other combinations. The choice of these five statistics needs stronger justification.

The prediction labels and confidence scores in Figure 5 appear to be repeated across different images. For example, panels (a), (k), and (u) display identical baseline and proposed-method annotations. Please check the figure and ensure that each annotation corresponds to its image.

Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.