acceptodds
Under review as a conference paper at ICLR 2027

Same Fit, Different Damage: Learning Order and Functional Dependence in Noisy-Label Memorization

Abstract

cause very different levels of semantic damage. We show that the difference is closely related to when individual errors are learned and how their final wrong predictions are functionally realized. To study this phenomenon, we introduce relational consistency, which describes how strongly an incorrect target is supported by surrounding training examples. Using a controlled intervention that preserves the corrupted examples and class transition statistics while changing only the assignment of wrong targets, we find that highly compatible errors are learned 27.64 epochs earlier on average than random matched errors under strong augmentation. Mid-training gradient alignment further predicts this ordering. Importantly, strong augmentation does not prevent eventual memorization. When both models have fitted 98% of the incorrect labels, the strongly augmented model still achieves 23.1 percentage points higher linear-probe accuracy for the true labels. Additional controls show that errors learned later preserve stronger true-label structure at comparable levels of noise fitting. Suffix rollback experiments further reveal that these later errors depend more strongly on deep network changes formed around their learning events, and removing these changes often restores the true prediction. These patterns persist across datasets, architectures, noise rates, and human annotation noise. Our results reveal that near-identical levels of noise fitting can coexist with substantially different semantic costs, systematically linked to learning order and the functional realization of individual errors.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Learning with Noisy Labels, Memorization, Learning Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7USbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies why nearly identical noisy-label fitting can incur different semantic costs. It defines sample-level stable learning events and model-level noise-fitting events. A target-pairing intervention preserves corrupted examples, wrong-target multisets, and class-transition counts to test how example–target relations affect learning order. Gradient alignment serves as an optimization diagnostic, while event-matched linear probes and neighborhood purity assess semantic preservation. The authors also keep example–wrong-target pairs fixed while changing supervision timing, and test functional dependence by retaining the final shallow prefix and rolling the deep suffix back around learning events. Strong augmentation does not prevent eventual noise fitting, but preserves more true-class structure. Later-learned errors depend more on late deep changes and more often recover the true class after rollback.

Critical strengths

The pairing intervention fixes noise rate, corrupted positions, and transition statistics, avoiding changes in error quantity as an explanation for relational effects. Matching noise-fitting events avoids relying only on same-epoch comparisons. At 98% noise fitting, true-label probe accuracies of 48.4% versus 71.5% clearly show that fitting alone does not characterize semantic damage. The fixed-pair crossover of early and delayed supervision provides additional intervention evidence beyond augmentation correlations.

Rollback is compared before and after individual learning events, and the analysis measures whether erased errors recover the true class. Final-margin matching, multivariate analysis, matched late-clean examples, and extensions to ViT and human noise provide complementary checks. I particularly appreciate the explicit restriction to intervention-defined functional suffix dependence, rather than claiming that erroneous information is uniquely stored in deep layers.

Critical weaknesses
  1. Equal total active-supervision epochs per example are an important control. However, the early and delayed windows occupy different portions of a cosine learning-rate schedule, and the subsets are evaluated at their own first q=0.98 checkpoints. Effective update magnitude, already learned representations, and total training time can therefore change together. The results support a contribution of when erroneous supervision is applied, not unique attribution to stable learning time itself. The authors already acknowledge this distinction; the wording should remain narrow, with sensitivity to learning rate or effective updates strengthening the interpretation.

  2. Combining the final prefix with an earlier suffix can be affected by prefix–suffix co-adaptation, normalization state, and the scale of parameter changes. Pre/post-event comparisons and true-class recovery reduce the concern that arbitrary earlier parameters simply disrupt predictions, while the ViT extension weakens a BatchNorm-only explanation. Nevertheless, the source and recalibration of running statistics, and controls matched by rollback lag or parameter-change magnitude, should be explicit. These are unresolved interpretation conditions, not established implementation errors. The conclusion should remain functional dependence under the specified hybrid-network intervention rather than unique localization or deletion of error memory.

Questions for authors

Q1. Does the fixed-pair timing effect persist with a constant learning rate or approximately matched effective update magnitude? Can you show both common-epoch and common-noise-fit views, including each subgroup’s q=0.98 time and cumulative active supervision before it? Q2. How are normalization parameters and running statistics handled during suffix rollback? Can matched-lag or approximately matched-parameter-change controls distinguish event specificity from cross-time module mismatch? Q3. Please add Table 3’s column headings and specify probe train/test splits and evaluation populations: which results concern all test examples, and which concern corrupted training examples? This would clarify distinctions among decodability, local geometry, and generalization.

Reviewer_E4FCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper establish a procedure to try to answer two questions: Form the paper: "what organizes the learning order of incorrect labels?" and "which training-time network changes the final wrong prediction functionally depend on?" or "why does learning order still affect the semantic cost of memorization?" The work studies the relationship between the order in which noise labels are learned and why the damage produced in the learned semantics is different depending on this order.

The conclusion of this work is that the order of learning is related to the class support, or how close the mislabeled examples are to examples of the real class. A stronger support is associated with an earlier learning and with low support examples are learned later. The semantics preservation is dependent on when and how the errors are learned in the network.

Critical strengths

None

Critical weaknesses

As far as I understand the paper creates a set of example-noise labels that by means of combination of the example to the wrong target generate three relationships between them. The low support, where wrong-labeled examples are distant from the target, so the semantic meaning is disimilar. High support, where the wrong labeled examples are matched with a more close target set, so they could support each other and a random selection where anything is randomly matched. According to the theory in the paper I was expecting in the experimental results, to find out that the low matched set and the high matched set have the biggest difference in epochs learned, however is the random matched set the one that generates the biggest difference in epochs needed to fit the random labels. I think this contradicts the claim that the support is what tells the order of learning, since low support should be the last to be learnt.

I think the paper also does not provide enough evidence to rule out the influence of the augmentation in the learned representation. In figure 3 the support effect between high and random, seems to be really low when no augmentation is used, it seems to suggest that when no augmentation is used, or too late applied, after the first phase of training, the effect of the support is almost gone.

Section 4.5 focus on the prediction of the learning order. The question here, is if the learning order is a consequence of the input data structure, changed by the example-wrong label assignations, that is also different under augmentation, rather than the other way around. Since the learning order cannot be predicted, but just observed, it could be the first observation possible of the representation that the network has encoded, in that case, it would not be a predictor but a reading of the representation.

Questions for authors

Could the authors provide the sets used in CIFAR 10 as low support, and high support? Can be the difference between low and random explained somehow with the support theory?

How can we be sure that the learning order is a predictor and not just a reading of the representation?

Figure 2 could be enhanced so the reader gets a better understanding of the experimental procedure, since in that figure the rollback is missing and the difference between low support and high support is in reality the difference between high support and random support.

Reviewer_VJ4ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the semantic damage of deep neural networks when learning with noisy labels. The authors study the order differences in learning noisy examples and propose using relational consistency to analyze them. Experiments on the CIFAR-10 dataset with ResNet-18 are conducted to analyze the representation during learning. The authors found that strong data augmentation can amplify these learning-order differences, and semantic information preservation is correlated with the learning order of noisy examples.

Critical strengths
  1. The study problem is interesting and meaningful. Although the deep network can memorize the same datasets with the same noise level, the levels of semantic damage can be different.
  2. The authors design controlled experiments to analyze how differences in learning order affect the semantic damage.
Critical weaknesses
  1. This paper finds that strong augmentation and other interventions can change the learning order and semantic preservation. It can indicate that there is a correlation between learning order and semantic preservation, but it cannot prove that learning order causes semantic preservation.
  2. The experiments are conducted on CIFAR-10 with ResNet-18. Whether the phenomenon exists in larger datasets and networks is unclear.
  3. Some key concepts in the paper are not explained clearly or defined when they first occur, for example, learning-order differences and semantic damage.
  4. The font sizes in the figures are small, making them hard to read.
Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.