acceptodds
Under review as a conference paper at ICLR 2027

Label Noise Forgotten in Loss Persists in Complexity

Abstract

Training loss can decline as a network memorizes incorrect labels. What remains observable about label noise after its initial loss response subsides? We identify an asymmetry in training trajectories: after controlled label-noise injections, loss returns toward its pre-injection course, while neural persistence, a statistic of normalized network weights, remains elevated through the recorded horizon. Matched clean additions do not produce the same sustained elevation. Separately, clean and noisy per-sample losses overlap strongly on real-world image data with injected label noise, despite clear separation on a simple synthetic dataset. We study a geometric mechanism for the complexity cost of fitting noise: maintaining a fixed positive output separation across nearby conflicting inputs requires increasing functional complexity. Explicit calibration and trajectory conditions connect this barrier to measured persistence and label recovery. We use the persistent signal in our Concept Recovery Test (CRT), combining classwise label-fit weights with a global complexity-level gate to aggregate training predictions. We prove finite-time recovery when the gate retains enough clean predictions, accounting for early errors, reference estimation, and the memorized tail. Across five benchmarks and four comparison methods, gated CRT improves rectification and detection over both the underlying method and loss-only CRT, and remains effective as oracle-tuned single-epoch loss thresholds deteriorate. These results show how a sustained structural response can complement loss-based evidence for label recovery.

Primary area:
learning theory
Keywords:
Label Noise Detection, Model Complexity, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hjw7OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether label noise remains observable after a network has learned to fit corrupted labels. Controlled experiments on MNIST show that noise injections produce a temporary increase in training loss but a sustained increase in neural persistence, a topological statistic computed from normalized network weights. Matched additions of clean examples do not produce the same sustained increase. The authors motivate this observation through a geometric argument: confidently assigning conflicting labels to nearby inputs requires increasing functional complexity. They then propose complexity-gated Concept Recovery Test (CRT), which aggregates predictions across training epochs using two weights. Experiments evaluate CRT as an augmentation to four methods across image and financial datasets, including naturally occurring annotation noise.

Critical strengths
  1. The authors present an interesting empirical observation: label noise can leave a persistent structural signature in network weights even after its effect on training loss subsides.

  2. The authors provide meaningful theoretical analysis of the complexity required to fit local label conflicts and the conditions under which complexity-gated prediction aggregation can recover clean labels.

  3. The numerical results demonstrate consistent improvements in label-noise detection and rectification across multiple datasets and baseline methods.

Critical weaknesses
  1. The calibration assumption in “Measured Persistence” may not hold in practice. Neural persistence can change under function-preserving reparameterizations, and the paper does not empirically validate its assumed relationship with functional complexity. This weakens the connection between the complexity-barrier analysis and the measured signal used by the proposed method.

  2. The architectural evaluation is limited, particularly by the reliance on MLP heads trained on frozen ResNet34 features in the MNIST/CIFAR experiments. Additional experiments with modern backbones, such as ViTs, and end-to-end fine-tuning would help establish whether the proposed persistence-based gate remains effective when representations evolve during training.

  3. While the paper reports aggregate rectification and detection metrics, it does not explicitly analyze prediction changes relative to the final model. Reporting how many final-model errors CRT corrects, and how many initially correct predictions it turns into errors, would clarify where the gains arise and whether they come at the expense of previously correct predictions.

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hjw7 (Part 1/2)

We thank the reviewer for the careful assessment and for identifying calibration, evolving representations, and individual prediction changes as the key issues. We have updated the paper with direct measured-score recovery conditions, calibration and reparameterization experiments, end-to-end training under symmetric and human annotation noise, and paired counts of repaired and newly introduced errors. The revision also makes the scope of the measured NP statistic and the conditional nature of the guarantees explicit. We address all three weaknesses below, in the review's order.

Critical weakness 1: calibration and reparameterization. We agree with the reviewer that function-preserving reparameterization prevents a universal identification of NP with functional complexity. Proposition 4 explicitly shows that neuron-wise ReLU rescaling can change NP without changing the function. Proposition 5 therefore states a conditional transfer requiring functional separation to exceed twice the combined calibration error. In 12 new one-dimensional ReLU runs, chronological held-out affine residuals reach 28.2–66.6% of the maximum held-out exact probability Lipschitz seminorm. Rescaling hidden weights/biases by 10 and output weights by 1/10 changes NP by 0.0032–0.0623 while preserving probabilities (maximum numerical discrepancy 1.4×10−141.4\times10^{-14}). These measurements support treating calibration as conditional.

The new Proposition 1 analyzes the measured penalties directly, without affine functional calibration. Let AyA_y be fitted vote mass and μy,Ry\mu_y,R_y the fitted-weighted mean and range of penalties qk=(Mk−M^ref)+q_k=(M_k-\widehat M_{\mathrm{ref}})_+. For Ac>0A_c>0, recovery of target cc is guaranteed when

\> \log\left(\frac{A_y}{A_c}\right)$$ for every positive-mass rival. This permits nonmonotone NP, overlapping penalty ranges, and intermittent target predictions. Favorable alignment remains a trajectory condition. Appendix B.11 gives the proof and successful/harmful-gating examples. Appendix B.12 adds decision stability: at fixed predictions, fitted weights, and sharpness, score perturbations bounded by $u$ and common-reference perturbations bounded by $v$ preserve a unique winner whose log score margin exceeds $\lambda(2u+v)$. **Positive affine changes of score units preserve the complete reference/median rule and votes.** Neuron-wise rescaling need not induce such a score transformation, and decision stability itself does not establish correctness. **Critical weakness 2: evolving representations.** The main setup now explicitly identifies the original MNIST/CIFAR trajectories as MLP heads on frozen ResNet34 features. **The revision distinguishes the head's feature-space bound from the full predictor's raw-input bound.** Remark 2 states that, for $f_k=h_k\circ\phi_k$ and pairwise output separation $\mu$, $$\operatorname{Lip}(h_k)\ge \frac{\mu}{\|\phi_k(x)-\phi_k(x')\|_2}, \qquad \operatorname{Lip}(f_k)\ge\frac{\mu}{\|x-x'\|_2}.$$ Feature learning can relax the head bound by increasing feature distance, while the full predictor still obeys the raw-input bound. Neither bound identifies NP with a functional seminorm. Architectural claims now explicitly follow the evaluated training protocols. We added six CIFAR-10 ResNet18 runs from random initialization, with all backbone and head weights trained under symmetric 40% and human `worse_label` noise. The gate measures exact NP of the three linear head layers. NP-level rectification error is $7.70\pm0.51\%$ and $16.52\pm0.37\%$, versus $7.28\pm0.15\%$ and $14.74\pm0.11\%$ for uniform voting and $7.40\pm0.18\%$ and $16.29\pm0.07\%$ for loss-only CRT. **The specified NP-level gate does not improve recovery in these end-to-end experiments.** The selector is active in 2/3 and 1/3 seeds, so fallback alone does not explain the outcome. This study addresses evolving representations with NP measured on the trained head; its scope does not extend to whole-network NP.
AuthorsOpenReview ↗

Response to Reviewer Hjw7 (Part 2/2)

Critical weakness 3: repaired errors versus new errors. We now directly count all four terminal-to-CRT transitions on the same examples: correct →\to correct, correct →\to wrong, wrong →\to correct, and wrong →\to wrong. With repaired-error fraction rr and newly introduced-error fraction hh, Appendix C.2 gives the exact identity

eT−eCRT=r−h.e_T-e_{\mathrm{CRT}}=r-h.

In the end-to-end study, NP-level voting repairs 11,590.3±494.911{,}590.3\pm494.9 terminal errors and harms 1,105.0±559.71{,}105.0\pm559.7 previously correct predictions per 50,000 examples under symmetric noise. Under human noise it repairs 9,627.7±40.19{,}627.7\pm40.1 and harms 609.3±153.4609.3\pm153.4. These are direct same-example counts. The full table also separates originally clean and noisy strata and reports every comparison method. Repairs exceed new errors in both conditions, while uniform voting repairs more net errors and has lower aggregate error.

The recovery guarantees are sample-conditional and permit harm elsewhere. Appendix B.11 explicitly demonstrates how excessive sharpness can overturn a correct aggregate despite favorable mean penalty ordering. The general response details the supplementary-metric correction; Appendix C.2 distinguishes detection, rectification, and terminal prediction changes.

Reviewer_PtcjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Concept Recovery Test (CRT) as an approach to label-noise learning. It leverages a novel observation about an asymmetry in the behaviour of the loss value and model complexity when the model is exposed to noisy samples: while the loss value increases but eventually returns to its natural trajectory, model complexity remains higher than the baseline. The authors suggest that, for the model to classify two nearby samples as clean and noisy respectively, it needs to increase its complexity. They call this the complexity barrier, which remains high throughout training. The evaluation setup explores symmetric and asymmetric noise, as well as real-world noise, across several datasets, including MNIST, CIFAR-10/100, and Clothing-1M.

Critical strengths
  • The paper provides a valuable contribution in terms of label-noise learning: the asymmetric behaviour of model loss and model complexity. This insight is well presented and clearly motivates the proposed method. Additionally, the provided results are competitive and surpass the included state-of-the-art competing methods.
  • The experimental study is comprehensive and illustrates the generalization of the approach across datasets and types of noise.
  • The mathematical framework provided theoretically supports the claims and justifies the proposed approach.
Critical weaknesses
  • The complexity barrier theoretically supports the observed increase in network complexity, and the suggested intuition that "fitting both labels requires a sharper transition" is consistent with the claims of the paper. However, the effect of model size, which reflects model learning capacity/expressivity, should be taken into consideration. While the observations presented in Subsection 3.1 seem to be model-size/model-architecture agnostic, a discussion of the effect that model size has, with respect to task complexity, on this observation would help clarify the generalization capabilities and limitations of the approach. The paper shows results across several architectures, but the main visualizations and experiments illustrating the neural complexity dynamics are based on small models and small datasets. This is particularly relevant for this study, since the proposed approach is based on the claim that noisy samples added to the training process increase the neural complexity of a model. This effect could be less noticeable when considering standard-sized models that are often overparameterized.
  • The paper provides good insights and a strong framework around an interesting observed mechanism: neural persistence/complexity increases and remains high after label noise is introduced. However, the mechanism by which the authors propose to improve label-noise training is less clear. The two proposed gates are intuitive: the loss gate assesses how well a prediction fits its label, while the complexity gate assesses the state of the model in terms of overfitting to noise. The first acts at the sample level, whereas the second acts at the model level. The interaction between the two is not entirely clear. The paper would benefit from a more intuitive explanation of the role of the complexity gate during the training process.
  • The temporal part of the model seems to require different configurations for the different types of models (MLPs, GNNs, and ResNet models). This should be discussed further. Please consider including the criteria used to design these configurations in the main text.

Minor and writing weaknesses:

  • "Detector" is not formally defined or introduced.
  • The F1 score and AUROC metrics refer to the ability of the loss to identify noisy samples. This is not explicitly mentioned in the text. Please consider reviewing the text so that this is explained and the reader can more easily follow the paper.
  • "Augmentation" for the proposed CRT is not defined or described.
  • It is not clear what Subsection 2.2 and Figure 3 are intended to motivate. Please review the text to ensure that the context is complete and the main motivation is thoroughly explained.
  • Is the reported "rectification error" in the tables the same quantity as the "accuracy" or "prediction error" usually reported in state-of-the-art label-noise papers? The description should be reviewed so that it is clear what values are being reported.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Ptcj (Part 1/2)

We thank the reviewer for the constructive assessment and for recognizing the loss–complexity observation and the motivation for temporal label recovery. We have updated the paper to clarify the effect of model size, the roles of the two gates, architecture-dependent reference selection, and the definitions of detection and rectification. The revision also adds width and end-to-end studies, sensitivity analyses, and theoretical stability results. We address all three critical weaknesses and all five minor/writing points below, in the review's order.

Critical weakness 1: model size and overparameterization. The functional lower bound is independent of parameter count, while the magnitude of the measured NP gap can depend on architecture and width. Maintaining output separation μ\mu across inputs at distance rr requires Hölder seminorm at least μ/rα\mu/r^\alpha. Width can change how weights realize this variation. Section 3.1 and Appendix C.2 now distinguish functional necessity from architecture-dependent structural measurement. Remark 2 additionally explains how learned feature distances can relax the head's bound while the full predictor still obeys the raw-input bound.

We added a fixed-feature width sweep with 35,242, 74,570, and 165,514 trainable parameters, 40% corruption, and three seeds per width. At the fixed 100-epoch horizon, observed-label error falls from 31.15±0.40%31.15\pm0.40\% to 9.20±0.39%9.20\pm0.39\% to 0.10±0.05%0.10\pm0.05\%, while final NP is 0.5895±0.02270.5895\pm0.0227, 0.6744±0.03460.6744\pm0.0346, and 0.6654±0.02820.6654\pm0.0282. Increased fitting capacity does not imply a monotone NP ordering. At width 128, paired noisy-minus-clean NP is negative at both 5% and 40% noise. Section 5.5 and Appendix D report these limits explicitly.

Critical weakness 2: intuitive roles of the gates. The fitted-loss gate measures classwise fit; the complexity gate selects training epochs through one shared structural score. We clarify that zk(y)z_k(y) is observed-class classification accuracy, and example xx receives zk(y^k(x))z_k(\hat y_k(x)). This downweights predictions from poorly fitted classes. The global complexity gate downweights epochs above the structural reference; each example's prediction history determines which labels receive the resulting votes. These complementary roles motivate the product, whose usefulness depends on trajectory alignment. Table 1 isolates the roles and compares level with increment gating: when late increments vanish, the rate gate returns to one, whereas a positive level excess continues to suppress votes. Proposition 1 quantifies the target/rival vote balance required for success.

Critical weakness 3: temporal settings across architectures. Section 4 and Appendix C.1 now give the complete reference-selection rule, architecture-specific smoothing, and fallback conventions. They specify a 10-epoch window, the empirical 25th percentile of signed increments, raw MLP/GNN increments, and a ResNet exponential moving average with half-life 5 epochs. The reference averages raw levels in the window. For positive signed median post-window excess dd, λ=log⁡(10)/d\lambda=\log(10)/d; otherwise, or without a valid window/post-window interval, CRT uses loss-only weighting. Tie and zero-vote conventions are explicit. The most localized sweep setting identifies its auxiliary low-noise reference.

The original sensitivity study varies windows over {5,10,20}\{5,10,20\} and sharpness over [λ∗/4,4λ∗][\lambda^*/4,4\lambda^*], with error within 0.15 points of the default in that setting. Appendix B.12 proves positive-affine score-unit invariance and conditional decision stability; these do not imply architecture-invariant tuning.

The new cross-entropy feature study exposes this distinction: the primary window-10 NP selector falls back in all 15 runs. The separately specified window-5 analysis activates it in 3/15 runs, with all seeds retained in the reported averages. Freezing the epoch-50 selector through epoch 100 preserves its fallback in all 15 runs. Appendix C.1 also quantifies the selector's requirement for temporally clustered low increments. We report uniform voting and every gate variant, including unsuccessful comparisons, and identify which outputs use loss-only fallback.

In six end-to-end ResNet18 runs, the primary NP selector is active in 2/3 symmetric-noise and 1/3 human-noise seeds. NP-level rectification error is 7.70±0.51%7.70\pm0.51\% and 16.52±0.37%16.52\pm0.37\%, compared with 7.28±0.15%7.28\pm0.15\% and 14.74±0.11%14.74\pm0.11\% for uniform voting. Window 5 changes the human-noise NP-level result to 15.81±0.77%15.81\pm0.77\% but remains a declared sensitivity, not a replacement for the primary setting. These outcomes show that architecture-specific reference behavior remains material.

AuthorsOpenReview ↗

Response to Reviewer Ptcj (Part 2/2)

Minor/writing point 1: “detector.” A detector makes a binary decision about whether an observed label is incorrect. Section 5 and Appendix C.2 now define this explicitly. CRT first recovers yi∗^\hat {y_i^*} and then flags 1[y^i∗≠yi]\mathbf{1}[\hat y_i^*\ne y_i].

Minor/writing point 2: F1 and AUROC. Corrupted observed labels are the positive class, and the revision distinguishes loss-based discrimination from CRT's disagreement decision. In Section 5.2/Appendix C.4, loss AUROC ranks corrupted versus clean examples and its oracle optimizes epoch/direction. The scalar-loss F1 oracle separately optimizes epoch, threshold, and direction on the fixed NLS trajectory. CRT F1 evaluates recovered-label disagreement. The comparator covers single-epoch scalar-loss thresholds. We also withdrew incompatible supplementary detection summaries following the consistency audit described in the general response; benchmark tables report rectification error.

Minor/writing point 3: “augmentation.” CRT augmentation adds temporal label aggregation and the disagreement decision to an underlying model trajectory. Section 4 now defines this term explicitly. Both loss-only and gated CRT are our variants; the latter multiplies the classwise fitted weight by the global complexity factor.

Minor/writing point 4: Section 2.2/Figure 3. Figure 3 motivates using information beyond a single per-example loss value. The revised explanation distinguishes the aggregate noise-intervention response in Section 2.1 from per-example loss separability in Section 2.2. With the same three-layer MLP, the synthetic populations separate early, whereas real-data trajectories overlap throughout 500 epochs. This observation motivates the subsequent comparison with single-epoch scalar-loss thresholds; it does not establish failure of all loss-based or temporal methods.

Minor/writing point 5: rectification versus prediction error. Rectification error measures recovered training labels against clean reference labels; held-out prediction error measures predictions on a separate test population. Sections 5 and C.2 define

eCRT=1N∑i=1N1[y^i∗≠yi∗].e_{\mathrm{CRT}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat y_i^*\ne y_i^*].

Rectification gains alone do not establish downstream accuracy gains. In the end-to-end study we report both quantities without retraining: NP-level temporal-ensemble test error is 10.88±0.50%10.88\pm0.50\% and 16.42±0.41%16.42\pm0.41\%, while uniform voting gives 10.71±0.38%10.71\pm0.38\% and 15.26±0.12%15.26\pm0.12\% under symmetric/human noise. Appendix C.2 gives

NeCRT=FP+FN+W,Ne_{\mathrm{CRT}}=\mathrm{FP}+\mathrm{FN}+W,

where WW counts correctly flagged corruptions assigned another wrong class. Detection and correct replacement are distinct.

Reviewer_HP7cOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether label noise leaves a persistent signal in neural networks after its initial effect on training loss diminishes. Through controlled noise-injection experiments, the authors observe that training loss gradually returns toward its pre-injection trajectory, while neural persistence, a topology-based measure of network weight structure, remains elevated. Motivated by this observation, the paper develops a theoretical analysis showing that fitting nearby samples with conflicting labels under a fixed positive output separation requires increased functional complexity. Additional trajectory and calibration assumptions are introduced to connect this theoretical result to measured neural persistence. Building on these observations, the authors propose a complexity-gated Concept Recovery Test (CRT), which aggregates predictions across training epochs using classwise label-fit weights and a global complexity-based gate to suppress unreliable early predictions and later noise memorization. The paper also provides a finite-time label recovery guarantee under specified assumptions. Experiments across five benchmarks and four baseline methods demonstrate improvements in label rectification and noise detection over the original methods and a loss-only CRT variant, particularly when single-epoch loss-based discrimination becomes less reliable.

Critical strengths
  1. The paper investigates a phenomenon where the effects of label noise on training loss diminish while neural persistence remains elevated in controlled experiments.

  2. The proposed CRT combines classwise label-fit information with a complexity-based gate to reduce the influence of early unreliable predictions and later noise memorization.

  3. Extensive experiments across multiple datasets, noise settings, and baseline methods demonstrate consistent improvements, supported by ablation studies.

Critical weaknesses
  1. Weak theoretical connection between neural persistence and functional complexity. The theoretical complexity barrier is established for functional complexity measures, whereas the proposed method relies on neural persistence (NP), a parameter-based topological statistic. The connection between them depends on an additional calibration assumption that is neither established for the evaluated neural networks nor empirically validated. Although the paper provides empirical motivation for using NP, the lack of evidence supporting this assumption leaves a substantial gap between the theoretical analysis and the proposed method.

  2. The explicit lower bound showing that fitting nearby inputs with conflicting labels under a fixed positive output separation requires higher Lipschitz or Hölder complexity follows directly from the definitions of these measures. Moreover, prior work has already investigated the relationship between label noise and neural network Lipschitz complexity [1]. The authors should clarify what additional nontrivial theoretical insights their complexity barrier provides beyond these basic properties and existing studies.

  3. The phenomenon that neural networks learn clean patterns before memorizing noisy labels has already been theoretically investigated in prior work [2, 3]. The proposed analysis assumes different convergence rates for clean and noise modes to characterize a similar temporal separation. However, it is unclear what additional theoretical insights this analysis provides beyond existing studies. Furthermore, the assumed differences in convergence rates and spatial complexity between clean and noise modes are not justified for the evaluated networks.

  4. Different parts of the theoretical analysis rely on restrictive conditions, including monotonically nondecreasing functional complexity, fixed positive output separation between nearby conflicting samples, and the alignment of sample-specific clean-prediction windows with a shared global complexity gate. It remains unclear to what extent these conditions hold in practical neural network training. Furthermore, the reversed NP ordering observed under low noise rates and early stopping raises concerns about the generality of the proposed mechanism. The recovery guarantee also targets the clean-model concept, which does not necessarily coincide with the ground-truth label.

  5. Although the paper evaluates Clothing1M and includes CIFAR-100N in the appendix, a substantial portion of the experiments relies on synthetic label noise. Since real-world annotation errors often exhibit complex instance-dependent patterns [4], additional evaluations on CIFAR-10N and other human-annotated noise settings would strengthen the empirical evidence. Furthermore, the main MNIST/CIFAR label-rectification experiments use MLP heads on pretrained features. Evaluations with end-to-end representation learning would help establish the broader applicability of the proposed method.

References

[1] Khromov, G., and Singh, S. P. Some Fundamental Aspects about Lipschitz Continuity of Neural Networks. ICLR, 2024.

[2] Han, A., Huang, W., Zhou, Z., Niu, G., Chen, W., Yan, J., Takeda, A., and Suzuki, T. On the Role of Label Noise in the Feature Learning Process. ICML, 2025.

[3] Liu, S., Niles-Weed, J., Razavian, N., and Fernandez-Granda, C. Early-Learning Regularization Prevents Memorization of Noisy Labels. NeurIPS, 2020.

[4] Wei, J., Zhu, Z., Cheng, H., Liu, T., Niu, G., and Liu, Y. Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR, 2022.

Questions for authors

See weaknesses. Given that the paper derives a functional complexity barrier using Hölder seminorms (including the Lipschitz seminorm), why does the proposed method use neural persistence (NP) rather than directly estimating these functional complexity measures?

2 comments
AuthorsOpenReview ↗

Response to Reviewer HP7c (Part 1/2)

We thank the reviewer for the detailed assessment and for emphasizing the distinction between functional complexity, neural persistence (NP), and label recovery. We have updated the paper with direct measured-score recovery guarantees, a maintained-confidence result, expanded discussion of prior work, and new calibration, model-width, and end-to-end experiments. The revised manuscript states the conditions and empirical limits of the structural interpretation explicitly. We address each of the five weaknesses and the additional question below, in the review's order.

Critical weakness 1: functional complexity–NP calibration. We agree with the reviewer that the functional barrier alone does not establish an NP barrier. Proposition 4 gives a function-preserving ReLU-rescaling counterexample; Proposition 5 states the conditional transfer and its required calibration margin. We now measure an exact probability-output Lipschitz seminorm in 12 fixed one-dimensional ReLU runs. Affine fits on the first half of each trajectory have maximum second-half residuals of 28.2–66.6% of the maximum second-half seminorm, despite positive fitted slopes. Thus correlation alone does not justify a tight calibration.

The new Proposition 1 establishes recovery conditions directly for the implemented penalties, without affine functional calibration. Let AyA_y be fitted vote mass, and μy,Ry\mu_y,R_y the fitted-weighted mean and range of penalties qk=(Mk−M^ref)+q_k=(M_k-\widehat M_{\mathrm{ref}})_+ for label yy. For target mass Ac>0A_c>0 and a positive-mass rival yy,

\le \frac{A_y}{A_c} \exp\left[-\lambda(\mu_y-\mu_c)+\frac{\lambda^2R_y^2}{8}\right].$$ Recovery follows when every positive-mass rival's bound is below one; zero-mass rivals receive no votes. This permits nonmonotone NP, overlapping ranges, and intermittent correct predictions. Favorable penalty alignment must still outweigh rival mass and dispersion. Appendix B.11 proves the result and gives examples of successful and harmful gating. **Critical weakness 2: contribution beyond the Hölder inequality.** We now present $\mu/r^\alpha$ as the geometric premise and discuss Khromov and Singh (ICLR 2024). **The additional theoretical contribution concerns temporal recovery, including vote-mass balance, feasible gate sharpness, and stability.** Theorem 2 compares retained target votes with early incorrect and late-tail mass; Corollary 5 requires one sharpness that suppresses the tail while preserving useful votes. Proposition 1 extends recovery beyond separated windows, and Proposition 11 quantifies stability to score/reference perturbations. These results address when a structural trajectory can support recovery beyond the pairwise inequality itself. **Critical weakness 3: prior clean-first/noise-later theory.** Related Work now explicitly discusses Han et al. (ICML 2025) and ELR (NeurIPS 2020). Our two-mode construction assumes a faster smooth mode and a slower localized mode. Under its stated spectral/spatial assumptions, Proposition 9 derives a unique prediction sign change and coexistence of a useful prediction window with persistent scalar-logit complexity excess. **We treat the spectral/spatial ordering as a model assumption; Proposition 1's recovery analysis is independent of this construction.** The subsequent vote analysis explains why a sustained level can suppress late votes after complexity increments become small.
AuthorsOpenReview ↗

Response to Reviewer HP7c (Part 2/2)

Critical weakness 4: assumptions and failure regimes. Proposition 7 establishes persistence of the functional lower bound under maintained confident fitting, without global complexity monotonicity. If a fixed conflicting pair at distance rr has both individual cross-entropies at most ℓ<log⁡2\ell<\log 2, its probability-output Hölder seminorm, using L1L^1 output distance, satisfies

∣fk∣α≥2(2e−ℓ−1)rα.|f_k|_\alpha \ge \frac{2(2e^{-\ell}-1)}{r^\alpha}.

Zero classification error or low aggregate loss alone does not imply this margin. In the exact study, three noisy neighboring pairs satisfy the maintained-confidence premise at recorded checkpoints despite nonzero total training error; all corresponding bounds hold. Remark 2 distinguishes fixed raw-input from changing feature coordinates.

The clean-trained target equals ground truth only where clean training is correct. Proposition 1 also permits reference clean labels as retrospective targets. We retain the low-noise/early-stopping NP reversal in Appendix C.5. When no valid window or positive median excess exists, CRT falls back to loss-only weighting. The guarantee is finite-horizon: bounded scores with fixed finite sharpness cannot suppress an indefinitely long memorized tail.

The new fixed-feature study further limits the structural claim: at width 128, final NP minus its paired clean control is −0.0532±0.0368-0.0532\pm0.0368 at 5% noise and −0.1042±0.0321-0.1042\pm0.0321 at 40% noise (three seeds). The primary NP selector falls back in all 15 width/noise runs. Section 5.5 reports these results explicitly: persistence ordering and successful automatic reference selection depend on the training protocol.

Critical weakness 5: natural noise and evolving representations. On Clothing1M, the retained rectification results show a 0.81–0.97-point reduction over loss-only CRT across four detectors; CIFAR-100N shows 3.35–4.35 points. These evaluate label recovery under natural annotation noise.

The main setup now explicitly identifies the original MNIST/CIFAR experiments as MLP heads on frozen ResNet34 features. We added six end-to-end CIFAR-10 ResNet18 runs from random initialization, with three seeds under exactly 40% symmetric corruption and three under CIFAR-10N's human worse_label annotations. All backbone and head weights train. NP is measured exactly for the three linear head matrices; the empirical scope is therefore head-based gating with evolving representations.

At epoch 100, uniform voting gives rectification error 7.28±0.15%7.28\pm0.15\% and 14.74±0.11%14.74\pm0.11\% under symmetric/human noise. NP-level voting gives 7.70±0.51%7.70\pm0.51\% and 16.52±0.37%16.52\pm0.37\%, versus 7.40±0.18%7.40\pm0.18\% and 16.29±0.07%16.29\pm0.07\% for loss-only CRT. The NP selector is active in 2/3 and 1/3 seeds. The specified NP gate does not improve this end-to-end setting; temporal voting itself remains substantially better than the terminal model. Remark 2 explains that learned feature distances can relax the head's bound, while the full predictor still obeys Lip⁡(f)≥μ/∥x−x′∥2\operatorname{Lip}(f)\ge\mu/\|x-x'\|_2. The general response details the supplementary-metric correction.

Question: why NP rather than a Lipschitz measure? NP is motivated by the controlled intervention evidence; the recovery rule accepts other structural scores. NP is computed from checkpoint weights and shows a sustained noise-injection response exceeding the matched clean-addition control. In the original NLS/MNIST 20%-noise ablation, weight displacement and path-norm increments give 4.10% and 4.15% rectification error, versus 3.98% for NP and 4.36% for loss-only CRT.

The new CIFAR-10 feature study directly compares a head-spectral-product gate: at width 128 and 40% noise, its error is 23.93±0.46%23.93\pm0.46\%, versus 27.12±0.39%27.12\pm0.39\% for NP's loss-only fallback and 23.27±0.32%23.27\pm0.32\% for uniform voting. In the end-to-end study the head-spectral selector falls back in all six runs, while NP is active in three; the NP-level output is worse than loss-only and uniform voting in both noise conditions. Thus NP is one measured structural choice; these comparisons establish neither its general superiority over Lipschitz estimates nor affine functional calibration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their careful assessments and constructive suggestions. We have substantially updated the paper, adding theoretical results, targeted experiments, and clearer explanations of the method and its scope. The reviewer-specific responses address every weakness point by point. The key updates are:

  • Recovery guarantees directly for measured trajectories. A new result establishes sufficient recovery conditions without affine calibration between neural persistence and functional complexity. It allows nonmonotone scores, overlapping penalty ranges, and intermittent correct predictions. We also provide stability guarantees for score and reference perturbations.

  • Weaker assumptions for persistent functional complexity. A new maintained-confidence result preserves the functional lower bound without requiring globally monotone complexity. The revision clarifies positive-margin requirements, learned-feature geometry, finite-horizon limits, and the relationship to prior work.

  • Thirty-three additional experimental runs. These comprise 12 exact calibration runs, 15 fixed-feature width/noise runs, and six end-to-end ResNet18 runs under symmetric and human annotation noise. They directly examine calibration, overparameterization, reference selection, and evolving representations. These studies also identify limits: calibration residuals can be substantial, reference selection can fail, and the specified NP gate does not improve upon uniform or loss-only voting in the new end-to-end setting.

  • Direct accounting of corrections and new errors. We now report paired prediction transitions on the same examples, separating repaired errors from previously correct predictions that become incorrect, including clean/noisy strata.

  • Clearer definitions and corrected reporting. We specify the gate-selection procedure and fallback rules, distinguish detection, label rectification, and held-out prediction error, and correct inconsistent supplementary detection summaries.

The revised paper makes the contribution and its conditions more precise: the controlled loss–persistence asymmetry motivates a structural trajectory signal, while its usefulness for recovery depends on alignment with informative predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.