- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether label noise remains observable after a network has learned to fit corrupted labels. Controlled experiments on MNIST show that noise injections produce a temporary increase in training loss but a sustained increase in neural persistence, a topological statistic computed from normalized network weights. Matched additions of clean examples do not produce the same sustained increase. The authors motivate this observation through a geometric argument: confidently assigning conflicting labels to nearby inputs requires increasing functional complexity. They then propose complexity-gated Concept Recovery Test (CRT), which aggregates predictions across training epochs using two weights. Experiments evaluate CRT as an augmentation to four methods across image and financial datasets, including naturally occurring annotation noise.
Critical strengths
-
The authors present an interesting empirical observation: label noise can leave a persistent structural signature in network weights even after its effect on training loss subsides.
-
The authors provide meaningful theoretical analysis of the complexity required to fit local label conflicts and the conditions under which complexity-gated prediction aggregation can recover clean labels.
-
The numerical results demonstrate consistent improvements in label-noise detection and rectification across multiple datasets and baseline methods.
Critical weaknesses
-
The calibration assumption in “Measured Persistence” may not hold in practice. Neural persistence can change under function-preserving reparameterizations, and the paper does not empirically validate its assumed relationship with functional complexity. This weakens the connection between the complexity-barrier analysis and the measured signal used by the proposed method.
-
The architectural evaluation is limited, particularly by the reliance on MLP heads trained on frozen ResNet34 features in the MNIST/CIFAR experiments. Additional experiments with modern backbones, such as ViTs, and end-to-end fine-tuning would help establish whether the proposed persistence-based gate remains effective when representations evolve during training.
-
While the paper reports aggregate rectification and detection metrics, it does not explicitly analyze prediction changes relative to the final model. Reporting how many final-model errors CRT corrects, and how many initially correct predictions it turns into errors, would clarify where the gains arise and whether they come at the expense of previously correct predictions.
Questions for authors
Please see the weaknesses.
2 comments
Response to Reviewer Hjw7 (Part 2/2)
Critical weakness 3: repaired errors versus new errors. We now directly count all four terminal-to-CRT transitions on the same examples: correct correct, correct wrong, wrong correct, and wrong wrong. With repaired-error fraction and newly introduced-error fraction , Appendix C.2 gives the exact identity
In the end-to-end study, NP-level voting repairs terminal errors and harms previously correct predictions per 50,000 examples under symmetric noise. Under human noise it repairs and harms . These are direct same-example counts. The full table also separates originally clean and noisy strata and reports every comparison method. Repairs exceed new errors in both conditions, while uniform voting repairs more net errors and has lower aggregate error.
The recovery guarantees are sample-conditional and permit harm elsewhere. Appendix B.11 explicitly demonstrates how excessive sharpness can overturn a correct aggregate despite favorable mean penalty ordering. The general response details the supplementary-metric correction; Appendix C.2 distinguishes detection, rectification, and terminal prediction changes.
Response to Reviewer Hjw7 (Part 1/2)
We thank the reviewer for the careful assessment and for identifying calibration, evolving representations, and individual prediction changes as the key issues. We have updated the paper with direct measured-score recovery conditions, calibration and reparameterization experiments, end-to-end training under symmetric and human annotation noise, and paired counts of repaired and newly introduced errors. The revision also makes the scope of the measured NP statistic and the conditional nature of the guarantees explicit. We address all three weaknesses below, in the review's order.
Critical weakness 1: calibration and reparameterization. We agree with the reviewer that function-preserving reparameterization prevents a universal identification of NP with functional complexity. Proposition 4 explicitly shows that neuron-wise ReLU rescaling can change NP without changing the function. Proposition 5 therefore states a conditional transfer requiring functional separation to exceed twice the combined calibration error. In 12 new one-dimensional ReLU runs, chronological held-out affine residuals reach 28.2–66.6% of the maximum held-out exact probability Lipschitz seminorm. Rescaling hidden weights/biases by 10 and output weights by 1/10 changes NP by 0.0032–0.0623 while preserving probabilities (maximum numerical discrepancy 1.4×10−14). These measurements support treating calibration as conditional.
The new Proposition 1 analyzes the measured penalties directly, without affine functional calibration. Let Ay be fitted vote mass and μy,Ry the fitted-weighted mean and range of penalties qk=(Mk−Mref)+. For Ac>0, recovery of target c is guaranteed when
\> \log\left(\frac{A_y}{A_c}\right)$$ for every positive-mass rival. This permits nonmonotone NP, overlapping penalty ranges, and intermittent target predictions. Favorable alignment remains a trajectory condition. Appendix B.11 gives the proof and successful/harmful-gating examples. Appendix B.12 adds decision stability: at fixed predictions, fitted weights, and sharpness, score perturbations bounded by $u$ and common-reference perturbations bounded by $v$ preserve a unique winner whose log score margin exceeds $\lambda(2u+v)$. **Positive affine changes of score units preserve the complete reference/median rule and votes.** Neuron-wise rescaling need not induce such a score transformation, and decision stability itself does not establish correctness. **Critical weakness 2: evolving representations.** The main setup now explicitly identifies the original MNIST/CIFAR trajectories as MLP heads on frozen ResNet34 features. **The revision distinguishes the head's feature-space bound from the full predictor's raw-input bound.** Remark 2 states that, for $f_k=h_k\circ\phi_k$ and pairwise output separation $\mu$, $$\operatorname{Lip}(h_k)\ge \frac{\mu}{\|\phi_k(x)-\phi_k(x')\|_2}, \qquad \operatorname{Lip}(f_k)\ge\frac{\mu}{\|x-x'\|_2}.$$ Feature learning can relax the head bound by increasing feature distance, while the full predictor still obeys the raw-input bound. Neither bound identifies NP with a functional seminorm. Architectural claims now explicitly follow the evaluated training protocols. We added six CIFAR-10 ResNet18 runs from random initialization, with all backbone and head weights trained under symmetric 40% and human `worse_label` noise. The gate measures exact NP of the three linear head layers. NP-level rectification error is $7.70\pm0.51\%$ and $16.52\pm0.37\%$, versus $7.28\pm0.15\%$ and $14.74\pm0.11\%$ for uniform voting and $7.40\pm0.18\%$ and $16.29\pm0.07\%$ for loss-only CRT. **The specified NP-level gate does not improve recovery in these end-to-end experiments.** The selector is active in 2/3 and 1/3 seeds, so fallback alone does not explain the outcome. This study addresses evolving representations with NP measured on the trained head; its scope does not extend to whole-network NP.