- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how structured test-label noise can reverse rankings of methods trained with noisy labels. It decomposes measured accuracy into clean accuracy and agreement with a class-confusion direction, then derives crossover thresholds. Experiments include synthetic noise and human annotations. SASR supplies a CLIP-based training method, while PAD estimates clean accuracy from two agreement rates under a known noise rate and a mutual-pair confusion map.
Critical strengths
-
The paper addresses a consequential evaluation problem. Holding predictions fixed while changing test labels isolates how annotation errors alter rankings, rather than conflating this effect with retraining.
-
Human-annotation and text experiments support the relevance of the phenomenon beyond synthetic visual confusion. Reporting cases where SASR loses to simpler methods and PAD fails also makes the empirical scope easier to assess.
Critical weaknesses
- The exact symmetric-noise analysis is wrong. Under uniform flips to the other K-1 classes, accuracy is (1-r)A + r(1-A)/(K-1), not (1-r)A + r/K. Expected rankings are preserved only below r=(K-1)/K, and accuracy ratios are not generally preserved. This may leave the reported controls unchanged at their tested noise rates, but Sections 3.3 and Appendix J.3 require correction. The finite-K partial-alignment analysis also needs revision. These errors weaken a central theoretical claim.
- PAD's validation falls short of the proposed practical use. It needs a noise-rate estimate and a closed mutual-twin subset; some results select that subset using true labels. Table 13 reports a deployable error of 0.043 at r=0.207, and subset accuracy cannot directly replace a full-benchmark score. Since the decomposition itself is elementary and evaluation instability is already known, convincing evidence under deployable conditions is important to the contribution. Default, short-schedule baselines and high-noise synthetic crossovers do not establish broad leaderboard applicability.
Questions for authors
- Section 3 and Appendices G/J: After correcting the symmetric-noise and finite-K partial-alignment formulas, do the control results and crossover estimates remain unchanged?
- Sections 8–9: Can PAD be evaluated using only observed labels for subset selection and an estimated noise rate, with uncertainty and contamination from outside the subset included?
- Sections 5–7: How are peak checkpoints selected? Do tuned baselines and the stronger CLIP controls support the same ranking conclusions under a consistent checkpoint protocol?
- Section 7.4 and Appendix I: Which CIFAR-100N field is used, given the description of its labels as majority votes? Please also reconcile the ImageNet recovery claim with the reported 0.013 error and provide the full evaluation results.
4 comments
Brief Addendum: Contribution Scope, Schedule/Tuning Controls, and Leaderboard Claims
Brief Addendum: Contribution Scope, Schedule/Tuning Controls, and Leaderboard Claims
To ensure every point in your Critical Weaknesses and Q3 has direct pointers to the revised 36-page PDF, we add three brief clarifications complementing our main reply above:
- Elementary decomposition vs. contribution scope (App D.3 L944–971, p. 18): We agree the two-term decomposition is elementary and qualitative test-set instability is known (Northcutt et al., 2021). App D.3 separates prior ingredients (Hui & Walter, 1980; Platanios et al., 2014–2017; Northcutt et al., 2021) from what this paper adds: (i) analyzing structured noise aligned with training noise via the closed-form crossover (Eq. 4), the exact symmetric boundary , and the limit; (ii) measuring the effect size on real crowd noise; and (iii) replacing classical error-independence with measured twin agreement for a single model, while bounding where deployable PAD fails (Sec 8 L342–343, Sec 10 L400–404—we report as the observed-label subset error, not a plug-and-play full-benchmark guarantee).
- Short schedule & baseline tuning (App A L532–539 & Table 3 L541–554, pp. 10–11; App H Table 24 L1675–1688, p. 32): Doubling the matched final-checkpoint schedule from 20 to 40 epochs (Table 3, seed 42, no tuning) leaves all pairwise rankings unchanged (on Cars196 coefficients move ; on Flowers102 CE gains and ELR in , while SASR still leads every baseline at vs. ). A 4-configuration hyperparameter search (Table 24) improves historical classical baselines by pts without overturning SASR (), though App A.1 (L592–593) notes Table 24 mixes historical single-seed peak/final metrics and the matched study (Table 1) uses implementation defaults rather than tuned upper bounds.
- High-noise synthetic crossovers vs. broad leaderboard applicability (Sec 10 L395–404, p. 8): Agreed—Sec 10 (L403–404) states explicitly that we do not infer existing fine-grained leaderboards are wrong (our blind audit of Flowers102 finds 6.7% twin flips, 95% CI [2.6%, 15.9%], below median , so we claim no inversion there). Empirical support for real inversions comes where crowd error is high (CIFAR-100N
noisy_labelat 40.2%, reversing pts to pts, , Table 25).
The revised finite- analysis and the matched three-seed results at the final checkpoint address my main concerns regarding the symmetric-noise control and baseline protocol.
My main remaining concern is the practical effectiveness of PAD. In Table 27, the point estimates rely on a reference-derived subset noise rate, while the uncertainty analysis varies the noise rate with the agreement rates fixed. This provides a useful sensitivity analysis, but does not demonstrate that PAD can effectively correct predictions when the noise rate must be estimated in practice. In addition, Appendix D.2 should explicitly state the uniform wrong-class assumption, since independence alone does not imply this condition.
Overall, the revision provides stronger evidence for the ranking-reversal result, but the practical effectiveness of PAD remains insufficiently validated. I therefore maintain my rating of 2.
Follow-Up Response to Reviewer 3oAN
We thank Reviewer 3oAN for the follow-up, and are glad the exact finite- analysis and matched 3-seed final-checkpoint study resolved your concerns on the symmetric-noise control and baseline protocol. Both remaining points are addressed in an updated 36-page PDF preserving all page breaks and margin line numbers (Lxxx) cited in prior replies.
1. End-to-End Unknown-Rate PAD & Joint Uncertainty (Revised Table 27, App K p. 35; Secs 8–10 L354–402)
You are right that the prior Table 27 was a sensitivity check: point estimates used reference and intervals held fixed. Because Eq. (6) debiases subset accuracy () of fixed predictions under noisy labels rather than altering , practical effectiveness means recovering and rankings when must be estimated.
In revised Table 27 we remove reference from all estimates and evaluate strictly on across all 5 saved CIFAR-10N models and all 3 ImageNet ReaL models (, 382 mutual-twin classes, , CPU rerun of the Table 12 recipe):
- Estimators & guard: on comes solely from (a) an -item audit on , or (b) with no verified audit labels () on CIFAR-10N, unsupervised 3-worker Dawid-Skene EM on crowd votes. An observable guard flags without reference labels.
- Joint resampling: across replicates we bootstrap (updating ) and re-estimate jointly, comparing all-model RMSEs and SASR's joint 95% CI against naive agreement and direct evaluation on the audited items (Direct-):
| Set | Estimator | Mean [95% CI] | Naive | Direct- | PAD RMSE | SASR 95% CI () |
|---|---|---|---|---|---|---|
| CIFAR maj. | audit | 0.109 [0.087, 0.134] | 0.0936 | 0.0115 | 0.0192 | [0.897, 0.963] (0.918) |
| CIFAR maj. | crowd DS () | 0.096 [0.083, 0.112] | 0.0936 | --- | 0.0117 | [0.894, 0.931] (0.918) |
| CIFAR 1-annot. | audit | 0.231 [0.200, 0.264] | 0.2056 | 0.0116 | 0.0559 | [0.903, 1.046] (0.921) |
| CIFAR 1-annot. | crowd DS () | 0.217 [0.202, 0.232] | 0.2056 | --- | 0.0196 | [0.917, 0.961] (0.921) |
| CIFAR worst | audit | 0.476 [0.438, 0.518] | 0.4231 | 0.0113 | abstain 90.0% | guard (0.922) |
| ReaL () | audit | 0.118 [0.060, 0.190] | 0.0535 | 0.0489 | 0.0236 | [0.419, 0.491] (0.451) |
| ReaL () | audit | 0.119 [0.080, 0.165] | 0.0535 | 0.0334 | 0.0177 | [0.426, 0.480] (0.451) |
| ReaL () | audit | 0.118 [0.092, 0.142] | 0.0535 | 0.0209 | 0.0133 | [0.429, 0.469] (0.451) |
Three findings delimit effectiveness. (1) Zero audit labels (): where Direct- is undefined, crowd-DS PAD cuts raw error by 87%–90% (0.0936→0.0117, matching Direct-500 at zero audit cost; 0.2056→0.0196, as DS slightly underestimates at 0.217 vs. 0.230, offsetting inflow bias). (2) Small audits (): Eqs. (13)–(15) give variance ratio plus squared inflow bias: near-ceiling CIFAR-10N (, , 0.104 inflow at ) favours Direct-500, whereas on 1000-class ReaL (, , ) PAD has a smaller factor, cutting Direct- RMSE by 36%–52% at every with Spearman 1.00 vs. 0.889/0.964/0.991. (3) Near : the guard abstains on 90.0% of replicates without clean labels.
Note on Round-1 pointers: Table 27 supersedes our Round-1 Binomial check (inflow 0.046/0.104/0.217, CIs /, ); Secs 8 & 10 still note inflow bias remains, Table 26 keeps known-rate errors, and pad_end2end_deploy.py + JSON logs are in the updated supplementary .zip.
2. Explicit Uniform Wrong-Class Assumption in Appendix D.2 (p. 18 L919–943)
Agreed: conditional independence suffices for (Hui & Walter, 1980, ), whereas for obtaining also requires a uniform wrong-class assumption over the non-true classes, for . App D.2 (Proposition 2 & proof, L919–943), Sec 3.4 (p. 4 L177–180), Sec 8 (p. 7 L358–364), and Apps D.3, F, L now state both conditions, showing semantic noise violates both independence and uniformity while PAD assumes neither by measuring directly.
Response to Reviewer 3oAN
We thank Reviewer 3oAN for the thorough review. Your critiques on exact finite-K formulas, deployable PAD, checkpoint protocols, and dataset fields drove major improvements in the revised PDF.
1. Sec 3 & Apps B.6/I.2/L.3: Exact Symmetric-Noise & Finite-K Partial-Alignment Formulas
r⋆(ρ)=(1+K−11−ρ)ΔA+ρΔNF(1)ΔA
Across ρ∈{0.25,0.5,0.75,1} and all 232 pair–alignment cases in (0,1), the max difference from ΔA+ρΔNF(1)ΔA is 0.00656 (median 0.00101), and Fig. 8 uses exact Eq. (17).
2. Secs 8–9 & App K: Observed-Label PAD, Contamination, and Rate Uncertainty
PAD) vs. observed-label selection (PAD, observed) on CIFAR-10N: at r=0.105,0.207, observed-label selection gives ∣A^−A∣=0.0099,0.0430 (vs. 0.0882,0.1803 uncorrected), while at r=0.444 (near 0.5) PAD is unusable (1.66 / 3.19).contam. = 0.046, 0.104, 0.217), 95% intervals ([0.899, 0.965]at rsub=0.109,[0.900, 1.053]at rsub=0.230), and ∣A^−Asub∣=0.009,0.041,2.328, showing budget alone does not remove inflow bias.3. Secs 5–7.1 & App A: Unified Final-Checkpoint Study & Stronger CLIP Controls
CLIP selection,CLIP pseudo-labels, andStandard ELR(λlog(1−⟨p,t⟩); App A.2 notes historical ELR† used −βlog⟨p,t⟩). Across the 9 trained methods (108 pairs), Spearman(A,NF) is −0.883,−0.983,−0.900, with 98/108 analytical crossings in (0,1), 85/108 in r⋆≤0.7, and 85/108 realized crossings on finite-sample noisy curves (Table 2). Historical peak sweeps remain in Tables 6, 22, 23 (r≤0.3 fits predict r=0.7 ordering on 102/108 pairs).4. Sec 7.2 & App G.2: CIFAR-100N Field and ImageNet ReaL Error
noisy_label(40.2% error vs. official CIFAR-100) andclean_label(verifying image-order alignment).