acceptodds
Under review as a conference paper at ICLR 2027

When Better Models Score Lower: Test-Set Label Noise Inverts Method Rankings in Learning with Noisy Labels

Abstract

Label noise in fine-grained recognition is not random: annotators confuse semantically adjacent classes, so errors concentrate on twin class pairs. This affects both the methods of learning with noisy labels (LNL) and the protocol used to compare them. (i) Evaluation. Test sets are annotated by the same procedure as training sets, so they carry the same structured noise, and a noisy test set does not merely lower scores: it inverts rankings. Measured accuracy obeys an exact two-term law, , where NF ("noise fitting") is a method-intrinsic propensity to agree with the noise direction and is anti-correlated with true quality. Across three fine-grained benchmarks and seven methods, 58 of 63 method pairs change order, with a median crossover at ; a symmetric-noise control shows NF at chance level and no inversion. The inversion is not an artefact of the injected noise: it also appears on real crowd re-annotation of CIFAR-100N, where no CLIP output enters the noise at any point and a method the clean test set rejects by a wide margin comes out ahead on the human-annotated test set of the same images. The closed-form crossover recovers, as its small- limit, the earlier observation that test-set errors can flip model selection. Nor is the law tied to our method, backbone or modality: it holds on a 1000-class human re-annotation of ImageNet val, where with no clean labels the correction still recovers the human-verified accuracy to within a point, and the rankings invert again on 20 Newsgroups with a bag-of-words model and no pretrained network. Given any class-level confusion partner map it predicts the accuracy curve of any model, with NF the only free quantity; and the threshold grows as when only a fraction of test errors shares the training direction. (ii) Methods. To measure this we need a method that is actually strong under semantic-confusion noise, so we build SASR — a vehicle for the measurement, not a claimed advance. Every remedy based on data-internal statistics fails here: kNN label propagation inverts and lands below the CLIP zero-shot baseline. SASR beats cross-entropy by 14–18 points on all three benchmarks. Against contemporary CLIP-based remedies the result is less flattering, and we report it: a plain CLIP hard-selection baseline matches SASR on CUB-200 and beats it on Cars196. (iii) Protocol. Correcting an accuracy estimate for an imperfect evaluator is classical, and the standard correction assumes that model errors and annotation errors are independent. LNL methods violate that assumption by construction, being optimised on the same noise that reappears in the test set. PAD (Pair-Agreement Debiasing) replaces the assumption with a measurement: using class-name semantics to obtain the noise direction, a single model's clean accuracy is recovered in closed form from two agreement rates on one noisy test set (error for ). PAD reduces exactly to the classical correction when independence holds, and as the two unknowns become unidentifiable — the distortion is worst exactly where it cannot be repaired. PAD needs no clean reference, no retraining and no model internals, so it can be applied post hoc to an existing leaderboard.

Primary area:
datasets and benchmarks
Keywords:
noisy labels, learning with noisy labels, test-set noise, benchmark evaluation, vision-language models, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3oANOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how structured test-label noise can reverse rankings of methods trained with noisy labels. It decomposes measured accuracy into clean accuracy and agreement with a class-confusion direction, then derives crossover thresholds. Experiments include synthetic noise and human annotations. SASR supplies a CLIP-based training method, while PAD estimates clean accuracy from two agreement rates under a known noise rate and a mutual-pair confusion map.

Critical strengths
  1. The paper addresses a consequential evaluation problem. Holding predictions fixed while changing test labels isolates how annotation errors alter rankings, rather than conflating this effect with retraining.

  2. Human-annotation and text experiments support the relevance of the phenomenon beyond synthetic visual confusion. Reporting cases where SASR loses to simpler methods and PAD fails also makes the empirical scope easier to assess.

Critical weaknesses
  1. The exact symmetric-noise analysis is wrong. Under uniform flips to the other K-1 classes, accuracy is (1-r)A + r(1-A)/(K-1), not (1-r)A + r/K. Expected rankings are preserved only below r=(K-1)/K, and accuracy ratios are not generally preserved. This may leave the reported controls unchanged at their tested noise rates, but Sections 3.3 and Appendix J.3 require correction. The finite-K partial-alignment analysis also needs revision. These errors weaken a central theoretical claim.
  2. PAD's validation falls short of the proposed practical use. It needs a noise-rate estimate and a closed mutual-twin subset; some results select that subset using true labels. Table 13 reports a deployable error of 0.043 at r=0.207, and subset accuracy cannot directly replace a full-benchmark score. Since the decomposition itself is elementary and evaluation instability is already known, convincing evidence under deployable conditions is important to the contribution. Default, short-schedule baselines and high-noise synthetic crossovers do not establish broad leaderboard applicability.
Questions for authors
  1. Section 3 and Appendices G/J: After correcting the symmetric-noise and finite-K partial-alignment formulas, do the control results and crossover estimates remain unchanged?
  2. Sections 8–9: Can PAD be evaluated using only observed labels for subset selection and an estimated noise rate, with uncertainty and contamination from outside the subset included?
  3. Sections 5–7: How are peak checkpoints selected? Do tuned baselines and the stronger CLIP controls support the same ranking conclusions under a consistent checkpoint protocol?
  4. Section 7.4 and Appendix I: Which CIFAR-100N field is used, given the description of its labels as majority votes? Please also reconcile the ImageNet recovery claim with the reported 0.013 error and provide the full evaluation results.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 3oAN

We thank Reviewer 3oAN for the thorough review. Your critiques on exact finite-KK formulas, deployable PAD, checkpoint protocols, and dataset fields drove major improvements in the revised PDF.

1. Sec 3 & Apps B.6/I.2/L.3: Exact Symmetric-Noise & Finite-KK Partial-Alignment Formulas

  • Symmetric-noise formula & controls (Sec 3.3 L148–157, Sec 7.1 L300–302, App B.6 L780–787): Under uniform flips to the remaining K−1K-1 classes, NFsym=1−AK−1\mathrm{NF_{sym}} = \frac{1-A}{K-1} and Accsym(r)=(1−r)A+r1−AK−1=(K−1)−rKK−1A+rK−1\mathrm{Acc_{sym}}(r) = (1-r)A + r\frac{1-A}{K-1} = \frac{(K-1)-rK}{K-1}A + \frac{r}{K-1} (App D.2 L922). Rankings are strictly preserved for r<K−1Kr < \frac{K-1}{K}, tie at K−1K\frac{K-1}{K}, and reverse above it; ratios are constant once rK−1\frac{r}{K-1} is subtracted. All controls hold:
    1. On Flowers102 (K=102K=102), CUB-200 (K=200K=200), and Cars196 (K=196K=196), K−1K∈{0.9902,0.9950,0.9949}≫0.7\frac{K-1}{K} \in \{0.9902, 0.9950, 0.9949\} \gg 0.7 and rK−1≤0.007\frac{r}{K-1} \le 0.007 over r≤0.7r \le 0.7.
    2. On 20 Newsgroups (K=20K=20, App G.1 L1219–1225, Table 15), NFsym\mathrm{NF_{sym}} matches 1−AK−1\frac{1-A}{K-1} within 0.0070.007 (0.0250.025 vs. 0.0310.031 for CE; 1/K=0.051/K=0.05 errs by 0.017–0.0450.017\text{–}0.045): no noisy method crosses the oracle below r=0.94r = 0.94 (K−1K=0.95\frac{K-1}{K} = \mathbf{0.95}; finite-sample noise puts CE–kkNN at r≈0.72r \approx 0.72, beyond r≤0.7r \le 0.7).
    3. Exact finite-KK partial alignment (App I.2 Eq. 16–17, L1735–1748; Fig. 8): With ΔA=Aa−Ab\Delta A = A_a - A_b and ΔNF(1)=NFb(1)−NFa(1)\Delta\text{NF}(1) = \text{NF}_b(1) - \text{NF}_a(1), Eq. (16) NF(ρ)=ρNF(1)+(1−ρ)1−AK−1\text{NF}(\rho) = \rho\text{NF}(1) + (1-\rho)\frac{1-A}{K-1} gives ΔNF(ρ)=ρΔNF(1)+(1−ρ)ΔAK−1\Delta\mathrm{NF}(\rho) = \rho\Delta\mathrm{NF}(1) + (1-\rho)\frac{\Delta A}{K-1} and the exact crossover (Eq. 17):

r⋆(ρ)=ΔA(1+1−ρK−1)ΔA+ρΔNF(1)r^\star(\rho) = \frac{\Delta A}{\left(1 + \frac{1-\rho}{K-1}\right)\Delta A + \rho\Delta\mathrm{NF}(1)}

Across ρ∈{0.25,0.5,0.75,1}\rho \in \{0.25, 0.5, 0.75, 1\} and all 232 pair–alignment cases in (0,1)(0,1), the max difference from ΔAΔA+ρΔNF(1)\frac{\Delta A}{\Delta A + \rho\Delta\text{NF}(1)} is 0.00656 (median 0.00101), and Fig. 8 uses exact Eq. (17).

2. Secs 8–9 & App K: Observed-Label PAD, Contamination, and Rate Uncertainty

  • Scope & mechanism limits (Abstract, Sec 8 L331–356, Sec 9): PAD estimates mutual-twin subset accuracy (29–52%29\text{–}52\% of classes), requires rsubr_{\mathrm{sub}} and compatible transition/selection assumptions, does not recover full-test accuracy in general, and is singular at r=0.5r=0.5. Sec 8 (L337–343) details why closed pairs (0/100/10 classes on CIFAR-10N), global matching, and naive r^1−r^N(y~=j)\frac{\hat{r}}{1-\hat{r}}N(\tilde{y}=j) subtraction (where N(y~=j)N(\tilde{y}=j) and joint counts u,vu,v are contaminated) do not yield a general closed-form inflow fix.
  • Table 26 (App K): Compares true-class (PAD) vs. observed-label selection (PAD, observed) on CIFAR-10N: at r=0.105,0.207r=0.105, 0.207, observed-label selection gives ∣A^−A∣=0.0099,0.0430|\hat{A}-A| = 0.0099, 0.0430 (vs. 0.0882,0.18030.0882, 0.1803 uncorrected), while at r=0.444r=0.444 (near 0.50.5) PAD is unusable (1.661.66 / 3.193.19).
  • Table 27 (App K): Simulates observed-label selection on CIFAR-10N (n=1000n=1000 Binomial annotations, r^∼Beta(k+1,n−k+1)\hat{r} \sim \text{Beta}(k+1, n-k+1)), reporting outside contamination (contam. = 0.046, 0.104, 0.217), 95% intervals ([0.899, 0.965] at rsub=0.109r_{\mathrm{sub}}=0.109, [0.900, 1.053] at rsub=0.230r_{\mathrm{sub}}=0.230), and ∣A^−Asub∣=0.009,0.041,2.328|\hat{A}-A_{\mathrm{sub}}| = 0.009, 0.041, 2.328, showing budget alone does not remove inflow bias.

3. Secs 5–7.1 & App A: Unified Final-Checkpoint Study & Stronger CLIP Controls

  • 90-run matched final-checkpoint study (Tables 1, 2, 4; Secs 5–7.1; App A): We ran a matched 3-seed study (seeds 42/43/44, 20 epochs, final checkpoints only, no peak selection) across all 10 methods, adding CLIP selection, CLIP pseudo-labels, and Standard ELR (λlog⁡(1−⟨p,t⟩)\lambda\log(1-\langle p,t\rangle); App A.2 notes historical ELR†\text{ELR}^\dagger used −βlog⁡⟨p,t⟩-\beta\log\langle p,t\rangle). Across the 9 trained methods (108 pairs), Spearman(A,NF)\text{Spearman}(A,\text{NF}) is −0.883,−0.983,−0.900-0.883, -0.983, -0.900, with 98/108 analytical crossings in (0,1)(0,1), 85/108 in r⋆≤0.7r^\star \le 0.7, and 85/108 realized crossings on finite-sample noisy curves (Table 2). Historical peak sweeps remain in Tables 6, 22, 23 (r≤0.3r \le 0.3 fits predict r=0.7r=0.7 ordering on 102/108 pairs).

4. Sec 7.2 & App G.2: CIFAR-100N Field and ImageNet ReaL Error

  • CIFAR-100N field (Sec 7.2 L312–316): Corrected—CIFAR-100N ships noisy_label (40.2%40.2\% error vs. official CIFAR-100) and clean_label (verifying image-order alignment).
  • ImageNet ReaL error (Abstract L027–029, App G.2 L1266, Table 12): We removed "within a point" from the Abstract and report exact ImageNet ReaL PAD errors (382/1000382/1000 mutual-twin classes, n=7193n=7193, rsub=0.1184r_{\mathrm{sub}}=0.1184) in Table 12, with max ∣A^−A∣=0.0128|\hat{A}-A| = \mathbf{0.0128} (1.28 pts).
AuthorsOpenReview ↗

Brief Addendum: Contribution Scope, Schedule/Tuning Controls, and Leaderboard Claims

Brief Addendum: Contribution Scope, Schedule/Tuning Controls, and Leaderboard Claims

To ensure every point in your Critical Weaknesses and Q3 has direct pointers to the revised 36-page PDF, we add three brief clarifications complementing our main reply above:

  • Elementary decomposition vs. contribution scope (App D.3 L944–971, p. 18): We agree the two-term decomposition is elementary and qualitative test-set instability is known (Northcutt et al., 2021). App D.3 separates prior ingredients (Hui & Walter, 1980; Platanios et al., 2014–2017; Northcutt et al., 2021) from what this paper adds: (i) analyzing structured noise aligned with training noise via the closed-form crossover (Eq. 4), the exact symmetric boundary K−1K\frac{K-1}{K}, and the r→12r \to \frac{1}{2} limit; (ii) measuring the effect size on real crowd noise; and (iii) replacing classical error-independence with measured twin agreement vv for a single model, while bounding where deployable PAD fails (Sec 8 L342–343, Sec 10 L400–404—we report 0.04300.0430 as the observed-label subset error, not a plug-and-play full-benchmark guarantee).
  • Short schedule & baseline tuning (App A L532–539 & Table 3 L541–554, pp. 10–11; App H Table 24 L1675–1688, p. 32): Doubling the matched final-checkpoint schedule from 20 to 40 epochs (Table 3, seed 42, no tuning) leaves all pairwise rankings unchanged (on Cars196 coefficients move ≤0.013\le 0.013; on Flowers102 CE gains +0.060+0.060 and ELR +0.037+0.037 in AA, while SASR still leads every baseline at 0.7490.749 vs. 0.6160.616). A 4-configuration hyperparameter search (Table 24) improves historical classical baselines by +0.9–11.9+0.9\text{–}11.9 pts without overturning SASR (0.769/0.557/0.5930.769 / 0.557 / 0.593), though App A.1 (L592–593) notes Table 24 mixes historical single-seed peak/final metrics and the matched study (Table 1) uses implementation defaults rather than tuned upper bounds.
  • High-noise synthetic crossovers vs. broad leaderboard applicability (Sec 10 L395–404, p. 8): Agreed—Sec 10 (L403–404) states explicitly that we do not infer existing fine-grained leaderboards are wrong (our blind audit of Flowers102 finds 6.7% twin flips, 95% CI [2.6%, 15.9%], below median r⋆=0.47–0.52r^\star = 0.47\text{–}0.52, so we claim no inversion there). Empirical support for real inversions comes where crowd error is high (CIFAR-100N noisy_label at 40.2%, reversing −13.4-13.4 pts to +3.8+3.8 pts, p≤0.009p \le 0.009, Table 25).

The revised finite-KK analysis and the matched three-seed results at the final checkpoint address my main concerns regarding the symmetric-noise control and baseline protocol.

My main remaining concern is the practical effectiveness of PAD. In Table 27, the point estimates rely on a reference-derived subset noise rate, while the uncertainty analysis varies the noise rate with the agreement rates fixed. This provides a useful sensitivity analysis, but does not demonstrate that PAD can effectively correct predictions when the noise rate must be estimated in practice. In addition, Appendix D.2 should explicitly state the uniform wrong-class assumption, since independence alone does not imply this condition.

Overall, the revision provides stronger evidence for the ranking-reversal result, but the practical effectiveness of PAD remains insufficiently validated. I therefore maintain my rating of 2.

AuthorsOpenReview ↗

Follow-Up Response to Reviewer 3oAN

We thank Reviewer 3oAN for the follow-up, and are glad the exact finite-KK analysis and matched 3-seed final-checkpoint study resolved your concerns on the symmetric-noise control and baseline protocol. Both remaining points are addressed in an updated 36-page PDF preserving all page breaks and margin line numbers (Lxxx) cited in prior replies.

1. End-to-End Unknown-Rate PAD & Joint Uncertainty (Revised Table 27, App K p. 35; Secs 8–10 L354–402)

You are right that the prior Table 27 was a sensitivity check: point estimates used reference rsubr_{\text{sub}} and intervals held (u,v)(u,v) fixed. Because Eq. (6) debiases subset accuracy (A^\hat{A}) of fixed predictions y^\hat{y} under noisy labels rather than altering y^\hat{y}, practical effectiveness means recovering AsubA_{\text{sub}} and rankings when rsubr_{\text{sub}} must be estimated.

In revised Table 27 we remove reference rsubr_{\text{sub}} from all estimates and evaluate strictly on Sobs={i:y~i∈M}\mathcal S_{\text{obs}}=\lbrace i:\tilde y_i\in M\rbrace across all 5 saved CIFAR-10N models and all 3 ImageNet ReaL models (K=1000K=1000, 382 mutual-twin classes, Nobs=7193N_{\text{obs}}=7193, CPU rerun of the Table 12 recipe):

  • Estimators & guard: r^sub\hat r_{\text{sub}} on Sobs\mathcal S_{\text{obs}} comes solely from (a) an mm-item audit on Sobs\mathcal S_{\text{obs}}, or (b) with no verified audit labels (m=0m=0) on CIFAR-10N, unsupervised 3-worker Dawid-Skene EM on crowd votes. An observable guard flags ∣r^sub−0.5∣≤0.05|\hat r_{\text{sub}}-0.5|\le0.05 without reference labels.
  • Joint resampling: across B=500B=500 replicates we bootstrap Sobs\mathcal S_{\text{obs}} (updating u^,v^\hat{u},\hat{v}) and re-estimate r^sub\hat r_{\text{sub}} jointly, comparing all-model RMSEs and SASR's joint 95% CI against naive agreement and direct evaluation on the mm audited items (Direct-mm):
SetEstimatorMean r^sub\hat r_{\text{sub}} [95% CI]NaiveDirect-mmPAD RMSESASR 95% CI (AsubA_{\text{sub}})
CIFAR maj.audit m=500m=5000.109 [0.087, 0.134]0.09360.01150.0192[0.897, 0.963] (0.918)
CIFAR maj.crowd DS (m=0m=0)0.096 [0.083, 0.112]0.0936---0.0117[0.894, 0.931] (0.918)
CIFAR 1-annot.audit m=500m=5000.231 [0.200, 0.264]0.20560.01160.0559[0.903, 1.046] (0.921)
CIFAR 1-annot.crowd DS (m=0m=0)0.217 [0.202, 0.232]0.2056---0.0196[0.917, 0.961] (0.921)
CIFAR worstaudit m=500m=5000.476 [0.438, 0.518]0.42310.0113abstain 90.0%guard (0.922)
ReaL (K=1000K=1000)audit m=100m=1000.118 [0.060, 0.190]0.05350.04890.0236[0.419, 0.491] (0.451)
ReaL (K=1000K=1000)audit m=200m=2000.119 [0.080, 0.165]0.05350.03340.0177[0.426, 0.480] (0.451)
ReaL (K=1000K=1000)audit m=500m=5000.118 [0.092, 0.142]0.05350.02090.0133[0.429, 0.469] (0.451)

Three findings delimit effectiveness. (1) Zero audit labels (m=0m=0): where Direct-mm is undefined, crowd-DS PAD cuts raw error by 87%–90% (0.0936→0.0117, matching Direct-500 at zero audit cost; 0.2056→0.0196, as DS slightly underestimates rsubr_{\text{sub}} at 0.217 vs. 0.230, offsetting inflow bias). (2) Small audits (m>0m>0): Eqs. (13)–(15) give variance ratio r(1−r)A(1−A)(A−NF1−2r)2\frac{r(1-r)}{A(1-A)}\bigl(\frac{A-\mathrm{NF}}{1-2r}\bigr)^2 plus squared inflow bias: near-ceiling CIFAR-10N (A≈0.92A\approx0.92, A(1−A)≈0.075A(1-A)\approx0.075, 0.104 inflow at rsub=0.23r_{\text{sub}}=0.23) favours Direct-500, whereas on 1000-class ReaL (A∈[0.45,0.61]A\in[0.45,0.61], A(1−A)≈0.25A(1-A)\approx0.25, rsub=0.118r_{\text{sub}}=0.118) PAD has a 6×6\times smaller factor, cutting Direct-mm RMSE by 36%–52% at every mm with Spearman 1.00 vs. 0.889/0.964/0.991. (3) Near rsub≈0.48r_{\text{sub}}\approx0.48: the guard abstains on 90.0% of replicates without clean labels.

Note on Round-1 pointers: Table 27 supersedes our Round-1 n=1000n=1000 Binomial check (inflow 0.046/0.104/0.217, CIs [0.899,0.965][0.899,0.965]/[0.900,1.053][0.900,1.053], ∣A^−Asub∣=0.009/0.041/2.328|\hat{A}-A_{\text{sub}}|=0.009/0.041/2.328); Secs 8 & 10 still note inflow bias remains, Table 26 keeps known-rate errors, and pad_end2end_deploy.py + JSON logs are in the updated supplementary .zip.

2. Explicit Uniform Wrong-Class Assumption in Appendix D.2 (p. 18 L919–943)

Agreed: conditional independence suffices for K=2K=2 (Hui & Walter, 1980, K−1=1K-1=1), whereas for K>2K>2 obtaining NFind=(1−A)/(K−1)\mathrm{NF_{ind}}=(1-A)/(K-1) also requires a uniform wrong-class assumption over the K−1K-1 non-true classes, P(y^=c∣y^≠y,y)=1/(K−1)\mathbb{P}(\hat{y}=c\mid\hat{y}\neq y,y)=1/(K-1) for c≠yc\neq y. App D.2 (Proposition 2 & proof, L919–943), Sec 3.4 (p. 4 L177–180), Sec 8 (p. 7 L358–364), and Apps D.3, F, L now state both conditions, showing semantic noise violates both independence and uniformity while PAD assumes neither by measuring vv directly.

Reviewer_j4nKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how test-label noise aligned with training noise can reward noise fitting and reverse method rankings. A class-confusion partner map τ defines structured corruption. Under the specified uniform random-flip model, measured accuracy is Acc(r)=(1−r)A+r·NF, where A is clean accuracy and NF is the probability of predicting the designated confusion partner. The decomposition yields a pairwise crossover threshold. SASR, a CLIP-guided learner, serves as an experimental vehicle rather than a claimed state-of-the-art advance. The paper studies fine-grained image benchmarks, human annotation noise, and text classification. PAD uses a noise-rate estimate and two observable agreement rates to recover clean accuracy on mutual-partner classes.

Critical strengths

Holding model predictions fixed while varying test labels separates measurement distortion from training changes. The held-out CIFAR-100N crowd-annotation experiment also shows that reversal is not confined to injected CLIP-based confusion: on the ten noisiest classes, SASR trails the oracle by 13.4 percentage points under clean labels but leads by 3.8 points under human-observed labels.

Separating true correctness from agreement with a designated error direction makes crossover conditions transparent. With mutual partners and a known noise rate, PAD’s two-equation inversion provides an explicit degeneracy condition. The disclosed selection trap, instability near r=0.5, and lack of uniform ranking improvement are useful negative findings that help delimit applicability.

Critical weaknesses
  1. Section 3.3 and Appendix J.3 define a flipped label as uniform over the remaining K−1 classes, but state NF→1/K and Acc(r)=(1−r)A+r/K. Under that definition, my derivation is that a correct prediction cannot match a forcibly incorrect label, while an incorrect prediction matches it with probability 1/(K−1). The following expressions should therefore replace the stated formula. Accsym(r)=(1−r)A+r(1−A)K−1\mathrm{Acc}_{\mathrm{sym}}(r) = (1-r)A + \frac{r(1-A)}{K-1}

ΔAccsym(r)=(1−KrK−1)ΔA\Delta \mathrm{Acc}_{\mathrm{sym}}(r) = \left(1-\frac{Kr}{K-1}\right)\Delta A

Rankings are preserved for r<(K−1)/K, tie at the boundary, and reverse above it. This need not overturn the many-class controls at r≤0.7, but “cannot invert” requires qualification; ratio invariance is not generally implied either. Appendix B.2 already uses (1−A)/(K−1), creating an internal inconsistency. Fixed A and NF also depend on the specified corruption mechanism: an arbitrary clean/compromised partition cannot retain their unconditional definitions without additional assumptions.

  1. Section 8 acknowledges that observed-label selection admits examples from outside the mutual-partner set, raising its flip rate to 0.306 at nominal r=0.20. Restricting injection to a closed subset avoids this, but does not solve arbitrary existing-test-set correction. PAD needs a noise-rate estimate and valid confusion map, and covers only some classes. Real-noise reversals do not validate all paired-flip assumptions. If the target is the true class, noisy/clean evaluation disagreement does not make the clean ranking erroneous. These qualifications should be consistent throughout. Also, 58/63 counts crossings predicted from fitted coefficients within r<1, not necessarily observed on the finite grid.
Questions for authors

Q1. Does symmetric corruption force a flip to one of K−1 other classes, or resample over all K classes? Please reconcile formulas, appendices, and plotting code, and state the implications for ratio invariance and ranking claims. Q2. Without true-class selection or restricted injection, what is PAD’s deployable error? Please compare closed-subset and observed-label-selected variants, specifying noise-rate estimation, class coverage, and selection bias. Q3. Which claims require uniform, instance-independent flipping, and which are conditional partition identities or empirical findings? Please distinguish a shared annotation workflow from identical transitions, and noisy-evaluation bias from clean evaluation of the true-class target.

2 comments
AuthorsOpenReview ↗

Response to Reviewer j4nK

We thank Reviewer j4nK for the incisive review. Your derivation of the exact symmetric-noise formula and questions on selection bias, finite-grid crossovers, and partition assumptions directly shaped the revised PDF.

Q1. Symmetric Corruption, Exact Formulas, and In-Grid Crossovers

  • Force-flipping & exact formula (Sec 3.3, Sec 7.1, App B.6, App L.3): Our code (run_testnoise2.sh), App D.2 (L922), and App I.2 (Eq. 16) force flipped labels uniformly onto the remaining K−1K-1 classes (y~∼Unif({1..K}∖{y})\tilde{y} \sim \text{Unif}(\{1..K\}\setminus\{y\})). We replaced the large-KK approximation in Sec 3.3 and App B.6/L.3 with your exact formula:

NF=1−AK−1\mathrm{NF} = \frac{1-A}{K-1}

Acc(r)=(1−r)A+r1−AK−1=[(K−1)−rKK−1]A+rK−1\mathrm{Acc}(r) = (1-r)A + r\frac{1-A}{K-1} = \left[\frac{(K-1)-rK}{K-1}\right]A + \frac{r}{K-1}

  • Ranking preservation: Since the coefficient of AA is >0>0 iff r<K−1Kr < \frac{K-1}{K}, expected rankings are strictly invariant for r<K−1Kr < \frac{K-1}{K}, tie at K−1K\frac{K-1}{K}, and reverse above it. On Flowers102 (K=102K=102), CUB-200 (K=200K=200), and Cars196 (K=196K=196), K−1K\frac{K-1}{K} is 0.9902,0.9950,0.99490.9902, 0.9950, 0.9949 (Sec 7.1 L301–302)—far above r≤0.7r \le 0.7. On 20 Newsgroups (K=20K=20, App G.1 L1219–1225, Table 15), NFsym\mathrm{NF_{sym}} lands within 0.0070.007 of 1−AK−1\frac{1-A}{K-1} (1/K=0.051/K = 0.05 errs by 0.017–0.0450.017\text{–}0.045): no noisy method crosses the oracle below r=0.94r = 0.94 (K−1K=0.95\frac{K-1}{K} = \mathbf{0.95}; finite-sample noise puts CE–kkNN at r≈0.72r \approx 0.72, beyond r≤0.7r \le 0.7).
  • Ratio invariance (App B.6 L783–787): Under the exact formula, only the ordering is strictly invariant for r<K−1Kr < \frac{K-1}{K}; the accuracy ratio is constant once the chance term rK−1\frac{r}{K-1} is subtracted (≤0.007\le 0.007 for K≥102K \ge 102 and r≤0.7r \le 0.7, explaining why SASR/oracle = 0.795 at both r=0r=0 and r=0.7r=0.7, differing by 6×10−46 \times 10^{-4}).
  • In-grid vs. extrapolated crossovers (Table 2 & App B.5):
    • Historical 7-method scan (App B.5 L752–756, Table 14): 50/63 pairs (79.4%) cross inside the sampled grid r≤0.7r \le 0.7 (Flowers 18/21, CUB 17/21, Cars 15/21) and 21/63 (33.3%) by r≤0.5r \le 0.5; 8 lie in (0.7,1.0)(0.7, 1.0) (58/63 for r⋆<1r^\star < 1).
    • New matched 3-seed final-checkpoint study (Sec 7.1, Table 2): Across 9 trained methods (108 pairs), 85/108 (78.7%) cross within r⋆≤0.7r^\star \le 0.7 and 55/108 (50.9%) within r⋆≤0.5r^\star \le 0.5 (98/108 in r⋆<1r^\star < 1), and finite-sample re-injection into the 81 saved models (realized in Table 2) crosses in the exact same 85/108 pairs.

Q2. Deployable PAD Without True-Class Selection or Restricted Injection (Sec 8 L333–356, App K Tables 26–27)

When all KK classes are corrupted along τ\tau, selecting samples by observed noisy labels y~∈M\tilde{y} \in M (M={k:τ(τ(k))=k}M = \{k:\tau(\tau(k))=k\}) admits outside samples (y∉M,τ(y)∈My \notin M, \tau(y) \in M), inflating the realized flip rate on Flowers102 from 0.200.20 to 0.3060.306 (signed errors A^−A\hat{A}-A of −0.108-0.108 at r=0.20r=0.20 and −0.161-0.161 at r=0.40r=0.40). Sec 8 (L337–343) analyzes three candidate remedies and their failure modes:

  1. In-degree-1 closed pairs: Mutual pairs with no incoming edges equate observed-label and true-class selection under deterministic twin corruption, but can be empty (0/100/10 classes on CIFAR-10N).
  2. Global matching: Max-weight matching on class names makes τ\tau a global involution (as on 20 Newsgroups), but does not ensure human errors follow it.
  3. Why naive inflow subtraction fails: Subtracting r^1−r^N(y~=j)\frac{\hat{r}}{1-\hat{r}}N(\tilde{y}=j) is not a general inflow fix because observed counts N(y~=j)N(\tilde{y}=j) may themselves contain inflow, and the joint counts defining u,vu, v also need correction. Accordingly, the Abstract (L031–036) and Secs 8–9 state that PAD is a subset-level estimator conditional on compatible transition and selection assumptions, and App K adds Table 26 (restricted vs. observed-label selection on CIFAR-10N: 0.00950.0095 vs. 0.00990.0099 at r=0.105r=0.105; 0.02830.0283 vs. 0.04300.0430 at r=0.207r=0.207) and Table 27 (n=1000n=1000 Binomial rate draws under observed-label selection with outside contamination shares).

Q3. Partition Identity vs. Constant (A,NF)(A, \text{NF}) & Clean-Test Target

  • Conditional partition identity (App B.7.1 L792–796, App G.3 L1277–1279): Acc=(1−r)P(y^=y∣uncompromised)+rP(y^=y~∣compromised)\text{Acc} = (1-r)\mathbb{P}(\hat{y}=y\mid \text{uncompromised}) + r\mathbb{P}(\hat{y}=\tilde{y}\mid \text{compromised}) is a total-probability identity that does not make either conditional term invariant in rr or equate the first with clean accuracy AA; reducing to constant (A,NF)(A, \text{NF}) requires independent constant-rate twin corruption.
  • Clean-test reference (Sec 1 L047–049, Sec 10 L399): Corrected—when the target is true class yy, clean-test evaluation is the correct reference, and shared noisy test labels introduce measurement bias that distort rankings.
AuthorsOpenReview ↗

Brief Addendum: Class Coverage, Shared Workflow vs. Alignment, and In-Grid Counts

Brief Addendum: Class Coverage, Shared Workflow vs. Alignment, and In-Grid Counts

For completeness on the remaining qualifications in your Critical Weaknesses and Q2–Q3, we note three explicit pointers in the revised PDF:

  • Class coverage, external rate input, and arbitrary test sets (Sec 8 L331–343, Sec 9 L373–377, Sec 10 L400–403): Mutual twins cover 29–52% of classes (30/10230/102 Flowers102, 88/20088/200 CUB-200, 102/196102/196 Cars196; 382/1000382/1000 on known-rate ImageNet ReaL where rsub=0.1184r_{\text{sub}}=0.1184 and ∣A^−A∣≤0.0128|\hat{A}-A|\le 0.0128, Table 12). PAD requires an external estimate of rsubr_{\text{sub}} (Table 16, Table 27), and Secs 8 & 10 state explicitly that neither restricted nor observed-label selection provides a general correction for an arbitrary existing test set.
  • Shared annotation workflow vs. identical transitions (Sec 7.2 L324–325, App I.2 L1727–1733 & L1751–1757, pp. 32–33): Agreed—a shared annotation procedure is neither necessary nor sufficient for identical transitions (App I.2 L1727–1730). We therefore parameterize directional alignment directly by ρ∈[0,1]\rho \in [0,1] (treating ρ=1\rho=1 as a synthetic control rather than a consequence of data provenance), and state that real-noise reversals on CIFAR-100N do not validate paired-flip assumptions beyond the tested benchmark (Sec 7.2 L324–325, App I.2 L1751–1752).
  • Predicted vs. observed crossings (Sec 7.1, Table 2, App B.5 L752–756): Both historical counts (58/6358/63 in r⋆<1r^\star < 1; 50/6350/63 in r⋆≤0.7r^\star \le 0.7) are analytical predictions from fitted (A,NF)(A,\text{NF}), whereas realized finite-sample crossings are reported separately in Table 2 (85/10885/108 realized vs. 85/10885/108 analytical in r⋆≤0.7r^\star \le 0.7).
Reviewer_vtN9OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that label noise in fine-grained recognition concentrates on semantically adjacent “twin” classes (defined via CLIP text embeddings), and that because test sets share this noise, it can invert the ranking of learning-with-noisy-labels methods. The authors decompose measured accuracy as Acc(r) = (1−r)A + r·NF, where NF is the rate at which a model predicts the twin class, and derive a closed-form crossover. Under injected twin noise, 58 of 63 method pairs reverse order on three fine-grained benchmarks, with no reversals under symmetric noise; reversals are also reported on CIFAR-100N and 20 Newsgroups. The paper further proposes PAD, a closed-form estimator of clean accuracy from a noisy test set, which becomes unidentifiable as r → 0.5. SASR, a CLIP-anchored method, serves only as an experimental vehicle.

Critical strengths
  • The paper highlights how noisy test labels can reverse method rankings, challenging the reliability of LNL evaluation.

  • The accuracy decomposition and closed-form crossover threshold provide a clear explanation of when and why ranking reversals occur.

  • Experiments across multiple datasets, noise settings, and real human annotations support the findings and demonstrate their generalizability.

Critical weaknesses
  1. The abstract is overly long and dense. It reads as a results summary rather than an abstract. Containing many numerical numbers, split into parts (i)–(iii).

  2. Misplaced cross-references.

    a) Appendix F consists of a heading only (Page 20). Section 1 (The perdataset crossovers and the remaining checks are in Appendix F.) and Section 7.2 (The individual crossovers … are in Appendix F) refer to it.

    b) Section 7.4. (The four stratum rates, the granularity check and the ImageNet ReaL validation of Aˆ are in Appendix I.) but Appendix I only covers 20 Newsgroups.

    c) Appendix I repeats E.1.

  3. Figure 1 is too small. Its crossover markers are difficult to read at print size.

  4. Table 1 is hard to interpret correctly. No standard deviations are reported for any row (lead to a 0.002 difference on CUB-200 uninformative). The caption of Table 1 (''Test accuracy under 50% semantic-confusion noise'') can be misread as a 50%-noisy test set.

  5. Overloaded notation. For example, \tau denotes both the twin map and the softmax temperature (Section 1,3, A.1, Table 3). q denotes both the GCE parameter (q = 0.7, Table 3) and the fraction of corrupted anchors (Table 9). r denotes both the noise rate and a correlation coefficient (r = 0.987, E.2 item 4).

  6. Inconsistent. The SASR−CE gap on Flowers102 is reported as 17.8, 17.4 and 17.3 depending on the run and rounding (Section 6.2) without STD. Section 7.2 describes the Flowers102 ordering at r = 0.7 as “oracle > ⋯ > SASR”, whereas Figure 1 shows the oracle lowest at noise level 0.7.

  7. PAD requires a known noise rate and mutual-twin class pairs, restricting its applicability to real-world datasets.

  8. Most ranking reversals occur at relatively high test-label noise rates, raising questions about their significance for benchmarks with cleaner annotations.

Questions for authors
  1. Weaknesses above

  2. On human-annotated test labels (CIFAR-10N/100N), how does the fitted NF compare with NF computed directly from the definition P(ŷ = τ(y))?

  3. Do the original test labels of Flowers102, CUB-200, and Cars196 exhibit twin-class noise, and at what rates?

  4. Are there any real benchmarks currently in use where test-set noise exhibits twin-class structure and is sufficiently large to cause ranking inversions?

  5. How does the proposed accuracy decomposition differ from existing confusion-matrix-based analyses of noisy-label evaluation? How can PAD be reliably applied when the true noise rate and confusion map are unknown?

4 comments
AuthorsOpenReview ↗

Response to Reviewer vtN9

We thank Reviewer vtN9 for catching the editorial, cross-reference, and notation defects, and for the questions on real benchmarks and PAD.

  • W1 (Abstract, L013–036): Condensed to one paragraph stating the decomposition law, historical 7-method scan (58/63 crossings; 50/63 at r≤0.7r \le 0.7), matched 3-seed study (98/108 crossings across 9 trained methods), and PAD's mutual-twin subset scope (29–52% of classes), rate input, selection assumptions, and r=0.5r=0.5 singularity; detailed numbers live in Secs 5–7.
  • W2 (Cross-references, Apps A–L): (a) Rebuilt Apps A–L with all floats placed under their sections; per-pair crossovers and extra curves/tables are in App H (Tables 14–24, Figs. 3–10). (b) Sec 7.2 (L325–326, old 7.4) points to App K (Table 25) for CIFAR-100N stratum rates, App G.2 (Table 12) for ReaL granularity/A^\hat{A}, and App G.3 (Table 13) for noise-agreement comparisons. (c) Duplicate 20 Newsgroups text is removed, appearing once in App G.1 (Table 15, Fig. 2).
  • W3 (Figure 1, p. 2): Redrawn as a vector PDF for the 5.5 in (396 pt) text block (400.6 pt natural width, scale 0.99), with 8.0–9.2 pt type/legend and larger crossover markers (Fig. 8 at 396.0 pt, scale 1.00).
  • W4 (Table 1, p. 4 L163–176): Caption reads "Matched final-checkpoint clean accuracy AA: mean ±\pm sample SD over seeds 42/43/44", stating 50% training noise, clean-split AA, and no test peaks. All rows carry SDs (sub-0.01 CUB-200 gaps sit inside seed SDs 0.0055–0.0207), with paired gaps/95% CIs in Sec 6.1 (L249–251) and App A (L523–530) plus a 90/90 predictions.npz audit (L579–583); historical peaks are in Table 6.
  • W5 (Notation): τ\tau is the twin map only (T=0.07T=0.07, Sec 4 L193, Table 10); qq is the GCE parameter only (broken-anchor share α∈[0,1]\alpha\in[0,1], Sec 6.2 L263, Table 19, Fig. 6); rr is the noise rate only (ratioi\text{ratio}_i in Sec 4 L192, App C.2; App G.4 L1308–1310 drops r=r= for the 0.987 off-diagonal Pearson correlation, distinct from ρ\rho).
  • W6 (Inconsistencies): (a) App B.2 (L722–724, Table 18) notes 17.8, 17.4, and 17.3 are the historical full-split SASR$-$CE gap computed three ways (Table 6, control's own run, and 3-seed mean; drift ≤0.5\le 0.5 pts), while the matched study gives paired gap +23.2±5.38+23.2 \pm 5.38 pts (App A L525–526). (b) App B.5 (L756–758) fixes the r=0.7r=0.7 Flowers102 typo: the oracle is last (0.286 vs. kkNN 0.411).
  • W7 (PAD scope): Agreed and stated (Abstract L031–036, Secs 8–9): PAD is a subset estimator needing rsubr_{\mathrm{sub}}, τ\tau, and compatible selection, does not recover full-test accuracy in general, and is singular at r=0.5r=0.5.
  • W8 (High noise rates): Agreed and quantified (Secs 7.2, 10): inversions occur at high structured error (Q3), not on cleaner curated sets (Q2).
  • Q1 (Fitted vs. direct NF, App G.3 L1273–1282, Table 13): Without injected noise, Table 13 separates direct NF=P(y^=τ(y))\text{NF}=\mathbb{P}(\hat{y}=\tau(y)) (0.030–0.034 on CIFAR-10N; 0.040–0.054 on CIFAR-100N), error-conditioned P(y^=τ(y)∣y^≠y)\mathbb{P}(\hat{y}=\tau(y)\mid\hat{y}\neq y) (0.37–0.42; 0.14–0.17), and human-error agreement B=P(y^=y~∣y~≠y)B=\mathbb{P}(\hat{y}=\tilde{y}\mid\tilde{y}\neq y) (0.128 GCE pool; 0.157 kNN50 top-10). As App G.3 (L1280–1282) states, the saved CIFAR log has no paired fitted NF\text{NF} on the same images (Table 4 computes direct NF\text{NF} on all 90 matched runs; Table 12 gives PAD NF^\widehat{\text{NF}} of 0.028–0.050 on ReaL).
  • Q2 (Original test noise, Sec 10 L395–398): A blind 2AFC audit of 200 Flowers102 test images from the 30 mutual-twin classes (4 exemplars/class, rows shuffled, label hidden) matches the official label on 191/200 (2 twin = 1.0%, 7 neither). On 60 images, three readers (0%, 15%, 6.7% twin; one pre-calibrated on a separate 20-image known-answer sheet, 20/20) give majority-vote 6.7% twin flips (95% CI [2.6%, 15.9%]) (13.3% upper bound with "neither")—below median r⋆=0.47–0.52r^\star = 0.47\text{–}0.52, so we claim no inversion on Flowers102 (CUB-200/Cars196 unmeasured, Sec 9 L385).
  • Q3 (Real inversions, Sec 7.2 L312–323, Table 25): CIFAR-100N noisy_label (40.2% crowd error; 0.538–0.763 across strata, Table 25) reverses a −13.4-13.4 pt clean deficit into a +3.8+3.8 pt lead (p≤0.009p \le 0.009); lower-noise ReaL (rsub=0.1184r_{\mathrm{sub}}=0.1184, Table 12) is a known-rate PAD check (∣A^−A∣≤0.0128|\hat{A}-A|\le 0.0128).
  • Q4 (Confusion analyses & unknown r,τr,\tau, App D.2, Tables 16, 21, 26–27): Hui & Walter (1980) assume NFind=1−AK−1\mathrm{NF_{ind}}=\frac{1-A}{K-1} (+0.101+0.101 bias vs. +0.007+0.007 for PAD, Table 21). With unknown r,τr,\tau, τ\tau comes from class names and rsubr_{\mathrm{sub}} needs external estimation (Table 16); Tables 26–27 give observed-label selection (0.0099, 0.0430 at r=0.105,0.207r=0.105, 0.207) and contamination/rate intervals.

I would like to thank the authors for their response. I have read through your response; however, I am not convinced by the points I mentioned above. These responses above seem "mechanistic" or "engineering" (you only list the number of results), which does not give insight or analysis to reader/ reviewer. I personally feel this is not a rebuttal; it is like a technical report. Therefore, I will keep my original rating and have no further questions.

AuthorsOpenReview ↗

Thank You for the Candid Feedback — Direct Analytical Answers to Q1–Q4

Thank You for the Candid Feedback — Direct Analytical Answers to Q1–Q4

Thank you for reading our response and for this candid critique. Looking back at our reply, your point is fair: in indexing twelve items with section and table pointers under the character limit, we wrote a dense changelog of numbers rather than explaining why the results behave as they do. Respecting that you have no further questions, we leave direct analytical answers to your four questions below for the record.

  • Q1 (How fitted NF\mathrm{NF} compares with direct P(y^=τ(y))P(\hat{y}=\tau(y)) on human-annotated CIFAR-10N/100N): They do not coincide on human labels, and the gap reveals how real crowd errors differ from synthetic class-level flips. Under synthetic twin noise swept across rr, flips occur independently of image difficulty, so the fitted slope equals unconditional P(y^=τ(y))P(\hat{y}=\tau(y)) identically. On a real crowd test set, rr is fixed; in Acc=(1−r)P(y^=y∣y~=y)+rP(y^=y~∣y~≠y)\mathrm{Acc} = (1-r)P(\hat{y}=y \mid \tilde{y}=y) + r P(\hat{y}=\tilde{y} \mid \tilde{y}\neq y), the effective counterpart of NF\mathrm{NF} is human-error agreement B=P(y^=y~∣y~≠y)B = P(\hat{y}=\tilde{y} \mid \tilde{y}\neq y). On CIFAR-100N, unconditional P(y^=τ(y))P(\hat{y}=\tau(y)) is 4%–5%, whereas BB is ~3×3\times higher (13%–16%). Two opposing effects explain why: (i) instance-difficulty coupling—unconditional P(y^=τ(y))P(\hat{y}=\tau(y)) is diluted across easy images that neither humans nor models miss, whereas conditioning on human errors (y~≠y\tilde{y}\neq y) isolates ambiguous images where model mistakes spike along the same confusion channel; and (ii) neighborhood spread—conversely, human errors spread over small clusters of similar classes rather than landing solely on the top-1 text twin τ(y)\tau(y).

  • Q2 (Twin-class noise and rates in original Flowers102, CUB-200, and Cars196 test sets): On Flowers102, our blind audit of the official test split confirms that residual errors concentrate on visually confusable twin pairs at an estimated 6.7% twin-flip rate (95% CI 2.6%–15.9%). On CUB-200 and Cars196, we did not re-annotate the official test splits (prior audits estimate overall residual error on curated fine-grained test sets at ~3%–6%). By r⋆=11+ΔNF/ΔAr^\star = \frac{1}{1 + \Delta\mathrm{NF}/\Delta A}, single-digit error sits far below the threshold needed to flip 15–35 point clean gaps, so we find no ranking inversion on these curated official test splits.

  • Q3 (Real benchmarks currently in use where test noise causes ranking inversions): On curated benchmarks with extra-cleaned test splits (standard CIFAR/ImageNet test sets, Flowers102, or ImageNet ReaL at ~12% subset error), no—error rates are too low to flip wide clean gaps, though r⋆=11+ΔNF/ΔAr^\star = \frac{1}{1 + \Delta\mathrm{NF}/\Delta A} implies closely matched contenders (1–2 point clean gaps with different noise memorization) can flip at ~6%–12% error. Real inversions arise when uncurated crowd or workflow annotations serve directly as the evaluation target: on held-out crowd labels from CIFAR-100N (40.2% overall error; 54%–76% in the hardest strata), human test noise reverses a 13.4-point clean deficit into a 3.8-point lead with zero synthetic corruption. This pinpoints where the hazard bites in practice: LNL model selection and domain benchmarks (crowd, clinical, ecological) splitting one noisy annotation pool into train and validation/test sets without a clean reference.

  • Q4 (Difference from confusion-matrix analyses, and applying PAD when rr and τ\tau are unknown): (i) Full K×KK\times K confusion-matrix analyses require O(K2)O(K^2) transition entries (ill-posed at large KK without gold labels) and give no closed-form reversal threshold, while classical latent-class corrections (Hui–Walter) assume classifier and label errors are conditionally independent and uniform across wrong classes—both violated by construction when LNL models train on the same confusion structure. Projecting onto τ(y)\tau(y) gives a two-scalar law (A,NF)(A,\mathrm{NF}) with closed-form r⋆r^\star, and lets PAD measure error coupling via v=P(y^=τ(y~))v=P(\hat{y}=\tau(\tilde{y})) instead of assuming independence. (ii) When τ\tau and rr are unknown, PAD is a targeted subset tool, not a universal clean-test replacement: zero-label lexical τ\tau works when confusions concentrate on semantic nearest neighbors (10/10 top-1 partners on CIFAR-10N, softening to 53%–77% on 1000-class ReaL), and rsubr_{\text{sub}} must come either at zero audit cost from crowd votes (Dawid–Skene) or from a small audit mm—beating direct audit accuracy on many-class mid-accuracy sets like ReaL (6×6\times smaller variance factor), losing to direct auditing on near-ceiling few-class sets, and abstaining near r→0.5r\to 0.5 or under outside-class inflow.

We appreciate your time and the reminder to foreground analytical insight over bookkeeping.

Thank you for your re-explanation. This version is much more readable and somehow convinces me partly of my concern. I will update my rating. I have no further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to the Area Chair and All Reviewers

We thank all three reviewers for their rigorous and constructive feedback. We are encouraged that all reviewers found our evaluation problem consequential, appreciated our experimental design (fixing model predictions while varying test labels to isolate evaluation distortion, alongside validations on real human annotations in CIFAR-100N/10N and text in 20 Newsgroups), and valued our candor in reporting negative results and method boundaries.

In the revised PDF, we address the theoretical, experimental, and editorial issues raised by the reviewers, and state explicitly where a limitation remains:

1. Matched 3-Seed Final-Checkpoint Study & Standard ELR (Reviewers 3oAN, vtN9)

  • New primary matched study (Tables 1, 2, 4; Secs 5–7.1; App A): To eliminate peak-vs-final checkpoint ambiguity, provide 3-seed standard deviations across all methods, and include stronger CLIP baselines (CLIP selection, CLIP pseudo-labels) on equal footing, we completed a 90-evaluation matched final-checkpoint study (81 trained runs + 9 zero-shot across Flowers102, CUB-200, Cars196, 10 methods, seeds 42/43/44, 20 epochs, no test-peak selection). Auditing our historical code also revealed that the archived ELR runs (now marked ELR†\text{ELR}^\dagger in Fig. 1 and historical Table 6) used −βlog⁡⟨p,t⟩-\beta\log\langle p,t\rangle; the new matched study uses the standard ELR loss λlog⁡(1−⟨p,t⟩)\lambda\log(1-\langle p,t\rangle) (App A.2, Tables 7–8).
  • 98/108 analytical and 85/108 in-range crossovers: Across the 9 trained methods (36 pairs/dataset, 108 total), Spearman(A,NF)\text{Spearman}(A,\text{NF}) is −0.883,−0.983,−0.900-0.883, -0.983, -0.900. Analytical crossings (0<r⋆<10<r^\star<1) occur in 98/108 pairs (85/108 [78.7%] within r⋆≤0.7r^\star \le 0.7, and 55/108 [50.9%] within r⋆≤0.5r^\star \le 0.5; medians 0.471–0.5260.471\text{–}0.526, Table 2). Re-injecting finite-sample noise into the 81 saved prediction vectors (realized column in Table 2) yields crossings in the exact same 85/108 pairs (median location discrepancy 0.00550.0055). In the historical 7-method study (App B.5, Table 14), 50/63 (79.4%) cross within the sampled grid r≤0.7r \le 0.7 (21/63 within r≤0.5r \le 0.5; 58/63 for r⋆<1r^\star < 1).

2. Exact Finite-KK Symmetric-Noise and Partial-Alignment Formulas (Reviewers j4nK, 3oAN)

  • Exact symmetric-noise formula (Sec 3.3, App B.6, App L.3): Under uniform flips to the remaining K−1K-1 classes, a correct prediction (y^=y\hat{y}=y, rate AA) matches the flipped label with probability 00, while an incorrect prediction (1−A1-A) matches it with probability 1K−1\frac{1}{K-1}. Thus NFsym=1−AK−1\mathrm{NF_{sym}} = \frac{1-A}{K-1} (matching App D.2 Eq. 11 and App I.2 Eq. 16), giving, under symmetric noise:

Acc(r)=(1−r)A+r1−AK−1=(1−KrK−1)A+rK−1\mathrm{Acc}(r) = (1-r)A + r\frac{1-A}{K-1} = \left(1 - \frac{Kr}{K-1}\right)A + \frac{r}{K-1}

ΔAcc(r)=(1−KrK−1)ΔA\Delta\mathrm{Acc}(r) = \left(1 - \frac{Kr}{K-1}\right)\Delta A

Expected rankings are strictly preserved for all r<K−1Kr < \frac{K-1}{K}, tie at r=K−1Kr = \frac{K-1}{K} (≥0.990\ge 0.990 for K≥102K \ge 102), and reverse only above it; accuracy ratios are constant once the chance term rK−1=O(r/K)\frac{r}{K-1} = O(r/K) (≤0.007\le 0.007 for K≥102K \ge 102 over r≤0.7r \le 0.7) is subtracted. On 20 Newsgroups (K=20K=20, App G.1, Fig. 2, Table 15), NFsym\mathrm{NF_{sym}} matches 1−AK−1\frac{1-A}{K-1} within 0.0070.007: no noisy method crosses the oracle below r=0.94r = 0.94 (matches K−1K=0.95\frac{K-1}{K} = 0.95; across all 6 pairs, finite-sample fit noise puts near-tied CE–kkNN at r≈0.72r \approx 0.72, still beyond the plotted r≤0.7r \le 0.7).

  • Exact finite-KK partial alignment (App I.2 Eq. 16–17, Fig. 8): With ΔA=Aa−Ab\Delta A = A_a - A_b and ΔNF(1)=NFb(1)−NFa(1)\Delta\text{NF}(1) = \text{NF}_b(1) - \text{NF}_a(1), retaining 1−AK−1\frac{1-A}{K-1} gives ΔNF(ρ)=ρΔNF(1)+(1−ρ)ΔAK−1\Delta\text{NF}(\rho) = \rho\Delta\text{NF}(1) + (1-\rho)\frac{\Delta A}{K-1} and the exact crossover:

r⋆(ρ)=ΔA(1+1−ρK−1)ΔA+ρΔNF(1).r^\star(\rho) = \frac{\Delta A}{\left(1 + \frac{1-\rho}{K-1}\right)\Delta A + \rho\Delta\text{NF}(1)}.

Across all 232 pair–alignment cases in (0,1)(0,1), the max difference from ΔAΔA+ρΔNF(1)\frac{\Delta A}{\Delta A + \rho\Delta\text{NF}(1)} is 0.00656 (median 0.001010.00101).

3. Recalibrated PAD Scope & Editorial Fixes (All Reviewers)

  • PAD scope and selection trap (Abstract, Secs 8–9, App K Tables 26–27): We explicitly scope PAD as a mutual-twin subset estimator requiring a subset noise rate rsubr_{\mathrm{sub}} and compatible transition/selection assumptions, analyze why closed pairs (0/100/10 on CIFAR-10N), global matching, and naive r^1−r^N(y~=j)\frac{\hat{r}}{1-\hat{r}}N(\tilde{y}=j) subtraction do not provide a universal inflow fix (Sec 8 L337–343), and report observed-label selection and simulated rate-estimation intervals in Tables 26–27.
  • Formatting & notation: All appendices (A–L) and floats are properly placed, duplicate text removed, overloaded symbols (τ,q,r\tau, q, r) disambiguated, and the Sec 7.2/B.5 ordering and CIFAR-100N noisy_label wording corrected.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.