acceptodds
Under review as a conference paper at ICLR 2027

Learning from Partially Corrected Labels: A Framework for Human-in-the-Loop Data Curation with Statistical Guarantees

Abstract

Large language models rely heavily on high-quality data, making annotation and curation central to their development. As models become more capable, curation is increasingly shifting from labeling from scratch toward reviewing, correcting, or rejecting existing outputs. These workflows naturally produce before/after correction records, a structure largely overlooked in conventional learning from noisy labels. We formalize this setting as *Learning from Partially Corrected Labels* (PCL), where every instance has a potentially noisy label and a subset additionally reveals its correction. We show that this additional observation fundamentally changes identifiability under instance-dependent label noise: while the noise process is generally non-identifiable from noisy labels alone, partial correction pairs make the underlying correction process identifiable under mild audit conditions. Building on this result, we develop a learning framework with finite-sample guarantees, achieving an correction-estimation rate with observed corrections. We evaluate PCL across image and text settings, including human-noise benchmarks and historical curation data, and further translate the approach into a real annotation pipeline through a prospective user study. Our results show that correction histories provide reusable supervision for improving downstream learning and data curation, suggesting a promising direction for future human-AI annotation systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Partially Corrected Labels, Human-in-the-loop Data Curation, Annotation Edits, Learning with Noisy Labels, Identifiability, Weak Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xusbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a noisy-label pool with reviewed clean labels available for a subset of examples. Under edits at random and positive review coverage, it identifies the correction posterior, clean-label distribution, noise transition, and clean risk. The method estimates this posterior from reviewed examples and propagates soft targets to the full pool. Validation includes simulations, human-noise benchmarks, and a prospective annotation-workflow study.

Critical strengths
  1. The observation model captures a useful curation setting and explicitly distinguishes random auditing from selective correction. This makes the conditions needed to transport reviewed-label information to the full pool clear.
  2. Estimating the correction posterior avoids transition-matrix inversion. The experiments cover several noise and label-space regimes, report failures as well as gains, and use family-disjoint historical evaluation to reduce leakage.
Critical weaknesses
  1. The contribution beyond existing validation-sample and alignment-set methods needs clearer evidence. EAR corresponds to MAR, and the identification result follows from conditional independence and Bayes’ rule. Tjandra and Wiens, “Leveraging an Alignment Set in Tackling Instance-Dependent Label Noise” (CHIL 2023), already use paired observed and ground-truth labels, but no direct comparison is provided. The Fresh-clean baseline also confounds pairing with the learning procedure: its examples retain noisy labels and could form correction pairs, whereas PCL additionally learns a correction model and propagates soft targets. A matched ablation with and without the original noisy-label input would help establish whether pairing itself explains the gains.
  2. The claimed guarantees and practical validation need tighter qualification. Appendix A.3 invokes regular parametric MLE theory without an explicit nonasymptotic result; its KL projection does not generally justify the stated best-L1 approximation term. The bounded-loss and independent-estimation assumptions also need a connection to the implementation. The prospective system predicts review actions rather than the correction posterior, changes selection and recommendations together, and uses item-level tests despite repeated participants and reported order effects. It supports system feasibility, but does not directly validate the theoretical learner or isolate its benefit.
Questions for authors
  1. Sections 1 and 4: What is the main advance over existing alignment-set learning? Can you provide a direct comparison with Tjandra and Wiens (2023)?
  2. Section 2.3 and Appendix A.3: Can you state the finite-sample conditions for the MLE rate, correct the misspecification term, and explain how the guarantee applies to cross-entropy and the implemented cross-fitting?
  3. Section 3.2: With the same reviewed subset, representation, propagation procedure, and training budget, how does g(Y|X) compare with g(Y|X,noisy Y)? This would directly test the value of retaining the original label.
  4. Section 3.3 and Appendix D: Do the quality gains remain after accounting for participant/question clustering and block order? What are the initial error rates in each condition, and how much of the gain is attributable to candidate selection?
6 comments
AuthorsOpenReview ↗

Initial Response Part 1

We thank the reviewer for their careful and constructive assessment. We appreciate their recognition of the practical relevance of our correction-based observation model, the methodological advantages of correction-posterior estimation, and the breadth of our experimental evaluation. We have taken their concerns regarding novelty, theoretical guarantees, and experimental validity seriously, and provide additional comparisons, controlled ablations, and clarified theoretical and empirical claims below.

Comparison with Tjandra & Wiens (2023) [W1, Q1]

Thank you for highlighting this closely related work. We have added Tjandra & Wiens (2023) as an alignment-set baseline in Table 1.

**Methodological distinction. **Tjandra & Wiens estimate the probability that the observed label is correct,β(x,y~)=P(Y=y~∣x,y~)\beta(x,\tilde y)=P(Y=\tilde y\mid x,\tilde y), and use it to reweight the training loss. In contrast, PCL estimates the full correction posterior g(y∣x,y~)g(y\mid x,\tilde y) and uses it to construct posterior-weighted training targets. This distinction is particularly relevant in multiclass settings, where knowing whether a noisy label is incorrect does not determine which alternative class is correct. Under the evaluated settings in Table 1, PCL outperforms the alignment-set baseline in 14 of 15 configurations, while PCL + DivideMix outperforms it in all 15.

MethodCIFAR-10N (Aggregate) 1%CIFAR-10N (Aggregate) 5%CIFAR-10N (Aggregate) 10%CIFAR-10N (Worst) 1%CIFAR-10N (Worst) 5%CIFAR-10N (Worst) 10%CIFAR-100N 1%CIFAR-100N 5%CIFAR-100N 10%AlleNoise 1%AlleNoise 5%AlleNoise 10%Clothing1M (paired train) 1%Clothing1M (paired train) 5%Clothing1M (paired train) 10%
Alignment set (Tjandra & Wiens, 2023)69.58 ± 1.3085.35 ± 0.9589.90 ± 0.1455.27 ± 0.6281.11 ± 0.3686.92 ± 0.2315.02 ± 6.0936.78 ± 1.4449.56 ± 0.7917.80 ± 1.9640.93 ± 0.6156.06 ± 0.2056.40 ± 1.1665.34 ± 0.4768.35 ± 0.82
PCL (ours)90.20 ± 0.2791.14 ± 0.1291.75 ± 0.0974.78 ± 1.5883.33 ± 0.9086.63 ± 0.0352.58 ± 0.7561.46 ± 0.2763.72 ± 0.3927.59 ± 1.5956.24 ± 0.2666.54 ± 0.1763.16 ± 1.2369.75 ± 0.8871.93 ± 0.42
PCL + DivideMix (ours)89.56 ± 0.7389.71 ± 0.2090.69 ± 0.4386.27 ± 0.2390.31 ± 0.2491.21 ± 0.4963.05 ± 0.6067.27 ± 0.1668.76 ± 0.2641.64 ± 0.7156.63 ± 0.4165.51 ± 0.1967.24 ± 1.2372.35 ± 0.5673.04 ± 0.26

Scope of our contribution. We acknowledge that paired noisy and clean labels have been studied previously, and that identification under EAR builds on the standard missing-at-random principle. Our contribution is to characterize this observation structure in the context of human correction records, including identifiability under explicit audit conditions, failure under selective review, and the separation of correction-estimation and downstream generalization errors. We have clarified the scope and assumptions of the corresponding guarantees in the revised manuscript.

AuthorsOpenReview ↗

Initial Response Part 2

Ablation: Conditioning on the Original Noisy Label [W1, Q3]

We thank the reviewer for highlighting the potential confounding factors in the Fresh-clean comparison. While Figure 2 compares PCL with Replace-corrected and Fresh-clean under matched clean-label budgets, differences in sample coverage and training procedures may also contribute to the observed performance gaps.

To more directly isolate the contribution of the original noisy label y~\tilde y, we conduct an additional ablation comparing three correction models: g^(y∣y~)\hat g(y\mid\tilde y), g^(y∣x)\hat g(y\mid x), and g^(y∣x,y~)\hat g(y\mid x,\tilde y), while keeping the audit budget and downstream corrected-risk learning procedure fixed.

MethodCIFAR-10N (Aggregate) 1%CIFAR-10N (Aggregate) 5%CIFAR-10N (Aggregate) 10%CIFAR-10N (Worst) 1%CIFAR-10N (Worst) 5%CIFAR-10N (Worst) 10%CIFAR-100N 1%CIFAR-100N 5%CIFAR-100N 10%
Correction target in the PCL risk
onehot(ỹ) (Noisy CE)89.33 ± 0.1989.33 ± 0.1989.33 ± 0.1966.72 ± 0.2766.72 ± 0.2766.72 ± 0.2753.69 ± 0.2553.69 ± 0.2553.69 ± 0.25
ĝ(y∣ỹ)89.69 ± 0.3790.09 ± 0.1690.33 ± 0.2265.80 ± 0.1168.68 ± 0.2070.76 ± 1.3750.49 ± 0.6352.68 ± 0.6155.31 ± 0.05
ĝ(y∣x)87.03 ± 2.9990.49 ± 0.3690.55 ± 0.1773.75 ± 5.6383.69 ± 0.5786.69 ± 0.3139.90 ± 0.5957.70 ± 0.3161.23 ± 0.87
ĝ(y∣x, ỹ) (PCL)90.29 ± 0.0691.20 ± 0.0791.62 ± 0.3875.09 ± 0.8083.62 ± 0.3886.71 ± 0.3252.79 ± 0.4661.65 ± 0.1563.95 ± 0.30
Reference (Table 1, as submitted)
PCL90.20 ± 0.2791.14 ± 0.1291.75 ± 0.0974.78 ± 1.5883.33 ± 0.9086.63 ± 0.0352.58 ± 0.7561.46 ± 0.2763.72 ± 0.39

As shown in the new ablation table, the full posterior g^(y∣x,y~)\hat g(y\mid x,\tilde y) outperforms g^(y∣x)\hat g(y\mid x) in 8 of 9 configurations and g^(y∣y~)\hat g(y\mid\tilde y) in all 9 configurations, although the differences are negligible for CIFAR-10N Worst at 5% and 10% audit rates. For example, on CIFAR-100N at a 5% audit rate, the full posterior achieves 61.65% accuracy, compared with 57.70% using g^(y∣x)\hat g(y\mid x) and 52.68% using g^(y∣y~)\hat g(y\mid\tilde y).

These results provide direct empirical evidence that conditioning on the original noisy label can improve downstream learning beyond using the audited clean labels alone, supporting the benefit of modeling the full correction posterior.

Theoretical Guarantees and Assumptions [W2, Q2]

Thank you for raising these concerns.

Theoretical guarantees. We have revised Theorem 4 and Appendix A.3 to clarify the scope of our guarantees. While the general finite-sample bound in Eq. (4) remains unchanged, we now explicitly state the additional finite-sample estimation condition required for the (O(dG/ne)O(\sqrt{d_{\mathcal G}/n_e})) rate, rather than deriving it directly from asymptotic MLE theory. We have also corrected the misspecification term to use the L1 distance between the population KL projection and the true correction posterior, rather than the best-L1 approximation error.

Theory–implementation gap. Theorem 4 assumes bounded losses and a correction-posterior estimator independent of the downstream training pool. Our practical implementation instead uses standard cross-entropy and two-fold cross-fitting, with representations pretrained on the noisy pool. While cross-fitting reduces direct reuse of audited labels, it does not fully satisfy the theorem's independence assumption. We have clarified these distinctions and explicitly acknowledged that the current guarantee does not directly cover the complete deep-learning implementation.

AuthorsOpenReview ↗

Initial Response Part 3

Prospective User Study: Scope and Limitations [W2, Q4]

We thank the reviewer for raising these important methodological concerns. In re-examining the original study records, we identified additional details about the deployed design and conducted further analyses to better characterize the observed differences and their limitations.

Study design and scope. We agree that the deployed system predicts review actions (accept/edit/reject), rather than directly estimating the theoretical correction posterior (g). Furthermore, the guided condition combines candidate prioritization with action recommendations, so their individual contributions cannot be isolated. We have clarified this distinction in the revised manuscript.

We also identified an important source of dependence: the guided condition presented the same fixed, confidence-ranked candidate list to all eight participants. Its 235 presentations therefore involved only 50 distinct questions, compared with 207 distinct questions in the baseline. We now explicitly disclose this design and account for its implications when interpreting the statistical results.

  1. Participant clustering and block order. The observed accepted-set error rate was 0/193 in the guided condition versus 13/210 in the baseline. Participant-stratified analyses yielded (p=0.0002), and participant-level comparisons favored the guided condition for six participants, with two ties (exact sign-flip (p=0.031)). A Bayesian logistic mixed model with participant and question random intercepts and an order term yielded an odds ratio of 0.04 (95% interval: 0.006–0.27), although this estimate is prior-sensitive because of complete separation.

    Importantly, the evidence is weaker when accounting for repeated questions: collapsing to distinct accepted questions gives 0/44 versus 13/174 ((p=0.076)). Moreover, nine of the thirteen baseline errors came from two participants. The observed direction was consistent across both block orders, but the study is underpowered for a reliable assessment of order effects. We therefore no longer interpret item-level significance as sufficient evidence of a robust causal effect.

  2. Initial candidate quality. The confirmed pre-review error rates were similar: 50/235 (21.3%) in the guided condition and 59/275 (21.5%) in the baseline. However, some initial states could not be determined, so these proportions do not establish equivalent input difficulty. Within the guided condition, EDIT-recommended candidates had substantially higher confirmed initial error rates than ACCEPT-recommended candidates, indicating that candidate difficulty varied across recommendation types.

  3. Candidate selection versus recommendations. We further examined the role of candidate prioritization. Within predicted-EDIT items, the guided condition had 0/75 incorrect accepted items versus 11/141 in the baseline. However, the guided condition preferentially selected high-confidence candidates, and the corresponding high-confidence region had very limited baseline coverage. In a broader confidence band, the baseline error rate was 2/74 versus 0/193 in the guided condition ((p=0.08)). Thus, confidence-based selection may explain part of the observed difference, but the separate effects of selection and recommendations remain unidentified.

Practical contribution and limitations. Despite these limitations, we believe the prospective study provides a useful demonstration of integrating models trained on historical curation records into an existing human-in-the-loop annotation workflow. Conducting the study within a real workflow imposed practical and ethical constraints on participant recruitment, annotation resources, and experimental design. These constraints motivate our feasibility-oriented evaluation, but do not remove the confounding factors identified above.

Accordingly, we have revised our main claim to emphasize feasibility and the observed difference under the combined guided condition, rather than a causal effect of individual system components or direct validation of the theoretical learner. Future work will require a larger, counterbalanced factorial study that independently varies candidate selection and recommendation display, as well as more direct integration of correction-posterior estimation into the workflow.

Reviewer_nwFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes PCL - a framework for learning from noisy labels when a subset of instances has been reviewed, so that both the original noisy label and its correction are observed for the same instance. It primarily targets the distribution of the clean label given the instance and its noisy label, which shows how a potentially noisy label should be corrected in context of this particular instance. The authors demosntrate that if instances are selected for review independently of their true label, these pairs identify the correction posterior, which is not possible from noisy labels alone. The proposed two-stage method fits a correction model on the reviewed pairs and uses its predictions as soft labels to train the classifier on the full dataset. The method is evaluated on a synthetic 1-D task, on CIFAR-10N/100N, Clothing1M and AlleNoise against trusted-data and noisy-label baselines, and in a user study on a Japanese MCQ-curation pipeline.

Critical strengths
  • The paper is well-written and easy to follow.
  • The paper is clearly motivated. The signal that comes from the label correction (where both the original and the corrected labels are available) is indeed valuable but often overlooked.
  • The experimental setup is extensive: the results include a controlled synthetic dataset, real-world noisy datasets, and a live human-in-the-loop setup. The experiments are conducted for images and texts.
Critical weaknesses
  • Novelty. The two-stage procedure (learning a mapping from noisy to verified labels conditioned on the input on a subset where both are observed, and using its outputs as targets for training on the full noisy set) was previously studied in the literature - closest one is [1], but also [2]. It's worth being discussed as well as included for comparison as baselines. In the context of those works, the current paper looks more like a formal analysis of this setup rather than the novel procedure.
  • The empirical gains of the proposed method are limited and don't consistently support the benefit of the approach. In Table 1, simple PCL outperforms all four baselines in only 5 (out of 15) settings, while DivideMix, which uses no audited labels at all, outperforms it in 8. Since most of the best results come from the PCL + DivideMix combination, it is hard to tell how much of the improvement is due to PCL. Moreover, in the low-audit regime PCL can perform worse than ordinary training on the noisy labels; the paper explains this through the estimation error of the correction posterior, but it offers no practical criterion for detecting when PCL will hurt.
  • Consider to increase fonts in Figures 1 & 2.

[1] Veit et al. Learning From Noisy Large-Scale Datasets With Minimal Supervision. CVPR 2017.

[2] Lee et al. CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise. CVPR 2018.

Questions for authors
  • I'm surprised by the scores for several baselines: e.g., DivideMix scores reported in Wei et al. 2022 are considerably higher than the ones provided in this paper (95.01, 92.56, and 71.13 on CIFAR-10N Aggregate, CIFAR-10N Worst, and CIFAR-100N), and those numbers would outperform all PCL variants on these datasets, incl. PCL + DivideMix. Are these differences explained by a different architecture or training schedule? Could you also report the baselines under their standard settings?
  • In the user study, how was the correctness of accepted questions determined?
4 comments
AuthorsOpenReview ↗

Initial Response Part 1

We thank the reviewer for their thoughtful evaluation and constructive feedback. We appreciate their recognition of the paper's clear motivation, the value of correction pairs as an often-overlooked source of supervision, and the extensive evaluation across image and text domains, including a real-world human-in-the-loop study. We address the reviewer's concerns regarding novelty, empirical performance, and experimental methodology below.

Novelty and Comparison with Prior Work [w1]

We thank the reviewer for highlighting these relevant works. We agree that the two-stage procedure itself is not new, and will clarify our contribution accordingly. Veit et al. [1] learn an input-dependent noisy-to-clean mapping using a residual label-cleaning network, while CleanNet [2] primarily learns image-label correctness for reweighting. Similarly, Tjandra & Wiens (2023) estimate label correctness rather than the full multiclass correction posterior.

Our main contribution is a statistical characterization of this previously explored but theoretically underanalyzed setting. Specifically, we establish (i) identifiability of the instance-dependent correction process under explicit audit conditions, including failure cases under selective review; and (ii) finite-sample excess-risk guarantees distinguishing the correction budget nen_e from the full dataset size N.

Following the reviewer's suggestion, we additionally implemented all three methods as baselines. Please refer to the revised manuscript for the full values. At a 10% audit rate, PCL achieves 86.63% on CIFAR-10N Worst versus 71.22%/82.85% for [1]/[2], and 71.93% on Clothing1M versus 57.52%/67.11%.

MethodCIFAR-10N (Aggregate) 1%CIFAR-10N (Aggregate) 5%CIFAR-10N (Aggregate) 10%CIFAR-10N (Worst) 1%CIFAR-10N (Worst) 5%CIFAR-10N (Worst) 10%CIFAR-100N 1%CIFAR-100N 5%CIFAR-100N 10%AlleNoise 1%AlleNoise 5%AlleNoise 10%Clothing1M (paired train) 1%Clothing1M (paired train) 5%Clothing1M (paired train) 10%
L2RW90.72 ± 0.3090.90 ± 0.1190.94 ± 0.1180.86 ± 0.1282.28 ± 0.3482.51 ± 0.1956.97 ± 0.2957.84 ± 0.1458.19 ± 0.4360.34 ± 0.1860.47 ± 0.0760.92 ± 0.1766.31 ± 0.3866.51 ± 0.4666.67 ± 0.72
DivideMix89.39 ± 0.8190.02 ± 0.1990.71 ± 0.3388.52 ± 0.8490.06 ± 0.3890.84 ± 0.5365.04 ± 0.9066.41 ± 0.2967.56 ± 0.1160.96 ± 0.2162.37 ± 0.0262.96 ± 0.0262.94 ± 1.0265.63 ± 0.2968.05 ± 0.35
BLTM (IDN transition)89.59 ± 0.2189.64 ± 0.1889.12 ± 0.3968.84 ± 1.9265.98 ± 1.1954.76 ± 3.4252.77 ± 0.8653.53 ± 0.5653.76 ± 0.4762.47 ± 0.2363.21 ± 0.4963.68 ± 0.2861.59 ± 0.1161.60 ± 0.6861.89 ± 0.24
PCL (ours)90.20 ± 0.2791.14 ± 0.1291.75 ± 0.0974.78 ± 1.5883.33 ± 0.9086.63 ± 0.0352.58 ± 0.7561.46 ± 0.2763.72 ± 0.3927.59 ± 1.5956.24 ± 0.2666.54 ± 0.1763.16 ± 1.2369.75 ± 0.8871.93 ± 0.42
PCL + DivideMix (ours)89.56 ± 0.7389.71 ± 0.2090.69 ± 0.4386.27 ± 0.2390.31 ± 0.2491.21 ± 0.4963.05 ± 0.6067.27 ± 0.1668.76 ± 0.2641.64 ± 0.7156.63 ± 0.4165.51 ± 0.1967.24 ± 1.2372.35 ± 0.5673.04 ± 0.26

Beyond theoretical analysis, we extend the empirical investigation beyond image classification to text classification and, importantly, real-world human-in-the-loop data curation, which is the central motivation of our work. In particular, we leverage historical human corrections to guide future annotation decisions and validate their practical utility through a prospective user study. We will revise the paper to more clearly position these theoretical and practical contributions relative to prior work.

[1] Veit et al. Learning From Noisy Large-Scale Datasets With Minimal Supervision. CVPR 2017.
[2] Lee et al. CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise. CVPR 2018.

AuthorsOpenReview ↗

Initial Response Part 2

DivideMix Baseline Clarification and Additional Ablations [w2 (fisrt-half), q1]

We thank the reviewer for raising this important point and giving us the opportunity to clarify our experimental setup and the role of PCL.

Clarification of DivideMix baselines.In the submitted Table 1, the method labeled "DivideMix" incorporated the available audited labels through a trusted-data adaptation. This was intended to ensure a matched-audit comparison with PCL, but we acknowledge that the distinction from vanilla DivideMix was not sufficiently clear. We apologize for this ambiguity.

We retain the matched-audit comparison in Table 1 and clarify the baseline configuration in the revised manuscript. To address the reviewer's concern about standard DivideMix performance, we additionally report audit-free DivideMix and training-recipe ablations in Appendix C.3.

Additional ablations. To further investigate the contribution of PCL, we conducted an additional ablation using both the original Table 1 setup (PreAct ResNet-18, 100 epochs) and the original DivideMix training recipe (ResNet-34, 300 epochs), following the backbone used by Wei et al. (2022).

Additional ablation results. The following results report clean test accuracy (%, mean ± std). Each cell contains results at 1% / 5% / 10% audit rates, respectively.

MethodCIFAR-10N (Aggregate) 1%CIFAR-10N (Aggregate) 5%CIFAR-10N (Aggregate) 10%CIFAR-10N (Worst) 1%CIFAR-10N (Worst) 5%CIFAR-10N (Worst) 10%CIFAR-100N 1%CIFAR-100N 5%CIFAR-100N 10%
Reported (Wei et al., 2022; ResNet-34)
DivideMix95.01 ± 0.7195.01 ± 0.7195.01 ± 0.7192.56 ± 0.4292.56 ± 0.4292.56 ± 0.4271.13 ± 0.4871.13 ± 0.4871.13 ± 0.48
Table 1 budget (PreAct-RN18, 100 epochs, batch 128, cosine LR)
DivideMix93.89 ± 0.2493.89 ± 0.2493.89 ± 0.2487.93 ± 0.6387.93 ± 0.6387.93 ± 0.6365.59 ± 0.2365.59 ± 0.2365.59 ± 0.23
DivideMix + adaptation (submitted)89.39 ± 0.8190.02 ± 0.1990.71 ± 0.3388.52 ± 0.8490.06 ± 0.3890.84 ± 0.5365.04 ± 0.9066.41 ± 0.2967.56 ± 0.11
PCL + DivideMix + adaptation (submitted)89.56 ± 0.7389.71 ± 0.2090.69 ± 0.4386.27 ± 0.2390.31 ± 0.2491.21 ± 0.4963.05 ± 0.6067.27 ± 0.1668.76 ± 0.26
Original DivideMix recipe (ResNet-34, 300 epochs, batch 64, step LR)
DivideMix95.79 ± 0.1895.79 ± 0.1895.79 ± 0.1893.53 ± 0.1793.53 ± 0.1793.53 ± 0.1770.11 ± 0.1270.11 ± 0.1270.11 ± 0.12
PCL + DivideMix (ĝ prior)95.76 ± 0.1295.63 ± 0.0495.61 ± 0.1889.07 ± 1.4492.18 ± 0.0693.60 ± 0.1766.86 ± 0.2971.47 ± 0.3272.07 ± 0.68
Original recipe + trusted-data adaptation
DivideMix + adaptation95.55 ± 0.2295.89 ± 0.2395.91 ± 0.1393.11 ± 0.3194.10 ± 0.3094.47 ± 0.1270.74 ± 0.3972.19 ± 0.3473.27 ± 0.28
PCL + DivideMix + adaptation95.71 ± 0.0895.71 ± 0.2695.58 ± 0.1189.37 ± 0.8992.41 ± 0.5093.88 ± 0.1867.60 ± 0.5472.57 ± 0.1773.48 ± 0.46

These results reveal both the benefits and limitations of PCL. On CIFAR-10N Aggregate, performance is already near saturation, and PCL provides little additional benefit. On CIFAR-10N Worst, PCL improves the submitted adapted DivideMix baseline at 5–10% audits, but the original ResNet-34 experiments show that trusted-data adaptation alone can perform better. On CIFAR-100N, PCL provides clearer benefits as the audit budget increases: with the original DivideMix recipe, PCL + DivideMix reaches 72.07% versus 70.11% for vanilla DivideMix at 10% audits. With trusted-data adaptation, PCL also improves accuracy at 5% (72.57% vs. 72.19%) and 10% (73.48% vs. 73.27%).

We therefore do not claim that PCL universally outperforms robust noisy-label methods. Rather, PCL provides an additional correction signal that can complement existing learning methods, particularly when sufficient paired corrections are available. The results also demonstrate its limitations under small audit budgets and when strong existing methods already perform well.

Since Wei et al. (2022) focused on CIFAR-N, we retain the original experimental backbone for the other benchmarks. The revised manuscript clarifies the matched-audit protocol used in Table 1, while Appendix C.3 provides the full standard-recipe replication and controlled ablations.

AuthorsOpenReview ↗

Initial Response Part 3

Practical Method Selection under Limited Audit Budgets [w2 (seconde-half)]

We acknowledge that PCL is not universally preferable, particularly when the available correction pairs are insufficient for reliable estimation. As shown in Table 1, robust learning methods may outperform vanilla PCL at low audit fractions, while PCL can also complement existing methods such as DivideMix.

To address how practitioners can select an appropriate method, we conduct an additional 5-fold cross-validation experiment on CIFAR-100N, using only the available audited clean labels for validation, without requiring additional human annotations. We compare PCL, Replace-corrected, DivideMix, and PCL + DivideMix under the same audit budget.

MethodCV 1%CV 5%CV 10%Test 1%Test 5%Test 10%
PCL51.33 ± 2.7260.29 ± 1.2062.91 ± 0.6052.83 ± 0.3661.69 ± 0.3063.89 ± 0.65
Replace-corrected53.73 ± 1.8654.47 ± 0.2455.14 ± 0.7853.59 ± 0.2754.79 ± 0.3255.78 ± 0.73
DivideMix65.27 ± 1.1066.16 ± 1.2565.60 ± 0.1665.88 ± 0.5065.53 ± 0.2465.60 ± 0.09
PCL + DivideMix62.07 ± 0.5867.64 ± 0.3668.36 ± 0.2663.81 ± 0.1567.96 ± 0.2769.18 ± 0.09

At audit fractions of 1%, 5%, and 10%, cross-validation selects DivideMix, PCL + DivideMix, and PCL + DivideMix, respectively. These choices match the best-performing methods on the independent test set in all three evaluated settings, achieving zero selection regret relative to the tested candidates.

These results suggest that cross-validation over existing audited data can provide a practical strategy for selecting among PCL and simpler alternatives, although selection may become less reliable when the audited subset is extremely small.

We will clarify the motivation for the audit-fraction experiments, the observed performance trade-offs, and the distinction between matched audit-label budgets and actual human annotation costs in the revised manuscript.

Ground-Truth Determination in the User Study [q2]

We thank the reviewer for this question. The correctness of accepted questions was independently evaluated by an annotator who did not participate in the user study. The evaluator verified the correctness of each final answer against the original source material. To ensure a fair evaluation, the evaluator was blinded to whether each question came from the model-guided or baseline condition.

We will clarify this evaluation procedure in the revised manuscript.

Figure Readability [w3]

Thank you for pointing this out. We have increased the font sizes in Figures 1 and 2 to improve readability.

AuthorsOpenReview ↗

Gentle Reminder

We would like to thank you again for your constructive feedback, and give a gentle reminder that the discussion period will close soon.

In case it is helpful before the deadline, here is a brief summary of how our response and revised manuscript address your comments:

Prior Correction-Pair Baselines (Table 1): Added direct comparisons with Veit et al. (2017), CleanNet (Lee et al., 2018), and Tjandra & Wiens (2023).

Standard DivideMix Ablations (Appendix C.3, Table 9): Clarified the matched-audit setup in Table 1 and added full ablations under the original ResNet-34 (300-epoch) DivideMix recipe.

Low-Audit Selection Criterion: Added a 5-fold cross-validation experiment using only existing audited labels on CIFAR-100N, achieving zero selection regret across 1%, 5%, and 10% audit rates.

User Study & Figure Readability (Section 3.3, Appendix D.3, Figures 1–2): Clarified that final-answer correctness was evaluated by an independent, condition-blinded annotator, and enlarged the fonts in Figures 1 and 2.

We would be happy to further discuss any unresolved questions that you may have!

Reviewer_sMvkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of learning from noisy labels. It points out that, in general, the posterior probabilities of clean labels cannot be identified using noisy labels alone. The authors propose PCL, a method that leverages pairs of noisy labels and their corrected counterparts, obtained from a subset of verified samples, to identify the correction posterior, clean-label posterior, and clean risk. Furthermore, the paper provides finite-sample error bounds for the proposed approach. Experimental results, spanning synthetic data, various real-world noisy-label benchmarks, and a Japanese multiple-choice question annotation workflow, demonstrate the effectiveness of the method.

Critical strengths
  1. By utilizing partially corrected label pairs, this paper establishes identifiability of the correction posterior, clean-label posterior, and clean risk under instance-dependent label noise, given the EAR and coverage conditions, without the need to recover or invert the noise transition matrix.

  2. Experiments on information matching demonstrate that corrected pairs for the same instance provide additional supervisory information. This advantage is particularly pronounced on the CIFAR-10N "Worst" dataset when a higher proportion of labels is verified.

Critical weaknesses
  1. The paper evaluates PCL by varying the audit fraction, but in practice, the audit fraction is typically determined by upstream processes.

  2. Increasing the audit fraction not only yields more correction pairs but also directly improves label quality. The paper does not sufficiently clarify when PCL should be chosen over simpler methods, particularly given a fixed audit budget and low audit fractions, nor does it provide a thorough analysis of the performance differences between various methods when held to the same audit cost.

Questions for authors
  1. Given a fixed budget for auditing, how can one determine whether PCL outperforms methods based on trusted data or robust learning baselines?
  2. Could the authors propose a practical validation scheme that accounts for both auditing costs and the impact of the top-16 approximation used for AlleNoise?
  3. When auditing propensities are unknown, how can data from random audits and voluntary error corrections be combined to mitigate selection bias and ensure identifiability?
  4. Could the authors further analyze the impact of random audit sample size on model performance, and distinguish between the conditions required for identifiability versus those needed for stable estimation?
5 comments
AuthorsOpenReview ↗

Initial Response Part 1

We thank the reviewer for their thoughtful evaluation and for recognizing the theoretical and empirical contributions of our work, particularly the identifiability guarantees enabled by partially corrected labels and the additional supervisory value of same-instance correction pairs. We appreciate the constructive feedback and address the concerns and questions below.

Motivation and Benefits of Paired Corrections [w1,w2]

We thank the reviewer for raising these practical concerns. We agree that the audit fraction is often determined by upstream annotation processes and may not be directly controlled by the learner. We would like to clarify that PCL is motivated by the availability of existing correction records, rather than by the need to optimize the upstream auditing process. Accordingly, the audit-fraction sweeps in Table 1 and Figure 2 are intended to characterize the performance and limitations of PCL under different amounts of available supervision, rather than to suggest that practitioners should adjust the audit fraction. PCL does not require an adjustable audit budget, and Theorem 4 characterizes how estimation error depends on the number of observed correction pairs.

We agree that increasing the audit fraction provides more clean labels. However, Figure 2 explicitly distinguishes this effect from the benefit of paired corrections by comparing PCL with Replace-corrected and Fresh-clean under matched clean-label budgets.

To further isolate the contribution of the observed noisy label y~\tilde y, we conduct an additional ablation comparing three correction targets: g^(y∣y~)\hat g(y\mid\tilde y), g^(y∣x)\hat g(y\mid x), and g^(y∣x,y~)\hat g(y\mid x,\tilde y), keeping the audit budget fixed.

MethodCIFAR-10N (Aggregate) 1%CIFAR-10N (Aggregate) 5%CIFAR-10N (Aggregate) 10%CIFAR-10N (Worst) 1%CIFAR-10N (Worst) 5%CIFAR-10N (Worst) 10%CIFAR-100N 1%CIFAR-100N 5%CIFAR-100N 10%
Correction target in the PCL risk
onehot(ỹ) (Noisy CE)89.33 ± 0.1989.33 ± 0.1989.33 ± 0.1966.72 ± 0.2766.72 ± 0.2766.72 ± 0.2753.69 ± 0.2553.69 ± 0.2553.69 ± 0.25
ĝ(y∣ỹ)89.69 ± 0.3790.09 ± 0.1690.33 ± 0.2265.80 ± 0.1168.68 ± 0.2070.76 ± 1.3750.49 ± 0.6352.68 ± 0.6155.31 ± 0.05
ĝ(y∣x)87.03 ± 2.9990.49 ± 0.3690.55 ± 0.1773.75 ± 5.6383.69 ± 0.5786.69 ± 0.3139.90 ± 0.5957.70 ± 0.3161.23 ± 0.87
ĝ(y∣x, ỹ) (PCL)90.29 ± 0.0691.20 ± 0.0791.62 ± 0.3875.09 ± 0.8083.62 ± 0.3886.71 ± 0.3252.79 ± 0.4661.65 ± 0.1563.95 ± 0.30
Reference (Table 1, as submitted)
PCL90.20 ± 0.2791.14 ± 0.1291.75 ± 0.0974.78 ± 1.5883.33 ± 0.9086.63 ± 0.0352.58 ± 0.7561.46 ± 0.2763.72 ± 0.39

The results show that incorporating both x and y~\tilde y generally improves downstream performance. For example, on CIFAR-100N at a 5% audit fraction, PCL achieves 61.65% accuracy, compared with 57.70% using g^(y∣x)\hat g(y\mid x) and 52.68% using g^(y∣y~)\hat g(y\mid\tilde y). These findings provide additional evidence that the original noisy label contains useful information for learning the correction posterior, beyond simply obtaining more clean labels.

AuthorsOpenReview ↗

Initial Response Part 2

Practical Method Selection under a Fixed Audit Budget [q1,q2(fisrt-half)]

We acknowledge that PCL is not universally preferable, particularly when the available correction pairs are insufficient for reliable estimation. As shown in Table 1, robust learning methods may outperform vanilla PCL at low audit fractions, while PCL can also complement existing methods such as DivideMix.

To address how practitioners can select an appropriate method, we conduct an additional 5-fold cross-validation experiment on CIFAR-100N, using only the available audited clean labels for validation, without requiring additional human annotations. We compare PCL, Replace-corrected, DivideMix, and PCL + DivideMix under the same audit budget.

MethodCV 1%CV 5%CV 10%Test 1%Test 5%Test 10%
PCL51.33 ± 2.7260.29 ± 1.2062.91 ± 0.6052.83 ± 0.3661.69 ± 0.3063.89 ± 0.65
Replace-corrected53.73 ± 1.8654.47 ± 0.2455.14 ± 0.7853.59 ± 0.2754.79 ± 0.3255.78 ± 0.73
DivideMix65.27 ± 1.1066.16 ± 1.2565.60 ± 0.1665.88 ± 0.5065.53 ± 0.2465.60 ± 0.09
PCL + DivideMix62.07 ± 0.5867.64 ± 0.3668.36 ± 0.2663.81 ± 0.1567.96 ± 0.2769.18 ± 0.09

At audit fractions of 1%, 5%, and 10%, cross-validation selects DivideMix, PCL + DivideMix, and PCL + DivideMix, respectively. These choices match the best-performing methods on the independent test set in all three evaluated settings, achieving zero selection regret relative to the tested candidates.

These results suggest that cross-validation over existing audited data can provide a practical strategy for selecting among PCL and simpler alternatives, although selection may become less reliable when the audited subset is extremely small.

We will clarify the motivation for the audit-fraction experiments, the observed performance trade-offs, and the distinction between matched audit-label budgets and actual human annotation costs in the revised manuscript.

Top-k Approximation on AlleNoise [q2(second-half)]

Thank you for this constructive suggestion. To examine the impact of the top-16 approximation, we conducted an additional ablation on AlleNoise with top-k values of 4, 8, 16, and 64, while keeping the audit budget and other training configurations fixed.

top-kAcc 1%Acc 5%Acc 10%Mass kept 1%Mass kept 5%Mass kept 10%
429.52 ± 1.5154.86 ± 0.1066.13 ± 0.200.0087 ± 0.00020.1742 ± 0.00150.6087 ± 0.0012
828.87 ± 1.5755.75 ± 0.1666.44 ± 0.100.0139 ± 0.00020.1963 ± 0.00150.6438 ± 0.0011
1627.59 ± 1.5956.24 ± 0.2666.54 ± 0.180.0229 ± 0.00020.2224 ± 0.00150.6772 ± 0.0011
6424.96 ± 1.4856.39 ± 0.3366.71 ± 0.250.0662 ± 0.00010.2974 ± 0.00210.7454 ± 0.0010

At moderate audit fractions (5% and 10%), downstream performance is relatively insensitive to the choice of k. Increasing k from 16 to 64 improves accuracy by only 0.15 and 0.17 percentage points, respectively, despite retaining more estimated posterior probability mass. This suggests that top-16 provides a reasonable trade-off between computational efficiency and downstream performance in these regimes.

At the lowest audit fraction (1%), we observe a different pattern: increasing k consistently decreases accuracy, from 29.52% at k=4 to 24.96% at k=64. Combined with the correction posterior estimation diagnostics in Appendix C.2, this suggests that the main difficulty in this regime is insufficient supervision for accurately estimating the correction posterior, rather than overly aggressive truncation. Retaining additional low probability predictions may introduce unreliable supervision rather than improve learning.

Overall, these results indicate that the effect of top-k truncation depends on the reliability of the estimated correction posterior. We will incorporate this analysis and clarify the associated computational and estimation trade-offs in the revised manuscript.

AuthorsOpenReview ↗

Initial Response Part 3

Unknown Audit Propensities and Selection Bias [q3]

Thank you for raising this important practical question. We agree that voluntary error corrections may introduce outcome-dependent selection bias, and that their propensities are often unknown in practice.

As discussed in Section 2.4 and Appendix A.6, voluntary corrections alone generally do not identify the population correction posterior when their selection propensities are unknown. However, a small, separately logged random-audit subset can serve as an unbiased anchor. Under EAR and positive coverage, the correction posterior remains identifiable from this subset, regardless of the unknown voluntary-correction mechanism (Theorem 3).

A practical approach is therefore to first estimate the correction posterior using random audits and then use voluntary corrections as auxiliary supervision with appropriate bias adjustment. When the two review sources are distinguishable and have sufficient overlap, their conditional label distributions can also be used to estimate relative selection propensities for reweighting voluntary corrections.

We will clarify this practical strategy and emphasize that identifiability is guaranteed by the random-audit stratum, while the finite-sample benefits of incorporating voluntary corrections depend on the accuracy of the bias-adjustment procedure.

Identifiability vs. Finite-sample Estimation [q4]

Thank you for this insightful suggestion. We agree that it is important to distinguish population-level identifiability from stable finite-sample estimation. Regarding identifiability, Theorem 3 establishes that the correction posterior is identifiable under EAR and positive coverage, without requiring a particular minimum audit fraction. However, this is a population-level result and does not guarantee accurate estimation from a small number of audited examples.

Regarding finite-sample estimation, Theorem 4 explicitly characterizes the dependence on the number of correction pairs nen_e, with an estimation-error term of order O(dG/ne)O(\sqrt{d_G/n_e}) under the stated regularity and model assumptions. Figure 1 provides empirical evidence consistent with this dependence. In addition, Table 1 and Appendix C.2 demonstrate that limited audit supervision can lead to poor downstream performance, particularly when the correction posterior is difficult to estimate in large label spaces.

Furthermore, Appendix B.4 examines audit allocation under a fixed budget, illustrating that the distribution of audited examples also matters for finite-sample estimation, beyond the total number of audits.

Comments on the paper

1 comment
Area_Chair_eisAOpenReview ↗

[Open Discussion] Thank you for your review and please check authors' rebuttal

Dear Reviewers sMvk and nwFS,

Thank you both for your thoughtful reviews and constructive feedback. Currently, it is still in the open discussion phase, so reviewers are welcome to have further discussions as well as the authors.

Could you please review the authors' rebuttal and share your updated opinions on whether your concerns have been adequately addressed? Please also let us know if you have any remaining questions or concerns that require further clarification.

Thank you again for your time and valuable contributions to the review process.

Best regards,
AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.