- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies a noisy-label pool with reviewed clean labels available for a subset of examples. Under edits at random and positive review coverage, it identifies the correction posterior, clean-label distribution, noise transition, and clean risk. The method estimates this posterior from reviewed examples and propagates soft targets to the full pool. Validation includes simulations, human-noise benchmarks, and a prospective annotation-workflow study.
Critical strengths
- The observation model captures a useful curation setting and explicitly distinguishes random auditing from selective correction. This makes the conditions needed to transport reviewed-label information to the full pool clear.
- Estimating the correction posterior avoids transition-matrix inversion. The experiments cover several noise and label-space regimes, report failures as well as gains, and use family-disjoint historical evaluation to reduce leakage.
Critical weaknesses
- The contribution beyond existing validation-sample and alignment-set methods needs clearer evidence. EAR corresponds to MAR, and the identification result follows from conditional independence and Bayes’ rule. Tjandra and Wiens, “Leveraging an Alignment Set in Tackling Instance-Dependent Label Noise” (CHIL 2023), already use paired observed and ground-truth labels, but no direct comparison is provided. The Fresh-clean baseline also confounds pairing with the learning procedure: its examples retain noisy labels and could form correction pairs, whereas PCL additionally learns a correction model and propagates soft targets. A matched ablation with and without the original noisy-label input would help establish whether pairing itself explains the gains.
- The claimed guarantees and practical validation need tighter qualification. Appendix A.3 invokes regular parametric MLE theory without an explicit nonasymptotic result; its KL projection does not generally justify the stated best-L1 approximation term. The bounded-loss and independent-estimation assumptions also need a connection to the implementation. The prospective system predicts review actions rather than the correction posterior, changes selection and recommendations together, and uses item-level tests despite repeated participants and reported order effects. It supports system feasibility, but does not directly validate the theoretical learner or isolate its benefit.
Questions for authors
- Sections 1 and 4: What is the main advance over existing alignment-set learning? Can you provide a direct comparison with Tjandra and Wiens (2023)?
- Section 2.3 and Appendix A.3: Can you state the finite-sample conditions for the MLE rate, correct the misspecification term, and explain how the guarantee applies to cross-entropy and the implemented cross-fitting?
- Section 3.2: With the same reviewed subset, representation, propagation procedure, and training budget, how does g(Y|X) compare with g(Y|X,noisy Y)? This would directly test the value of retaining the original label.
- Section 3.3 and Appendix D: Do the quality gains remain after accounting for participant/question clustering and block order? What are the initial error rates in each condition, and how much of the gain is attributable to candidate selection?
6 comments
Initial Response Part 2
Ablation: Conditioning on the Original Noisy Label [W1, Q3]
We thank the reviewer for highlighting the potential confounding factors in the Fresh-clean comparison. While Figure 2 compares PCL with Replace-corrected and Fresh-clean under matched clean-label budgets, differences in sample coverage and training procedures may also contribute to the observed performance gaps.
To more directly isolate the contribution of the original noisy label , we conduct an additional ablation comparing three correction models: , , and , while keeping the audit budget and downstream corrected-risk learning procedure fixed.
| Method | CIFAR-10N (Aggregate) 1% | CIFAR-10N (Aggregate) 5% | CIFAR-10N (Aggregate) 10% | CIFAR-10N (Worst) 1% | CIFAR-10N (Worst) 5% | CIFAR-10N (Worst) 10% | CIFAR-100N 1% | CIFAR-100N 5% | CIFAR-100N 10% |
|---|---|---|---|---|---|---|---|---|---|
| Correction target in the PCL risk | |||||||||
| onehot(ỹ) (Noisy CE) | 89.33 ± 0.19 | 89.33 ± 0.19 | 89.33 ± 0.19 | 66.72 ± 0.27 | 66.72 ± 0.27 | 66.72 ± 0.27 | 53.69 ± 0.25 | 53.69 ± 0.25 | 53.69 ± 0.25 |
| ĝ(y∣ỹ) | 89.69 ± 0.37 | 90.09 ± 0.16 | 90.33 ± 0.22 | 65.80 ± 0.11 | 68.68 ± 0.20 | 70.76 ± 1.37 | 50.49 ± 0.63 | 52.68 ± 0.61 | 55.31 ± 0.05 |
| ĝ(y∣x) | 87.03 ± 2.99 | 90.49 ± 0.36 | 90.55 ± 0.17 | 73.75 ± 5.63 | 83.69 ± 0.57 | 86.69 ± 0.31 | 39.90 ± 0.59 | 57.70 ± 0.31 | 61.23 ± 0.87 |
| ĝ(y∣x, ỹ) (PCL) | 90.29 ± 0.06 | 91.20 ± 0.07 | 91.62 ± 0.38 | 75.09 ± 0.80 | 83.62 ± 0.38 | 86.71 ± 0.32 | 52.79 ± 0.46 | 61.65 ± 0.15 | 63.95 ± 0.30 |
| Reference (Table 1, as submitted) | |||||||||
| PCL | 90.20 ± 0.27 | 91.14 ± 0.12 | 91.75 ± 0.09 | 74.78 ± 1.58 | 83.33 ± 0.90 | 86.63 ± 0.03 | 52.58 ± 0.75 | 61.46 ± 0.27 | 63.72 ± 0.39 |
As shown in the new ablation table, the full posterior outperforms in 8 of 9 configurations and in all 9 configurations, although the differences are negligible for CIFAR-10N Worst at 5% and 10% audit rates. For example, on CIFAR-100N at a 5% audit rate, the full posterior achieves 61.65% accuracy, compared with 57.70% using and 52.68% using .
These results provide direct empirical evidence that conditioning on the original noisy label can improve downstream learning beyond using the audited clean labels alone, supporting the benefit of modeling the full correction posterior.
Theoretical Guarantees and Assumptions [W2, Q2]
Thank you for raising these concerns.
Theoretical guarantees. We have revised Theorem 4 and Appendix A.3 to clarify the scope of our guarantees. While the general finite-sample bound in Eq. (4) remains unchanged, we now explicitly state the additional finite-sample estimation condition required for the () rate, rather than deriving it directly from asymptotic MLE theory. We have also corrected the misspecification term to use the L1 distance between the population KL projection and the true correction posterior, rather than the best-L1 approximation error.
Theory–implementation gap. Theorem 4 assumes bounded losses and a correction-posterior estimator independent of the downstream training pool. Our practical implementation instead uses standard cross-entropy and two-fold cross-fitting, with representations pretrained on the noisy pool. While cross-fitting reduces direct reuse of audited labels, it does not fully satisfy the theorem's independence assumption. We have clarified these distinctions and explicitly acknowledged that the current guarantee does not directly cover the complete deep-learning implementation.
Initial Response Part 3
Prospective User Study: Scope and Limitations [W2, Q4]
We thank the reviewer for raising these important methodological concerns. In re-examining the original study records, we identified additional details about the deployed design and conducted further analyses to better characterize the observed differences and their limitations.
Study design and scope. We agree that the deployed system predicts review actions (accept/edit/reject), rather than directly estimating the theoretical correction posterior (g). Furthermore, the guided condition combines candidate prioritization with action recommendations, so their individual contributions cannot be isolated. We have clarified this distinction in the revised manuscript.
We also identified an important source of dependence: the guided condition presented the same fixed, confidence-ranked candidate list to all eight participants. Its 235 presentations therefore involved only 50 distinct questions, compared with 207 distinct questions in the baseline. We now explicitly disclose this design and account for its implications when interpreting the statistical results.
-
Participant clustering and block order. The observed accepted-set error rate was 0/193 in the guided condition versus 13/210 in the baseline. Participant-stratified analyses yielded (p=0.0002), and participant-level comparisons favored the guided condition for six participants, with two ties (exact sign-flip (p=0.031)). A Bayesian logistic mixed model with participant and question random intercepts and an order term yielded an odds ratio of 0.04 (95% interval: 0.006–0.27), although this estimate is prior-sensitive because of complete separation.
Importantly, the evidence is weaker when accounting for repeated questions: collapsing to distinct accepted questions gives 0/44 versus 13/174 ((p=0.076)). Moreover, nine of the thirteen baseline errors came from two participants. The observed direction was consistent across both block orders, but the study is underpowered for a reliable assessment of order effects. We therefore no longer interpret item-level significance as sufficient evidence of a robust causal effect.
-
Initial candidate quality. The confirmed pre-review error rates were similar: 50/235 (21.3%) in the guided condition and 59/275 (21.5%) in the baseline. However, some initial states could not be determined, so these proportions do not establish equivalent input difficulty. Within the guided condition, EDIT-recommended candidates had substantially higher confirmed initial error rates than ACCEPT-recommended candidates, indicating that candidate difficulty varied across recommendation types.
-
Candidate selection versus recommendations. We further examined the role of candidate prioritization. Within predicted-EDIT items, the guided condition had 0/75 incorrect accepted items versus 11/141 in the baseline. However, the guided condition preferentially selected high-confidence candidates, and the corresponding high-confidence region had very limited baseline coverage. In a broader confidence band, the baseline error rate was 2/74 versus 0/193 in the guided condition ((p=0.08)). Thus, confidence-based selection may explain part of the observed difference, but the separate effects of selection and recommendations remain unidentified.
Practical contribution and limitations. Despite these limitations, we believe the prospective study provides a useful demonstration of integrating models trained on historical curation records into an existing human-in-the-loop annotation workflow. Conducting the study within a real workflow imposed practical and ethical constraints on participant recruitment, annotation resources, and experimental design. These constraints motivate our feasibility-oriented evaluation, but do not remove the confounding factors identified above.
Accordingly, we have revised our main claim to emphasize feasibility and the observed difference under the combined guided condition, rather than a causal effect of individual system components or direct validation of the theoretical learner. Future work will require a larger, counterbalanced factorial study that independently varies candidate selection and recommendation display, as well as more direct integration of correction-posterior estimation into the workflow.
Initial Response Part 1
We thank the reviewer for their careful and constructive assessment. We appreciate their recognition of the practical relevance of our correction-based observation model, the methodological advantages of correction-posterior estimation, and the breadth of our experimental evaluation. We have taken their concerns regarding novelty, theoretical guarantees, and experimental validity seriously, and provide additional comparisons, controlled ablations, and clarified theoretical and empirical claims below.
Comparison with Tjandra & Wiens (2023) [W1, Q1]
Thank you for highlighting this closely related work. We have added Tjandra & Wiens (2023) as an alignment-set baseline in Table 1.
**Methodological distinction. **Tjandra & Wiens estimate the probability that the observed label is correct,β(x,y~)=P(Y=y~∣x,y~), and use it to reweight the training loss. In contrast, PCL estimates the full correction posterior g(y∣x,y~) and uses it to construct posterior-weighted training targets. This distinction is particularly relevant in multiclass settings, where knowing whether a noisy label is incorrect does not determine which alternative class is correct. Under the evaluated settings in Table 1, PCL outperforms the alignment-set baseline in 14 of 15 configurations, while PCL + DivideMix outperforms it in all 15.
Scope of our contribution. We acknowledge that paired noisy and clean labels have been studied previously, and that identification under EAR builds on the standard missing-at-random principle. Our contribution is to characterize this observation structure in the context of human correction records, including identifiability under explicit audit conditions, failure under selective review, and the separation of correction-estimation and downstream generalization errors. We have clarified the scope and assumptions of the corresponding guarantees in the revised manuscript.