- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares correction counts and worst-group accuracy after retraining under equal label verification budgets. Controlled experiments examine which samples are corrected, and a first-order approximation analyzes the resulting model changes. The theory shows that knowing which labels are wrong can still leave the best correction allocation unidentified when group labels are unavailable. The paper also tests class quotas and policy selection using validation gains.
Critical strengths
- The question is well aimed. Data cleaning usually measures progress by how many label errors are found, and the paper uses worst-group accuracy after retraining instead, with 80 of 510 paired comparisons going the other way.
- The theory is solid. Theorem 1 is an information-theoretic lower bound, and without group annotations the latent partition cannot be identified, so even a perfect detector carries worst-case regret approaching 50 pp.
Critical weaknesses
- Table 19 illustrates the paper's central finding that higher correction recall need not yield higher WGA. On CivilComments with 20% uniform noise and a 10% budget, NoiseScore corrects 59.32% of the initial worst group's errors and reaches 23.24% WGA, while CB-Loss corrects 41.69% and reaches 34.59%. Although Appendix E discusses allocation differences and evaluates a first-order approximation, it does not directly establish which changes in corrected samples account for this specific WGA gap. A direct comparison of the resulting group accuracies and prediction changes would strengthen the explanation.
- The paper recommends an equal budget per observed class as protection for the vulnerable subgroup inside each class. That subgroup can occupy a small part of a class, and the paper does not say whether an even split reaches it. Waterbirds does not support the recommendation: at the 10% budget CB-Loss reaches 54.26% WGA while the uncorrected model reaches 60.33%.
Questions for authors
- In Table 19, which group ends up worst under NoiseScore and which under CB-Loss? Do the corrected samples alone explain the gap between 23.24% and 34.59%?
- After the budget is split evenly across classes, how many corrections does each group receive? Does that number explain why CB-Loss ends at 54.26% on Waterbirds while the uncorrected model reaches 60.33%?
- The first-order analysis predicts group cross-entropy. Applied to the CivilComments 10% budget, does it rank NoiseScore and CB-Loss the same way the measured WGA does?
4 comments
Q2 & W2: Waterbirds subgroup corrections and bottleneck switching
Table 6 (Section 6, p. 9; full three-method breakdown in Appendix Table 25, p. 25) reports the four-group correction counts and accuracies on Waterbirds under the 5-epoch diagnostic protocol (, seeds 0–9):
| Subgroup | Available Errors | CB-Loss Corrections (Recall %) | Baseline Accuracy (%) | Final Accuracy (%) | Worst Group Count (/10) |
|---|---|---|---|---|---|
| Landbird / Land (Majority) | 709.1 | 238.3 (33.65%) | 99.32 | 98.41 | 0 |
| Landbird / Water (Vulnerable Minority) | 36.1 | 0.2 (0.59%) | 74.02 | 54.67 | 9 |
| Waterbird / Land (Initial Worst) | 12.1 | 6.9 (56.58%) | 60.33 | 69.88 | 1 |
| Waterbird / Water (Majority) | 204.0 | 187.2 (91.84%) | 92.17 | 95.42 | 0 |
The group-level numbers show why overall WGA drops from 60.33% to 54.26%:
- The initial worst group (, Waterbird on Land) improves: CB-Loss corrects 56.58% of its label errors, raising accuracy from 60.33% to 69.88% ().
- Intra-class skew shifts the bottleneck to (Landbird on Water): Within the query quota (mislabeled landbirds), majority group takes 238.3 corrections while minority group gets only 0.2 corrections (0.59% recall). After retraining, accuracy drops from 74.02% to 54.67%, making the new worst group in 9 out of 10 seeds.
- As we note in Section 6 (p. 9) and Appendix E.1 (p. 24), because retraining on these queries changes both within-class error coverage and the global decision boundary, this table captures their combined effect and does not establish that low coverage alone causes the full decline. Rather, it illustrates a cross-group accuracy trade-off and shows that equal class quotas do not ensure subgroup coverage. Under longer training (30 epochs) or end-to-end retraining, CB-Loss WGA recovers to 65.23% and 67.39% (Tables 2 and 7, pp. 5, 9; Appendix Table 16, p. 20).
Q3: First-order ranking vs. measured WGA on CivilComments
On CivilComments, first-order continuous cross-entropy (CE) loss predictions rank the methods in the opposite order of empirical WGA—but evaluating discrete 0–1 accuracy at the first-order parameter step recovers the exact refit WGA ordering across all 10 seeds.
Appendix Table 29 (p. 28) tests this directly using an exact L-BFGS convex logistic model on CivilComments (, seeds 0–9, ; under all three methods have 0.00% WGA):
| Method | Predicted CE Gain | Actual CE Gain | Linearized 0–1 WGA (%) | Refit WGA (%) |
|---|---|---|---|---|
| CB-Loss | 27.92% | 30.04% | ||
| Loss | 19.38% | 23.71% | ||
| NoiseScore | 19.02% | 18.96% |
- Continuous CE loss vs. 0–1 WGA: The first-order Taylor expansion accurately tracks empirical cross-entropy reduction on (predicting nats for NoiseScore vs. nats for CB-Loss, matching the actual CE gain ordering ). Yet NoiseScore has the lowest refit 0–1 WGA (18.96% vs. 30.04% for CB-Loss).
- Linearized 0–1 step recovers the true ranking: When we instead evaluate thresholded 0–1 accuracy at the first-order parameter update (), the linearized WGA ordering is CB-Loss (27.92%) > Loss (19.38%) > NoiseScore (19.02%), matching the refit WGA ordering in all 10 seeds.
- We see the same phenomenon in the 5-epoch neural network (Table 5, p. 7): NoiseScore achieves lower test cross-entropy on than CB-Loss (0.9269 vs. 0.9627), yet trails by 11.35 pp in 0–1 accuracy. Because one-sided corrections alter classification margins unevenly, reducing average log-loss on a subgroup does not guarantee more examples cross the zero decision threshold.
Thanks for the response. The per-group results answer my questions on Table 19 and on Waterbirds, and the claim about class quotas now matches what the results show. These results make clear where the gap appears, though not yet why, and the first-order analysis does not predict WGA either, so I'd suggest stating this clearly in the paper. As my main questions have been answered, I'm raising my score to 3.
Thank you for taking the time to review our responses and for raising your score! We completely agree with both of your points and have stated them explicitly in the revised manuscript:
- Observational ledgers show where the gap appears rather than isolating why: In Section 4.3 (p. 7), Section 6 (p. 9), and Appendix E.1 (p. 24), we explicitly state that policy comparisons (such as NoiseScore vs. CB-Loss on CivilComments and the four-group ledger on Waterbirds) jointly change correction counts, label-transition directions, and sample identities, and therefore reflect combined effects rather than isolating a single causal factor on its own.
- First-order group-loss approximation does not predict WGA: In Section 4.3 (p. 7) and Appendix E.2 (p. 28, Table 29), we state directly that the first-order continuous group-loss predictor does not recover the observed WGA ordering on CivilComments (favoring NoiseScore over CB-Loss), and highlight this as a limitation of continuous loss surrogates for thresholded 0–1 subgroup accuracy. Thank you again for your constructive feedback, which has significantly sharpened the precision of our claims.
Thank you for the detailed questions. Below we provide the test prediction transition counts on CivilComments, the four-group correction breakdown on Waterbirds, and the convex first-order influence diagnostics.
Q1 & W1: CivilComments final worst group and sample-level prediction changes
Across all 10 seeds, the final worst group after retraining is G2 (Toxic, Black indicator = 0) for Loss, NoiseScore, and CB-Loss alike.
The 59.32% and 41.69% recalls quoted in our earlier discussion averaged over whichever group was worst at baseline in each seed. When evaluated strictly on the fixed final worst group G2, error recall is 58.05% for NoiseScore (3,183.2 corrections) versus 40.70% for CB-Loss (2,231.9 corrections). Even though NoiseScore fixes nearly a thousand more errors in G2, CB-Loss achieves an 11.35 pp higher post-retraining accuracy on G2 (34.59% vs. 23.24%).
Table 5 (Section 4.3, p. 7; Appendix Table 26, p. 26) tracks exact test-set prediction flips between NoiseScore (NS) and CB-Loss (CB) across all four groups:
Out of the 13,687 test examples in G2, CB-Loss turns 1,557.2 incorrect NoiseScore predictions into correct ones while flipping only 3.5 in the opposite direction—a net gain of +1,553.7 correct predictions (+11.35 pp). Because NoiseScore queries exclusively in Y~=0 (fixing 0 errors in G0 and G1) whereas CB-Loss splits queries across both classes, CB-Loss trades a modest accuracy reduction on the non-toxic groups G0 (−2.06 pp) and G1 (−5.91 pp) for a large boundary shift in favor of the toxic groups G2 and G3.