acceptodds
Under review as a conference paper at ICLR 2027

Detection Is Not Repair: Evaluating Budgeted Label Verification for Group Robustness

Abstract

Does uncovering more label errors necessarily improve worst-group accuracy after retraining? In budgeted data cleaning, standard practice prioritizes detection precision—identifying as many corruptions as possible within a fixed query budget. In this work, we investigate this premise when group annotations are inaccessible during data acquisition, and uncover a fundamental decoupling between error detection and downstream group robustness. Across extensive empirical evaluations on vision and text benchmarks, we document widespread empirical inversions where acquisition policies that correct substantially fewer errors achieve markedly superior worst-group accuracy compared to high-precision detectors. This divergence arises because high-precision selection systematically starves underrepresented, vulnerable subgroups of corrections, skewing the retrained decision boundary unfavorably. Mathematically, we formalize this structural failure: without group annotations, even an idealized oracle detector incurs a worst-case regret approaching 50 percentage points due to the inherent non-identifiability of latent group structure. Finally, we explore practical remedies in the absence of group metadata, showing that simple observed-class quotas safeguard against decision boundary destabilization, while tracking post-retraining clean validation improvement guides policy selection in balanced vision domains. Our results reposition data cleaning from passive noise filtering to an active intervention on classifier decision boundaries, underscoring that label verification must be evaluated by downstream subpopulation robustness rather than raw detection metrics.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
budgeted label verification, data-centric AI, group robustness, worst-group accuracy, subpopulation shift, data cleaning, label noise, minimax regret

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TA3SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares correction counts and worst-group accuracy after retraining under equal label verification budgets. Controlled experiments examine which samples are corrected, and a first-order approximation analyzes the resulting model changes. The theory shows that knowing which labels are wrong can still leave the best correction allocation unidentified when group labels are unavailable. The paper also tests class quotas and policy selection using validation gains.

Critical strengths
  1. The question is well aimed. Data cleaning usually measures progress by how many label errors are found, and the paper uses worst-group accuracy after retraining instead, with 80 of 510 paired comparisons going the other way.
  2. The theory is solid. Theorem 1 is an information-theoretic lower bound, and without group annotations the latent partition cannot be identified, so even a perfect detector carries worst-case regret approaching 50 pp.
Critical weaknesses
  1. Table 19 illustrates the paper's central finding that higher correction recall need not yield higher WGA. On CivilComments with 20% uniform noise and a 10% budget, NoiseScore corrects 59.32% of the initial worst group's errors and reaches 23.24% WGA, while CB-Loss corrects 41.69% and reaches 34.59%. Although Appendix E discusses allocation differences and evaluates a first-order approximation, it does not directly establish which changes in corrected samples account for this specific WGA gap. A direct comparison of the resulting group accuracies and prediction changes would strengthen the explanation.
  2. The paper recommends an equal budget per observed class as protection for the vulnerable subgroup inside each class. That subgroup can occupy a small part of a class, and the paper does not say whether an even split reaches it. Waterbirds does not support the recommendation: at the 10% budget CB-Loss reaches 54.26% WGA while the uncorrected model reaches 60.33%.
Questions for authors
  1. In Table 19, which group ends up worst under NoiseScore and which under CB-Loss? Do the corrected samples alone explain the gap between 23.24% and 34.59%?
  2. After the budget is split evenly across classes, how many corrections does each group receive? Does that number explain why CB-Loss ends at 54.26% on Waterbirds while the uncorrected model reaches 60.33%?
  3. The first-order analysis predicts group cross-entropy. Applied to the CivilComments 10% budget, does it rank NoiseScore and CB-Loss the same way the measured WGA does?
4 comments

Thank you for the detailed questions. Below we provide the test prediction transition counts on CivilComments, the four-group correction breakdown on Waterbirds, and the convex first-order influence diagnostics.

Q1 & W1: CivilComments final worst group and sample-level prediction changes

Across all 10 seeds, the final worst group after retraining is G2G_2 (Toxic, Black indicator = 0) for Loss, NoiseScore, and CB-Loss alike.

The 59.32% and 41.69% recalls quoted in our earlier discussion averaged over whichever group was worst at baseline in each seed. When evaluated strictly on the fixed final worst group G2G_2, error recall is 58.05% for NoiseScore (3,183.2 corrections) versus 40.70% for CB-Loss (2,231.9 corrections). Even though NoiseScore fixes nearly a thousand more errors in G2G_2, CB-Loss achieves an 11.35 pp higher post-retraining accuracy on G2G_2 (34.59% vs. 23.24%).

Table 5 (Section 4.3, p. 7; Appendix Table 26, p. 26) tracks exact test-set prediction flips between NoiseScore (NS) and CB-Loss (CB) across all four groups:

Subgroup (Y,A)(Y, A)Test Instances NtestN_{\text{test}}Incorrect →\to Correct (NS wrong, CB right)Correct →\to Incorrect (NS right, CB wrong)Net Correct PredictionsΔAcc\Delta\text{Acc} (pp) [95% CI]Test Cross-Entropy (NS vs. CB)
G0 (0,0)G_0\ (0, 0) Non-toxic, Black ind = 0115,2237.92,386.0−2,378.1-2,378.1−2.06  [−2.15,−1.98]-2.06\;[-2.15, -1.98]0.3179 vs. 0.2435
G1 (0,1)G_1\ (0, 1) Non-toxic, Black ind = 13,3350.6197.7−197.1-197.1−5.91  [−6.18,−5.64]-5.91\;[-6.18, -5.64]0.4501 vs. 0.4208
G2 (1,0)G_2\ (1, 0) Toxic, Black ind = 0 (Worst)13,6871,557.23.5+1,553.7+1{,}553.7+11.35  [+10.92,+11.78]+11.35\;[+10.92, +11.78]0.9269 vs. 0.9627
G3 (1,1)G_3\ (1, 1) Toxic, Black ind = 11,537198.70.6+198.1+198.1+12.89  [+12.02,+13.76]+12.89\;[+12.02, +13.76]0.8362 vs. 0.8306

Out of the 13,687 test examples in G2G_2, CB-Loss turns 1,557.2 incorrect NoiseScore predictions into correct ones while flipping only 3.5 in the opposite direction—a net gain of +1,553.7+1{,}553.7 correct predictions (+11.35 pp+11.35\text{ pp}). Because NoiseScore queries exclusively in Y~=0\tilde Y=0 (fixing 0 errors in G0G_0 and G1G_1) whereas CB-Loss splits queries across both classes, CB-Loss trades a modest accuracy reduction on the non-toxic groups G0G_0 (−2.06 pp-2.06\text{ pp}) and G1G_1 (−5.91 pp-5.91\text{ pp}) for a large boundary shift in favor of the toxic groups G2G_2 and G3G_3.


Q2 & W2: Waterbirds subgroup corrections and bottleneck switching

Table 6 (Section 6, p. 9; full three-method breakdown in Appendix Table 25, p. 25) reports the four-group correction counts and accuracies on Waterbirds under the 5-epoch diagnostic protocol (B=480B=480, seeds 0–9):

Subgroup (Y,A)(Y, A)Available ErrorsCB-Loss Corrections (Recall %)Baseline Accuracy (%)Final Accuracy (%)Worst Group Count (/10)
G0 (0,0)G_0\ (0, 0) Landbird / Land (Majority)709.1238.3 (33.65%)99.3298.410
G1 (0,1)G_1\ (0, 1) Landbird / Water (Vulnerable Minority)36.10.2 (0.59%)74.0254.679
G2 (1,0)G_2\ (1, 0) Waterbird / Land (Initial Worst)12.16.9 (56.58%)60.3369.881
G3 (1,1)G_3\ (1, 1) Waterbird / Water (Majority)204.0187.2 (91.84%)92.1795.420

The group-level numbers show why overall WGA drops from 60.33% to 54.26%:

  • The initial worst group (G2G_2, Waterbird on Land) improves: CB-Loss corrects 56.58% of its label errors, raising G2G_2 accuracy from 60.33% to 69.88% (+9.55 pp+9.55\text{ pp}).
  • Intra-class skew shifts the bottleneck to G1G_1 (Landbird on Water): Within the Y~=1\tilde Y=1 query quota (mislabeled landbirds), majority group G0G_0 takes 238.3 corrections while minority group G1G_1 gets only 0.2 corrections (0.59% recall). After retraining, G1G_1 accuracy drops from 74.02% to 54.67%, making G1G_1 the new worst group in 9 out of 10 seeds.
  • As we note in Section 6 (p. 9) and Appendix E.1 (p. 24), because retraining on these queries changes both within-class error coverage and the global decision boundary, this table captures their combined effect and does not establish that low G1G_1 coverage alone causes the full decline. Rather, it illustrates a cross-group accuracy trade-off and shows that equal class quotas do not ensure subgroup coverage. Under longer training (30 epochs) or end-to-end retraining, CB-Loss WGA recovers to 65.23% and 67.39% (Tables 2 and 7, pp. 5, 9; Appendix Table 16, p. 20).

Q3: First-order ranking vs. measured WGA on CivilComments

On CivilComments, first-order continuous cross-entropy (CE) loss predictions rank the methods in the opposite order of empirical WGA—but evaluating discrete 0–1 accuracy at the first-order parameter step θ0+vQ\theta_0 + v_Q recovers the exact refit WGA ordering across all 10 seeds.

Appendix Table 29 (p. 28) tests this directly using an exact L-BFGS convex logistic model on CivilComments (B=10%B=10\%, seeds 0–9, λ=10−4\lambda=10^{-4}; under λ=10−2\lambda=10^{-2} all three methods have 0.00% WGA):

MethodPredicted G2G_2 CE GainActual G2G_2 CE GainLinearized 0–1 WGA (%)Refit WGA (%)
CB-Loss+0.0884+0.0884+0.0616+0.061627.92%30.04%
Loss−0.0723-0.0723−0.1356-0.135619.38%23.71%
NoiseScore+0.1037+0.1037+0.0975+0.097519.02%18.96%
  1. Continuous CE loss vs. 0–1 WGA: The first-order Taylor expansion accurately tracks empirical cross-entropy reduction on G2G_2 (predicting +0.1037+0.1037 nats for NoiseScore vs. +0.0884+0.0884 nats for CB-Loss, matching the actual CE gain ordering +0.0975>+0.0616+0.0975 > +0.0616). Yet NoiseScore has the lowest refit 0–1 WGA (18.96% vs. 30.04% for CB-Loss).
  2. Linearized 0–1 step recovers the true ranking: When we instead evaluate thresholded 0–1 accuracy at the first-order parameter update θ0+vQ\theta_0 + v_Q (vQ=−H0−1δgQv_Q = -H_0^{-1}\delta g_Q), the linearized WGA ordering is CB-Loss (27.92%) > Loss (19.38%) > NoiseScore (19.02%), matching the refit WGA ordering in all 10 seeds.
  3. We see the same phenomenon in the 5-epoch neural network (Table 5, p. 7): NoiseScore achieves lower test cross-entropy on G2G_2 than CB-Loss (0.9269 vs. 0.9627), yet trails by 11.35 pp in 0–1 accuracy. Because one-sided corrections alter classification margins unevenly, reducing average log-loss on a subgroup does not guarantee more examples cross the zero decision threshold.

Thanks for the response. The per-group results answer my questions on Table 19 and on Waterbirds, and the claim about class quotas now matches what the results show. These results make clear where the gap appears, though not yet why, and the first-order analysis does not predict WGA either, so I'd suggest stating this clearly in the paper. As my main questions have been answered, I'm raising my score to 3.

Thank you for taking the time to review our responses and for raising your score! We completely agree with both of your points and have stated them explicitly in the revised manuscript:

  • Observational ledgers show where the gap appears rather than isolating why: In Section 4.3 (p. 7), Section 6 (p. 9), and Appendix E.1 (p. 24), we explicitly state that policy comparisons (such as NoiseScore vs. CB-Loss on CivilComments and the four-group ledger on Waterbirds) jointly change correction counts, label-transition directions, and sample identities, and therefore reflect combined effects rather than isolating a single causal factor on its own.
  • First-order group-loss approximation does not predict WGA: In Section 4.3 (p. 7) and Appendix E.2 (p. 28, Table 29), we state directly that the first-order continuous group-loss predictor does not recover the observed WGA ordering on CivilComments (favoring NoiseScore over CB-Loss), and highlight this as a limitation of continuous loss surrogates for thresholded 0–1 subgroup accuracy. Thank you again for your constructive feedback, which has significantly sharpened the precision of our claims.
Reviewer_SZujOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies budgeted label verification for group robustness when group annotations are unavailable during acquisition.

It questions whether selecting labels by detection precision (finding as many corrupted labels as possible within a fixed budget) also improves worst-group accuracy (WGA) after retraining. Across Waterbirds, CelebA, CivilComments and CIFAR-10N, it reports cases where lower-precision policies yield higher WGA. It attributes them to high-precision selection "starving" underrepresented subgroups of corrections.

A two-world construction (Theorem 1) shows that, without group information, even an oracle detector can incur a worst-case regret approaching 50 pp. Finally, the paper explores two group-blind remedies: equal verification quotas per observed class (CB-Loss), and policy selection by clean, class-balanced validation gain.

Critical strengths

1. A practically relevant question. Ranking candidates by their likelihood of being mislabeled is a natural and common way to spend a verification budget. Asking whether this actually improves the weakest subgroup after retraining is a worthwhile reframing of how data cleaning should be evaluated.

2. Careful and transparent empirical work. The paper's empirical practice is careful throughout:

  • comparisons are paired across seeds, splits and retraining runs, with multiplicity control;
  • a matched control holds the correction count and flip direction fixed;
  • a full reproduction package is provided.
Critical weaknesses

1. Under the paper's own group-blind constraint, the problem and its remedies reduce to class-imbalance handling, which limits novelty. Because group annotations are unavailable during acquisition, the only structure a policy can act on is the observed class. The paper's diagnosis and remedies both operate at that level:

  • Diagnosis. High-precision ranking concentrates the budget in a single observed class (88–>99.9% of queries for Loss; Table 19). Correcting those labels therefore shifts the class balance of the training labels in one direction.
  • First remedy. CB-Loss is class-balanced sampling applied to the query budget.
  • Second remedy. Selection by class-balanced validation accuracy is standard model selection under imbalance.

These are well-known tools. Prior work has shown that simple class balancing alone yields competitive worst-group accuracy on standard group-robustness benchmarks, without group labels (Idrissi et al., "Simple data balancing achieves competitive worst-group-accuracy", CLeaR 2022). However, The class-imbalance literature is neither cited nor discussed; the word "imbalance" appears once, as a post-hoc explanation in Sec. 6. Even granting that budgeted verification is a new setting, the paper should explain what is specific to label verification beyond applying class balancing to the query budget. Without this, the contribution reads as a known imbalance effect observed in a new pipeline.

2. The proposed remedies do not direcly address the problem the paper poses, and the paper does not explain when or why they would. The problem is framed around latent subgroups being starved of corrections. The remedies, however, act only on observed classes. The paper itself states that they cannot solve the posed problem in general: "no group-blind policy can eliminate worst-case regret in the worst case" (Sec. 6). It also labels them exploratory (Sec. 6 title).

Yet the contributions present the remedies as general ("prevent boundary destabilization across noise regimes", Sec. 1; "zero-cost safeguard", Sec. 6). What is missing is a rationale for when balancing observed classes protects latent groups. This is exactly the question Theorem 1 raises. Theorem 1 does not help here. Its construction makes groups independent of every observable by design, so the impossibility is close to definitional. It also does not explain the empirical phenomenon the paper reports, namely why lower-precision policies win. The theory and the remedies therefore remain disconnected from the posed problem.

Questions for authors

Please refer to the weaknesses

3 comments

Thank you for the thoughtful critique. Below we clarify how our work connects to the class-imbalance literature, present direct evidence that sample-level repair goes beyond marginal class counts, and explain how Theorem 1 relates to observed-class quotas.

W1: Relationship to class-imbalance literature and evidence beyond marginal balancing

We agree that CB-Loss and class-balanced validation selection build directly on established class-balancing principles. In the revised paper (Sections 1, 6, and 7), we cite and discuss Idrissi et al. (CLeaR 2022) and make clear that we treat these balancing rules as group-blind baselines rather than new algorithmic contributions.

At the same time, withholding group annotations AA during acquisition does not restrict a policy to observed class labels alone. Under our problem formulation (Section 2.1, p. 3), group-blind policies can use feature representations, training loss dynamics, and unannotated validation data (for instance, neither RepairValue nor adapted D3M uses group labels) to distinguish examples within the same observed class. Two experiments show how sample-level verification differs from marginal class balancing:

1. Matched-correction control experiment (Section 4.2, Figure 3, p. 6; Appendix Table 18, p. 21)

On Waterbirds, we constructed paired query sets that hold the query budget, total corrected errors, already-correct queries, and exact label-flip counts (0→10\to 1 and 1→01\to 0) strictly constant across arms, varying only whether the corrected errors lie inside or outside the initial worst group:

Control SettingSeedsPaired ΔWGA\Delta\text{WGA} (pp) [95% CI]Signs (+/-/0)
Initial worst group: Inside vs. Outside (Matched Control)20+3.24  [+1.56,+5.22]+3.24\;[+1.56, +5.22]17/3/0
50% Inside vs. Outside20+0.92  [+0.01,+2.03]+0.92\;[+0.01, +2.03]14/6/0

Because all arms start from the same noisy label vector and apply the exact same number of 0→10\to 1 and 1→01\to 0 flips, the repaired datasets have identical post-repair marginal class histograms. Even so, placing those corrections inside the bottleneck group yields +3.24 pp+3.24\text{ pp} higher WGA than placing them outside (positive in 17 of 20 seeds). As our first-order analysis shows (Section 4.3, p. 7), fixing correction counts and flip directions fixes only the scalar sum ∑i∈Q(y~i−yi)\sum_{i\in Q}(\tilde y_i - y_i), leaving the feature-weighted perturbation ∑i∈Q(y~i−yi)xi\sum_{i\in Q}(\tilde y_i - y_i)x_i and its alignment with subgroup loss gradients unconstrained.

2. Verified label repair vs. unverified data balancing (Section 6, Table 7, p. 9; Appendix Table 27, p. 27)

We also compared 10%-budget CB-Loss against unverified inverse-frequency reweighting and subsampling on the noisy labels (B=0%B=0\%; Idrissi et al., 2022):

DatasetEpochsNoise RegimeBaseline WGA (%)Reweight (0% Verify)Subsample (0% Verify)CB-Loss (10% Verify)
Waterbirds5epuniform2060.3335.8234.9154.26
Waterbirds30epuniform2060.8152.4446.7565.23
Waterbirds30epminority_high_4048.6235.0442.0556.52
CelebA30epuniform2076.28—81.8287.22
CivilComments30epuniform209.2024.5823.8832.23
CivilComments30epminority_high_406.5741.6140.0821.22

The two approaches succeed in different regimes:

  • On Waterbirds, upweighting or subsampling unverified labels amplifies the 20%–40% corruptions inside minority slices, causing WGA to drop below the uncorrected baseline (34.69%–52.44%) and trailing 10%-budget CB-Loss by 7.90 to 21.48 pp.
  • Conversely, on CivilComments under 40% minority noise and 30-epoch training, full-dataset unverified balancing (40.08%–41.61%) outperforms 10%-budget verification (21.22%), where a 10% budget can only clean a fraction of the dataset.

W2: Role of Theorem 1 and operating boundaries of observed-class quotas

Theorem 1 establishes an information-theoretic lower bound: without group annotations, even an oracle detector with 100% precision faces worst-case regret approaching 50 pp. Its role is to prove formally that error detection alone cannot guarantee downstream group repair. Importantly, Theorem 1 does not provide a theoretical guarantee for CB-Loss: in our two-environment construction, both candidate error blocks share the same observed label (Y~=0\tilde Y=0), so splitting budget across latent blocks in the proof is distinct from balancing queries across observed classes (Section 5, p. 8).

Writing the corrections received by subgroup gg as Cg(Q)=B1−y⋅P^i∈Q(1−y)(Gi=g,  Yi≠Y~i)C_g(Q) = B_{1-y} \cdot \hat{P}_{i \in Q(1-y)}\bigl(G_i = g,\; Y_i \ne \tilde{Y}_i\bigr) makes the practical scope and limits of equal class quotas clear:

  • When skew is across observed classes (effective): On CivilComments, unconstrained detectors like NoiseScore put 100% of their queries into observed class Y~=0\tilde Y=0. Enforcing equal class quotas (Bc=B/2B_c = B/2) restores corrections in both directions and improves worst-group G2G_2 accuracy by +11.35 pp+11.35\text{ pp} (see response to TA3S Q1).
  • When skew is within an observed class (ineffective): Class quotas fix B1−yB_{1-y} but cannot control which subgroup wins queries inside that class. On Waterbirds, within the Y~=1\tilde Y=1 quota, majority group G0G_0 absorbs 238.3 corrections while minority G1G_1 receives just 0.2 corrections (0.59% recall). Retraining drops G1G_1 accuracy from 74.02% to 54.67%, making it the new worst group in 9 of 10 seeds (Table 6, p. 9). Because retraining changes both within-class coverage and the global decision boundary, Table 6 reflects their combined effect rather than isolating low G1G_1 coverage as the sole cause of the drop (Section 6, p. 9; Appendix E.1, p. 24). What it demonstrates is that observed-class quotas cannot guarantee subgroup coverage and can induce cross-group trade-offs.

This contrast also explains why validation-guided selection is useful in Section 6: whereas fixed class quotas are blind to intra-class skew on Waterbirds, clean validation monitoring detects the difference and selects adapted D3M (+8.27 pp over CB-Loss).

Dear Reviewer SZuj,

As the discussion period progresses, we wanted to briefly check in and see if our responses and new experiments have addressed your concerns regarding the connection to class-imbalance methods and the scope of Theorem 1. Please let us know if any questions remain or if there is any further analysis we can provide!

Reviewer_4mPEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Here, the authors study whether correcting more label errors leads to better worst-group accuracy after retraining. The paper shows that correcting fewer errors can give better results, depending on which examples are corrected. It provides a very good example of the precision of the methods and their influence on worst-group accuracy, and discusses that tradeoff.

Critical strengths
  • The authors investigate an interesting problem and offer a useful insight because they show that finding more incorrect labels does not necessarily help the groups on which the model performs poorly.
  • I enjoyed reading the paper, and the motivating example makes the problem clear.
Critical weaknesses
  • W1. I felt that some claims should be weakened that the paper makes, especially because they show that higher detection precision does not necessarily lead to better worst-group accuracy. However, the alternatives do not consistently perform better across settings. For example, the end-to-end Waterbirds comparison in Table 2 has a confidence interval that includes zero. I think the claims and statements they make should reflect this more clearly.
  • W2. Can the authors include CB-Loss in the policy selection comparison and evaluate it under end-to-end retraining? Especially because it performs well, but Appendix H excludes it. For me, this leaves an important practical question unanswered: how much does historical validation-based selection help compared to this baseline?
  • W3. For me, it was surprising that TracIn corrects none of the initial worst group's errors across the seeds. Do the authors have an intuition or insight into why this happens?
  • W4. I was wondering if sampling from a larger candidate pool helps avoid this concentration?
Questions for authors

See weaknesses.

3 comments

Thank you for the constructive feedback. Below we address each point and summarize the corresponding revisions.

W1: Statistical bounds and claim calibration (Waterbirds Table 2)

We agree that the Waterbirds end-to-end difference is not statistically separated from zero. Across the 20 evaluation seeds (1000–1019; Section 3.2, Table 2, p. 5; Appendix Tables 14–16, pp. 19–20), NoiseScore achieves a higher point estimate than Loss (+5.91 pp vs. +4.68 pp; paired difference +1.23 pp, 13 wins, 6 losses, 1 tie), but its 95% bootstrap CI spans zero ([−0.38,+2.78][-0.38, +2.78], sign test p=0.167p=0.167). We now state explicitly in Section 3.2 (p. 5) that this difference is not statistically significant, and we revised the Abstract, Introduction, Section 6, and Section 8 to remove overbroad terms (such as "zero-cost safeguards" or "guaranteed boundary stabilization"), keeping our claim strictly tied to the evidence: high error-detection precision does not guarantee better post-retraining worst-group accuracy (WGA).


W2: Evaluating CB-Loss in policy selection and end-to-end retraining

We added fixed CB-Loss to our 13-method selection pool (300 runs across 3 datasets, 2 noise regimes, 5 budgets, and 10 seeds) and evaluated it under 20-seed ResNet-50 end-to-end retraining (seeds 1000–1019).

1. Policy selection across budgets (Section 6, Table 8, p. 9; Appendix Table 32, p. 34)

Held-out WGA differences (pp; 95% seed-cluster bootstrap CIs, 10,000 draws) holding out the evaluation seed (Within) or both budget and seed (Cross):

DatasetProtocolVal13\text{Val}_{13} − Prec13\text{Prec} _ {13}Val13−Fixed CB-Loss\text{Val}_{13} - \text{Fixed CB-Loss}
WaterbirdsWithin / Cross+11.23 [+9.35,+13.25]+11.23\ [+9.35, +13.25] / +11.00 [+9.46,+12.80]+11.00\ [+9.46, +12.80]+8.27 [+6.99,+9.67]+8.27\ [+6.99, +9.67]
CelebAWithin / Cross+9.87 [+9.24,+10.64]+9.87\ [+9.24, +10.64] / +9.46 [+8.93,+10.12]+9.46\ [+8.93, +10.12]−0.05 [−0.23,+0.35]-0.05\ [-0.23, +0.35] / −0.46 [−0.58,−0.04]-0.46\ [-0.58, -0.04]
CivilCommentsWithin / Cross+2.39 [+2.30,+2.52]+2.39\ [+2.30, +2.52] / +2.44 [+2.33,+2.55]+2.44\ [+2.33, +2.55]−0.04 [−0.06,+0.00]-0.04\ [-0.06, +0.00] / +0.00 [+0.00,+0.00]+0.00\ [+0.00, +0.00]
Three-Dataset MeanWithin / Cross—+2.73 [+2.32,+3.23]+2.73\ [+2.32, +3.23] / +2.60 [+2.19,+3.10]+2.60\ [+2.19, +3.10]
  • Vs. Prec13\text{Prec}_{13}: Selection by clean, class-balanced validation recovery (Val13\text{Val}_{13}) outperforms precision selection on all three datasets (+2.39 to +11.23 pp).
  • Vs. fixed CB-Loss: On Waterbirds, where intra-class skew causes a bottleneck switch under equal class quotas (see TA3S Q2), validation selection picks AUTO-D3M-Q in all 100/100 folds (+8.27 pp over CB-Loss). On CelebA and CivilComments, where class quotas work well, it selects CB-Loss in most folds (75/100 Within and 73/100 Cross on CelebA; 90/100 Within and 100/100 Cross on CivilComments; Appendix H.1, p. 33), matching fixed CB-Loss within 0.5 pp0.5\text{ pp}. Fixed CB-Loss is a single-run rule requiring no prior trials, whereas validation selection searches across candidate policies using historical validation trajectories.

2. End-to-end retraining (Section 3.2, Table 2, p. 5; Appendix Tables 14–16, pp. 19–20)

ResNet-50 retraining on the 20 held-out seeds (1000–1019; 20% uniform noise, 10% budget; 50,000 seed-bootstrap 95% CIs):

DatasetPolicyPrecision (%)ΔWGA\Delta\text{WGA} vs. Baseline (pp) [95% CI]Paired Difference ΔWGA\Delta\text{WGA} (pp) [95% CI]Paired Signs (+/-/0)
WaterbirdsLoss (πprec\pi_{\mathrm{prec}})91.61+4.68  [+2.33,+7.18]+4.68\;[+2.33, +7.18]— (Reference)—
NoiseScore (πWGA\pi_{\mathrm{WGA}})71.41+5.91  [+3.03,+9.12]+5.91\;[+3.03, +9.12]+1.23  [−0.38,+2.78]+1.23\;[-0.38, +2.78] (vs. Loss)13/6/1
CB-Loss (Fixed Quota)76.43+5.99  [+2.36,+9.93]+5.99\;[+2.36, +9.93]— (Descriptive comparison)15/5/0 (vs. Baseline)
CelebATracIn-CP (πprec\pi_{\mathrm{prec}})99.76−0.32  [−0.66,+0.02]-0.32\;[-0.66, +0.02]— (Reference)—
RepairValue (πWGA\pi_{\mathrm{WGA}})27.12+0.44  [+0.16,+0.72]+0.44\;[+0.16, +0.72]+0.76  [+0.51,+1.04]+0.76\;[+0.51, +1.04] (vs. TracIn)18/0/2
CB-Loss (Fixed Quota)62.57+0.14  [−0.10,+0.37]+0.14\;[-0.10, +0.37]— (Descriptive comparison)11/6/3 (vs. Baseline)

On CelebA, RepairValue (27.12% precision) beats high-precision TracIn-CP (99.76%) by +0.76 pp+0.76\text{ pp} (winning 18/20 seeds). Supplementary fixed CB-Loss gains +5.99 pp+5.99\text{ pp} on Waterbirds and +0.14 pp+0.14\text{ pp} on CelebA over independently reproduced uncorrected baselines (which match the archived baselines across all 20 seeds on aggregate metrics; Appendix C, p. 18).


W3: Why TracIn corrects zero errors in CelebA's worst group

Subgroup query breakdown for TracIn-CP across the 10 E2E development seeds (B=16,277B=16{,}277, seeds 10–19; Appendix C, Table 13, p. 18):

Subgroup (Y,A)(Y, A)Mean QueriesBudget Share (%)Corrections
G0 (0,0)G_0\ (0, 0) No Eyeglasses / Female (Majority)10,393.163.85%10,393.1
G1 (0,1)G_1\ (0, 1) No Eyeglasses / Male (Majority)5,850.335.94%5,850.3
G2 (1,0)G_2\ (1, 0) Eyeglasses / Female (Bottleneck Worst)4.90.03%0.0
G3 (1,1)G_3\ (1, 1) Eyeglasses / Male (Minority)28.70.18%0.0
Majority Slices Total (G0+G1G_0+G_1)16,243.499.79%16,243.4 (Weak group corrections = 0)

TracIn-CP ranks candidates by the negative inner product between their last-layer gradients and the uncertainty-weighted clean validation gradient summed over j∈Dvalj \in D_{\mathrm{val}} with weights wj,k=1−max⁡cqjc(k)w_{j,k}=1-\max_c q^{(k)}_{jc} (Appendix A.3, Eq. 6, p. 16). Because this sum is not normalized by class or subgroup, majority-class examples dominate the aggregate validation gradient under CelebA's class skew. Mislabeled majority examples (G0,G1G_0, G_1) strongly oppose this gradient and absorb 99.79% of the budget, whereas minority slices (G2,G3G_2, G_3) receive only 33.6 queries—all landing on clean examples and yielding zero corrections.


W4: Would sampling from a larger candidate pool help?

Exact conditional hypergeometric expectations for uniform sampling of B=16,277B=16{,}277 queries from the top-KK archived CelebA TracIn rankings (N=162,770N=162{,}770, seeds 0–9; Appendix D.1, Table 19, p. 21):

Candidate Pool Size KKPool Weak ErrorsExpected CorrectionsExpected Weak Recall (%)Pr⁡(At least 1 hit)\Pr(\text{At least 1 hit})Expected Precision (%)
K=BK=B (Top 10%, 16,277)0.00.000.00%0.0000099.36%
K=2BK=2B (Top 20%, 32,554)0.00.000.00%0.0000091.99%
K=5BK=5B (Top 50%, 81,385)12.32.460.56%0.9049437.44%
K=NK=N (All data, 162,770)440.244.0210.00%1.0000019.97%

Across all 10 seeds, the highest-ranked worst-group error appears at rank 38,942–42,425 (24th–26th percentile). Thus, doubling the pool to K=2BK=2B (top 20%) still yields 0.00% expected recall, while expanding to K=5BK=5B (top 50%) recovers only 0.56% recall (2.46 errors) at the cost of cutting precision from 99.36% to 37.44%. Moderate pool expansion with uniform sampling cannot resolve starvation when minority errors rank outside the top quartile.

I thank the authors for their thorough response and the additional clarifications. As my score was already high, I will maintain it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Manuscript Revisions

We thank all reviewers for their constructive feedback. While we have not yet had the opportunity to hear back from Reviewer SZuj following our November 11 responses and revised PDF, we briefly summarize how the new experiments resolve every initial concern:

  1. Verification Beyond Marginal Class Balancing (Addressing SZuj W1):

    • Matched-Correction Control (Sec. 4.2, Fig. 3, Table 18): Under strictly identical post-repair marginal class histograms, placing corrections inside vs. outside the bottleneck group still improves WGA by +3.24 pp ([+1.56,+5.22][+1.56, +5.22], 17/20 seeds).
    • Vs. Unverified Balancing (Idrissi et al., 2022; Sec. 6, Table 7, App. Table 27): Reweighting/subsampling noisy labels (B=0%B=0\%) amplifies minority noise on Waterbirds, dropping WGA to 34.69%34.69\%–52.44%52.44\% and trailing 10%10\%-budget CB-Loss by 7.90–21.48 pp.
  2. Boundaries of Class Quotas & First-Order Diagnostics (Confirmed with TA3S Q1–Q3; Also Addressing SZuj W2):

    • Cross- vs. Intra-Class Skew (Tables 5–6, App. Tables 25–26): Equal class quotas mitigate across-class concentration on CivilComments (+1,553.7 net test flips / +11.35 pp on worst group G2G_2), but fail under intra-class skew on Waterbirds (minority G1G_1 receives 0.59%0.59\% recall and drops −19.35-19.35 pp, causing a bottleneck switch in 9/10 seeds).
    • Causal & First-Order Caveats (Sec. 4.3, Sec. 6, App. Table 29): As confirmed with Reviewer TA3S, the main text explicitly states that group ledgers reflect combined effects and that first-order continuous CE loss does not predict 0–1 WGA rankings (whereas linearized 0–1 accuracy at θ0+vQ\theta_0+v_Q recovers the exact refit ordering in 10/10 seeds).
  3. CB-Loss Selection, E2E Runs & TracIn Audits (Addressing 4mPE W1–W4):

    • 13-Method Selection & E2E (Tables 2 & 8, App. Tables 16 & 32): Validation selection beats precision selection across all datasets (+2.44 to +11.00 pp), outperforms fixed CB-Loss by +8.27 pp on Waterbirds (selecting AUTO-D3M-Q 100/100 times), and matches it on CelebA/CivilComments. We added 20-seed E2E CB-Loss runs (+5.99 pp Waterbirds, +0.14 pp CelebA) and calibrated all non-significant claims.
    • TracIn Starvation & Pool Expansion (App. Tables 13 & 19): Majority slices absorb 99.79% of CelebA TracIn queries (00 worst-group corrections); hypergeometric audits show worst-group errors rank past the 24th24\text{th} percentile, yielding 0.00% recall at K=2BK=2B and 0.56% at K=5BK=5B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.