- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
FAIR-DS extends in-run data attribution to group-aware validation objectives. It accumulates each training example’s gradient contribution to a selected fairness objective, provides bounds for aggregate attribution, and tests detection and repair of synthetically biased labels on three datasets.
Critical strengths
The method targets a specified fairness objective while reusing the realized training trajectory, avoiding repeated subset retraining for attribution.
Comparisons across datasets and inspection budgets show strong top-ranked detection; the analysis distinguishes smooth fairness surrogates from hard-prediction parity.
Critical weaknesses
The theoretical bound concerns aggregate attribution, not the accuracy or causal effect of each individual score; good aggregate fit could coexist with misleading top-ranked examples.
The repair protocol restores labels only for selected samples known to be synthetically flipped. This oracle-assisted correction does not measure the cost or errors of real human inspection.
The experiments rely on injected label bias and a clean group-labeled validation set. Sensitivity to naturally occurring errors, noisy protected attributes, and small subgroups remains unclear.
Questions for authors
At a fixed annotation budget, how does the method perform when an annotator must review every selected example, including uncorrupted ones?
Do top-ranked examples cause the predicted fairness change under actual leave-one-out or targeted retraining on a tractable subset?
How do noisy validation labels and rare protected groups change the rankings and repair outcome?