Reference-Aware Fairness Auditing: Separating Label Artifacts from Model Bias

Fairness audits compare model errors across subgroups, but rely on the assumption that evaluation labels are equally accurate for every subgroup. When label noise differs across subgroups, part of the reported disparity can come from the labels rather than the model. We show that the two can be separated when label errors are not systematically related to the model's own errors, and propose Reference-Aware Fairness Auditing (RAFA): estimate the label noise per subgroup, subtract its variance from the measured error, and report the adjusted disparity alongside the conventional one. In simulation, unequal label noise alone makes a fair model appear unfair, and RAFA removes 93% of the label-induced disparity. In controlled experiments on real lung-nodule and chest X-ray predictions, we inject label-noise and model-error disparities of known size. RAFA removes the label-induced part and preserves the known model part: adjusted and known model gaps differ by 0.003–0.004 on the nodules and below 0.002 on the X-rays, while the injected gaps reach 0.9 and 0.6. We then apply the full procedure to fetal-weight estimation from ultrasound (31,386 examinations, 17 hospitals), where the evaluation label is birth weight projected back to scan time, so its noise grows with the scan-to-delivery interval, as a growth model predicts. RAFA attributes 73% of the measured disparity to unequal label noise (bootstrap median 64%, 95% CI 37–83%). Finally, we give three explicit conditions that determine when RAFA applies and show how to check them in practice, using committee-labeled nodules and crowd counting as diagnostic examples.