- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Reference-Aware Fairness Auditing (RAFA), which aims to separate the fairness disparity caused by label noise from the actual model error. The method is based on a variance decomposition under the assumption that model errors and label noise are uncorrelated. The authors conduct synthetic noise injection experiments on LIDC-IDRI and CheXpert, and apply RAFA to fetal-weight estimation. They also discuss failure cases on crowd-counting datasets.
Critical strengths
-
The paper studies a meaningful problem. Evaluation labels are not always reliable, and different label quality across subgroups can indeed affect fairness evaluation.
-
The paper is clearly written, and the motivation is easy to understand.
-
The authors discuss the assumptions and limitations of the method, including failure cases where model errors and label noise are correlated.
Critical weaknesses
I have one major concern for this article:
The experiments do not sufficiently validate recovery against independently established ground truth.
The authors conduct controlled noise injection on LIDC-IDRI and CheXpert, but the original labels are already potentially noisy. The experiments only show that RAFA can remove artificially injected independent noise, while the actual model error against the true target remains unknown. In the fetal-weight experiment, the true target is also unavailable, making it difficult to verify whether the adjusted gap is correct. I think a more convincing evaluation should start from a task with reliable ground truth, introduce controlled subgroup-dependent label noise, and directly compare the RAFA-adjusted disparity with the true model disparity. It is also important to test correlated label and model errors, since this is likely to happen in real applications.
Other concerns:
- The main real-world experiment divides samples according to the scan-to-delivery interval, rather than demographic or protected attributes. This is more like an analysis of measurement reliability across different conditions. Also, RAFA only considers the variance of continuous prediction errors, while common fairness metrics such as FNR and AUC are not covered. The general significance for fairness auditing is therefore limited.
Questions for authors
See the weakness.
2 comments
Response to Reviewer nSEq, Part 2:
This continues our response to Reviewer nSEq.
4. What supports the fetal application?
Direct scan-time weight is unavailable. We therefore tested whether the result remains when we change how noise is estimated.
We added a fetal-weight predictor by fine-tuning the UltraDINO ultrasound foundation model (Ambsdorf et al., 2025). All evaluated women were excluded from fine-tuning and model selection. We call this predictor DINO below.
The three approaches are a supervised image model, a biometry formula and a fine-tuned foundation model. We estimated noise using only Hadlock and DINO, then audited DL. DL contributed nothing to its own noise estimate.
Both comparisons below use the same 31,346 examinations with all three predictions:
| Noise estimator | DL gap before → after | Point reduction | Bootstrap median [95% interval] |
|---|---|---|---|
| DL–Hadlock | 0.237 → 0.070 | 70% | 61% [32–82%] |
| Hadlock–DINO | 0.237 → 0.050 | 79% | 61% [11–88%] |
We used 1,000 paired bootstrap repeats, resampling whole women and refitting the noise law. The estimated reduction remains when DL is excluded, although its magnitude is uncertain.
We also reversed the roles and audited DINO using noise estimated from DL and Hadlock. Its gap decreased from 0.350 to 0.248, a 29% point reduction with a bootstrap interval of 13–38%. A substantial gap remained: the adjustment does not produce the same reduction for every predictor.
These findings support robustness to estimator choice. They do not establish independent predictor errors or replace validation against directly measured scan-time weight.
5. Connection to demographic fairness
We agree that STD groups alone are not demographic groups. Their role is to examine a source of unequal label reliability that can also affect comparisons between patient groups.
The controlled CheXpert experiments already use race and age groups. We have now also applied the fetal audit to maternal-age groups and added exploratory BMI and pixel-spacing comparisons.
The table reports the highest-minus-lowest group error SD. Brackets give 95% intervals from 500 bootstrap repeats, resampling whole women and refitting the pooled noise law.
| Grouping | Before adjustment | After adjustment |
|---|---|---|
| Maternal age | 0.060 [0.019–0.118] | 0.099 [0.029–0.206] |
| BMI | 0.078 [0.039–0.121] | 0.145 [0.076–0.222] |
| Pixel spacing | 0.071 [0.037–0.112] | 0.045 [0.013–0.121] |
The age analysis includes 20,877 of 31,386 examinations. Age groups have similar mean STD, 37.9–39.1 days, and therefore similar estimated noise under the pooled law. Removing equal independent noise variance can widen an SD gap because common noise compresses differences on that scale.
Pixel spacing describes acquisition rather than a demographic attribute. Its groups have different mean STD, 32.1–47.4 days, and their point-estimated gap decreases. BMI describes patient groups, but remains exploratory because covariance checks suggest differences the pooled timing law may not capture.
These examples show how the audit extends beyond timing groups and that adjustment can increase or decrease a gap. Their interpretation remains conditional on the noise model. The intervals describe each gap separately, not the before–after change.
The practical question is: how does a group comparison change when evaluation-label reliability is accounted for? The controlled experiments test recovery where the answer is known; the natural-data applications provide conditional estimates with uncertainty.
6. Scope of the fairness metrics
As already stated in our limitations, RAFA currently addresses group differences in continuous signed-error spread. FNR and AUC do not follow the same variance decomposition; extending the correction to these metrics requires separate development and remains future work.
We will also report mean signed errors alongside the variance audit so that systematic over- or underprediction remains visible.
We hope these analyses address your concerns and clarify the contribution and its scope.
Reference: Ambsdorf et al. (2025), General Methods Make Great Domain-specific Foundation Models: A Case-study on Fetal Ultrasound. https://arxiv.org/abs/2506.19552
Response to Reviewer nSEq, Part 1:
Thank you for recognising the importance of label reliability in fairness evaluation and the value of reporting assumptions and failure cases. Your main concern is whether RAFA recovers the correct model disparity. We clarify below what is known in each experiment and distinguish controlled validation from the clinical application.
1. Recovery when the answer is known
In the fully synthetic simulation, the target and model errors are known by construction. This provides a direct test of recovery against known truth.
The real-data experiments then test whether RAFA can distinguish two controlled changes: increased label noise and increased model-error disparity. We vary these separately across a 7 × 7 grid. The original labels define the controlled reference, and the injected noise variance is supplied to RAFA.
The adjusted gap tracks the known controlled model gap, with mean recovery errors of 0.003–0.004 on LIDC and below 0.002 on CheXpert. These experiments therefore test both removal of added label noise and preservation of model differences.
Your distinction concerning the original labels is valid: the grids do not recover unknown errors already present in those labels. They establish recovery relative to the controlled reference, not independently measured biological truth. We will make this distinction explicit.
2. Estimating noise before correction
The paper also tests the estimation step. Appendix B estimates noise from repeated corrupted labels rather than supplying its variance. Using real LIDC predictions and equal model behaviour across four synthetic groups, the gap decreases from 0.370 to 0.094 with estimated noise, compared with 0.077 with exact noise, over 300 seeds.
Thus, the existing experiments provide three complementary tests: recovery with synthetic truth, preservation of controlled model gaps on real predictions, and correction after estimating noise. We will make these roles clearer in the main text.
An additional real-world task with independently established ground truth would strengthen validation. We do not present the fetal analysis as supplying that ground truth.
3. Correlated model and label errors
We added a sensitivity analysis that explicitly varies the assumed within-group correlation between model error and label noise, holding estimated label-noise variances fixed.
Starting from the original fetal estimate, the interpretation that more than half the gap is explained stops holding at approximately 0.18.
This directly examines correlation rather than only changing noise magnitude. It quantifies how the clinical interpretation depends on the assumption; it is not an estimate of the actual correlation or a known-truth recovery experiment.
The crowd-counting analysis provides a separate empirical example where shared difficulty undermines the conditions for adjustment. Reporting this case is part of the intended use of RAFA: an unsupported noise estimate should not be used to explain away an observed disparity.
Part 2 presents the additional clinical robustness checks and demographic comparisons.