- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces a fairness metric involving the (signed) residual distribution of a binary probabilistic classifier. Their main contribution is showing the Wasserstein distance between the protected attribute conditional residual distributions is related to averaged absolute gaps of familiar aggregate statistics of (group) FPR and FNR. In their experiments, they investigate several related questions concerning their introduced metric, such as the effect of model calibration and group prevalence, as well as whether their metric ranks models between the validation and test set more accurately than alternatives such as equalized odds at a fixed threshold.
Critical strengths
- Authors introduce a creative and intuitive object, the residual distribution, and connect it with the familiar classification metrics of FPR and FNR via prevalence rates. This distributional audit has a nice geometric and intuitive interpretation, which the authors investigate thoroughly.
- The experiments explore natural follow up questions, and they provide analyses with a diverse range of datasets including tabular and text classification.
Critical weaknesses
-
The authors do not sufficiently justify the use of their introduced metric. It is unclear why their approach is preferable to other distributional fairness metrics, e.g., comparing a models' EO and FPR and FNR across (non fixed) thresholds, or other methods mentioned in their related work section-- justification for their method is argued by comparing against a classifier with a fixed threshold, but this is not the natural baseline to compare against, as distributional metrics naturally aggregate information across thresholds.
-
Beyond the identity in Equation (3), the overall contribution seems limited. The authors do point out that existing literature for distribution based metrics exist (Becker et al., 2024), but do not directly compare against it (e.g., for a task with model selection). They do provide experiments for model selection, but these primarily concern the stability of their introduced metric (measured via ranking and Spearman correlation), and provide limited evidence that this metric is preferable relative to existing alternatives.
Questions for authors
-
What additional practical benefit does your introduced residual formulation provide over existing distributional aware fairness measures (e.g., Becker et al. (2024))? For example, can this benefit be demonstrated in a concrete model selection experiment?
-
On a related note, could the authors add empirical results that rank the models according to overall classification accuracy for various thresholds, rather than the FN cost gap between groups?
-
(Future work) Could the metric be extended to multiple classes? And can your residual distribution formulation motivate a post-processing algorithm that adjusts the scores to improve (group) fairness?
We thank the reviewer for recognizing the intuitive interpretation of residual distributions and the connection between our formulation and familiar classification error measures.
1. Additional practical value over existing distributional fairness measures
Our formulation provides a complementary auditing perspective by explicitly incorporating group prevalence into the distributional analysis of classification errors.
While outcome-conditional Wasserstein measures characterize differences in score distributions within each label class, residual Wasserstein distance captures the resulting prevalence-aware group-level FN/FP burdens.
This distinction becomes important when group prevalences differ, because similar conditional error disparities can correspond to different proportions of individuals experiencing classification errors.
We demonstrate this through two complementary analyses.
First, in a controlled synthetic construction, two models have the same conditional Wasserstein disparity (E=0.20) but different residual distances (0.16 and 0.04) at prevalence 0.20. The residual formulation distinguishes how positive- and negative-label disparities contribute to group-level error burden.
Second, our ACSIncome experiments reveal differences in demographic group-pair rankings. For instance, Black–White changes from rank 2 under conditional Wasserstein distance to rank 5 under residual distance, while Asian–Two or More Races changes from rank 6 to rank 3 (ascending disparity rankings).
These findings demonstrate that residual-based auditing provides a different prioritization of group disparities when the auditing objective incorporates group prevalence.
2. Model selection and predictive utility
We additionally evaluated model selection under different fairness objectives while monitoring predictive accuracy.
On Adult, residual-based selection produces a prevalence-weighted false-negative gap of 0.057, compared with 0.064 under fixed-threshold EO selection, with the selected models remaining within a comparable predictive accuracy range of approximately 0.871.
This illustrates how the choice of fairness measure influences model selection according to the type of error disparity being prioritized.
The FN/FP decomposition further allows these contributions to be examined separately, making the relationship between fairness objectives and predictive utility more interpretable.
3. Extensions to multiclass classification and post-processing
Our current formulation considers binary probabilistic classification, where signed residuals connect naturally to threshold-dependent false-negative and false-positive errors.
A multiclass extension could examine class-specific residual distributions and their corresponding prevalence-weighted error contributions.
Likewise, the decomposition provides a foundation for investigating post-processing procedures that target specific group-level error burdens.
In summary, the residual formulation contributes an interpretable, prevalence-aware perspective on distributional fairness, connecting classification scores, group prevalence, and the FN/FP burdens experienced by different demographic groups across decision thresholds.