- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Counterfactual AUC Fairness (CF-AUC), which compares factual and counterfactual ranking accuracy over fixed positive–negative pair populations under joint sensitive-attribute interventions. It examines relationships with existing fairness criteria, decomposes observational AUC disparities, and formulates constrained learning using estimated counterfactuals. Experiments investigate fairness–utility trade-offs on synthetic data and Adult.
Critical strengths
- Fixing the evaluation population provides a clear distinction between population differences and intervention-induced ranking changes.
- The paper explicitly distinguishes average AUC stability from pairwise ranking stability.
- The error analysis highlights how counterfactual-generation error interacts with scorer sensitivity and margins near the ranking boundary.
Critical weaknesses
- Limited fairness guarantees. CF-AUC allows opposing ranking changes to cancel and only evaluates joint attribute flips. A small gap therefore provides limited assurance about individual ranking stability.
- Unclear generator assumptions. The criterion requires counterfactual distributions conditional on factual labels, but Adult uses a label-free generator. The main text does not establish the assumptions connecting these two.
- Potential surrogate degeneracy. Shrinking score magnitudes can reduce smooth fairness gaps while leaving hard CF-AUC gaps unchanged. How the method prevents this behavior requires clarification.
- Limited empirical validation. Real-data fairness is evaluated under the fitted generator, while robustness tests mainly use Gaussian noise in a linear SCM. Evidence under alternative causal models or systematic generation errors remains limited.
Questions for authors
- Why restrict the criterion to joint attribute flips? Can you report individual-side intervention results and groupwise pairwise reversal rates?
- Do the main conclusions remain stable under alternative plausible generators or systematic group-dependent generation errors?
- How do you prevent score-scale shrinkage from satisfying the smooth constraints without improving hard CF-AUC gaps?
9 comments
Official Comment by Authors (2/5)
(Continue with W1.)
Theoretical bounds also extend to this setting. In particular, under the assumptions of Proposition 4.1, the generation-error transfer bound becomes
$|\Delta_{ab}^{+}-\widetilde\Delta_{ab}^{+}| \leq \mathbb P_{\mathcal P_{ab}} (|\widetilde M^{+}|\leq r) +\frac{e_{ab}^{+}(\theta)}{r},$ where $e_{ab}^{+}(\theta)=L_\theta, \mathbb E_{\mathcal P_{ab}} \bigl[|\widetilde X_i^\kappa-X_i^\kappa|\bigr].$
Compared with the joint-flip case, the generation-error budget contains only the intervened individual's error, since the other member retains its factual score. The same ranking-boundary argument underlying Theorem 4.1 also applies to the unilateral margin. The negative member only case follows symmetrically. We provide these extensions in the Appendix F.5. We provide the additional instability results in the experiments, and discuss the intervention-policy scope and unilateral theoretical extensions. While the current experiments evaluate joint flips, the underlying framework is not restricted to them. For more detailed results, please refer to the answer to Question 1.
W2. Unclear generator assumptions.
We thank the reviewer for raising this concern.
First, label-free generation is compatible with evaluation conditional on factual labels. Our implementation uses a label-free DCEVAE, with descendant latent posterior . For each observed individual, we infer the latent variables from factual covariates, generate counterfactual descendants under the flipped sensitive attribute, and retain the observed non-descendants and factual outcome label. Thus, although is not used by the generator, the resulting counterfactual scores are evaluated within the original factual strata.
More formally, let denote the generator-induced distribution of counterfactual descendants under . The corresponding label-conditional distribution is $\widehat P(X(c)\in B\mid A=a,Y=y)= \int \widehat K_c(B\mid a,w,x) ,dP(w,x\mid A=a,Y=y).$ Thus, conditioning on factual labels is achieved through the evaluation population and does not require the generator itself to take as input.
Second, we distinguish this construction from identification of the true counterfactual distribution. A sufficient condition for the induced distribution to recover the true SCM target is that the generator correctly estimates the factual-record-conditional counterfactual kernel and that . More generally, it suffices that the generator induces the correct counterfactual score distributions within the factual group-label strata. Our Adult experiments therefore evaluate CF-AUC under the fitted counterfactual generator, rather than claiming nonparametric identification of the true counterfactual distribution from observational data. This interpretation is also consistent with our analysis of counterfactual generation error in Section 4.
We have clarified the generator inputs, the label-conditioned evaluation procedure, and the additional assumptions required for identification in the revised version.
W3. Potential surrogate degeneracy.
We thank the reviewer for raising this concern.
Our implementation addresses this scale degeneracy through score normalization within the optimization objective and constraints as shown in the Appendix. Specifically, both the ranking loss and smooth CF-AUC constraints are computed using scores standardized by their factual-training mean and standard deviation. These statistics are recomputed for each candidate scorer during optimization, and the same transformation is applied to factual and counterfactual scores. Consequently, common positive rescaling of raw scores leaves the normalized surrogate unchanged, up to numerical stabilization. For more detailed formulation and empirical evidence, please refer to the answer to Question 3.
We have clarified the role of score normalization in preventing global score shrinkage in the Appendix F.2.
Official Comment by Authors (3/5)
W4. Limited empirical validation.
We thank the reviewer for this observation. To complement the existing Gaussian-noise experiment, we have added a sensitivity experiment involving systematic, group-dependent counterfactual generation errors in the nonlinear SCM.
Specifically, we consider both 30% underestimation and 30% overestimation of the mediator's intervention effect. These perturbations induce systematic errors that vary across individuals and have opposite signs across factual sensitive groups. We retrain the CF-AUC scorers using the misspecified generators, while retaining the original optimization procedure, training-tolerance grid, and model-selection protocol. The resulting scorers are evaluated using the true SCM.
At training tolerance 0.025, the mean test CF-AUC gaps under the true SCM are 0.0430 and 0.0234 for 30% underestimation and overestimation, respectively, compared with 0.0226 under the correctly specified generator. All remain below the unconstrained gap of 0.1218, while factual AUC remains around 0.81 across the three settings (five seeds). These results extend the empirical validation beyond Gaussian perturbations in a linear setting, while showing that the achieved fairness remains sensitive to systematic generation errors. We have included the experimental setup and quantitative results in Appendix G.6.
Official Comment by Authors (1/5)
We thank the reviewer for the detailed and constructive feedback. In response, we have clarified the counterfactual generator assumptions and the treatment of score normalization, and supplemented the experiments with unilateral intervention evaluations, groupwise pairwise reversal rates, and systematic generation-error sensitivity analysis. We address each point below. We have revised our draft accordingly.
We thank the reviewer for pointing out these concerns. We would like to clarify both the motivation for the proposed fairness criterion and the scope of its intervention-specific guarantees.
1. Cancellation of opposing ranking changes.
CF-AUC and pairwise ranking stability capture different fairness requirements. CF-AUC measures changes in ranking accuracy within each fixed factual group-pair population, whereas Rab measures the total amount of pairwise comparison change.
To assess the extent of remaining pairwise changes, we have additionally evaluated Rab using ground-truth counterfactuals in our existing synthetic experiments, without retraining or reselecting the original models. Representative low-gap results are shown below (test mean ± standard deviation over ten runs):
These results show that stringent CF-AUC constraints can also yield low pairwise instability. We report the results in the supplementary material.
Importantly, our choice of CF-AUC reflects the fairness objective of interest, rather than an inability to impose stronger pairwise stability constraints. For example, consider clinical prioritization, where patients are ranked according to an accepted benchmark of adverse-outcome risk. If the fairness concern is whether the scoring model maintains its ability to distinguish patients with different clinical priorities under sensitive-attribute interventions, group-pair ranking accuracy stability provides a meaningful criterion. This does not necessarily require every individual pairwise comparison to remain unchanged.
To clarify the distinction, in the absence of ties, let pabloss denote the probability that a correctly ordered factual positive–negative pair becomes incorrectly ordered after intervention, and pabgain the probability of the reverse change. Then
$\Delta_{ab}= |p_{ab}^{\mathrm{loss}}-p_{ab}^{\mathrm{gain}}|, \qquad R_{ab}= p_{ab}^{\mathrm{loss}}+p_{ab}^{\mathrm{gain}}.$
Thus, Rab penalizes all pairwise changes, including corrections of previously incorrect rankings, while CF-AUC measures their net effect on group-pair ranking accuracy. Since Δab≤Rab, the pairwise stability constraint is stronger and may impose additional factual ranking utility costs. The suitability of two objectives are therefore depends on the application.
2. Restriction to joint attribute flips.
We agree that fairness under simultaneous joint flips does not automatically imply fairness under unilateral interventions.
However, our focus on joint flips is deliberate: for binary sensitive attributes, simultaneous flipping maps a factual group pair (a,b) to its complementary assignment (1−a,1−b). This provides a natural symmetric construction for the analysis in Section 3.2, connecting CF-AUC with observational xAUC parity and within-group AUC parity.
Meanwhile, unilateral interventions can be directly incorporated into the proposed framework by changing the intervention-specific pair margin. For example, when only the positive member is intervened upon, the counterfactual margin becomes Mij+=Siκ−SjF, instead of Mijκ=Siκ−Sjκ. The CF-AUC definition and constrained learning formulation apply with this substituted margin.