acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual AUC Fairness: Learning Fair Bipartite Ranking under Intervention

Abstract

Causal fairness and AUC fairness have largely developed separately: the former studies how predictions respond to interventions on sensitive attributes, whereas the latter compares ranking performance across observed groups. We bridge these perspectives by introducing counterfactual AUC fairness (CF-AUC), which asks whether ranking accuracy within each positive–negative pair population remains stable when both individuals' sensitive attributes are counterfactually intervened. By fixing the pair population, CF-AUC isolates intervention-induced ranking changes from changes in population composition. We decompose observational AUC disparities into causal and population components and show that observational parity can be achieved by introducing counterfactual ranking changes while leaving factual AUC unchanged. We further characterize CF-AUC's relation to its pairwise and population variants and distinguish it from fixed-threshold decision invariance and score invariance. We then formulate fair ranking as maximizing factual AUC subject to groupwise CF-AUC constraints, providing a differentiable surrogate formulation, and bound the discrepancy between true and estimated CF-AUC gaps in terms of counterfactual-generation error, scorer sensitivity, and pairwise margin concentration near the ranking boundary. Experiments on synthetic and real data demonstrate favorable fairness–utility trade-offs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Algorithmic fairness, AUC, Fair ranking, Bipartite ranking, Causality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YdjtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Counterfactual AUC Fairness (CF-AUC), which compares factual and counterfactual ranking accuracy over fixed positive–negative pair populations under joint sensitive-attribute interventions. It examines relationships with existing fairness criteria, decomposes observational AUC disparities, and formulates constrained learning using estimated counterfactuals. Experiments investigate fairness–utility trade-offs on synthetic data and Adult.

Critical strengths
  1. Fixing the evaluation population provides a clear distinction between population differences and intervention-induced ranking changes.
  2. The paper explicitly distinguishes average AUC stability from pairwise ranking stability.
  3. The error analysis highlights how counterfactual-generation error interacts with scorer sensitivity and margins near the ranking boundary.
Critical weaknesses
  1. Limited fairness guarantees. CF-AUC allows opposing ranking changes to cancel and only evaluates joint attribute flips. A small gap therefore provides limited assurance about individual ranking stability.
  2. Unclear generator assumptions. The criterion requires counterfactual distributions conditional on factual labels, but Adult uses a label-free generator. The main text does not establish the assumptions connecting these two.
  3. Potential surrogate degeneracy. Shrinking score magnitudes can reduce smooth fairness gaps while leaving hard CF-AUC gaps unchanged. How the method prevents this behavior requires clarification.
  4. Limited empirical validation. Real-data fairness is evaluated under the fitted generator, while robustness tests mainly use Gaussian noise in a linear SCM. Evidence under alternative causal models or systematic generation errors remains limited.
Questions for authors
  1. Why restrict the criterion to joint attribute flips? Can you report individual-side intervention results and groupwise pairwise reversal rates?
  2. Do the main conclusions remain stable under alternative plausible generators or systematic group-dependent generation errors?
  3. How do you prevent score-scale shrinkage from satisfying the smooth constraints without improving hard CF-AUC gaps?
9 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/5)

We thank the reviewer for the detailed and constructive feedback. In response, we have clarified the counterfactual generator assumptions and the treatment of score normalization, and supplemented the experiments with unilateral intervention evaluations, groupwise pairwise reversal rates, and systematic generation-error sensitivity analysis. We address each point below. We have revised our draft accordingly.

W1. Limited fairness guarantees.

We thank the reviewer for pointing out these concerns. We would like to clarify both the motivation for the proposed fairness criterion and the scope of its intervention-specific guarantees.

1. Cancellation of opposing ranking changes.

CF-AUC and pairwise ranking stability capture different fairness requirements. CF-AUC measures changes in ranking accuracy within each fixed factual group-pair population, whereas RabR_{ab} measures the total amount of pairwise comparison change.

To assess the extent of remaining pairwise changes, we have additionally evaluated RabR_{ab} using ground-truth counterfactuals in our existing synthetic experiments, without retraining or reselecting the original models. Representative low-gap results are shown below (test mean ± standard deviation over ten runs):

SCMTraining toleranceFactual AUC ↑Δmax⁡\Delta_{\max} ↓Rmax⁡R_{\max} ↓
Linear0.0010.8529 ± 0.00320.0016 ± 0.00060.0112 ± 0.0032
Linear0.0030.8646 ± 0.00370.0045 ± 0.00130.0226 ± 0.0045
Linear0.0050.8718 ± 0.00360.0070 ± 0.00210.0340 ± 0.0064
Nonlinear0.0010.7558 ± 0.00630.0005 ± 0.00040.0014 ± 0.0012
Nonlinear0.0030.7713 ± 0.01710.0020 ± 0.00170.0198 ± 0.0285
Nonlinear0.0060.7837 ± 0.01840.0050 ± 0.00310.0390 ± 0.0354

These results show that stringent CF-AUC constraints can also yield low pairwise instability. We report the results in the supplementary material.

Importantly, our choice of CF-AUC reflects the fairness objective of interest, rather than an inability to impose stronger pairwise stability constraints. For example, consider clinical prioritization, where patients are ranked according to an accepted benchmark of adverse-outcome risk. If the fairness concern is whether the scoring model maintains its ability to distinguish patients with different clinical priorities under sensitive-attribute interventions, group-pair ranking accuracy stability provides a meaningful criterion. This does not necessarily require every individual pairwise comparison to remain unchanged.

To clarify the distinction, in the absence of ties, let pablossp_{ab}^{\mathrm{loss}} denote the probability that a correctly ordered factual positive–negative pair becomes incorrectly ordered after intervention, and pabgainp_{ab}^{\mathrm{gain}} the probability of the reverse change. Then

$\Delta_{ab}= |p_{ab}^{\mathrm{loss}}-p_{ab}^{\mathrm{gain}}|, \qquad R_{ab}= p_{ab}^{\mathrm{loss}}+p_{ab}^{\mathrm{gain}}.$

Thus, RabR_{ab} penalizes all pairwise changes, including corrections of previously incorrect rankings, while CF-AUC measures their net effect on group-pair ranking accuracy. Since Δab≤Rab\Delta_{ab}\leq R_{ab}, the pairwise stability constraint is stronger and may impose additional factual ranking utility costs. The suitability of two objectives are therefore depends on the application.

2. Restriction to joint attribute flips.

We agree that fairness under simultaneous joint flips does not automatically imply fairness under unilateral interventions.

However, our focus on joint flips is deliberate: for binary sensitive attributes, simultaneous flipping maps a factual group pair (a,b)(a,b) to its complementary assignment (1−a,1−b)(1-a,1-b). This provides a natural symmetric construction for the analysis in Section 3.2, connecting CF-AUC with observational xAUC parity and within-group AUC parity.

Meanwhile, unilateral interventions can be directly incorporated into the proposed framework by changing the intervention-specific pair margin. For example, when only the positive member is intervened upon, the counterfactual margin becomes Mij+=Siκ−SjF,M_{ij}^{+}=S_i^\kappa-S_j^F, instead of Mijκ=Siκ−SjκM_{ij}^{\kappa}=S_i^\kappa-S_j^\kappa. The CF-AUC definition and constrained learning formulation apply with this substituted margin.

AuthorsOpenReview ↗

Official Comment by Authors (2/5)

(Continue with W1.)

Theoretical bounds also extend to this setting. In particular, under the assumptions of Proposition 4.1, the generation-error transfer bound becomes

$|\Delta_{ab}^{+}-\widetilde\Delta_{ab}^{+}| \leq \mathbb P_{\mathcal P_{ab}} (|\widetilde M^{+}|\leq r) +\frac{e_{ab}^{+}(\theta)}{r},$ where $e_{ab}^{+}(\theta)=L_\theta, \mathbb E_{\mathcal P_{ab}} \bigl[|\widetilde X_i^\kappa-X_i^\kappa|\bigr].$

Compared with the joint-flip case, the generation-error budget contains only the intervened individual's error, since the other member retains its factual score. The same ranking-boundary argument underlying Theorem 4.1 also applies to the unilateral margin. The negative member only case follows symmetrically. We provide these extensions in the Appendix F.5. We provide the additional instability results in the experiments, and discuss the intervention-policy scope and unilateral theoretical extensions. While the current experiments evaluate joint flips, the underlying framework is not restricted to them. For more detailed results, please refer to the answer to Question 1.

W2. Unclear generator assumptions.

We thank the reviewer for raising this concern.

First, label-free generation is compatible with evaluation conditional on factual labels. Our implementation uses a label-free DCEVAE, with descendant latent posterior qG^,(UX∣X,A)q_{\hat G,}(U_X\mid X,A). For each observed individual, we infer the latent variables from factual covariates, generate counterfactual descendants under the flipped sensitive attribute, and retain the observed non-descendants and factual outcome label. Thus, although YY is not used by the generator, the resulting counterfactual scores are evaluated within the original factual (A,Y)(A,Y) strata.

More formally, let K^c(⋅∣a,w,x)\widehat K_c(\cdot\mid a,w,x) denote the generator-induced distribution of counterfactual descendants under A←cA\leftarrow c. The corresponding label-conditional distribution is $\widehat P(X(c)\in B\mid A=a,Y=y)= \int \widehat K_c(B\mid a,w,x) ,dP(w,x\mid A=a,Y=y).$ Thus, conditioning on factual labels is achieved through the evaluation population and does not require the generator itself to take YY as input.

Second, we distinguish this construction from identification of the true counterfactual distribution. A sufficient condition for the induced distribution to recover the true SCM target is that the generator correctly estimates the factual-record-conditional counterfactual kernel and that X(c)⊥Y∣A,W,XX(c)\perp Y\mid A,W,X. More generally, it suffices that the generator induces the correct counterfactual score distributions within the factual group-label strata. Our Adult experiments therefore evaluate CF-AUC under the fitted counterfactual generator, rather than claiming nonparametric identification of the true counterfactual distribution from observational data. This interpretation is also consistent with our analysis of counterfactual generation error in Section 4.

We have clarified the generator inputs, the label-conditioned evaluation procedure, and the additional assumptions required for identification in the revised version.

W3. Potential surrogate degeneracy.

We thank the reviewer for raising this concern.

Our implementation addresses this scale degeneracy through score normalization within the optimization objective and constraints as shown in the Appendix. Specifically, both the ranking loss and smooth CF-AUC constraints are computed using scores standardized by their factual-training mean and standard deviation. These statistics are recomputed for each candidate scorer during optimization, and the same transformation is applied to factual and counterfactual scores. Consequently, common positive rescaling of raw scores leaves the normalized surrogate unchanged, up to numerical stabilization. For more detailed formulation and empirical evidence, please refer to the answer to Question 3.

We have clarified the role of score normalization in preventing global score shrinkage in the Appendix F.2.

AuthorsOpenReview ↗

Official Comment by Authors (3/5)

W4. Limited empirical validation.

We thank the reviewer for this observation. To complement the existing Gaussian-noise experiment, we have added a sensitivity experiment involving systematic, group-dependent counterfactual generation errors in the nonlinear SCM.

Specifically, we consider both 30% underestimation and 30% overestimation of the mediator's intervention effect. These perturbations induce systematic errors that vary across individuals and have opposite signs across factual sensitive groups. We retrain the CF-AUC scorers using the misspecified generators, while retaining the original optimization procedure, training-tolerance grid, and model-selection protocol. The resulting scorers are evaluated using the true SCM.

At training tolerance 0.025, the mean test CF-AUC gaps under the true SCM are 0.0430 and 0.0234 for 30% underestimation and overestimation, respectively, compared with 0.0226 under the correctly specified generator. All remain below the unconstrained gap of 0.1218, while factual AUC remains around 0.81 across the three settings (five seeds). These results extend the empirical validation beyond Gaussian perturbations in a linear setting, while showing that the achieved fairness remains sensitive to systematic generation errors. We have included the experimental setup and quantitative results in Appendix G.6.

Reviewer_dt46OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces counterfactual AUC fairness (CF-AUC), a causal fairness extension for the bipartite ranking problem. It asks whether a scorer’s ranking accuracy on the same positive–negative pairs stays stable under counterfactual fairness setting. The authors show that this criterion is distinct from observational AUC parity, fixed-threshold decision invariance, and score invariance. They then propose learning a ranker that maximizes AUC subject to groupwise CF-AUC constraints using estimated counterfactuals, with a bound on how counterfactual generation error affects the fairness estimate. Experiments on synthetic data and Adult show better fairness–utility trade-offs than the baselines on the CF-AUC metric.

Critical strengths
  1. The authors extend prior work on AUC-based ranking fairness by considering counterfactual reasoning. The main novelty is a causal perspective on fairness in bipartite ranking. This may provide a meaningful extension of existing fairness notions
  2. The paper clearly relates CF-AUC to existing fairness criteria and draws a useful finding that for observational AUC parity and counterfactual ranking fairness, simply achieving one cannot guarantee the other as they are distinct
  3. The paper is easy to follow, and the methodology design is directly driven by the theoretical analysis, making it intuitive why standard classification or score invariance fails for ranking.
Critical weaknesses
  • Experiments are somehow limited. The evaluation is close to self-referential, and comparable baselines are missing. The main plots judge every method on the metric only CF-AUC constrains, making CF-AUC wins easier to achieve
  • As the authors acknowledge, CF-AUC only measures average ranking accuracy, so changes in individual pairwise rankings may cancel each other out. As a result, a small CF-AUC gap does not necessarily guarantee pairwise ranking stability under counterfactual interventions. If so, what cannot we directly optimize the stronger pairwise ranking stability criterion instead?
Questions for authors

See above

4 comments

We thank the reviewer for the thoughtful assessment and constructive suggestions. We have clarified the specific fairness objective addressed by CF-AUC and its relationship to observational fairness and stronger pairwise ranking stability. We also provide additional pairwise stability evaluations to better illustrate the distinction between group-level fairness and individual ranking changes. We respond to the concerns below.

W1. Experiments are somehow limited.

We thank the reviewer for this comment.

The main fairness--utility plots address a specific question: when an application requires controlling groupwise counterfactual AUC gaps, how much factual ranking utility can different fairness constraints retain? Our intention is not to suggest that methods with larger CF-AUC gaps fail to satisfy their own fairness criteria. Importantly, our experiments include directly related causal alternatives in addition to observational fairness baselines. In particular, population-level CF-AUC examines whether controlling an aggregated gap is sufficient to control groupwise disparities, while counterfactual score invariance imposes a stronger pointwise requirement that implies CF-AUC fairness. These baselines therefore provide meaningful comparisons between different levels of counterfactual fairness constraints and their associated utility costs.

W2. what cannot we directly optimize the stronger pairwise ranking stability criterion instead?

We thank the reviewer for raising this question. We agree that directly optimizing pairwise ranking stability is a valid alternative, particularly when the application requires individual positive–negative pair comparisons to remain unchanged under sensitive-attribute interventions. Our choice of CF-AUC is motivated by a different fairness objective rather than by an inability to optimize the stronger criterion.

First, the two criteria capture different fairness requirements. CF-AUC controls the change in ranking accuracy within each group-pair population, whereas RabR_{ab} measures the total amount of pairwise comparison change.

Consider the clinical prioritization setting discussed in our revised Introduction, where patients are ranked for follow-up care according to their risk of an adverse outcome. When the fairness concern is whether the scorer can reliably recognize the same clinical priority distinctions under sensitive-attribute interventions, stability of group-pair ranking accuracy provides a natural objective. This does not necessarily require every individual positive–negative pair comparison to remain unchanged. In particular, RabR_{ab} also penalizes changes that correct previously incorrect rankings.

To see the distinction, in the absence of ties, let pablossp_{ab}^{\mathrm{loss}} and pabgainp_{ab}^{\mathrm{gain}} denote the probabilities of correct-to-incorrect and incorrect-to-correct ordering changes, respectively. Then

$\Delta_{ab}=|p_{ab}^{\mathrm{loss}}-p_{ab}^{\mathrm{gain}}|, \qquad R_{ab}=p_{ab}^{\mathrm{loss}}+p_{ab}^{\mathrm{gain}}.$

Thus, CF-AUC measures the net change in ranking accuracy, while RabR_{ab} measures total pairwise instability. Neither is universally preferable: the appropriate criterion depends on whether the application prioritizes group-level accuracy stability or invariance of individual pairwise comparisons.

Second, pairwise stability imposes a stronger constraint. Since Δab≤Rab\Delta_{ab}\leq R_{ab}, requiring Rab≤εR_{ab}\leq\varepsilon for every group pair implies ε\varepsilon-CF-AUC fairness, but not conversely. The feasible set under the pairwise constraint is therefore a subset of that under the corresponding CF-AUC constraint. Consequently, maximizing factual AUC subject to pairwise stability may incur additional utility costs, although the difference need not be strict.

Third, the criteria differ in the counterfactual information required for identification. Under independent pair sampling, CF-AUC for a fixed scorer is determined by the conditional counterfactual score distributions within the factual group-label strata. In contrast, RabR_{ab} depends on whether the same factual pair's ordering changes across worlds, and therefore generally requires the joint distribution of factual and counterfactual scores for each individual, conditional on their factual stratum. These joint distributions are not determined by the marginal score distributions sufficient for CF-AUC. This reflects the broader challenge of identifying cross-world counterfactual dependence (Shpitser and Pearl, 2008; Nasr-Esfahany et al., 2023). Importantly, this does not preclude estimating or optimizing RabR_{ab}: a sufficiently specified SCM or counterfactual generator can provide the required coupling, albeit with additional modeling assumptions.

We have clarified these distinctions in Section 3.1. We emphasize that pairwise ranking stability is an appropriate stronger objective when individual pairwise invariance is required. CF-AUC instead targets the stability of group-level ranking accuracy for a fixed factual population, providing a less restrictive alternative when that is the fairness requirement of interest.

Thanks for your response. I will maintain my positive score, good luck!

Thanks for strengthening our paper, and we greatly appreciate your efforts!

Reviewer_s7FDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes CF-AUC fairness, which measures changes in ranking accuracy under interventions on sensitive attributes while keeping factual labels and pair populations fixed. It develops a constrained learning method and evaluates fairness–utility trade-offs.

Critical strengths
  • Section 4.1 explicitly distinguishes smooth training constraints from the original evaluation metric. This is an important distinction that is easy to overlook.
  • Figure 3 shows substantial reductions in CF-AUC gaps with relatively small changes in factual AUC, which is an encouraging result.
Critical weaknesses
  • Motivation and novelty: I’m not yet convinced when CF-AUC should be preferred over existing fairness criteria. The mathematical distinctions in Sections 3.2–3.3 are useful, but a concrete application would help establish the contribution beyond combining counterfactual invariance with AUC-based evaluation.
  • Computational cost: Direct evaluation of the pairwise objective in Section 4.1 involves all positive–negative comparisons. Although counterfactuals are cached, repeated pairwise optimization could still be expensive. Runtime, memory usage, and the effect of pair sampling on constraint accuracy need discussion.
  • I found the notation in Section 3.1 difficult to follow. Expressions such as AUCabF\mathrm{AUC}_{ab}^{F}, AUCabκ\mathrm{AUC}_{ab}^{\kappa}, and HijκH_{ij}^{\kappa} combine several superscripts and subscripts, making the equations visually dense and hard to read.
Questions for authors
  1. If a counterfactual sample has an exact match in the observed data, why not use that observation directly and reserve SCM-based generation for cases without an observed match? Is there any discussion of this possibility in the paper?

  2. Section 3.1 notes that opposite ordering changes can cancel, so zero groupwise CF-AUC gaps do not guarantee unchanged rankings. Could you report RabR_{ab} alongside CF-AUC to show how much pairwise ranking change remains even when the fairness gaps are close to zero?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank the reviewer for the careful reading and constructive feedback. In response, we have strengthened the motivation with a concrete clinical prioritization example, clarified the notation and computational implementation, and added quantitative diagnostics of pairwise ranking changes. We address each concern below and have revised our draft accordingly.

W1. Motivation and Novelty.

We thank the reviewer for this helpful suggestion.

Consider a concrete example of clinical prioritization under potentially biased patient recording. Consider a healthcare service that ranks patients for follow-up based on their risk of an adverse outcome. While clinical need provides the accepted basis for prioritization, the recorded information used by the model may be affected by sensitive attributes. This concern is supported by empirical evidence of racial disparities in clinical documentation (Sun et al., 2022), while the American Medical Association's ethical guidance recommends prioritization based on medical need rather than nonmedical characteristics. In this setting, CF-AUC captures a specific fairness requirement: the probability of correctly recognizing the same clinical priority distinctions should remain stable under interventions on sensitive attributes. Observational AUC parity cannot directly assess this requirement because it compares different factual patient populations. Conversely, counterfactual score invariance may be unnecessarily restrictive, since numerical scores can change without altering ranking accuracy. We emphasize that CF-AUC is appropriate when the relevant fairness objective is group-level counterfactual stability of ranking accuracy for a factual priority benchmark. Its contribution goes beyond directly combining counterfactual invariance with AUC evaluation: it explicitly separates the factual pair population from the counterfactual scoring world, characterizes the resulting distinctions from observational fairness, and provides a learning framework with guarantees relating counterfactual approximation error to ranking stability.

We have clarified this motivation in the revised Introduction with the corresponding causal graph.

W2. Computational cost

We thank the reviewer for this comment.

Computational cost. Rather than enumerating all positive–negative pairs, we use fixed sampled pairs (12,000 per group for the linear SCM and 20,000 for the nonlinear SCM), reused throughout optimization. Thus, pairwise computation and storage scale linearly with the sampling budget. Our recorded median runtimes are 6.82s (linear) and 64.93s (nonlinear) per seed. These runtimes cover all fairness methods and tolerance settings for each seed.

Sampling accuracy. We evaluated the sampling error of the smooth CF-AUC constraints using fixed scorers. In the linear SCM, increasing the sampled pairs per group from 1,000 to 12,000 reduced the mean maximum constraint-estimation error from 0.00130 to 0.00039. Importantly, validation and test CF-AUC gaps are computed exactly over all empirical pairs, independently of training-pair sampling.

We have added these implementation details and sampling-accuracy results to the Appendix F.1.

W3. notation in Section 3.1

We thank the reviewer for pointing out the problem in following the notation in Section 3.1.

We have reorganized Section 3.1 to introduce CF-AUC through three steps: selecting a fixed factual positive–negative pair population, evaluating its scores under factual and counterfactual attribute assignments, and comparing the resulting ranking accuracies. To reduce notational complexity, we first define the individual scores SiFS_i^{\mathrm F} and SiκS_i^\kappa, and then express the pairwise comparisons compactly as Hijw=h(Siw−Sjw)H_{ij}^{w}=h(S_i^w-S_j^w), for w∈{F,κ}w\in\{\mathrm F,\kappa\}. We also explicitly clarify that the subscripts (a,b)(a,b) identify the factual pair population, while the superscript specifies the scoring world. The factual and counterfactual AUCs are evaluated over the same population.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Question 1. If a counterfactual sample has an exact match in the observed data, why not use that observation directly and reserve SCM-based generation for cases without an observed match?

We thank the reviewer for this suggestion. The main difficulty is that the counterfactual sample is not known before generation, so we cannot determine whether an exact match exists in the observed data beforehand. If we first generate the counterfactual and then search for an exact match, the generation step has already been performed. Therefore, the proposed matching strategy would not eliminate the need for counterfactual generation.

Question 2. Could you report alongside CF-AUC to show how much pairwise ranking change remains even when the fairness gaps are close to zero?

We thank the reviewer for this helpful suggestion.

We have additionally evaluated the pairwise ranking instability RabR_{ab} for the existing synthetic experiments using ground-truth counterfactuals. We report Rmax⁡=max⁡(a,b)∈GRabR_{\max}=\max_{(a,b)\in\mathcal G}R_{ab} alongside factual AUC and Δmax⁡\Delta_{\max}. The following table presents representative results in the low-gap region (mean ± standard deviation over 10 runs).

SCMTraining toleranceFactual AUC ↑Δmax⁡\Delta_{\max} ↓Rmax⁡R_{\max} ↓
Linear0.0010.8529 ± 0.00320.0016 ± 0.00060.0112 ± 0.0032
Linear0.0030.8646 ± 0.00370.0045 ± 0.00130.0226 ± 0.0045
Linear0.0050.8718 ± 0.00360.0070 ± 0.00210.0340 ± 0.0064
Nonlinear0.0010.7558 ± 0.00630.0005 ± 0.00040.0014 ± 0.0012
Nonlinear0.0030.7713 ± 0.01710.0020 ± 0.00170.0198 ± 0.0285
Nonlinear0.0060.7837 ± 0.01840.0050 ± 0.00310.0390 ± 0.0354

The results show that tight CF-AUC constraints can also yield low pairwise instability, particularly at the strictest tolerance. As the constraints are relaxed, the two metrics provide complementary information about changes in ranking accuracy and individual pairwise comparisons. We have included these results in the Appendix G.3.

Dear Reviewer,

Thank you again for your constructive feedback. We have addressed your concerns in our rebuttal and updated the manuscript with additional clarifications and experimental results. As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and let us know whether they adequately address your concerns. We would also be happy to clarify any remaining questions.

Thank you for your time and consideration.

Dear Reviewer s7FD,

Thank you again for your careful assessment and constructive feedback. We appreciate your positive assessment of the paper's soundness and your recognition of the useful mathematical distinctions in Sections 3.2–3.3. We are also encouraged that you found the substantial reductions in CF-AUC gaps with relatively small losses in factual AUC to be a promising result.

We would like to emphasize that our contribution goes beyond combining counterfactual invariance with AUC-based evaluation. By holding factual pair populations fixed across counterfactual worlds, CF-AUC isolates intervention-induced changes in ranking accuracy from observational population differences, addressing a fundamental distinction in causal ranking fairness. Our theoretical analysis establishes why observational AUC parity and counterfactual score or decision invariance do not capture the same fairness objective. We further develop a constrained learning framework and establish boundary-sensitive bounds on counterfactual generation errors.

In response to your suggestion, we have strengthened the practical motivation with a concrete clinical prioritization example and an accompanying causal graph. This example illustrates how race-dependent clinical documentation may influence risk-based prioritization and why maintaining ranking accuracy under sensitive attribute interventions is a meaningful fairness requirement. Similar considerations arise in other bipartite ranking applications discussed in our Introduction, including lending, criminal justice, and hiring.

We have also improved the presentation of Section 3.1, clarified the use of standard pair sampling during optimization, and added evaluations of pairwise ranking stability as requested.

As the discussion period is nearing its end, we would greatly appreciate it if you could let us know whether these revisions have addressed your concerns. We would be happy to clarify any remaining questions.

Thank you very much for your time and consideration.

Authors of 43024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.