- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the intervention attribution of fairness-aware graph neural networks (GNNs). The paper claims that existing studies always compare the fairness method with the reference baseline in terms of fairness performance to verify the effectiveness of the fairness method. However, such an evaluation pipeline neglects the impact of other factors like optimizer and model architecture. Consequently, the paper conducts matched comparisons between intervention-disabled and intervention-enabled versions of the same method. Then, the paper treats these comparisons as conditional on the protocol for training, testing, and evaluating the model. Based on this new comparison pipeline, the paper conducts evaluations across eleven methods and 36 cells, demonstrating that intervention-level conclusions depend on the evaluated specification.
Critical strengths
-
The topic of the paper is fairness in GNNs, which is an increasingly important topic within trustworthy AI.
-
The paper provides a detailed audit result. Specifically, the paper records the intervention boundary of each fairness method and released-code limitations. Then, the paper checks strong baselines and statistical sensitivity under a limited number of data splits.
-
The paper conducts experiments across eleven graph fairness methods, including the most representative graph fairness methods.
Critical weaknesses
-
The motivation of the attribution framework is weak, and the methodological gap beyond standard ablation studies is insufficiently established. In a unified code framework, such as PyGDebias (https://github.com/yushundong/PyGDebias) and GraphFairness (https://github.com/ZzoomD/GraphFairness), a standard ablation study can easily achieve the intervention attribution defined in this paper. In this regard, although the core contribution of the paper focuses on the formulation of protocol-conditional attribution and audit of existing fairness methods, the motivation of the paper is weak due to the above-mentioned reason. Meanwhile, existing studies, such as [1] and [2], have already discussed base-model confounding, fairness-control hyperparameters, and training dynamics. Thus, the contribution of the paper should be improved through additional findings.
-
The main component comparison needs more empirical evidence to obtain robust conclusions. After the external GCN baseline model has completed 1,000 training epochs across the various datasets, the number of units for which the absolute change in fairness due to the surrounding environment exceeds the change resulting from the intervention decreases: for the DP metric, this count drops from 26/36 to 19/36, while for the EO metric, it falls from 29/36 to 20/36. This demonstrates that this finding highly relies on the settings of baselines. Furthermore, some cases counted as opposite directions have an uncertain intervention effect. For example, when NIFTY is used on the German dataset, the point estimate is -0.006 with an interval of [−0.042, +0.032]. Thus, it cannot be determined whether the intervention improves or worsens fairness.
-
Improvements in fairness need to be explained in combination with actual classification results. The AUC metric used in the paper primarily reflects the ranking of predicted scores. It does not directly indicate how the model performs once a fixed classification threshold is applied. For instance, a model could predict the negative class for every single sample, resulting in zero disparity between groups in terms of DP and EO, while simultaneously failing to identify any true positive cases.
-
The experiments on training duration and model selection criteria are not sufficiently comprehensive. Among 16 sets of systematic comparisons regarding training duration, 15 focus on FairSIN, with training epochs reduced from 200 to 100–150. There is limited investigation into whether the conclusions hold when other methods are trained for longer periods. Furthermore, the criteria used for selecting the final model focus primarily on predictive performance. It is recommended to evaluate longer training durations across a broader range of methods, and to adopt model selection approaches that account for both predictive performance and fairness. These experiments will determine whether the conclusions concerning the contribution of the interventions remain robust.
References:
[1] Unprocessing Seven Years of Algorithmic Fairness
[2] FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
Questions for authors
Refer to Critical Weaknesses for more details.