acceptodds
Withdrawn from ICLR 2027

Protocol-Conditional Attribution in Fairness-Aware Graph Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fairness-aware graph neural networks (GNNs) are typically evaluated by comparing a complete method with an external baseline, even when the scientific claim concerns an intervention intended to reduce bias. Such comparisons show what the complete method does, but not how much of the observed change is associated with the claimed intervention. We separate the complete-method contrast from a matched intervention contrast between intervention-disabled and intervention-enabled versions of the same method, and treat this intervention attribution as conditional on the protocol used to train, select, and evaluate the model, an approach we call protocol-conditional attribution. Across 36 controlled cells spanning eleven methods, the surrounding package has a larger absolute contrast than the claimed intervention in 26/36 cells for demographic parity and 29/36 for equal opportunity (about one half against a more strongly trained baseline), and the intervention contrast is resolved in only 8/36 and 9/36 cells, with most remaining fairness contrasts inconclusive rather than negligible. Changes in method configuration and checkpoint selection frequently alter the magnitude and resolution of intervention attribution, although resolved directions are rarely reversed. Targeted analyses show that longer training can change the intervention contrast at later checkpoints and that an explicit fairness correction can contribute little relative to preceding components. Claims about why a system is fairer therefore require matched intervention evidence under an explicit protocol. Our code is available at https://anonymous.4open.science/r/FairGNN-Eval-ICLR.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fairness, graph neural networks, fairness-aware GNNs, evaluation methodology, intervention attribution

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_vthGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the intervention attribution of fairness-aware graph neural networks (GNNs). The paper claims that existing studies always compare the fairness method with the reference baseline in terms of fairness performance to verify the effectiveness of the fairness method. However, such an evaluation pipeline neglects the impact of other factors like optimizer and model architecture. Consequently, the paper conducts matched comparisons between intervention-disabled and intervention-enabled versions of the same method. Then, the paper treats these comparisons as conditional on the protocol for training, testing, and evaluating the model. Based on this new comparison pipeline, the paper conducts evaluations across eleven methods and 36 cells, demonstrating that intervention-level conclusions depend on the evaluated specification.

Critical strengths
  1. The topic of the paper is fairness in GNNs, which is an increasingly important topic within trustworthy AI.

  2. The paper provides a detailed audit result. Specifically, the paper records the intervention boundary of each fairness method and released-code limitations. Then, the paper checks strong baselines and statistical sensitivity under a limited number of data splits.

  3. The paper conducts experiments across eleven graph fairness methods, including the most representative graph fairness methods.

Critical weaknesses
  1. The motivation of the attribution framework is weak, and the methodological gap beyond standard ablation studies is insufficiently established. In a unified code framework, such as PyGDebias (https://github.com/yushundong/PyGDebias) and GraphFairness (https://github.com/ZzoomD/GraphFairness), a standard ablation study can easily achieve the intervention attribution defined in this paper. In this regard, although the core contribution of the paper focuses on the formulation of protocol-conditional attribution and audit of existing fairness methods, the motivation of the paper is weak due to the above-mentioned reason. Meanwhile, existing studies, such as [1] and [2], have already discussed base-model confounding, fairness-control hyperparameters, and training dynamics. Thus, the contribution of the paper should be improved through additional findings.

  2. The main component comparison needs more empirical evidence to obtain robust conclusions. After the external GCN baseline model has completed 1,000 training epochs across the various datasets, the number of units for which the absolute change in fairness due to the surrounding environment exceeds the change resulting from the intervention decreases: for the DP metric, this count drops from 26/36 to 19/36, while for the EO metric, it falls from 29/36 to 20/36. This demonstrates that this finding highly relies on the settings of baselines. Furthermore, some cases counted as opposite directions have an uncertain intervention effect. For example, when NIFTY is used on the German dataset, the point estimate is -0.006 with an interval of [−0.042, +0.032]. Thus, it cannot be determined whether the intervention improves or worsens fairness.

  3. Improvements in fairness need to be explained in combination with actual classification results. The AUC metric used in the paper primarily reflects the ranking of predicted scores. It does not directly indicate how the model performs once a fixed classification threshold is applied. For instance, a model could predict the negative class for every single sample, resulting in zero disparity between groups in terms of DP and EO, while simultaneously failing to identify any true positive cases.

  4. The experiments on training duration and model selection criteria are not sufficiently comprehensive. Among 16 sets of systematic comparisons regarding training duration, 15 focus on FairSIN, with training epochs reduced from 200 to 100–150. There is limited investigation into whether the conclusions hold when other methods are trained for longer periods. Furthermore, the criteria used for selecting the final model focus primarily on predictive performance. It is recommended to evaluate longer training durations across a broader range of methods, and to adopt model selection approaches that account for both predictive performance and fairness. These experiments will determine whether the conclusions concerning the contribution of the interventions remain robust.

References:

[1] Unprocessing Seven Years of Algorithmic Fairness

[2] FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods

Questions for authors

Refer to Critical Weaknesses for more details.

Reviewer_9zzmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This manuscript points out that existing studies typically fail to specify how much of the improvement in fairness stems from the fairness interventions declared by the method. The authors propose constructing two versions of the same method—one with the intervention enabled and one with it disabled—under fixed training, model selection, and evaluation protocols, decomposing the differences in the complete methods, and treating the attribution conclusions as conditional on the nature of the specific specifications. The authors suggest that claims regarding fairness interventions must be accompanied by evidence supporting the intervention, a statement of the training schedule and selection rules, and evidence of robustness under at least one alternative specification.

Critical strengths
  1. The experimental setup is fair and comprehensive, and the descriptions are thorough and clear.
  2. There are plenty of comparison methods.
  3. The research questions are precisely defined, clearly distinguishing between “Is the complete method fairer?” (RQ1) and “Did the stated intervention contribute to this change?” (RQ2).
Critical weaknesses
  1. The introduction lacks persuasiveness. In fact, after reading the paper, I still have some reservations. Why couldn’t the ablation experiments achieve this? What was lacking?
  2. There are a bit too many made-up terms, which makes it tiring for readers to get through.
  3. The explanation for the phenomenon of “minimal intervention contribution and reliance on protocols” is insufficient and unclear.
Questions for authors

Please refer to the “Weaknesses” section. I believe this manuscript should first clearly explain its objectives and contributions. Also, can this method be applied to fairness checks in the field of computer vision? If this is a general-purpose method, its technical contribution would be even greater.

Reviewer_HCL9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the fairness improvements of fairness-aware GNN methods come from their proposed fairness components or from other parts of the methods, such as training choices. The authors compare each method with and without its fairness intervention under a consistent evaluation protocol, including controlled training settings. The experiments mainly focus on group fairness across commonly used datasets and fairness-aware GNN methods in literature. The results suggest that the proposed fairness interventions do not always explain the fairness improvements achieved by the complete methods.

Critical strengths
  1. The paper studies an important question: Do the fairness improvements of fairness-aware GNN methods actually come from their proposed fairness components, or are they influenced by other factors, such as training procedures and model configurations?

  2. The authors conduct a systematic empirical evaluation by considering different configurations, training epochs, and checkpoint selection methods. They also analyze the uncertainty in their findings across different data splits and training settings.

Critical weaknesses
  1. The main idea is closely related to existing ablation studies and fairness benchmarking. The mathematical decomposition of fairness contributions appears straightforward. Therefore, the paper is mainly an empirical evaluation rather than a new algorithmic or methodological contribution.

  2. Although the paper provides a systematic evaluation of fairness interventions under different training settings, epochs, and data splits, it is not sufficiently clear how its findings differ from those of existing ablation studies. In particular, the paper does not directly compare its conclusions with the ablation results reported in the original papers. As a result, it is difficult to identify which findings are genuinely new.

  3. In practice, training is often stopped based on performance on a validation set rather than after a fixed number of epochs. The paper does not sufficiently justify why fixed training epochs are preferred over validation-based early stopping. This raises questions about whether the main findings would remain consistent under commonly used training procedures.

Presentation Issues

In Section 3.1, line 152, the authors define the graph as G = (V, E, X), but the individual symbols V, E, and X are not explicitly explained.

In Equation (8), the authors introduce P^{(0)} and P^{(1)} without explicitly defining them.

Questions for authors
  1. Most fairness-aware GNN papers already include ablation studies to evaluate the contribution of their proposed components. Although the authors explain how their evaluation differs from standard ablation studies, it is still unclear what additional findings are obtained. Can the authors directly compare their conclusions with the ablation results reported in the original papers under their published training settings and clarify which findings are new or different?

  2. Why do the authors prefer fixed training epochs over validation-based early stopping, which is commonly used in practice? Would the main conclusions remain similar if validation-based early stopping were used instead of a fixed training horizon?

  3. The evaluation mainly considers group fairness metrics (DP and EO). However, the proposed framework appears applicable to individual fairness metrics as well. Could the authors discuss whether the contribution of a fairness intervention might differ when evaluated using group fairness versus individual fairness?

Reviewer_Gz6DOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at an important gap in how fairness-aware Graph Neural Networks are evaluated. Most existing studies focus on the first research question: does the full method improve fairness compared with a reference system? But this does not tell us whether the proposed fairness intervention is actually responsible for the improvement. The authors seperate these two questions through a framework called protocol-conditional attribution. The idea is straightforward, they compare the same method with its fairness intervention switched on and off, keeping everything else fixed.

The evaluation covers 36 controlled experimental settings across 11 methods and leads to a couple of notable findings. First, the surrounding package, including architectural changes, preprocessing and training tricks, often produces a larger absolute contrast than the fairness intervention itself. Second, many intervention contrasts remain unresolved, either because the results are too noisy or because they fall below a pre-set threshold. These findings raise questions about how much of the reported fairness improvement can actually be attributed to the proposed intervention.

Critical strengths

The authors put considerable effort into the evaluation, covering 36 controlled cells, 11 methods, and several datasets, including German, Bail, and Credit. Their use of hierarchical bootstrapping and 30 matched realizations per cell, with careful handling of data splits and repeated runs, is a particular strength. It allows them to examine variability across splits and runs more thoroughly than is typical in this literature.

Critical weaknesses

(1) Presenting a standard ablation as a new framework: The main idea is to compare a method with and without its fairness intervention, while holding everything else fixed. This is essentially a standard component ablation. The additive decomposition and evaluation across different protocols give the analysis more structure, but it is not clear that they amount to a fundamentally new framework. The authors seem to overstate the novelty of what is, at its core, a familiar diagnostic check.

(2) Reading too much into small intervention effects: The authors emphasize that explicit fairness corrections, such as those in FMP or NIFTY, contribute little compared with earlier components or additional training. However, this interpretation may overlook interactions between stages of the model. In some graph debiasing methods, graph propagation may first need to produce stable representations before a fairness regularizer can work effectively. A near-zero marginal gain could therefore reflect an unsuitable intermediate representation or a problem with the optimization sequence. It does not, by itself establish that the fairness intervention has little value

Questions for authors

(1) Figures are difficult to read: Figures 1, 2, 5, 7, 8, and 9 try to show too much at once. The combination of insets, symlog axes, tiny scatter points, and multiple panels makes it hard to identify the main patterns without zooming in considerably. Simplifying these figures or splitting them into separate plots would help readers follow the results.

(2) Too many essential details are left to the appendices: Important information about method implementation, score conventions, and baseline settings is scattered across 11 appendices, covering more than 30 pages. Readers have to move back and forth between the main text and the supplement to understand the analysis. Bringing the most relevant details into the main paper would make the argument easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.